Converting a pdf to csv using linux shell script

linux script to extract data from pdf and create a csv. The regular expressions for sed are rather different from the Perl like ones i am used to in java. So \d is not allowed, + needs to be escaped, etc.

Below, we iterate thru pdfs, use pdftk to get the uncompressed version that has text, use strings to extract string data, use tr to remove newlines, apply sed on it to extract particular fields that we want, assign those to variables, and echo the variables to a csv file.

---

rm pdf.csv
for FILE in *.pdf
do
  echo $FILE
  pdftk "$FILE" output - uncompress | strings | grep ")Tj" | tr '\n' ' ' | sed -e 's/)Tj /) /g'  > temptocsv.txt 
  AMOUNT=`sed -e 's/.*(Rs \:) \([0-9]\+\).*/\1/' temptocsv.txt`
  CHLDATE=`sed -e 's/.*(Date of) (challan :) (\([^)]\+\)).*/\1/' temptocsv.txt`
  SBIREFNO=`sed -e 's/.*(SBI Ref No. : ) (\([^)]\+\)).*/\1/' temptocsv.txt`
  CHLNO=`sed -e 's/.*(Challan) (No) (CIN) \(.*\) (Date of).*/\1/' temptocsv.txt`
  echo $FILE,$CHLDATE,$SBIREFNO,$CHLNO,$AMOUNT >> pdf.csv
done

---

USB HID based IO utils

This project was started with the idea of turning a USB device into a kind of IO Swiss-knife, so that many things could be done with a single program. The use-case was to implement a remote-control, first to read the durations, and next to write them back. I had success in the read, by adjusting for code-delays. (The writes require more accurate timings, and i wrote a separate app for that. See  chaukasalshi.blogspot.in/2015/03/pic-micro-remote-for-tata-sky.html ) This project is based on Microchips Custom HID MLA project. Not all files are provided, only the main ones that were changed. The app_device_hid_custom.* files replaced with app_device_hid_io.*. Actually, i wanted to abstract the io routines into pin_io module, which would be independent of protocol, i.e USB/Serial/Other. And to create it as a standalone project with as few files as possible. Some such items are TODO, i may not get time to finish them. i have used the 18F4550 for this project. Probably, the project is quite primitive, considering the level of expertise on these forums, but i hope it may help somebody to begin.
The project files are available here : github.com/manojmo/pic_micro/tree/master/hid_io

Firing the IO commands :

This is done using the pyusb framework. The hid_io_test.py file has the code to send the commands and receive the results. Just as a POC, the results can be saved to a .JS file, and viewed as a chart in mychart.html, using the Chart.js framework. That needs to be downloaded separately. The script also has a facility to adjust results, based on expected timing inaccuracies. For the sampling commands, num_samples are specified, and the script will calculate the number of packets and loop to receive them.

Common features

The framework is generic, and the commands accept input like which pin to use, what delay intervals, etc. (As a result, accurate timing is an issue. ). There is a debug feature that can return you the actual time taken for an interval. (not very exact) Routines are provided to measure and execute 2^24 cycle delays. Timings can be specified in uS or mS. There are flags for each command, to provide additional control. All sampling commands also specify an IDLE state/value, and sampling does to start until the input changes from the IDLE state/value. This is useful for manually-triggered sampling, like reading from a remote.
The commands are

WP : Write Pin :

This allows us to write a pattern to the <pin_num>. A pattern is an array of 2-byte durations, starting from HI. Currently, upto ~ 60 durations are supported. They end with a zero duration marker. There is a facility to make the HIGHs pulsed at a carrier-freq, as needed for IR codes. A reset flag, mS/uS flag, and a repeat flag are also provided.

RP : Read Pin :

This allows us to read <num_samples> of HIGH/LOW states from <pin_num> at <sampleInterval> intervals. Not well tested. TODO : store the result in bits instead of bytes. A debug flag and mS/uS flag are also provided.

RPD : Read Pin durations :

This allows us to read <num_samples> durations of HIGH/LOWs by sampling <pin_num> at intervals of <sampleInterval>. durations upto FFFF cycles are currently supported. A debug flag, idle-state flag and mS/uS flag are also provided. This was used to read durations of IR-code from tata-sky remote, but i needed to adjust the values for code-exec delays. i have blogged about it here : chaukasalshi.blogspot.in/2015/03/pic-micro-remote-for-tata-sky.html

RA : Read ADC :

This allows us to read <num_samples> samples from ADC at intervals of <sampleInterval>. A debug flag, idle-value and mS/uS flag are also provided. The results can be viewed as a chart.

PIC micro remote for tata sky

Background

My entry into the world of micro-controllers was out of my desire to create a USB device. I wanted to create a general purpose IO framework that could be used to do many useful things, and one of my use cases was to add infra-red capability to my mobile which had none, and use it like a TV remote. While that framework is under way, and i may blog about it later, what i realised was that the runtime overheads of generic code might prevent me from achieving the exact timings that are required for remote protocols like RC6. Hence i fell back to writing a more basic/specific code for this purpose.

I have tested this code on the PIC18F4550.

Basics

My work would be in two phases, first find the codes, and second, transmit them. Since i did not find any reference for codes for the tata sky settop box, i had to find out the codes myself. i.e. what was the tata-sky remote transmitting. My starting idea was that it would be a series of ON & OFFs, with specific durations. To capture them, i would need an infra red sensor.  i searched the net and found the following links : 

https://learn.adafruit.com/ir-sensor/making-an-intervalometer

This is a project for arduino, and describes how to capture the IR codes for a Nikon camera, and how to play them back. Since i was programming with PICs, this project would not work as is for me and i had to start on my own code.

http://www.sbprojects.com/knowledge/ir/index.php is also a good reference for knowledge of IR protocols. 

One basic thing to understand, is that the HIGHs in the code are usually transmitted as a series of pulses at high freq, called the carrier frequency. This is so that ambient light/disturbances are not mistaken for a code. Also, it lets the IR diode cool, since it takes rather high current, 50- 100mA. 

There are 2 kinds of sensors, a simple IR sensitive transistor/diode that would capture the signal as is, i.e. even the carrier pulses. This is available for 4-5 Rs. The other filters out the carrier pulses, so we directly get the highs and lows of the code, it costs slightly more, around 25 Rs. i got both, and experimented with them. The plain receiver has low sensitivity, and also, i did not have a oscilloscope to measure the waveform exactly. The one with the filter had good sensitivity so i decided to go with it. 

The one i got was TSOP1730( carrier freq 30Khz). Note that its output is inverted, i.e. it is usually HIGH and when it receives IR input, it goes low.
i also got an IR diode for transmitting for 4-5 Rs.

Trial and error

i added a facility to the read code to define an IDLE state and wait until it changes. so we can run the program, it will wait till input is in IDLE state, in our case, HIGH. when we press the remote, the state changes and now it starts taking the reading, i.e. measuring the durations of ones and zeroes. 

Why measure durations, why not just measure the sequence of 1s and 0s ? The reason being that in protocols like RC5/6, both 0s and 1s are represented using both the HIGH and LOW states.  HIGH first, LOW next could mean 1 , and LOW first, HIGH next could mean 0 etc. The durations are important and need to match the protocol.

One of the problems with the TSOP1730 sensor is that it can get randomly triggered by surrounding disturbances, so sometimes the program would start taking readings without my pressing the remote. Its suggested to tie the TSOP output pin to VDD using a > 10K resistor, and putting a capacitor betw the TSOP's VDD and GND to reduce power-supply interference. i also tried to tie the output to GND instead, thru a 175K resistor, and used a .1uF capacitor betw the TSOP's VDD and GND. None of the methods proved totally fool-proof however, and the random triggering still happens sometimes. 

i started to look at the readings i was getting on pressing the remote. then i would try to transmit those durations, inverted now, (since out sensor was inverted ), and see if it worked, but no luck. i did not even know what protocol my remote used, so i had no idea if my readings where right or not.

i searched the net again, came up with link link : http://forum.arduino.cc/index.php?topic=166536.0, that somebody else had built a remote for tata-sky and the protocol was like RC6, tho no details were provided. The guy had built it from the arduino link i have given at the start, so i started looking at it again. one thing i realised was that timing need to be accurate. 

at this time, i was using a generic io framework, where one could specify in the request which pin to use, what carrier-freq, the durations etc. i started measuring the timings of my transmit and read using the PIC's timer, and realised that the generic nature of the framework was adding a lot of overhead over direct hardcoding. So i changed the generic code to bare and specific code for the task in hand. One more problem was that my readings did not fit the RC6 protocol exactly; they extended some bits longer. i decided to try with both options, i.e. once stopping at RC6 length, and once with my readings as is.

i started to test again, however, still no success. The settop box links red when it receives a valid code, and this was not happening. i went back to the article to check what else i was missing, and there it was. The article said that we need to send the code twice, not once. i added the repeat with a gap of 65ms, and the set-top box finally responded by changing the channel. yay ! It turned out that the readings that i had got were correct, even tho longer than the RC6 length.

The strange part was that my readings did not show the code being repeated, so how did the remote work ? maybe the remote encodes some more info that i have missed.

The timings of my readings were another issue, they did not match RC6 timings, where one unit is of 444 uS. So i wrote a python script to change my readings to RC6.

def fit_to_rc6( durations) :
    rc6durations = []
    for loop in range( 0,len(durations) ):
        duration = durations[loop]
        newduration = duration
        if  duration < 600 :
            newduration = 444
        elif duration < 1000 :
            newduration = 889
        elif duration > 1800 and duration < 3500 :
            newduration = 2666
        rc6durations.append( newduration)
    return rc6durations


arr_durations = fit_to_rc6(
[3015,474,435,400,408,400,408,852,408,839,855,407,408,407,408,413,408,407,408,393,428,407,408,413,408,407,408,413,408,407,408,413,408,407,408,407,968,744,848,407,408,400,408,852,408,413,402]
) 

Here is a sample of the readings i got :

the program has a max wait of  65535 us. after the signal ends,we get all max-durations i.e. 65535. Also the sensor gives inverted output,so S=0 is actually HIGH input to the sensor
Starting..                      S:0,D:2662,S:1,D:656,S:0,D:495,S:1,D:265,S:0,D:528,S:1,D:224,S:0,D:535,S:1,D:663,S:0,D:522,S:1,D:656,S:0,D:948,S:1,D:218,S:0,D:542,S:1,D
:224,S:0,D:535,S:1,D:224,S:0,D:542,S:1,D:224,S:0,D:535,S:1,D:224,S:0,D:542,S:1,D:224,S:0,D:535,S:1,D:224,S:0,D:535,S:1,D:224,S:0,D:542,S:1,D:224,S:0,D:535,S:1
,D:224,S:0,D:542,S:1,D:224,S:0,D:535,S:1,D:224,S:0,D:542,S:1,D:224,S:0,D:535,S:1,D:224,S:0,D:542,S:1,D:224,S:0,D:955,S:1,D:656,S:0,D:522,S:1,D:231,S:0,D:535,S
:1,D:224,S:0,D:955,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:6
5535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:655
35,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535
,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S:1,D:65535,S
:1,D:65535,S:1,D:65535,Done..

we can see that the reading vary quite a bit, from 224 to 542 are all actually 444 us, and from 656 to 948 are all 889 us ? how do i know ? by getting a bunch of readings, and seeing which part repeats most, and comparing with the Rc6 protocol. The remote_codes.txt file contains codes for some common keys.

Most of them have been implemented in the sample code too.

about the transmitter, its a simple transistor circuit, with the base driven from the PIC, and the IR diode with a 100R resistor in series with the collector. emitter to ground.

Code

The code is available at https://github.com/manojmo/pic_micro. The read_pin_durations.c is for the reading of the remote codes. The tata_sky_remote_serial.c is for transmitting the codes. it uses a uart to accept a code, and looks it up from an array and transmits it. The array contains the durations of ON/OFFs. Ideally, since we know the protocol and timings, we could use a more compact format to store the codes, rather than actual durations. e.g, we could keep the start-bit of 2666 and 889 separate, and represent the rest with 444 as 0 and 889 as one, and hold the entire code in just an int. Also, no keyboard interfacing is done. One could use a multiplexer, to get 16 lines from 4 pins, or implement it in the PIC. a good link to understand a keyboard is
http://pcbheaven.com/wikipages/How_Key_Matrices_Works/

Other references

Ken sheriff has a IR library for Arduino, which can capture and play-back remote code on the go ! cool, no ? http://www.righto.com/2009/08/multi-protocol-infrared-remote-library.html

Another project for PICs is http://www.compendiumarcana.com/irwidget/

Generic UART for any microcontroller

Communicating with a micro-controller at runtime, is a very useful facility. It can be used for debugging, or to send commands etc. One of the easiest available protocol to communicate is UART. It requires a pin for transmit and a pin for receive, and the GROUND connection. There are many available programs that allow us to communicate using UART on the serial port, or using a usb-serial adapter. e.g. hyperterminal, minicom, etc.

However, not all microcontrollers have UART ports in built.

i was taking a look at the UART protocol, and it seemed pretty simple to implement.  So i gave it a go, and was zapped when it just worked the first time :).

Below is a little app that accepts 2 byte commands. Toggles LATD1 when it receives the command "TG", and echoes back the command after its done. Invert flag is provided. ( Inversion may be needed if we are not working directly with a real serial port). Tested with minicom @9600 baud, with hardware-flow-control OFF, so that it sends the chars we type.

It is tested on the PIC18F4550, will need modifications to run on other microcontrollers.

Its also available as a module to include at https://github.com/manojmo/pic_micro

/*
 * File:   main.c
 * Author: manoj
 * Writing to uart
 * Created on September 27, 2014, 6:01 PM
 */

#include 
#include 
#include 

// PLL with pre-scaler and post-scale options is a way to derive multiple
// freq from the same source, e.g. for low=6Mhz/high=48Mhz speed USB and for MCU clock
// Input to PLL has to be 4 Mhz and its output is 96 MHz
// So, for e.g. if we are using exernal 20 MHz osc, its o/p has to be
// divided by 5 to get 4 Mhz input for PLL
// PLLDIV is prescaler, o/p has to be 4MHz
// CPUDIV and USBDIV are postscalers, input is 96Mhz

#pragma config PLLDIV   = 5         // (20 MHz crystal on PICDEM FS USB board)
#pragma config CPUDIV   = OSC1_PLL2
#pragma config USBDIV   = 2         // Clock source from 96MHz PLL/2
#pragma config FOSC     = HSPLL_HS

#pragma config IESO = OFF
#pragma config WDT = OFF
#pragma config STVREN = ON
#pragma config LVP = ON
#pragma config BOR = ON
#pragma config MCLRE = ON
#pragma config PWRT = OFF
#pragma config PBADEN = OFF

// Effective CPU Freq, considering PLL and CPUDIV. Needed for the delay routines
#define _XTAL_FREQ 48000000

#define UART_BIT_TIME 104 // us. inverse gives the bit-rate
#define PIN_UART_TX LATD4 // output
#define PIN_UART_RX PORTDbits.RD5 // input

uint8_t UART_INVERTED = 0; // If inverted, low and high states will be inverted
uint8_t HIGH_STATE = 1;
uint8_t LOW_STATE = 0;

// flexible way to set some params.
void uart_init( uint8_t is_inverted){
    UART_INVERTED = is_inverted;
    if( is_inverted){
        HIGH_STATE = 0;
        LOW_STATE = 1;
    }
}

// start of transmission
void uart_start_tx(){
    PIN_UART_TX = HIGH_STATE;
}

// end of transmission
void uart_end_tx(){
    PIN_UART_TX = HIGH_STATE;
}

// write a char
void uart_write( char c){

    // start bit
    PIN_UART_TX = LOW_STATE;
    __delay_us( UART_BIT_TIME);

    uint8_t bit_pos = 0;
    // transmit bits, LSB first
    while (bit_pos < 8) {
        uint8_t currbit =  c & 0x01;
        PIN_UART_TX = UART_INVERTED ? ! currbit : currbit;
        __delay_us( UART_BIT_TIME);
        bit_pos++;
        c = c >> 1;
    }
    // stop bit
    PIN_UART_TX = HIGH_STATE;
    __delay_us( UART_BIT_TIME);
}

// Read specified number of chars and put them into holder array
void uart_read( uint8_t len, char holder[] ){
    uint8_t i;
    uint8_t bit_pos;

    // loop and read len number of chars.
    for( i=0; i< len; i++){

        // Wait for idle state to change, i.e. start bit
        while( PIN_UART_RX == HIGH_STATE);

        // start bit
        __delay_us( UART_BIT_TIME);

        // read bits, LSB first
        bit_pos = 0;
        char c = 0;
        while (bit_pos < 8) {
            uint8_t currbit = UART_INVERTED ? ! PIN_UART_RX : PIN_UART_RX;
            c = c | (currbit << bit_pos);
            __delay_us( UART_BIT_TIME);
            bit_pos++;
        }
        holder[i] = c;


        // stop bit
        __delay_us( UART_BIT_TIME);
    }
}

// write a string, optionally of specified len.
// if null terminated, len can be -1
void uart_writestr( char str[], int len ){
    int i;
    for( i=0; i< len || len == -1; i++){
        char curr = str[i];
        if( curr == '\0'){
            break;
        }
        uart_write( curr);
    }
}

// process the command we received
void process_cmd(char cmd[] ){
    if( strncmp( cmd, "TG", 2) == 0){
        LATD1 ^= 1;
    }
}

/*
 *
 */
int main(int argc, char** argv)
{
    TRISDbits.RD4 = 0; //TX pin set as output
    TRISDbits.RD5 = 1; //RX pin set as input
    TRISDbits.RD1 = 0; // command output

    uart_init(1); // invert
    uart_start_tx();
    char cmd[2]; // array to hold received command.
    while(1){
       uart_read( 2, cmd); // read a command, specified bytes long
       process_cmd( cmd);
       uart_writestr( cmd, 2); // echo it back
    }
}

Controlling Microchips 18F4550 as HID Custom USB device using python pyUSB

The MLA( Microchip Libraries for Applications ) ships with a MPLABX project for 18F4550 that makes the chip behave like a HID Custom device. i built the project, burnt the hex file, and was also able to test it using the plug_and_play executable provided for Linux

As per this project, there is 1 interface with 2 endpoints.
One is IN, other is OUT. Both are of type INTERRUPT.
( See usb_descriptors.c )

For the commands, see app_device_custom_hid.c.
The commands are :

• COMMAND_TOGGLE_LED = 0x80 -- LED on RD0
• COMMAND_GET_BUTTON_STATUS = 0x81 -- Button on RB4, pressing should bring it to Ground
• COMMAND_READ_POTENTIOMETER = 0x37 -- ADC_CHANNEL_0

Commands should be sent on the OUT endpoint, and data received on the IN.

i wanted to write my own code to control the device. came across the pyusb framework, which looked easy to use.

Here is working code for the first 2 commands :

import usb.core
import usb.util
import sys

# find our device
dev = usb.core.find(idVendor=0x04d8, idProduct=0x003f)

# was it found?
if dev is None:
    raise ValueError('Device not found')

interface = 0

if dev.is_kernel_driver_active(interface) is True:
            print "but we need to detach kernel driver"
            dev.detach_kernel_driver(interface)

# set the active configuration. With no arguments, the first
# configuration will be the active one
dev.set_configuration()

# get an endpoint instance
cfg = dev.get_active_configuration()
intf = cfg[(0,0)]

for cfg in dev:
    sys.stdout.write("ConfigurationValue:" + str(cfg.bConfigurationValue) + '\n')
    for intf in cfg:
        #print vars(intf)
        sys.stdout.write('\t' + \
                         "InterfaceNumber:" + str(intf.bInterfaceNumber) + \
                         ',' + \
                         "AlternateSetting:" + str(intf.bAlternateSetting) + \
                         '\n')
        for ep in intf:
            #print vars(ep)
            sys.stdout.write('\t\t' + \
                             "EndpointAddress:" + str(ep.bEndpointAddress) + \
                             '\n')

assert ep is not None

# toggle the led. COMMAND_TOGGLE_LED
# args : ep, value, timeout
dev.write( 1, chr(128), 100)

# read the button state. COMMAND_GET_BUTTON_STATUS
# first send the command on EP_OUT
dev.write( 1, chr(129), 100)
# read on EP_IN
ret = dev.read( 129, 100)
print "Received: %d,%d", ret[0], ret[1]
print "Button Status : " + "Released" if ret[1] == 1 else "Pressed"

PIC Programmer using a USB to Serial Converter - HVP

Background

This is a follow up to my earlier post by the same name, for LVP. Please read that first, since it discusses the basics. In this one, i will discuss HVP.

Current Status

I am able to program the PIC12F675 in HVP mode. As expected its quite slow, takes around 2-3 mins to burn a 500 byte program.

Hardware

Obtaining a High voltage > 9V from USB

i wanted to investigate whether a high voltage could be obtained from the USB port. There are voltage-pump/ charge-pump ICs like LM2623, MAX660,

mc34063a etc. I could obtain ADM660 for Rs 175. Later i also found mc34063a which is cheaper, but i don't know if it is locally available. With just two 1uF capacitors, ADM660 delivers double the input voltage, and i was able to get 10V from the 5V of the USB port.

The problem with Txd and flip-flop at 10V

i planned to supply the 10V to the flip-flop as supply, and use the flip-flops output to drive the VPP. However, i ran into an issue here. Txd, which drove the CLK of the flip-flop, was still at 5V, and while it could reset the flip-flop, it could not set it. Hence i kept the flip-flop at 5V, and added a NPN transistor to provide the 10V.

The VCC of the transistor was 10V from the ADM660. Output of the flip-flop was fed as input to base. emitter to ground. The Output from collector now provided the high voltage to VPP. Rbase=Rload=10k.

The working circuit

The circuit is powered by the usb port on the right. The Flip-flop with the LED indicating its output state is on the left. Next comes the transistor which changes the fip-flop output from 5V to 10V. Next comes the ADM660 voltage-doubler which provides the VCC of 10V for the transistor. Last is the PIC12F675, which gets its VPP from the transistor output.

Software

This time, i tried out a new software : pickle( old version is called k8048) from kewl.org, written in C. (This site also has a nice list of PICs that support LVP.) It supports a lot of PICs and modes, and is simple and easy to modify. I had to modify it, since it depends on the usb-serial-port to support the BREAK functionality, which my adapter does not.

The change is to introduce a new config param( bit-rule) VPP_TOGGLE_ON_PULSE, and alter the set_vpp() method so that it when VPP_TOGGLE_ON_PULSE is true, it sends a single 0 instead of the BREAK functionality.

The .k8048 file should be created in the home directory, and has the config.

the ktest command helps to check whether the pins have correct voltages.

 

For serial-port option, the connections are fixed in the code, not in the .k8048 file. So DTR->PGD, PGD->CTS,RTS->CLK and Txd->Vpp. You need to follow these exactly. 

 

The common reason for "device not detected" is that Vpp is not properly applied. Confirm that you have a Vpp > VDD + 3.5 V. Check whether it turns on and off correctly. Put an LED with a 10k resistor as an indicator on it. It should be low before programming, and will go high/low during programming, and go low after its finished.

 

The command to use for programming depends on the instruction-set of the chip. For 12f675, i used k14, since it has a 14-bit instructions set.

For validating device id, i use "k14 id".

For programming, k14 program <hexfile>

Do not set any flags(bitrules), unless you understand them.

 

The zip with the modified files is available as k8048_vpp_pulsed.zip under http://milunsagle.in/webroot/downloads/

More to come...

PIC Programmer using a USB to Serial Converter - LVP

Background

Nowadays, computers do not have serial/parallel ports. Neither has my laptop. Since my interest in PIC programming was temporary and for hobby purposes, i did not want to get a commercial USB based PIC programmer. Besides, where's the fun in that ? I decided to get a cheap USB-Serial converter, and see if it could make it work.

Important : This hack with the flip-flop is needed only if your USB-Serial converter does not support the BREAK functionality. If you want to buy a new one, i have been told that Cp2102 is a better option than pl2303.

Current Status

I used the Bafo Db9 USB-Serial converter(cost 200 Rs), and tried to program the PIC18F4550, and its a success. I had initially a problem reading the EEPROM, this was solved by a resistor between PGD and DTR. Its pretty slow, takes around 1 min to program a 350 byte hex file, verifying only 10 bytes instead of the entire code.

Disclaimer : This is an experiment, do it at your own risk.

PIC Programmer requirements

There are 2 modes for programming PICs :

1. HVP - High voltage programming in which the MCLR pin is raised above Vdd( typically to > 9V) to make the PIC enter into programming mode. 
2. LVP - Low voltage programming, in which the PIC can be programmed with the same voltage as VDD. Advantage is that high voltage is not needed, one can use the USB port power supply to program the PIC. Not all PICs support LVP, and LVP support can be turned on/off using a configuration setting, but this has to be done in HVP mode. There are 2 modes in LVP :

• PGM : A separate PIN, PGM is raised high, to VDD,  to make the PIC enter into programming mode. Disadvantage is a separate PIN blocked for this. My experiment is in LVP mode. 
• KEY : A special sequence of bytes fed into the PIC, to make it enter programming mode, usually with VPP held low.

My experiment is in the LVP-PGM mode. After the PIC enters the programming mode, we feed it commands/program-bytes on the DATA line( which is bidirectional on the PIC),  in sync with the clock signal provided on the clock line. This is Microchip's ICSP( In-circuit-serial-programming) protocol. For programming specs of various PICs, see http://www.microchip.com/TechDoc.aspx?type=Programming

Below is a (rather dirty)  diagram showing pins needed for PIC programming :

 

As we see 5 pins would be required of our programmer :

4 Outputs - MCLR, DATA_OUT, PGM, Clock

1 Inputs - DATA_IN

If our programmer is something with GPIOs, like a Raspberry Pi, then a GPIO pin can work as both input and output, so two separate pins will not be needed for DATA.

However, this post is going to be about using the USB to Serial Converter.

Take a look at http://www.db9-pinout.com/ to learn about a serial port pins.

Now, a serial port DB9 connector has only 3 lines which can be outputs :

RTS, DTR  and Txd. There are multiple candidates for input, usual one is CTS.

So we are one output pin short. However, as per research on the web, e.g. : http://artjoy.narod.ru/eng/index_eng.html, we see that the MCLR pin can simply be tied to VDD in LVP mode. So now, we seem to have met the pin requirements of the programmer. 

So if we build a simple JDM like programmer e.g. http://www.instructables.com/id/Simple-JDM-PIC-Programmer/, controlling PGM instead of MCLR from Txd, tie MCLR to VDD, and use some of the programming softwares available like PicPgm, it should work, right ?

Well, sadly, no.

Our usb-serial port is not a real serial port. It differs in two important respects :

1. Voltage levels : The Txd pin on a real serial port has low level voltage between 3-15 V, and a high level voltage between -3 to -15V. This is made use of in most serial programmer circuits to generate an HVP voltage. Whereas the usb-serial port usually has the TTL range of 0-5V. Well thats Ok you say, 5V is sufficient for LVP isn't it ? Yes, but there is another significant difference. In a real serial port, the Txd pin is held high in idle state, so it can provide a non-zero voltage in idle state, whereas in the usb-serial case, the txd pin is at zero-level in idle state, and goes high only when transmitting data.
2. Timing issues : USB is a scheduling based protocol, where data exchange occurs in time-frames controlled by the USB-controller. There is a limit on how much time can be allocated in a frame, to what kind of data(interrupt,bulk,isochronous). What this means is, if i want to toggle a line( e.g. RTS/DTR ) high or low for clock or data, it may not occur immediately. First the request is sent to USB controller. That will encode my request into a packet format. Maybe it will get sent in the next time frame, since the current one is full. Then the USB device at the other end will decrypt the packet, analyze my request, see that it means to raise the RTS line, and then execute it. Similar for a read. For full-speed USB normally used in such converters, frame time is 1 ms, and if say 20/30% of it is reserved for protocol handling/other kinds of data, it means that our frequency cannot be greater than 30% of 1ms. In practice, it can be even less, as we will see. Thus USB protocol is not really suited for real time applications.
3. Missing functionality. e.g. setting BREAKs. The Txd line of a real serial port is held high normally to show its active, and this can be brought low by transmitting all zeroes for some specific period, or indefinitely, by using the BREAK functionality. Usually, in a usb-serial adapter with 0-5v levels, the 0s have the opposite effect, of raising the line high. This is used by many of the programming softwares to manipulate the Txd line, and this can be missing in some USB-serial converters, like mine. See setbreak() in TERMIOS, EscapeCommFunction() in Windows  ioctl(handle, TIOCCBRK, 0) and ioctl(m_handle, TIOCSBRK, 0)) on Linux.

 Lets try to solve these issues :

Making the Txd Pin stay high for PGM

Why have we marked Txd for PGM ? Because we want exact and frequently changing 0-1 control on DATA and CLOCK pins, whereas Txd deals with bytes and not bits, and data is sent with start,stop,parity bits, so that even if we send 0, a 1 will also result due to the stop bit etc. PGM, on the other hand, just needs to be held high during LVP, and may be achieved more easily. Now lets see some options :

1. Using a Flip-flop that will change state when a pulse is received from Txd. The output is fed to PGM. This is the approach i have finalised. To set PGM, we send a '0' on  Txd, which results in a high pulse. Since its a toggle, next time we send the pulse, PGM will be reset. See the setPGMState() method in burnGPIO.py for how this is done. The schematic is at http://www.electronicshub.org/jk-flip-flop-using-cd4027/. NOTE : It worked best for me with just the 10k resistor(R3) connected from clock to GND. This circuit should be first tested with a push button, and once that works, try it with the Txd pin. Also, having a LED may load the output, make sure the output high voltage is above 4.5 volts. The LED should go high in LVP mode,( i.e when the program is executed) and stay low otherwise. Ensure that it is low before you run the program.
2. Using a simple latch as described here : http://homemadecircuitsandschematics.blogspot.in/2011/12/simple-and-useful-transistor-latch.html. So we turn on the latch to make PGM high at the start of the program. The disadvantage is that we cannot make it low programatically, it has to be done manually using a switch that connects the base to the ground. Another caveat is that the input should not go to ground after the pulse, but Txd is going to do so, so we prevent it by adding a diode in series before the base the the 1st transistor.
3. Since the BREAK functionality is missing in my usb-serial converter, another option is to continuously keeping Txd high by transmitting on it, in a separate thread. This does not work as is, probably because Txd does not rise fast/high enough for PGM. Does not work with driving a transistor either. The extra thread might also cause timing issues for the main operation. See the burnGPIO_txdOnThread.py for this.

Currently, option 1 seems the best one, and its my default.

The USB timing issue

We know that USB is going to introduce a delay, but we need to find exactly how much, so that we can use that for our clock operation. Thankfully, the clock rate for ICSP is not fixed, and can be varied as per the programmer.

What we do, is connect the DATA_OUT(RTS/DTR) to DATA_IN(CTS/DSR), output a 1 on DATA_OUT, and try to read it on DATA_IN, with a delay in between. Similarly for a 0 on DATA_OUT. And we adjust the delay until we can successfully read what we wrote. As per my tests, this delay turns out to be 5ms. See testSerial.py for this test.

The final hardware circuit

This is the final circuit used. The board on the right bottom contains the flip-flop circuit. It receives the Txd(3) line as the input, and its output is used to drive the PGM pin. RTS(7) is used as data-out, CTS(8) as data-in, DTR(4) as clock. VPP/MCLR is tied to VDD thru a pull-up-resistor of 10k. The crystal and capacitors connected to the bottom side of the PIC are not needed here, they are used when running the PIC as a USB device. The 3 transistors at the top-left are also not needed here, they are level shifters between the Raspberry Pi(3.3V) and the PIC(5V). i initially programmed the PIC using the Raspberry Pi, and they were used then. See http://electronics.stackexchange.com/questions/82104/single-transistor-level-up-shifter and http://elinux.org/RPi_GPIO_Interface_Circuits if interested.

Note : A resistor of 1-5K, between DTR and PGD is a must. It prevents a short between these two when PGD is in output mode and DTR is low, and ensures that a proper signal reaches CTS. Optionally, it can also be added to the RTS and CTS lines. Thanks to Darron from kewl.org for pointing this out.

Either of DTR/RTS can be used for PGD/PGC. My earlier schematic was using RTS for PGD, but i am reverting to DTR, since its more commonly used.

So now we have resolved these two issues. Next is, what software to use for programming ?

The Programming Software

Now we need a software to program the PIC. This software will read the .hex file we want to program into the PIC, and manipulate the clock, data. pgm lines to achieve this. There are many such softwares on the net, e.g. PicPgm, WinPic. Some allow a custom interface to be defined. However, none seem to handle the Txd line and delay like we want. Not being a C/C++ guy, I was looking for something in Java or a scripting language, that i could easily understand and modify. Then i came across a software in python - burnLVP, written for the Raspberry pi. See https://github.com/danjperron/burnLVP. Thanks Dan ! i was able to use it to program my PIC using the Rpi. Then i decided to see if i could make it work for USB-serial. There were references to the Rpi PINs in multiple places. I encapsulated all into the burnGPIO.py, which now represents the hardware interface. Currently, there are 2 versions of it, one for the USB-Serial and one for the Rpi. The one we want to use, should be copied as burnGPIO.py. The rest of the program remains the same. So if somebody want to use some other hardware interface for programming, they can create an appropriate version of burnGPIO.py and use that. Note that this software does not have a UI, its run from the command line as "python burnLVPx.py <hexfilename>". The intelhex library for python needs to be installed.

Due to the 5*2ms clock, the execution is quite slow. It takes a minute to burn

a small hex file of around 300 bytes. Since verification of the entire program memory would take a long time, i have introduced a parameter to those methods specifying how many bytes to check. Currently, i use 10. And for some reason i don't know yet, reading the EEPROM is not working. (always returns 0). So that is bypassed with a "True or" condition.

Another change in burnLVPx.py is the PIC families to be used. Currently i have kept only the one for the 4550. See the "AllCpuFamily" variable.

While the software has been named burnLVP, i believe that with minor modifications, it should work for HVP as well. The hardware will change, a >9V MCLR will be needed, and PGM is not needed, it should be tied to ground.

Update : i got HVP to work too. See my next post at http://chaukasalshi.blogspot.in/2015/01/pic-programmer-using-usb-to-serial.html

Sample output :

python burnLVPx.py Test.X.production.hex

File " Test.X.production.hex " loaded
Scan CPU
check  PIC18F2XXX ...
4610
Cpu Id   = 0x1200
Revision =  0x2
Found  PIC18F4550 from Cpu Family  PIC18F2XXX
Cpu Id: 0x1200  revision: 2
Bulk Erase  ..... Done!
Program code [ 32768 ] blank check.Passed!
Writing Program.Done!
Program check .Passed!
Writing EEPROM data[ 256 ]........Done!
Writing ID  ... Done!
ID Check . ... Passed!
CONFIG Burn  ... Done!
Config Check . ... Passed!
Program verification passed!

Trouble shooting

1. The flip-flop state should be off before programming, On during, and off again. Test this circuit separately first.
2. If you get the CpuTag as 0, most probably PGM is not being applied correctly.
3. PICs other than 4550 have not been tested, they may have issues.

In case somebody wants to do this in java, see https://code.google.com/p/java-simple-serial-connector/, which works nicely as a serial-port api.

Download the modified code

The modified code is available as burnLVP*.tar.gz file at http://milunsagle.in/webroot/downloads/or from the repository at https://github.com/danjperron/burnLVP/tree/usb2ser

Other open source programming softwares :

• Kewl : http://wiki.kewl.org/dokuwiki/interfaces. A nice, simple command line tool, easy to modify, supports many PICs
• WinPic : http://www.qsl.net/dl4yhf/winpicpr.html( No longer maintained )
• wxpic : a derivative of WinPic : http://wxpic.free.fr/