Testing Data Cleaning Python Scripts in Linux

If you want to retain the input data file’s format:

cat [input-file-location]/[input-file-name].txt | python [script-location]/[script-name].py > [output-file-location]/[output-file-name].txt

If you want to change the data file’s format (i.e.: comma delimited to tab delimited)

First create a script which will go line by line and convert from comma delimited to tab delimited. Say the script is named csv_to_tab.py

Now on to testing the data cleaning python script with the output file being made tab delimited:

cat [input-file-location]/[input-file-name].csv | [script-location]/csv_to_table.py |  python [script-location]/[script-name].py > [output-file-location]/[output-file-name].txt 

Now the input data file will run through csv_to_table.py script and convert to tab delimited before running through the data cleaning script and generating an output file.

Leave a Reply

Your email address will not be published. Required fields are marked *