Before you can work with data sets, it is important to first clean and preprocess the data using the scripting language/software of your choice. However, how do you know what needs to be cleaned? And further, how do you find issues that could not be resolved through data cleaning and need to be addressed with the source of your data or resolved via an authorized third party?

Here are some data quality checks to perform when dealing with healthcare data. These are of course meant for HIPAA-compliant companies and employees.

Check Format of ID Numbers

The healthcare data will often contain alphanumeric or numeric ID fields – such as subscriber IDs. It is important to make sure that if the values are numeric, that they are not in scientific notation. This can be fixed quickly in Excel, but if you are expecting to work with many data files, it is better to automate the fix in Python.

Check Format of Zip Codes

The healthcare data will often include zip codes – whether that of the patient, guarantor, or provider. If many values are varied in format within one column, this could be an issue and should be addressed in the data cleaning step.

For example, some values might include 4 digit zip4 and others might not. If you are expecting only 5 digit zip5 values, make sure to truncate the zip code field to 5 characters. If you have 2 columns for zip5 and zip4 and have some or all zip4 values in the zip5 column, make sure to move the zip4 values into the correct zip4 column.

Another issue could be the absence of a leading zero for a zip code. Somewhere in the data collection and transformation process, a leading zero might have been removed from a zip code – causing it to become 4 digits and having the appearance of a zip4 value. Make sure to research the associate city and state and see if this zip code with a 0 added to the front is a possible zip code for the city and state. If so, add the leading 0 in the data cleaning step.

Check for Missing Important Information

If the patient name, DOB, SSN, ID number, etc. fields are missing many values or are completely blank, you may not be able to work with this data set depending on your purposes. If few values are blank, you could simply drop the records if you could afford it. If your company has authorized access to a third party to populate any missing values, that would be a solution. Otherwise, you would have to request this information from the source of your data.