Data engineering with Python

In the first wave of the covid-19 pandemic I came across a site called DataCamp. They offered a huge discount on online data science courses mainly focusing on matplotlib and pandas framework. So I jumped in. Pandas is a great tool handling large databases in json, csv, sql, or even excel. You can easily analize, them, and extract the desired part. With matplotlib we can do visualisations of such data streams. As I finished the tutorial, It was quite obvious to use it for analizing the current trends of the pandemic.That way I could test and practice my new skills. By the way this sort of helped dealing with the lockdown situation psychologically too. Let’s see how it goes. I am not gonna go in detail on how to install Python packages. It can be very tricky, and there are many ways doing it. Easiest solution is to make a new Python project in an IDE (Integrated Development Environment) such as Pycharm for example. There you create a new Python file and simply add:

  
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

,then click on the error next to the code to install the packages. It is as easy as that. The rest of the heavy-lifting is handled by the IDE. Next thing we need is a data source. You can search Github for any kind of data APIs, just make a search on the desired data, and choose the best one for you. I stuck with the Our World In Data website’s data in comma separated values. So I went:

 data = pd.read_csv("https://raw.githubusercontent.com/owid/covid-19-data/master/public/data/owid-covid-data.csv") 

Next you have to analize the data somehow. Try to look inside. In the

 print(data.head()) 

That will try to print out the header and the first five rows of the data. We can see that the rows are representing the country datas column by column, and each country has rows for each day passed since the outbreak of the pandemic. The problem is, this data is too huge to see all the columns. Try to get all the data we have for each country:

  
for c in data:
    print(c) 
  

Now you get all the column names. Just out of curiosity, count the columns. let’s make a counter variable and iterate through them.

 
  counter = 0
  for c in data:
      counter += 1

Then:

print(counter)
#61

Now we know that we are dealing with 61 columns of data. How we know, how many entries are there for all the countries? It is easier just print out the shape of the database.

print(data.shape)
#(77549, 61)

Ok, now we know there are 61 columns for 77549 rows. Makes sense, because that includes more than a year of daily data for each country on Earth. We can clearly see, that dealing with a data of that scale would be almost impossible without writing scripts. Not to mention visualising it. Without going in all the details you can do with these frameworks, let’s just see a very simple sample visualizing some segment of our data. When we printed out all the columns, we saw that there is one column for location. That means we can filter this whole database for one countries data.

 
uk_data   = data[data['location'] == 'United Kingdom']

One interesting column have the name new_cases. That is representing the daily registered new infections. Let’ get that:

new_cases = uk_data['new_cases']

In order to be able to visualise this data, we need to tell pyplot, what are the x values for the chart. That is gonna be the axis for the days passed. In other words that is gonna be an array of dates.

dates = uk_data['date']

Now we have the necessary variables for our chart. Add

plt.plot(uk_dates, new_cases)
plt.show()

At this point our code looks like this.

import pandas as pd 
import matplotlib.pyplot as plt 
import numpy as np 

data = pd.read_csv("https://raw.githubusercontent.com/owid/covid-19-data/master/public/data/owid-covid-data.csv")

uk_data = data[data['location'] == 'United Kingdom']
new_cases = uk_data['new_cases']
uk_dates = uk_data['date']

plt.plot(uk_dates, new_cases)
plt.show()

Run the project. You are going to see a plot chart of the daily covid-19 cases in the UK. The dates below the chart are just a black line. This is because the dates are written out in yyyy-MM-dd form, and there are too big number of them. Let’s solve the problem by diplaying only every n-th date.

sample_frequency = 11
uk_dates = uk_data['date'][:-1: sample_frequency]

This means we want all the entries to the last one, by step dfined in the sample_frequency variable. Currently we set it to step by 11 days. Let’s display a shorter version of the date strings and store them in an array. This can be solved by writing a function called trimmedDates.

def trimmedDates(array):
    resultArray = []
    for date in array:
        date = date[-5:]
        resultArray.append(date)
    return resultArray

This function is taking an array argument and returning an array of date strings, 5 digits each. We can use them in the xtics function like the following:

plt.plot(uk_data['date'], new_cases)
plt.xticks(np.arange(0, len(uk_data['date']), sample_frequency), trimmedDates(uk_dates))

Note: we are using all the uk dates in the plot function, we only filtered for the x tick labels as uk_dates. Let’s run the project now. Better, but the dates are still barely visible. Lets rotate them, and add them a smaller fontsize!

plt.xticks(np.arange(0, len(uk_data['date']), sample_frequency), trimmedDates(uk_dates), fontsize=5, rotation=75)

At this point our code should look something like this:

import pandas as pd 
import matplotlib.pyplot as plt 
import numpy as np 

data = pd.read_csv("https://raw.githubusercontent.com/owid/covid-19-data/master/public/data/owid-covid-data.csv")

uk_data = data[data['location'] == 'United Kingdom']
new_cases = uk_data['new_cases']
uk_data['date']

sample_frequency = 11
uk_dates = uk_data['date'][:-1: sample_frequency]

def trimmedDates(array):
	resultArray = []
	for date in array:
		date = date[-5:]
		resultArray.append(date)
	return resultArray

plt.plot(uk_data['date'], new_cases)
plt.xticks(np.arange(0, len(uk_data['date']), sample_frequency), trimmedDates(uk_dates), fontsize=5, rotation=75)
plt.show()

Run the project, and you should get something like the following: Thank You for reading! Happy Coding!

• 2023 1
• 2021 2