Introduction to Pandas¶
- It is a python 3rd party library
- Used for data analysis and visualization
- Part of Anaconda python distribution
- Best used with Jupyter notebook, can be used with regular python programs
- Main feature is the Data Frame
In [1]:
# Load the pandas library to let python know you will use it
import pandas as pd
What is a Data Frame?¶
- Its a data structure, like lists and dictionaries
- Consists of rows and columns, similar to SQL tables and excel spreadsheets
- Columns are attributes or variables
- Rows are records or single observations
- Operations are typically performed on columns
- Has both numeric and named indexing
Tidy Data¶
- Standard form of organizing data in dataframe such that:
- Each variable forms a column
- Each row forms a row
- Each table is an observational unit (level of analysis)
- Required reading: Tidy Data, by Hadley Wickham
Importance of Tidy Data¶
- Most tools we will use assume that data is tidy
- Collected data is likely messy or non-tidy, need to learn how to reshape it
- We will start with tidy data in our analysis
- Will learn how to reshape when we start data collection
Loading data into a data frame¶
- Data is usually loaded/collected from an external source, like an api, website, csv, or excel file, and might not be well organized.
- We will start with well organized and behaved data to get to know how to use Pandas
- Download the weather data set from vega-dataset (right click and save as)
- Place it in the same directory as the jupyter notbook you are working on
In [5]:
# load the data using pandas library
# do you remember what was pd?
pd.read_csv("weather.csv")
# Jupter notebook tip:
# type: pd.
# then hit tab, see what happens
# try also: pd.read_ (then hit tab)
Out[5]:
Loading File From URL¶
If the CSV file is downloadable from a url, you can put the URL in place of the file name:
In [37]:
my_df = pd.read_csv("https://raw.githubusercontent.com/vega/vega-datasets/gh-pages/data/weather.csv")
my_df
Out[37]:
Now it is your turn¶
Download airport.csv then load it into the notebook
Remember: Right click on the link and select save target as
In [ ]:
# Your turn
# Load: https://github.com/vega/vega-datasets/raw/gh-pages/data/cars.json
# Into: cars_df
How to work with the data?¶
- You must place it in a variable so you can refer to it
- The current data was displayed and not assigned to a variable, so you cannot use it
- Assign it to a variable named my_df
In [3]:
my_df = pd.read_csv("weather.csv")
In [ ]:
# Your turn: Load airports.csv into airports_df
Let us discover how the data looks like¶
We examine the top and bottom records of the dataframe to get an idea of what the data looks like
In [9]:
my_df.head()
Out[9]:
In [ ]:
# You can pass a number in the head() method to show more data
# show 10 items (try it)
# do the same for airports_df
In [13]:
# To know which columns are available use the columns attribute
my_df.columns
Out[13]:
In [ ]:
# Your turn: explore the columns for airports_df
Data types¶
- Each column will have its own data type
- Remember, variables will be in columns
- Observations in rows
- Use dtypes attribute of to discover columns and datatypes
- OOP: What is the difference between a function, a method, an attribute, and a variable?
In [18]:
my_df.dtypes
Out[18]:
In [ ]:
# Your turn: Find out the data types for the airports_df column
In [18]:
# Pandas uses data types provided by numpy
# load numpy
import numpy as np
# convert the column to datetime
my_df.date.astype(np.datetime64)
Out[18]:
In [5]:
# an alternative way to do it is using
pd.to_datetime(my_df.date).head() # do you remember head method?
Out[5]:
In [6]:
# now let us examine the date column
my_df.date.head()
# why is it still of type object?
# How to fix it?
Out[6]:
In [ ]:
# just like the dataframe, the command creates a copy
# but does not store it
# We need to replace the old date column with the new one
my_df.date = my_df.date.astype(np.datetime64)
In [23]:
# check the types
my_df.dtypes
Out[23]:
In [2]:
# Your turn: examine the airports_df dataframe
# are there any date columns that you can convert?
# Check then numeric columns, what should their data type be?
Why convert an object column into a date column?¶
- As you will find out later, pandas can do more fancy things if it knows the column is a date
- For example:
- Sort
- Filter based on date range
- Date arethmatic
- Always make sure date/time columns have the correct data type
Indexing Columns¶
- Using square brackets [ ]
- Using dot notation .
In [7]:
# a single column is known as a series
my_df['location']
Out[7]:
In [8]:
# Some methods that work on Dataframes also work on Series
my_df['location'].head()
Out[8]:
In [13]:
# Dot notation to access series
my_df.location.head()
Out[13]:
In [ ]:
# Your turn: Try to index the columns for airports_df using square brackets and dot notation
# Use head() to get an idea of what the data is
In [14]:
# Descriptive statistics
my_df['location'].describe()
Out[14]:
In [15]:
# works also on dataframe
my_df.describe()
Out[15]:
In [26]:
# Different data types will have different descriptives
my_df['date'].describe()
Out[26]:
In [21]:
my_df.precipitation.describe()
Out[21]:
In [ ]:
# Your turn: Use describe() on airports_df
# Which columns are included in describe?
# Try it on the columns that were excluded:
# Why were these columns excluded?
You can also plot a dataframe¶
- Pandas will try to show it in the best way possible
- Plotting from dataframe is very simplistic and used for quick univariate exploration
In [ ]:
my_df.plot()
In [29]:
# You need to tell pandas that you want to display plots in the notebook
%matplotlib inline
In [31]:
# now try it
my_df.plot()
Out[31]:
Don't forget!¶
Always include in your notebook:
# this is the first cell in your notebook
import pandas as pd
%matplotlib inline # dont forget this
In [32]:
# It's more meaningful to plot Series
my_df['precipitation'].plot()
Out[32]:
In [9]:
# Remember plots show change from one observation to the next
my_df['wind'].plot()
# in some cases it might not be useful
Out[9]:
In [40]:
# you can try a histogram
my_df['wind'].hist()
# Which is useful to know distributions
Out[40]:
How can you find out if percipitation is usually high in the year or low?¶
In [ ]:
# Your turn:
In [45]:
# Sometime pandas cannot plot it
my_df['location'].plot()
In [ ]:
# Your turn: try to plot the columns in airport_df using either plot() or hist()
# What can you find out about the data?
In [49]:
# Such variables are usually categorical and you can get frequencies like so
my_df['location'].value_counts()
Out[49]:
In [3]:
# Your turn: Examine the columns for airports_df
# what would be the best columns to check frequencies for?
# try it:
# The best columns are:
# The reason frequencies is best calculated on them is because:
# What did you find out about your data?
# try to plot value_count(), how do you do it? what do you get?
Are data frames immutable?¶
- Yes, however, all operations that change values will produce a copy and not change the original
- You have to use assignment to change columns or dataframes
- So be careful!
Finding Missing Value (Nulls or Nans)¶
- Missing values are usually represented by:
- The Python Null if the value doesn't exists
- Numpy nan if the value is Not a Number (like zero division)
- Use isnull() value to find if any value is null in the DataFrame:
In [26]:
my_df.isnull()
Out[26]:
In [ ]:
# Its a Dataframe containing boolean values
# Use describe()
# or use value_counts() on each column to count the Nulls
Call any() on isnull() to see which columns have null values¶
In [11]:
my_df.isnull().any()
Out[11]:
Call any() again to see if the dataframe has any null value¶
In [14]:
my_df.isnull().any().any()
Out[14]:
Duplicates¶
- Search for any repeated values
- use duplicated()
- Note that this looks at all the columns in the record
- You can pass a list of column names to check duplication based on
In [28]:
my_df.duplicated()
Out[28]:
In [33]:
# Try to pass location as the duplication column and see what happens
my_df.duplicated("location").value_counts()
# what is value_counts?
Out[33]:
In [34]:
# now try it with both location and weather columns
my_df.duplicated(["location", "weather"]).value_counts()
Out[34]: