Introduction to Pandas¶
- It is a python 3rd party library
- Used for data analysis and visualization
- Part of Anaconda python distribution
- Best used with Jupyter notebook, can be used with regular python programs
- Main feature is the Data Frame
In [1]:
# Load the pandas library to let python know you will use it
import pandas as pd
What is a Data Frame?¶
- Its a data structure, like lists and dictionaries
- Consists of rows and columns, similar to SQL tables and excel spreadsheets
- Columns are attributes or variables
- Rows are records or single observations
- Operations are typically performed on columns
- Has both numeric and named indexing
Loading data into a data frame¶
- Data is usually loaded from an external source, like a csv or excel file.
- Download the weather data set from vega-dataset (right click and save as)
- Place it in the same directory as the jupyter notbook you are working on
In [5]:
# load the data using pandas library
# do you remember what was pd?
pd.read_csv("weather.csv")
# Jupter notebook tip:
# type: pd.
# then hit tab, see what happens
# try also: pd.read_ (then hit tab)
Out[5]:
Now it is your turn¶
Download airport.csv then load it into the notebook
Remember: Right click on the link and select save target as
In [ ]:
How to work with the data?¶
- You must place it in a variable so you can refer to it
- The current data was displayed and not assigned to a variable, so you cannot use it
- Assign it to a variable named my_df
In [3]:
my_df = pd.read_csv("weather.csv")
In [ ]:
# Your turn: Load airports.csv into airports_df
Let us discover how the data looks like¶
In [9]:
my_df.head()
Out[9]:
In [ ]:
# You can pass a number in the head() method to show more data
# show 10 items (try it)
# do the same for airports_df
In [13]:
# To know which columns are available use the columns attribute
my_df.columns
Out[13]:
In [ ]:
# Your turn: explore the columns for airports_df
Data types¶
- Each column will have its own data type
- Remember, variables will be in columns
- Observations in rows
- Use dtypes attribute of to discover columns and datatypes
- OOP: What is the difference between a function, a method, an attribute, and a variable?
In [18]:
my_df.dtypes
Out[18]:
In [ ]:
# Your turn: Find out the data types for the airports_df column
In [18]:
# Pandas uses data types provided by numpy
# load numpy
import numpy as np
# convert the column to datetime
my_df.date.astype(np.datetime64)
Out[18]:
In [5]:
# an alternative way to do it is using
pd.to_datetime(my_df.date).head() # do you remember head method?
Out[5]:
In [6]:
# now let us examine the date column
my_df.date.head()
# why is it still of type object?
# How to fix it?
Out[6]:
In [ ]:
# just like the dataframe, the command creates a copy
# but does not store it
# We need to replace the old date column with the new one
my_df.date = my_df.date.astype(np.datetime64)
In [23]:
# check the types
my_df.dtypes
Out[23]:
In [ ]:
# Your turn: examine the airports_df dataframe
# are there any date columns that you can convert?
# Check then umeric columns, what should their data type be?
Why convert an object column into a date column?¶
- As you will find out later, pandas can do more fancy things if it knows the column is a date
- For example:
- Sort
- Filter based on date range
- Date arethmatic
- Always make sure date/time columns have the correct data type
Indexing Columns¶
- Using square brackets [ ]
- Using dot notation .
In [7]:
# a single column is known as a series
my_df['location']
Out[7]:
In [8]:
# Some methods that work on Dataframes also work on Series
my_df['location'].head()
Out[8]:
In [13]:
# Dot notation to access series
my_df.location.head()
Out[13]:
In [ ]:
# Your turn: Try to index the columns for airports_df using square brackets and dot notation
# Use head() to get an idea of what the data is
In [14]:
# Descriptive statistics
my_df['location'].describe()
Out[14]:
In [15]:
# works also on dataframe
my_df.describe()
Out[15]:
In [26]:
# Different data types will have different descriptives
my_df['date'].describe()
Out[26]:
In [21]:
my_df.precipitation.describe()
Out[21]:
In [ ]:
# Your turn: Use describe() on airports_df
# Which columns are included in describe?
# Try it on the columns that were excluded:
# Why were these columns excluded?
You can also plot a dataframe¶
Pandas will try to show it in the best way possible
In [ ]:
my_df.plot()
In [29]:
# You need to tell pandas that you want to display plots in the notebook
%matplotlib inline
In [31]:
# now try it
my_df.plot()
Out[31]:
Don't forget!¶
Always include in your notebook:
# this is the first cell in your notebook
import pandas as pd
%matplotlib inline # dont forget this
In [32]:
# It's more meaningful to plot Series
my_df['precipitation'].plot()
Out[32]:
In [9]:
# Remember plots show change from one observation to the next
my_df['wind'].plot()
# in some cases it might not be useful
Out[9]:
In [40]:
# you can try a histogram
my_df['wind'].hist()
# Which is useful to know distributions
Out[40]:
How can you find out if percipitation is usually high in the year or low?¶
In [ ]:
# Your turn:
In [45]:
# Sometime pandas cannot plot it
my_df['location'].plot()
In [ ]:
# Your turn: try to plot the columns in airport_df using either plot() or hist()
# What can you find out about the data?
In [49]:
# Such variables are usually categorical and you can get frequencies like so
my_df['location'].value_counts()
Out[49]:
In [ ]:
# Your turn: Examine the columns for airports_df
# what would be the best columns to check frequencies for?
# try it:
# The best columns are:
# The reason frequencies is best calculated on them is because:
# What did you find out about your data?
Are data frames immutable?¶
- Yes, however, all operations that change values will produce a copy and not change the original
- You have to use assignment to change columns or dataframes
- So be careful!