Seminar 005: Working with Pandas¶

---

Notes¶

In [ ]:

Working with Pandas¶

In [ ]:

import pandas as pd

import pandas as pd

In [ ]:

# Fetch a dataset from the web
url = "https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv"

# Fetch a dataset from the web url = "https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv"

You can display a dataset as a beautiful table inside a Jupyter Notebook by writing the variable name as shown below.

In [ ]:

# Dataframe has to be the last line and without a print statement

# Dataframe has to be the last line and without a print statement

Data analysis¶

Pandas alone already provides many options to analyse your data. You can calculate various statistics and even plot data (although not visually pleasing). Most of the time, when working with numerical data, using the df.describe() method already gives a great overview.

In [ ]:

# General description
df.describe()

# General description df.describe()

In [ ]:

# Correlation matrix (Pearson)
df.corr("pearson", numeric_only=True)

# Correlation matrix (Pearson) df.corr("pearson", numeric_only=True)

In [ ]:

# Correlation matrix (Spearman)
df.corr("spearman", numeric_only=True)

# Correlation matrix (Spearman) df.corr("spearman", numeric_only=True)

Slicing and conditional sorting¶

Pandas provides a convenient way to extract specific data points by using logical operators within square brackets.

In [ ]:

# Only extract data for "setosa" species

# Only extract data for "setosa" species

In [ ]:

# Filter data to only include rows where sepal_length is greater than 7 and petal_length is greater than 6

# Filter data to only include rows where sepal_length is greater than 7 and petal_length is greater than 6

Visualisation¶

For a quick visualization of data, Pandas can be sufficient, but bear in mind that the visualizations are basic. On the other hand, Pandas is capable of Parallel Coordinates, which is very powerful!

In [ ]:

# Short-hand box plot from a dataframe.plot

# Short-hand box plot from a dataframe.plot

In [ ]:

# Multi-dimensional plotting (Parallel coordinates)

from pandas.plotting import parallel_coordinates

# Multi-dimensional plotting (Parallel coordinates) from pandas.plotting import parallel_coordinates

Exporting data¶

Aside from csv Pandas can also read XLSX (Excel) and many other formats to load and write a dataset.

In [ ]:

# Viewable in the browser
df.to_html("Dataset.html")

# Viewable in the browser df.to_html("Dataset.html")

In [ ]:

# Well known file format
df.to_xml("Dataset.xml")

# Well known file format df.to_xml("Dataset.xml")

In [ ]:

# Can be used and edited in Excel
df.to_excel("Dataset.xlsx", index=False)

# Can be used and edited in Excel df.to_excel("Dataset.xlsx", index=False)

In [ ]:

# ... and can be read again
pd.read_excel("Dataset.xlsx", sheet_name="Sheet1")

# ... and can be read again pd.read_excel("Dataset.xlsx", sheet_name="Sheet1")