Data Understanding

Before starting any machine learning project, such as predicting house prices, identifying cats and dogs in images, or recommending movies you might like, we first need to face one of the most basic and critical steps:Data Understanding。

We can imagine data understanding as the process of a detective carefully examining all clues and files before investigating a case. If we do not understand the context, authenticity, and meaning of the clues (data), any subsequent reasoning (modeling) may be built on a wrong foundation.

Data understanding is the cornerstone of the entire machine learning process. It determines how we later clean the data, choose models, and ultimately affects the success or failure of the model.

What is Data Understanding?

Data understanding, as the name suggests, means gaining an in-depth understanding of the dataset in your hands. Its core goal is to answer the following questions:

  • What data do I have?(Structure and types of the data)
  • What is the data quality like?(Is the data clean, complete, and reliable?)
  • What is the data saying?(What patterns, relationships, and distributions are hidden in the data?)

This process does not involve complex code and algorithms; it is more about gaining an understanding of the data through observation, statistics, and visualization.

Core Steps and Tools for Data Understanding

We will use the most popular data analysis library in PythonPandasand the visualization libraryMatplotlib/Seabornfor demonstration.

Before starting, please install the following dependency libraries:

pip install pandas matplotlib seaborn

Description of Python's three major data analysis libraries:

  • pandas
    Core library for data processing, specializing in tabular data reading/writing, cleaning, filtering, and statistical calculations; supports Excel/CSV reading and provides the DataFrame table structure.
  • matplotlib
    Low-level basic plotting library, the foundation of Python visualization; can draw various basic charts such as line, bar, and scatter plots, with high flexibility but somewhat verbose code.
  • seaborn
    A high-level plotting library based on matplotlib, designed for statistical charts; comes with nice color schemes and can draw distribution plots, heatmaps, and box plots with one line of code.

Step 1: First Meeting - Loading and Overview

First, we need to load the data into the program and quickly browse its overall appearance.

To begin, first download the test data and put it in the same directory as the Python test code:

Test data download URL:https://raw.githubusercontent.com/uiuc-cse/data-fa14/gh-pages/data/iris.csv

If the githubusercontent network download fails, you can click the link directly to download:https://static.jyshare.com/download/iris.csv。

Example

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

# 1. Load data (here using the classic Iris dataset as an example; you can also load your own CSV file)
# Load from local file
df = pd.read_csv('iris.csv')

# Load from network
# url = "https://raw.githubusercontent.com/uiuc-cse/data-fa14/gh-pages/data/iris.csv"
# df = pd.read_csv(url)

# 2. View the first few rows of data - first impression
print("The first 5 rows of data:")
print(df.head())
print("\n" + "="*50 + "\n")

# 3. View overall information of the data: number of rows, number of columns, data types, memory usage
print("Basic information of the dataset:")
print(df.info())
print("\n" + "="*50 + "\n")

# 4. View the shape of the data (how many rows, how many columns)
print(f"Dataset shape: {df.shape}") # Output (rows, columns)
print(f"There are {df.shape)

Code Explanation：

  • df.head(): Like flipping through a book's table of contents, quickly view the first few rows of the data to understand what the data looks like.
  • df.info(): This is the "physical examination report" of the data. It will tell you:
    • The name of each column (Column）
    • The number of non-null values (Non-Null Count), so you can immediately discover whether there are missing values
    • Data types (Dtype), such asint64(integer),float64(float),object(text or mixed type)
  • df.shape: Directly obtain the dimensions of the data table.

After running the above code, the output is similar to the following:

数据的前5行：
   sepal_length  sepal_width  petal_length  petal_width species
0           5.1          3.5           1.4          0.2  setosa
1           4.9          3.0           1.4          0.2  setosa
2           4.7          3.2           1.3          0.2  setosa
3           4.6          3.1           1.5          0.2  setosa
4           5.0          3.6           1.4          0.2  setosa

==================================================

数据集的基本信息：
<class 'pandas.DataFrame'>

Step 2: Quality Check - Detecting Missing Values and Anomalies

Data is rarely perfect. Common "data diseases" includemissing values(some positions are empty) andoutliers(some numbers are abnormally large or abnormally small).

Example

# 1. Check missing values
print("Number of missing values per feature:")
print(df.isnull().sum())
print("\n" + "="*50 + "\n")

# If there are many missing values, you can calculate the missing proportion
missing_ratio = df.isnull().sum() / len(df) * 100
print("Missing value proportion per feature (%):")
print(missing_ratio)
print("\n" + "="*50 + "\n")

# 2. Check the statistical summary of numeric features - can reveal clues to outliers
print("Statistical description of numeric features:")
print(df.describe())

Code Explanation：

  • df.isnull().sum(): Calculate the total number of null values (NaN) in each column.
  • df.describe(): Generate a statistical summary of numeric columns, including:
    • count: count (can be used to double-check missing values)
    • mean: mean
    • std: standard deviation (how much the data fluctuates)
    • min: minimum value
    • 25%, 50%(median),75%: quartiles
    • max: maximum value
    • By observingminandmax, you can preliminarily judge whether there are outliers(for example, an age column containing 200 years old).

Step 3: In-depth Insight - Visualizing Distributions and Relationships

Text and numbers are abstract, while charts allow us to intuitively "see" the data. This is the most interesting part of data understanding.

Example

# Set chart style
sns.set(style="whitegrid")

# 1. Univariate distribution - understand the distribution of each feature itself
fig, axes = plt.subplots(2, 2, figsize=(12, 8)) # Create a 2x2 canvas
features = ['sepal_length'， 'sepal_width'， 'petal_length'， 'petal_width']
colors = ['skyblue'， 'lightgreen'， 'salmon'， 'gold']

for i, (ax, feature, color) in enumerate(zip(axes.flat, features, colors)):
    # Draw histogram (distribution) and kernel density estimate curve
    sns.histplot(df[feature], kde=True, ax=ax, color=color, bins=20)
    ax.set_title(f'Distribution of {feature}'， fontsize=14)
    ax.set_xlabel(feature)
    ax.set_ylabel('Frequency')

plt.tight_layout()
plt.show()

# 2. Box plot - view data distribution and outliers (more intuitive)
plt.figure(figsize=(10, 6))
# Select numeric columns to draw box plot
df_box = df.drop(columns=['species']) # Assume 'species' is a text label column, remove it first
sns.boxplot(data=df_box)
plt.title('Box plot of each numeric feature (view distribution and outliers)'， fontsize=14)
plt.xticks(rotation=45)
plt.show()

# 3. Relationships between variables - scatter plot matrix
print("\nDrawing scatter plot matrix of feature relationships... (This can help us discover associations between features.)")
# Use Seaborn's pairplot, the hue parameter can color by category (e.g., Iris species)
sns.pairplot(df, hue='species'， height=2.5)
plt.suptitle('Feature relationship scatter plot matrix (colored by species)'， y=1.02, fontsize=16)
plt.show()

# 4. Correlation heatmap - quantify linear relationships between features
plt.figure(figsize=(8, 6))
# Calculate the correlation coefficient between numeric features
numeric_df = df.select_dtypes(include=['float64'， 'int64'])
correlation_matrix = numeric_df.corr()
sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm'， center=0, square=True)
plt.title('Feature correlation heatmap'， fontsize=14)
plt.show()

Chart Explanation：

  • Histogram: Shows how the values of a feature (such as petal length) are distributed. Are they concentrated in a certain interval, or scattered?
  • Box plot：
    • The line in the middle of the box represents themedian。
    • The upper and lower boundaries of the box represent the25th (Q1) and 75th (Q3) percentiles。
    • The "whiskers" extending upward and downward typically represent the reasonable range (Q1 - 1.5 IQR to Q3 + 1.5 IQR).
    • Individual pointsare likelyoutliers！
  • Scatter plot matrix: View the relationship between any two features simultaneously. If the points form a band-like distribution, they may be correlated.
  • Correlation heatmap: Precisely represents the degree of linear correlation between two features using colors and numbers (-1 to 1).
    • 1: Perfect positive correlation (one increases, the other also increases)
    • -1: Perfect negative correlation (one increases, the other decreases)
    • 0: No linear relationship

Output of Data Understanding: A Data Investigation Report

After completing the above steps, you should be able to summarize a clear report about the current dataset, for example:

Investigation Report on the Iris Dataset

Data Overview: A total of 150 samples, 5 features (4 numeric features: sepal/petal length and width; 1 category label: species).

Data Quality: No missing values; all numeric features are within reasonable biological ranges; no obvious outliers found.

Data Insights：

  • Petal length (petal_length) and petal width (petal_width) have extremely high correlation (>0.96), which may indicate information redundancy.
  • Different Iris species are clearly distinguishable by petal size, and the scatter plot shows clear clustering.
  • Sepal width (sepal_width) distribution is approximately normal.
Other Extensions