Cleaning data

What does "clean" means¶

Data cleaning makes a dataset more consistent and useful for a specific question. Poor quality can distort a campaign comparison, hide an operational issue, or lead a team to act on the wrong customers.

Cleaning is not about making every column look tidy. It is about documenting decisions so someone else can understand what changed and why.

Check before changing¶

Inspect a real campaign file¶

Use Polars to check the schema, the number of records, missing values, and whether the send ID is unique. The file includes expected missing values in later-stage fields such as time_spent and transaction_id.

import polars as pl

campaigns = pl.read_csv(
    "docs/kedge_data_analytics_and_AI/data/marketing_campaigns.csv",
    try_parse_dates=True,
)

print(f"Shape: {campaigns.shape}")
print(f"send_id unique: {campaigns['send_id'].n_unique() == campaigns.height}")
print("Missing values in selected columns:")
print(
    campaigns.select(
        pl.col("time_spent").null_count(),
        pl.col("visit_id").null_count(),
        pl.col("transaction_id").null_count(),
    )
)

Shape: (7000, 18)
send_id unique: True
Missing values in selected columns:
shape: (1, 3)
┌────────────┬──────────┬────────────────┐
│ time_spent ┆ visit_id ┆ transaction_id │
│ ---        ┆ ---      ┆ ---            │
│ u32        ┆ u32      ┆ u32            │
╞════════════╪══════════╪════════════════╡
│ 6016       ┆ 6016     ┆ 6838           │
└────────────┴──────────┴────────────────┘

The missing counts are not automatically errors. Compare them with the event flow: a recipient must click before time spent is recorded, and a purchase is less common than a message send.

Practice on deliberately messy rows¶

This small example includes a repeated send, inconsistent channel labels, a meaningful blank, and a suspicious time value. We standardise the field names and channel labels, flag the suspicious value for review, and remove only the repeated send_id.

import polars as pl

raw = pl.DataFrame(
    {
        "Send ID": ["S001", "S002", "S002", "S003"],
        "Channel": ["Email", " e-mail ", "email", "SOCIAL"],
        "Time spent (seconds)": [120, None, None, 9999],
        "Clicked": [True, False, False, True],
    }
)

prepared = (
    raw.rename(
        {
            "Send ID": "send_id",
            "Channel": "channel",
            "Time spent (seconds)": "time_spent_seconds",
            "Clicked": "clicked",
        }
    )
    .with_columns(
        pl.col("channel")
        .str.strip_chars()
        .str.to_lowercase()
        .replace({"e-mail": "email"}),
        (pl.col("time_spent_seconds") > 3600).alias("time_needs_review"),
    )
    .unique(subset=["send_id"], keep="first", maintain_order=True)
)

print(prepared)

shape: (3, 5)
┌─────────┬─────────┬────────────────────┬─────────┬───────────────────┐
│ send_id ┆ channel ┆ time_spent_seconds ┆ clicked ┆ time_needs_review │
│ ---     ┆ ---     ┆ ---                ┆ ---     ┆ ---               │
│ str     ┆ str     ┆ i64                ┆ bool    ┆ bool              │
╞═════════╪═════════╪════════════════════╪═════════╪═══════════════════╡
│ S001    ┆ email   ┆ 120                ┆ true    ┆ false             │
│ S002    ┆ email   ┆ null               ┆ false   ┆ null              │
│ S003    ┆ social  ┆ 9999               ┆ true    ┆ true              │
└─────────┴─────────┴────────────────────┴─────────┴───────────────────┘

The example flags a value for review rather than deleting it. The duplicate rule uses send_id because that field identifies one send in this example. If records describe visits, use a visit key instead.

Name and code variables consistently¶

Prefer clear names such as sent_date, discount_rate, and time_spent_seconds. Keep one case convention, avoid spaces and unexplained abbreviations, and include units where they affect interpretation.

For categorical variables, choose one spelling per category. A small codebook could define email, ads, and social as values for channel; it should also define whether a value such as unknown means missing, unavailable, or not applicable.

Continue¶

Apply the checks in a practical lab