Advanced: Using AI as a Data Science Pair Programmer

Everything in this section has been about one-shot prompts. The final shift is treating AI as a continuous collaborator — present from project kickoff to deployment, with shared context, a shared style guide, and explicit roles. This topic brings the lessons together into a working pair-programming practice.

1. Introduction

Pair programming with a human colleague has rules: a driver and a navigator, regular role swaps, shared keyboard etiquette. Pair programming with AI has its own rules, and they are different. The AI never tires, never gets defensive, and is happy to rewrite the same function ten different ways. But it has no memory between sessions, no awareness of your codebase, and no intuition about which conventions your team follows. To make AI an effective pair, you must give it a persistent context and clear roles — and this topic shows you exactly how to do that across an entire data science project.

2. The Concept Explained

A productive AI pair programmer relationship has three pillars: shared context, defined roles, and tight feedback loops. Shared context is everything the AI needs to know about your project: the codebase layout, the dataset schemas, the conventions, the recent decisions. Defined roles tell the AI whether to be the driver (writing code) or the navigator (reviewing, questioning, suggesting alternatives) at any given moment. Tight feedback loops mean every block of generated code gets reviewed, tested, and refined in conversation before the next one.

An analogy: think of AI as a brilliant graduate student joining your team for a six-month project. On day one they know everything about Pandas and nothing about your domain. By month two — if you brief them well, give them context, and give them feedback — they are operating at near-senior level on your specific problems. The difference between a great graduate and a frustrating one is rarely raw ability; it is how well they have been onboarded.

The project context document

The single highest-leverage habit is maintaining a "context document" for each project. It includes the dataset schemas, the modelling target, the coding conventions, the libraries in use, the directory layout, and a running log of recent decisions. Paste the document into the first prompt of every working session. The AI immediately operates as if it had been on the project for weeks.

3. The Problem Without This Technique

Help me with my data science project.

Act as my senior data science pair programmer.

PROJECT CONTEXT
- Repo layout:
  /src/data        (loaders, validators)
  /src/features    (feature builders)
  /src/models      (training, evaluation)
  /notebooks       (exploration only — not committed)
  /tests           (pytest)
- Stack: Python 3.11, Pandas 2.1, scikit-learn 1.4,
  XGBoost 2.0, dbt 1.7, Airflow 2.8, BigQuery.
- Style: PEP8, ruff, black-formatted (line length 100),
  type hints required on public functions, Google-style
  docstrings.
- Target dataset: customers_df (schema attached below).
- Modelling target: churn_within_90d (binary).
- Recent decisions:
   * Switched primary metric from ROC-AUC to PR-AUC
     last week (class imbalance ~7%).
   * Moved feature_engineering.py to use polars-lazy
     for the rolling aggregates.

YOUR ROLES (use whichever fits the moment):
- DRIVER: write code that fits the project conventions
  exactly; emit one file at a time with explicit path.
- NAVIGATOR: review my code; flag risks, suggest
  alternatives, ask clarifying questions.
- REVIEWER: critique a PR diff for bugs, leakage,
  test gaps, and style.

For this session, please START as NAVIGATOR.
I will paste a draft feature builder. You critique it
before we write any new code together.

4. The Solution

The pattern is: project context document → defined roles → tight loop → fresh context for each new session. The role assignment is the most underused technique. When you tell the AI to be the navigator, it stops generating code and starts asking the questions that catch bugs early. When you tell it to be the reviewer, it produces critiques that read like senior PR feedback. Default to driver only when you genuinely want code.

For longer projects, maintain a short AI_NOTES.md file in the repository — a running log of decisions and conventions. Paste it at the start of each session. Over weeks, it becomes the canonical context that turns AI into a senior team member who never forgets a decision.

5. Step-by-Step Breakdown

1. Create a project context document. Repo layout, stack, style guide, target dataset schema, modelling target, recent decisions. Keep it under 300 words.
2. Define explicit roles. Driver (writes code), navigator (reviews and challenges), reviewer (critiques PRs). Switch roles deliberately within a session.
3. Start each session with the context plus the role. One paste, no shortcuts. The five seconds it takes saves you twenty minutes of re-explaining.
4. Iterate in tight loops. Generate one file or one function, review it, refine it, commit, then start the next. Never accept five files in one shot.
5. Use AI to write tests, not just code. "Write pytest tests for this function. Include three positive cases, three edge cases, and one failure case." Tests are where AI shines and where humans lose patience.
6. Capture decisions back into the context document. When you make a non-obvious choice, ask the AI to draft a one-line entry for AI_NOTES.md. Future sessions inherit the wisdom.

Tip: When debugging together, paste both the traceback and the relevant file. Then prompt: "Diagnose this failure. List three plausible root causes in order of likelihood, and for each, the smallest experiment to confirm or rule it out." This turns AI into a calm, systematic debugging partner instead of a guess-and-check engine.

6. Practice Exercises

7. Key Takeaways