Fabric-Pandas-Migrator

Overview

A GitHub-ready, review-first codemod for bulk migration of Python files that use pandas toward PySpark workloads in Microsoft Fabric. Pandas and PySpark have different execution models; no codemod can safely infer every assumption. The tool surfaces risky patterns for manual review.

Review-first design

1. discovers every .py file
2. detects pandas usage
3. copies the repository to a separate output directory
4. applies deterministic conversions
5. adds a Fabric Spark bootstrap where needed
6. produces JSON, CSV and Markdown reports
7. flags unsupported/risky patterns for manual review
8. optionally fails CI when unresolved findings remain

Migration modes

pandas-api (recommended first pass): Changes import pandas as pd to import pyspark.pandas as pd. Preserves pandas-shaped API while moving execution to Spark.

native: Converts common patterns to native PySpark DataFrame code, e.g. read_csv to spark.read.csv, df.head(n) to df.limit(n), df.to_parquet to df.write.mode("overwrite").parquet(path).

Quick start

python -m venv .venv
pip install -e .
fabric-pandas-migrate scan ./legacy-app
fabric-pandas-migrate migrate ./legacy-app --output ./fabric-app --mode native
fabric-pandas-migrate check ./fabric-app --fail-on-review
# Dry run:
fabric-pandas-migrate migrate ./legacy-app --mode native --dry-run

Output & CI

The output root contains transformed source files, migration-report.json, migration-report.csv and MIGRATION_REPORT.md. Exit codes: 0 success, 1 operational error, 2 unresolved review findings with --fail-on-review.

Run fabric-pandas-migrate check ./fabric-app --fail-on-review in CI to block merges with unresolved findings.

Safety & Microsoft Fabric deployment

• Default behavior never overwrites the source tree
• Generated code is formatted by LibCST and remains parseable
• The tool never executes source files
• Deploy to Fabric: create a Lakehouse and Notebook/Spark Job Definition, attach Lakehouse, upload migrated modules as notebook resources or build a wheel and add to a Fabric Environment, use OneLake/Lakehouse paths and Delta tables

License

MIT. See LICENSE.