Python / Spark / AI Developer

Our client is seeking an experienced Python/Spark/AI Developer to help modernise data platforms by replatforming legacy T-SQL workloads onto Spark and Delta Lake pipelines. The successful candidate will work across data engineering, API development and AI-enabled capabilities, with a strong emphasis on type-safe, well-tested and specification-driven Python development.

Skills Required

• 4+ years of professional Python development experience.
• 2+ years of production experience building Spark or comparable distributed data pipelines.
• Strong Python 3.12 knowledge, including type hints, Pydantic, ABC-based provider patterns and modern packaging tools such as uv or Poetry.
• Experience with Click or similar CLI frameworks.
• Strong Apache Spark/PySpark experience, including DataFrame API, Spark SQL, partitioning, performance tuning and driver/executor architecture.
• Understanding of Spark Connect.
• Production experience with Delta Lake or equivalent lakehouse formats such as Iceberg or Hudi.
• Strong SQL skills and the ability to faithfully translate legacy T-SQL and stored-procedure logic into Spark SQL.
• Strong understanding of hashing, surrogate keys and set-based deduplication.
• Hands-on pytest experience, including fixtures, markers and unit, integration and end-to-end testing.
• Experience developing REST APIs using FastAPI or an equivalent framework.
• Practical experience with Docker and Compose-based development environments.
• Strong Git and CI practices, including PR workflows, linting, type checking and automated test gates .

Should you meet the requirements for this position, please email your CV to ***email_hidden***. You can also contact us on 031 350 4019 or alternatively you can visit our website www.stand-outstaffing.co.za

Should you not hear from us within 3 days, please consider your application unsuccessful.