Data Engineering Lead

Our client is seeking an experienced Data Engineering Lead to design, build, and lead enterprise-scale data platforms that enable business intelligence, advanced analytics, reporting, and data-driven decision-making.

The successful candidate will provide technical leadership across the full data engineering lifecycle, ensuring scalable, secure, reliable, and high-performing data solutions. This role requires close collaboration with business stakeholders, solution architects, BI teams, governance specialists, and software engineering teams to deliver trusted and analytics-ready data assets.

The ideal candidate is a hands-on technical leader with extensive experience in cloud data platforms, ETL/ELT development, data architecture, SQL optimization, and modern data engineering practices.

Key Responsibilities

Data Engineering Leadership

Lead the design, development, implementation, and continuous improvement of enterprise-grade data engineering solutions.

Architect scalable, secure, reusable, and high-performance data pipelines supporting batch and near real-time processing.

Provide technical leadership across multiple data engineering initiatives and mentor junior and intermediate engineers.

Establish and enforce enterprise data engineering standards, frameworks, and best practices.

Drive innovation and continuous improvement across the organisation's data platform capabilities.

Data Platform Architecture

Design robust ingestion frameworks for structured, semi-structured, and unstructured data.

Develop scalable storage architectures supporting data warehousing and lake house environments.

Implement efficient ETL and ELT processing frameworks.

Optimize cloud-native data architectures for performance, scalability, reliability, and cost efficiency.

Design resilient solutions supporting analytics, reporting, AI, and machine learning initiatives.

Data Engineering & Development

Build, maintain, and optimize enterprise data pipelines.

Develop complex SQL queries, stored procedures, and transformation logic.

Design and maintain reusable data integration components.

Perform data transformation, cleansing, enrichment, and validation processes.

Ensure consistent data availability and integrity across enterprise platforms.

Data Governance & Quality

Embed data quality controls throughout the engineering lifecycle.

Implement automated validation, monitoring, lineage, metadata management, and observability.

Ensure compliance with organizational governance frameworks and regulatory requirements.

Secure sensitive and confidential information through appropriate engineering controls.

Source-to-target mappings

Data dictionaries

Data lineage

Technical specifications

Data models

Pipeline documentation

Stakeholder Management

Business stakeholders

Business Intelligence teams

Data Analysts

Solution Architects

Application Development teams

Governance teams

Infrastructure teams

Translate business requirements into scalable technical solutions.

Communicate technical concepts clearly to both technical and non-technical audiences.

Provide regular project updates, technical recommendations, and risk assessments.

Operational Excellence

Monitor production data platforms.

Troubleshoot pipeline failures and performance issues.

Optimize processing times and infrastructure costs.

Lead root cause analysis and incident resolution.

Support production deployments and platform upgrades.

Requirements

Minimum Requirements

Education

Bachelor's Degree in one of the following

Computer Science

Information Systems

Information Technology

Software Engineering

Data Science

Engineering

Mathematics

Statistics

Or a related discipline

Relevant postgraduate qualifications will be advantageous.

Experience

8–10+ years' experience in Data Engineering.

Minimum 3 years leading technical data engineering teams.

Experience designing enterprise-scale cloud data platforms.

Proven experience building scalable ETL/ELT solutions.

Experience working within enterprise environments.

Strong experience delivering complex data integration solutions.

Experience supporting reporting, analytics, and data science initiatives.

Experience implementing data governance frameworks.

Experience in Financial Services, Mining, Retail, Telecommunications, or Consulting environments will be advantageous.

Technical Knowledge

Modern Data Engineering architectures

Enterprise Data Platforms

Data Warehousing

Lake house Architecture

ETL / ELT Frameworks

Data Integration

Cloud Computing

SQL Performance Optimization

Data Modelling

Metadata Management

Data Governance

Master Data Management

Data Quality

Data Lineage

CI/CD

Infrastructure as Code

Software Development Lifecycle (SDLC)

Information Security

Data Privacy Regulations

Technical Skills

Cloud Platforms

Experience with one or more

AWS

Microsoft Azure

Snowflake

Databricks

Programming & Query Languages

Advanced SQL

Python

Spark

PySpark

Data Engineering Technologies

Databricks

Azure Data Factory

AWS Glue

Snowflake

Apache Spark

Delta Lake

Airflow

Data Pipeline Orchestration Tools

Business Intelligence Tools

Experience with one or more

Power BI

Qlik Sense

QlikView

Tableau

Database Technologies

SQL Server

PostgreSQL

Oracle

Snowflake

Azure SQL

Amazon Redshift

Additional Technical Skills

Git

Azure DevOps

Terraform

CI/CD Pipelines

API Integration

REST Services

Data Security

Data Encryption

Behavioural Competencies

The successful candidate should demonstrate

Technical Leadership

Strategic Thinking

Analytical Thinking

Strong Problem-Solving Ability

Excellent Communication Skills

Stakeholder Management

Decision-Making Ability

Continuous Improvement Mindset

Innovation

Collaboration

Accountability

Attention to Detail

Adaptability

Results Orientation

Planning and Organizing

Mentoring and Coaching

Preferred Certifications

Microsoft

Microsoft Certified: Azure Data Engineer Associate

Azure Solutions Architect Expert

AWS

AWS Certified Data Engineer – Associate

AWS Certified Solutions Architect – Associate / Professional

Databricks

Databricks Certified Data Engineer Associate

Databricks Certified Data Engineer Professional

Snowflake

SnowPro Core Certification

Other

Certified Data Management Professional (CDMP)

Informatica Certification

TOGAF (Advantageous)

Key Success Measures

The successful candidate will be measured on

Delivery of scalable, secure data platforms.

Data pipeline reliability and availability.

Data quality and integrity.

Platform performance and optimization.

Successful project delivery within agreed timelines.

Stakeholder satisfaction.

Reduction in operational incidents.

Engineering automation and efficiency improvements.

Compliance with enterprise governance standards.

Mentoring and development of engineering team members.

Why Join This Opportunity?

This is an exciting opportunity to lead the development of modern enterprise data platforms within a collaborative and innovative environment. You will play a key role in shaping the organisation's data engineering capability, enabling business intelligence, advanced analytics, AI initiatives, and digital transformation through scalable, secure, and high-performing data solutions.