Senior Data Engineer

Job Description

• Data Warehouse, Lake and Lakehouse architecture patterns.
• Distributed data processing using frameworks such as Spark or Flink.
• Designing and supporting batch and near-real-time ingestion pipelines.
• Building incremental, idempotent, and fault-tolerant data pipelines.
• Data quality, reconciliation, and observability practices.
• Metadata, lineage, governance, and access control concepts.
• Analytical data modelling and efficient data structures for warehouse and large-scale query workloads.
• Medallion architecture such as Bronze, Silver, and Gold layers.
• Open table formats such as Iceberg.
• Schema evolution, partitioning strategies, file optimisation, and storage layout tuning.
• Event-driven or streaming platforms such as Kafka, Pulsar, or Redpanda.
• Columnar or high-performance analytical platforms such as ClickHouse.
• CI/CD pipelines, deployment automation, and engineering standards for data workloads.
• Experience improving reliability, performance, and scalability across production data platforms.
• Familiarity with monitoring, alerting, observability, and operational support practices.
• Exposure to containerised or clustered environments such as Kubernetes, OpenShift, or similar platforms.
• Strong debugging capability across data, pipeline, compute, and platform layers.
• Strong sense of ownership and accountability.
• Comfortable making technical trade-offs while remaining pragmatic and hands-on.
• Excellent problem-solving and communication skills.
• Able to collaborate effectively with technical and non-technical stakeholders.
• Passionate about building scalable, maintainable, and well-documented systems.
• Committed to sharing knowledge and raising engineering standards across the team.

Responsibilities

Data Platform Engineering:

• Design, build, and improve critical batch and near-real-time data pipelines that support enterprise analytics and operational use cases.
• Develop reusable engineering patterns for ingestion, transformation, storage, and serving layers across the platform.
• Own important platform components and ensure they are reliable, scalable, and supportable in production.
• Contribute directly to the modernisation of legacy data workflows into stronger platform-aligned solutions.

Lakehouse Architecture & Scalable Processing

• Contribute to the design and evolution of the enterprise data lake or lakehouse platform.
• Implement and refine engineering standards for storage layout, transformation patterns, and data processing frameworks.
• Optimise partitioning, schema evolution, and file organisation to improve performance and maintainability.
• Build and support distributed data processing solutions using modern frameworks and platform tooling.

Data Quality, Reliability & Governance

• Design and implement data quality, reconciliation, and observability controls for critical datasets and platform flows.
• Ensure key pipelines and datasets meet expectations for freshness, completeness, accuracy, and recoverability.
• Strengthen metadata, lineage, and documentation practices across the platform.
• Work with governance and security stakeholders to support compliant, well-controlled data management practices.

Technical Leadership & Collaboration

• Partner with BI, analytics, software engineering, product, and business stakeholders to understand and support data use cases.
• Translate business and platform requirements into scalable, well-engineered technical solutions.
• Provide hands-on technical leadership during planning, design, implementation, and operational improvement.
• Mentor intermediate and junior data engineers through code reviews, design guidance, and practical knowledge-sharing.

Continuous Improvement & Innovation

• Identify opportunities to improve platform performance, resilience, scalability, and cost efficiency.
• Drive automation, standardisation, and maintainability across the data platform.
• Evaluate new tools or patterns where they provide clear value to the platform or team.
• Help grow the maturity of BET Software's data engineering capability over time.

Tech Environment

• The platform may include a combination of established and modern technologies such as: SQL Server, Python, Spark, Flink, Airflow, Object storage, Open format Tables, Kafka / Redpanda, ClickHouse or similar columnar analytical stores, Git and CI/CD tooling, Kubernetes, OpenShift, or similar runtime environments.

Qualifications

• Degree or diploma in IT, Computer Science, Engineering, or a related technical discipline.
• 6+ years of experience in data engineering, ETL/ELT development, or data platform engineering.
• Strong hands-on SQL expertise, including advanced performance tuning, query optimisation, indexing strategies, and efficient analytical data design.
• Proven experience building and operating modern data platform components in production environments.
• Strong experience working with object storage in cloud or on-premises environments.
• Experience with workflow orchestration platforms such as Airflow, SQL Server Agent, or similar.

Technical & Architectural Skills

• Data Warehouse, Lake and Lakehouse architecture patterns.
• Distributed data processing using frameworks such as Spark or Flink.
• Designing and supporting batch and near-real-time ingestion pipelines.
• Building incremental, idempotent, and fault-tolerant data pipelines.
• Data quality, reconciliation, and observability practices.
• Metadata, lineage, governance, and access control concepts.
• Analytical data modelling and efficient data structures for warehouse and large-scale query workloads.

Experience In The Following Areas Is Highly Valuable

• Medallion architecture such as Bronze, Silver, and Gold layers.
• Open table formats such as Iceberg.
• Schema evolution, partitioning strategies, file optimisation, and storage layout tuning.
• Event-driven or streaming platforms such as Kafka, Pulsar, or Redpanda.
• Columnar or high-performance analytical platforms such as ClickHouse.
• CI/CD pipelines, deployment automation, and engineering standards for data workloads.

Platform & Engineering Practices

• Experience improving reliability, performance, and scalability across production data platforms.
• Familiarity with monitoring, alerting, observability, and operational support practices.
• Exposure to containerised or clustered environments such as Kubernetes, OpenShift, or similar platforms.
• Strong debugging capability across data, pipeline, compute, and platform layers.

Personal Attributes

• Strong sense of ownership and accountability.
• Comfortable making technical trade-offs while remaining pragmatic and hands-on.
• Excellent problem-solving and communication skills.
• Able to collaborate effectively with technical and non-technical stakeholders.
• Passionate about building scalable, maintainable, and well-documented systems.
• Committed to sharing knowledge and raising engineering standards across the team.

Living the Spirit

• Engages in cross-functional collaboration and problem solving while contributing to an inclusive team culture.
• Supports a culture of adaptability, accountability, and shared success across the department and wider business.
• Shows up authentically and contributes to team outcomes by working effectively with diverse colleagues and perspectives.
• Approaches challenges as opportunities to learn, improve, and help others grow.