Data Consultant (Databricks)

Department: Data Engineering

Employment Type: Full Time

Location: Cape Town, South Africa

Reporting To: Tom Bruce

Description

As a Senior Data Consultant specialising in Databricks, you’ll lead the way in delivering advanced solutions on Databricks while engaging with senior stakeholders to drive value for our clients.

You’ll play a pivotal role in our migration project to Databricks and build new, innovative use cases on the platform. You’ll also mentor team members, contribute to solution architecture, and spearhead internal initiatives to enhance Snap’s capabilities.

What you'll be doing...

Strategy & Solution Design

• Lead solution architecture design, contributing to proposal creation and project planning for Databricks-based initiatives.
• Proactively identify customer needs and propose innovative solutions leveraging Databricks to drive business impact.
• Build and nurture relationships with senior stakeholders, ensuring Snap remains a trusted partner.
  
  

Technical Delivery

• Drive the migration of legacy data workflows and pipelines to Databricks, ensuring seamless transition and minimal disruption.
• Design and implement efficient, scalable Databricks-based data solutions, leveraging Delta Lake and Spark for data processing.
• Develop robust data models (e.g., star schema, snowflake schema) tailored for Databricks use cases and optimize ETL/ELT processes.
• Build, automate, and monitor Databricks pipelines for managing structured and unstructured data at scale.
  
  

Performance Optimisation & Governance

• Analyse and optimize the performance of Databricks workflows and Spark jobs to ensure high availability and cost efficiency.
• Develop scalable, secure data architectures aligned with governance and compliance best practices.
  
  

Leadership & Collaboration

• Manage teams, fostering collaboration and delivering high-quality outcomes.
• Mentor junior engineers, enhancing their Databricks expertise and contributing to their professional growth.
• Collaborate with clients, data scientists, and business teams to align Databricks solutions with business objectives.
  
  

Innovation & Internal Initiatives

• Spearhead internal projects or products to enhance Snap’s Databricks capabilities and service offerings.
• Stay ahead of emerging Databricks technologies, advocating for innovative approaches to meet evolving client needs.
  
  

What you'll need to succeed...

Experience

• 5+ years of professional experience as a data engineer, with expertise in Databricks and cloud-based data warehousing.
• Proven experience in data migration projects, particularly transitioning legacy systems to Databricks.
• A Data Engineer Professional certification from Databricks is a strong plus.
  
  

Technical Skills

• Hands-on expertise with Databricks, Delta Lake, and Spark for data engineering.
• Proficiency in building and optimising ETL/ELT pipelines for Databricks.
• Strong understanding of data modelling principles and their application within Databricks environments.
• Experience integrating Databricks with cloud platforms (e.g., AWS, Azure, GCP).
• Familiarity with CI/CD pipelines, version control, and orchestration tools like Jenkins, Git, or Airflow.
  
  

Soft Skills

• Excellent communication and stakeholder management skills, with the ability to translate technical concepts to non-technical audiences.
• A proven track record of leadership, with the ability to inspire and guide teams to success.
  
  

So, what's in it for you?

• A collaborative and inspiring environment that encourages growth and knowledge sharing.
• The opportunity to work on a cutting edge Databricks analytics platform for one of the largest organisations in the world utilising near real-time architecture.
• The chance to work directly with Databricks Resident Solution Architects and learn from their expertise and grow the internal capability and expertise at Snap.