CY

Databricks Data Engineer

Cyber Security Employer London Posted 16 Nov 2026 Closing: 14 Dec 2026
Permanent Hybrid SC Cleared IT
£75000 - £95000/annum Hybrid Working

Role Description & Technical Scope

Job Ref: #225840441

We are seeking an experienced Senior Databricks Data Engineer with extensive hands-on experience designing, developing and implementing enterprise-scale data engineering solutions using Azure Databricks. The successful candidate will have strong experience delivering complex data platforms within UK Public Sector environments, working with large datasets, sensitive information and secure cloud-based data architectures.

Essential Technical Skills

  • 5+ years of hands-on experience with Databricks.
  • Expert-level knowledge of Azure Databricks.
  • Advanced PySpark, Spark SQL and Python programming.
  • Strong experience developing complex ETL/ELT pipelines.
  • Deep understanding of Delta Lake, Delta Tables and Delta Live Tables / Lakeflow Declarative Pipelines.
  • Experience implementing Medallion Architecture (Bronze, Silver and Gold).
  • Strong knowledge of Apache Spark architecture, optimisation and performance tuning.
  • Hands-on experience with Unity Catalog, access controls and data governance.
  • Experience implementing incremental data loads, CDC and SCD Type 1/Type 2.
  • Strong SQL development and query optimisation skills.
  • Experience integrating Azure Data Factory with Databricks.
  • Knowledge of Azure Data Lake Storage Gen2.
  • Experience with CI/CD pipelines, Git, Azure DevOps and automated deployments.
  • Experience implementing data quality, validation and reconciliation frameworks.

Public Sector Experience

  • Previous experience delivering data engineering projects within UK Government, Central Government, Local Government, NHS, DWP, HMRC, Home Office or other public sector organisations.
  • Understanding of public sector data governance, security and information management.
  • Experience working with sensitive and confidential datasets.
  • Knowledge of UK GDPR, Data Protection Act 2018 and relevant government data security standards.
  • Experience working within Agile delivery teams and government digital transformation programmes.

Security Clearance

Requirements
  • Be eligible and willing to undergo SC clearance.
  • Previous experience working on UK Government or Central Government projects is highly desirable.
  • Must be willing to undergo the required UK Government security vetting and background checks.

Responsibilities

  • Design and develop scalable Databricks data pipelines.
  • Build and maintain enterprise-grade Lakehouse platforms.
  • Optimise Spark workloads and Databricks clusters.
  • Implement robust data quality and governance controls.
  • Collaborate with architects, analysts and business stakeholders.
  • Support production deployments and troubleshoot complex data issues.
  • Ensure solutions meet public sector security and compliance
requirements

Ready to apply for this vacancy?

You are viewing Databricks Data Engineer at Cyber Security Employer. Click below to proceed directly to the official employer or verified application portal.

Direct official application • No candidate registration required

Similar Cyber Security Vacancies

Explore related InfoSec, SOC, Engineering, and Security Operations roles

Lead Cyber Security Architect

Eden James Consulting Limited • London
Competitive View Role

Cyber Security Consultant

Eden James Consulting Limited • London
Competitive View Role

Principal Cyber Security Consultant

Electus Recruitment Solutions • Somerset
Competitive View Role

Senior Software Engineer (Embedded Security Engineer)

Northern Recruitment Group Limited • Tyne And Wear
£60000.00 - £60000.00 per annual View Role

Security Engineer (Tooling & SIEM)

Networking People (UK) Limited • South East
not provided View Role

Cyber Security GRC Specialists - 3 Vacancies

University of Oxford • Oxfordshire
Competitive View Role

Vacancy Summary

  • Date Posted: 16 Nov 2026
  • Location: London
  • Job Type: Permanent
  • Workplace: Hybrid
  • Security Clearance: SC Cleared
  • Industry: IT
100% FREE CV BUILDER
Need an ATS-Ready Cyber CV?

Create an industry-standard cybersecurity CV with technical templates.

Build Free CV Now