Data Engineer

Data Engineer - Medical & Public Health Data Transformation

A skilled Data Engineer is required to support a major data‑transformation workstream within a clinical screening and diagnostics environment. The project focuses on modernising an end‑to‑end screening service through modern, data‑driven digital capabilities. You will join a small specialist team responsible for analysing complex datasets spread across multiple system instances, resolving data‑quality challenges, and shaping future data models and structures.
The core responsibility is to build secure, repeatable ingestion and transformation pipelines, apply robust data‑cleansing logic, and produce auditable, reproducible outputs.

Essential Skills

* Experience establishing import/export patterns, including handling data extracts, schema discovery, incremental loading and normalising data across multiple source systems.
* Strong capability in data‑transformation‑heavy pipelines covering profiling, cleansing, standardisation, conformance and final data publishing.
* Advanced SQL expertise including profiling, joins/merges, deduplication, anomaly detection and performance tuning.
* Practical Python scripting experience for automation, parsing, rules engines and data‑quality checks, using libraries such as Pandas/Polars, scikit‑learn or matplotlib.
* Experience with modern data tooling (e.g., Spark, Azure Data Factory) or the ability to deliver equivalent functionality in code‑based environments.
* Proven experience working with geospatial datasets (vector, raster, GeoJSON, shapefiles), including coordinate systems, spatial data handling and geospatial analysis workflows.
* Ability to interpret geographical context and aggregate/upscale local or regional geospatial insights into coherent national‑ or region‑level datasets.
* Experience working with publicly available official datasets (e.g., census boundaries, geographic lookups, deprivation indices, population estimates).
* Capability to design rules for completeness, validity and consistency, and to implement exception handling and reconciliation flows.
* Ability to build version‑controlled pipelines with deterministic transformations, logging, lineage and full traceability of data changes.
* Comfortable working in a secure environment with least‑privilege access principles, secure storage/transfer practices and handling of sensitive personal data.

Soft Skills

Technical skills alone are not enough-success in this role requires:
* Strong communication and collaboration skills.
* A team‑focused, cooperative approach.
* Enthusiasm, engagement and a positive attitude.
* Proactivity-comfortable working independently without constant direction.
* Ability to handle ambiguity and adapt to change effectively.

Nice‑to‑Have Skills

* Experience working with healthcare or medical‑sector datasets, including patient/episode‑style records or longitudinal histories.
* Experience building automated data‑profiling dashboards or reporting frameworks.

Additional Requirements

Candidates must be eligible for UK Security Clearance due to the sensitive nature of the data involved.

BPSS eligible also