Data Engineer

Description

Summary: Join as a Data Engineer to build a trustworthy and analytics-ready data lake for a regulated UK & Ireland credit and lending company. Highlights: 1. Foundational data engineering role on a regulated data estate 2. Opportunity to build a harmonized semantic layer 3. Focus on data protection and reproducibility **About the project** ***(description, duration, stage)*** ========================================================== Join Neurons Lab as a **Data Engineer** on a new engagement with a **regulated UK \& Ireland credit and lending company**. The client has lifted data from multiple business entities into a newly centralized, **anonymized data lake**, but lacks the data\-engineering depth to make it trustworthy and analytics\-ready: current pipelines were assembled quickly (partly AI\-assisted), and the descriptive statistics **cannot yet be validated or reproduced**. You put that foundation on solid ground so the Data Science Lead can model on it with confidence — validate and re\-engineer the pipelines, build the **harmonization / semantic layer** across entities, enforce data quality and lineage, and prepare clean, feature\-ready datasets. This is a **foundational data\-engineering role on a regulated data estate**; data protection and reproducibility are the primary constraints on every decision. **Full\-time engagement preferable.** **What you'll actually do** ***(example tasks)*** ================================================= * **Reproduce a descriptive\-statistics report end\-to\-end** so any figure traces back to raw source — closing the gap the client admitted (numbers they can't currently defend). * Profile and **reconcile differing source schemas** across acquired entities: map differing field names, types, encodings and business definitions for the same concept into one conformed model. * Build **dbt staging intermediate mart models** with tests; codify the harmonized definitions the Data Science Lead specifies. * Write **Great Expectations suites** (null / range / uniqueness / referential checks) and wire them into the pipeline so bad data fails loudly rather than silently corrupting analysis. * Implement **entity / identity resolution** (deterministic \+ fuzzy matching) where there is no clean shared key for the same customer or account across sources. * Implement and **verify anonymization / pseudonymization** (hashing / tokenization / k\-anonymity) and evidence that re\-identification risk is controlled for the client's IT / compliance team. * **Optimize Spark / Glue jobs over tens of millions of rows** — partitioning, file formats (Parquet), incremental loads, cost control. * Orchestrate with **Airflow / Step Functions**; build repeatable, scheduled pipelines rather than one\-off scripts. * Prepare **clean, documented, feature\-ready datasets** for the PD / delinquency models. * Document **runbooks** so the offshore team can operate the pipelines and handover takes days, not weeks; help scope onboarding of the remaining (Ireland \+ additional) sources. **Skills** ---------- * Strong **SQL** and **Python** for large\-scale data processing * **AWS data stack**: S3, Glue, Lake Formation, Athena / Redshift, EMR / Spark, Step Functions / Airflow * **Data modeling \& semantic layer** (dbt or equivalent); dimensional modeling * **Entity resolution / record linkage** across heterogeneous sources * **Data\-quality \& testing** frameworks (Great Expectations, dbt tests) and data lineage * **Anonymization / pseudonymization** techniques and their analytical trade\-offs * Big\-data processing (Spark) with performance and cost optimization at scale * Clear written / verbal English; documents for handover and works well with a distributed team **Knowledge** ------------- * **GDPR** fundamentals as applied to anonymized / pseudonymized financial data and UK / EU data residency * **AWS Well\-Architected** (Analytics, Security) for BFSI * Awareness of credit / risk data structures and what downstream modeling consumers need — a plus **Experience** -------------- * **4\+ years** in data engineering, with strong **AWS \+ Spark / SQL at scale** * Demonstrated experience **harmonizing / integrating data across multiple source systems** * Experience building **validated, reproducible pipelines in a regulated environment** (BFSI, healthcare, government) — strong plus * Comfortable stepping into a **messy, partly\-built data estate** and bringing it up to standard * Comfortable as the sole or lead data engineer on a small (3–4 person) delivery pod