Data Engineer — Streaming & Ingestion

Description

Summary: Hands-on engineer focused on migrating streaming data sources, managing EventHub consumers, and ensuring reliable real-time data flow with emphasis on streaming patterns and data quality. Highlights: 1. Hands-on engineer focused on streaming ingestion migration 2. Implement and tune EventHub consumer groups and data transformations 3. Build and execute data validation frameworks for output parity **Role Summary** Hands\-on engineer focused on the streaming ingestion side of the migration — connecting Telelog data sources, managing EventHub consumers, and ensuring reliable real\-time data flow into the target architecture. Emphasis on streaming patterns, data quality at ingestion, and validation against legacy outputs. Responsibilities **Migrate existing Telelog streaming pipelines into the target architecture (EventHub → Databricks)** Implement and tune EventHub consumer groups, partitioning, and checkpointing **Implement data transformations in PySpark/Databricks for different regions and head unit types** Build and execute data validation frameworks to ensure output parity between old and new pipelines **Write tests and ensure data quality across streaming ingestion stages** Collaborate with the Tech Lead on design decisions and with the Batch Engineer on handoff points Required Skills **Mid to senior\-level experience (3\+ years) in data engineering with streaming focus** Strong experience with streaming platforms (Kafka, EventHubs, Spark Structured Streaming) **PySpark / Databricks experience** Azure fundamentals (ADLS, EventHubs, KeyVaults) **Data quality \& testing — validation strategies, reconciliation** Infrastructure\-as\-Code awareness (Terraform, Bicep) **FinOps awareness — understanding cost implications of streaming resource decisions** Python **Git (GitLab), Jira, Confluence** Fluent English Nice to Have **Experience with streaming\-to\-batch migrations** Legal / compliance awareness (GDPR, PII handling, EU Data Act) **Java (for understanding legacy Telelog code)** Job Type: Full\-time Pay: 30,000\.00€ \- 45,000\.00€ per year Language: * ingles (Required) Work Location: Remote