Senior Data Engineer

Description

Job Summary: We are seeking a Senior Data Engineer specialized in Spark, with solid production experience in Scala and Apache Spark, to optimize data pipelines and design efficient distributed processing. Key Highlights: 1. Solid production experience with Scala + Apache Spark 2. Advanced Spark optimization and performance tuning 3. Stable project with high impact and a technically demanding environment **Senior Data Engineer (Scala & Apache Spark – Production)** ------------------------------------------------------------- At Excelia, we are a leading multinational firm in Consulting, Technology, and Professional Services, with over 25 years of history and presence in more than 50 countries. **We are looking for a Senior Data Engineer specialized in Spark** **Requirements (Mandatory)** ---------------------------- * Solid production experience with **Scala + Apache Spark** * In-depth knowledge of **Spark architecture and optimization** * Ability to analyze performance and refactor code * High technical autonomy **Key Experience** --------------------- * **Advanced Spark optimization** + Catalyst optimizer (logical and physical plans) + DAG inspection and bottleneck detection + Tungsten (efficient memory management and serialization) * **Performance tuning** + Cache/persist, checkpointing, and memory management + Shuffle elimination and join optimization (broadcast joins, co-partitioning) + Advanced partitioning, salting, and skew correction * **Complex distributed processing** + Advanced mapPartitions usage + Processing hierarchical structures (trees, forests) + Distributed traversal without costly iterations * **Advanced Snowflake** + Clustering, micro-partitions, pruning, and pushdown + Optimization of the Spark-Snowflake connector **What You Will Do?** --------------- * Optimize Spark pipelines in production * Refactor performance-impacted code * Design efficient large-scale distributed processing * Reduce costs, latency, and resource consumption **What We Offer** ------------- Stable, high-impact project Competitive salary Technically demanding environment Senior team