Data Engineer, Hybrid

Description

Job Summary: We are seeking a Data Engineer with experience in Big Data ecosystems to participate in an international project migrating Hadoop infrastructure to cloud environments based on Kubernetes. Key Highlights: 1. International Big Data environment project 2. Participation in data platform modernization projects 3. Modern technology environment Data Engineer We are seeking a Data Engineer with experience in Big Data ecosystems to participate in an international project focused on migrating Hadoop infrastructure to Kubernetes-based cloud environments. You will join a data engineering team responsible for designing, developing, and automating data pipelines, working with technologies such as Spark, Scala, Airflow, and CI/CD tools within an agile environment. Responsibilities Migrate Hadoop infrastructure to cloud using Kubernetes Engine, COS, Spark as a Service, and Airflow as a Service. Develop data transformation and data quality processes to ensure consistency and accuracy. Implement data pipelines using Scala, SQL, and Apache Spark. Automate processes using Airflow and orchestration tools. Create and maintain CI/CD pipelines for automated deployment and testing. Develop unit tests and validate data processes. Prepare technical and operational documentation. Collaborate with business and technology teams to design scalable data solutions. Technical Requirements Experience with Apache Spark and Scala Experience with Hadoop ecosystem Knowledge of SQL and NoSQL databases Experience with Apache Airflow Experience with HDFS Experience with CI/CD (GitLab, Jenkins or similar) Knowledge of S3 / COS Storage Experience working with Parquet and ORC Additional Desirable Skills Kubernetes / containerization Oozie Shell scripting Dremio Elasticsearch / Kibana Kafka or streaming processing What We Offer International Big Data environment project Hybrid work model in Madrid (1 day onsite) Participation in data platform modernization projects Modern technology environment Spark, Scala, SQL, NoSQL, CI/CD, S3, COS Storage