Summary: Seeking a Senior AI QA Engineer to define quality for intelligent systems, ensuring predictability, safety, and consistency across frontier AI systems. Highlights: 1. Own quality, safety, and reliability across frontier AI systems 2. Define what “quality” means for intelligent systems 3. Own end-to-end AI QA strategy across diverse AI components We’re looking for a **Senior AI QA Engineer** to own **quality, safety, and reliability** across frontier AI systems. You’ll be the final line of defense for **agentic workflows, RAG pipelines, LLM integrations, APIs, and data systems**, ensuring they behave predictably, safely, and consistently in real\-world environments. This is not traditional QA, you’ll define **what “quality” means for intelligent systems**, designing evaluation frameworks, stress tests, red\-team scenarios, and drift monitoring that keep experimental AI production\-ready. What You’ll Do * Own end\-to\-end **AI QA strategy** across UI, backend, data, retrieval, and agents * Build **LLM, RAG, and agent evaluation suites** (behavioral, regression, scenario\-based) * Develop **Python\-based test automation** for agents, APIs, and pipelines * Validate **data quality, embeddings, retrieval accuracy, and ranking performance** * Design **edge\-case and adversarial tests** (safety, robustness, compliance) * Define **observability metrics** for latency, cost, errors, and behavior drift * Run defect triage with clear root\-cause analysis and crisp reporting What You Bring * Strong **Python** skills for test automation and evaluation harnesses * Experience testing **AI/ML, LLMs, RAG, or agentic systems** * Understanding of **vector databases, retrieval pipelines, and embeddings** * Experience with **CI/CD, DevOps, and observability tooling** * Sharp instincts for **edge cases, failure modes, and nondeterministic systems** * Excellent communication and ownership mindset Nice to Have * Red\-teaming or safety testing experience * ML evaluation or benchmarking frameworks * Prior work on production AI systems What do we offer you? * Attractive salary * Large freedom and real influence * No unhealthy competition, team approach to meeting challenges * Remote\-first, flexible working culture * Company apartments in cool cities across Europe: work and enjoy a memorable getaway About Us We are a software house with 18 years of experience and a global portfolio of projects. We help businesses modernize, scale, and innovate through custom software solutions. Our team embraces unconventional ideas and new technologies, delivering solutions with real impact. If you value professionalism, creativity, and a strong engineering culture, you'll feel at home here. Job Type: Full\-time Pay: 54,000\.00€ \- 120,000\.00€ per year Experience: * QA: 5 years (Required) * AI: 3 years (Required) Work Location: Remote