Agentic AI Quality Engineer

Over de vacature

Overview

In this role you will advance AI quality across evaluation, safety, and deployment. You will work with cross-functional teams to define measurable quality criteria, deploy rigorous testing, and embed evaluation into CI/CD. Your work focuses on robust, safe, and compliant agentic AI systems at scale. This is an opportunity to influence the assurance of AI applications used by clients in complex, real-world contexts.

Arbeidsvoorwaarden / Benefits

• 4x9 Workweek
• Hybrid Working
• Competitive Base Salary
• Annual Bonus
• Health Insurance Scheme
• Mental Health Support

Verantwoordelijkheden

• Automate evaluation of multi-turn conversations and long-horizon tasks using metrics for reasoning accuracy, grounding, plan execution, latency, token usage, and recovery
• Apply schema validation, tracing, and calibrated LLM-as-a-Judge frameworks to detect inaccurate tool calls, loops, and misuse
• Implement filters and golden datasets to test harmful content, policy violations, hallucinations, bias, and brand drift
• Use human-in-the-loop review for high-risk edge cases
• Run red-team, prompt-injection, jailbreak, privacy, and PII-leak tests across agent workflows
• Analyze failures in model logic and API integrations to identify root causes and controls
• Embed test harnesses in CI/CD and monitor production telemetry for regressions, behavioral drift, and recovery
• Maintain evaluation data lineage, versioning, curation, and golden responses
• Translate business risks into measurable quality criteria, practical controls, and release recommendations
• Align stakeholders on user journeys, edge cases, thresholds, mitigations, testability, observability, governance, and evaluation strategy
• Communicate findings and residual risks for go/no-go decisions; coach teams and share reusable assets and standards

Belangrijkste vereisten

• 3+ years in machine learning QA, AI evaluation engineering, or MLOps engineering

• Dutch - B2+; English - C2

• Agentic AI Expertise: evaluating autonomous agents, multi-step workflows, or RAG applications

• Strong Python proficiency

• Experience with AI evaluation tools such as MLflow, Galileo, PyRIT

• Deep knowledge of prompt engineering, vector databases, tool schema design, and execution telemetry

• Hands-on experience in adversarial testing, threat modeling, or model safety guardrails

• Leadership: strong communication, technical leadership, growth mindset, continuous innovation

• communication

• collaboration

• growth mindset

• Python proficiency

• MLflow

• Galileo

Advertentie Cv voorbeelden voor ICT & Automatisering: zo ziet een sterk cv eruit Bekijk het voorbeeld en maak je eigen versie. Bekijk voorbeeld Geen account nodig