Ai quality engineer agentic ai evaluation Vacatures

Over de vacature

Overview

In this role you will advance AI quality across evaluation, safety, and deployment. You will work with cross-functional teams to define measurable quality criteria, deploy rigorous testing, and embed evaluation into CI/CD. Your work focuses on robust, safe, and compliant agentic AI systems at scale. This is an opportunity to influence the assurance of AI applications used by clients in complex, real-world contexts.

Arbeidsvoorwaarden / Benefits

• 4x9 Workweek
• Hybrid Working
• Competitive Base Salary
• Annual Bonus
• Health Insurance Scheme
• Mental Health Support

Verantwoordelijkheden

• Automate evaluation of multi-turn conversations and long-horizon tasks using metrics for reasoning accuracy, grounding, plan execution, latency, token usage, and recovery
• Apply schema validation, tracing, and calibrated LLM-as-a-Judge frameworks to detect inaccurate tool calls, loops, and misuse
• Implement filters and golden datasets to test harmful content, policy violations, hallucinations, bias, and brand drift
• Use human-in-the-loop review for high-risk edge cases
• Run red-team, prompt-injection, jailbreak, privacy, and PII-leak tests across agent workflows
• Analyze failures in model logic and API integrations to identify root causes and controls
• Embed test harnesses in CI/CD and monitor production telemetry for regressions, behavioral drift, and recovery
• Maintain evaluation data lineage, versioning, curation, and golden responses
• Translate business risks into measurable quality criteria, practical controls, and release recommendations
• Align stakeholders on user journeys, edge cases, thresholds, mitigations, testability, observability, governance, and evaluation strategy
• Communicate findings and residual risks for go/no-go decisions; coach teams and share reusable assets and standards

Belangrijkste vereisten

• 3+ years in machine learning QA, AI evaluation engineering, or MLOps engineering

• Dutch - B2+; English - C2

• Agentic AI Expertise: evaluating autonomous agents, multi-step workflows, or RAG applications

• Strong Python proficiency

• Experience with AI evaluation tools such as MLflow, Galileo, PyRIT

• Deep knowledge of prompt engineering, vector databases, tool schema design, and execution telemetry

• Hands-on experience in adversarial testing, threat modeling, or model safety guardrails

• Leadership: strong communication, technical leadership, growth mindset, continuous innovation

• communication

• collaboration

• growth mindset

• Python proficiency

• MLflow

• Galileo

Over de vacature

Overview

As a Senior AI Consultant at SQLI, you guide clients through AI-driven modernization, translating complex needs into strategic, measurable AI solutions. You will lead engaging workshops, shape proposals, and drive organisational change to adopt AI at scale. You act as the senior SQLI voice in client rooms, collaborating with architects, engineers, and sales to deliver high-impact outcomes. You'll help shape new accelerators and reusable IP, contributing to SQLI's leadership in AI offerings.

Arbeidsvoorwaarden / Benefits

• Hybrid policy work (2 days office)
• Competitive salary
• Work Schedule Monday to Friday

Verantwoordelijkheden

• Analyze client context to identify AI-driven value and root causes
• Design AI solutions with benchmarks and credible paths to measurable outcomes
• Advise on strategic/operational decision-making and long-term impact of AI adoption
• Lead discovery workshops, strategy sessions, and executive discussions
• Support change management, engagement, and skills transfer around AI adoption
• Contribute to commercial proposals and technical storytelling across the sales-to-delivery lifecycle
• Collaborate with Solution Architects, Data Specialists, and Engineering teams to validate feasibility
• Advise on AI applications (RAG, AI agents, MCP ecosystems, workflow automation)
• Assess AI solution quality using evaluation frameworks and governance best practices
• Contribute to new AI offerings, accelerators, and reusable assets
• Ensure delivery excellence and maintain trusted-advisor relationships with clients

Belangrijkste vereisten

• Strong consulting mindset with ability to translate objectives into practical AI solutions

• Experience identifying AI opportunities and delivering measurable results

• Excellent communication and stakeholder management for executive audiences

• Practical understanding of Claude Code, AI agents, MCP connectors, plugins, skills, orchestration layers, and agentic workflows

• Hands-on experience with RAG-based solutions, retrieval strategies, vector databases, enterprise search, and knowledge management

• Experience with AI agents and multi-agent systems, orchestration patterns, memory management, tool calling, and workflow automation

• Experience assessing AI solution quality via evaluation platforms (e.g., Harness, LangSmith, Promptfoo, Humanloop)

• Technical background in enterprise architecture, integrations, APIs, security, data flows, and solution design

• Strong ability to balance business ambition with implementation reality

• Native-level Dutch and Fluent English

• Excellent communication

• Stakeholder management

• Problem solving and critical thinking

• Claude Code

• AI agents

• MCP connectors