AI QA Automation Engineer (Python)

Luxoft Ukraine is hiring an AI QA Automation Engineer to build automated evaluation frameworks for AI agents, RAG, and prompts, and to test hallucination, grounding, and multi-agent coordination.

• Build automated evaluation frameworks for agents, RAG, prompts, tools, workflows, and model outputs.
• Create benchmark datasets, golden responses, regression suites, evaluation metrics, and release thresholds.
• Test hallucination, grounding, citation accuracy, prompt injection, tool selection, and multi-agent coordination.
• Develop agents that generate test cases, validate API and workflow outputs, and support intelligent UAT.
• Build UI, API, integration, performance, end-to-end automation, and CI/CD quality gates.
• Monitor production quality, drift, reliability, and recurring failure patterns.

• Bachelor's degree in Computer Science, Software Engineering, Information Systems, or a related field.
• 5+ years of software quality engineering, test automation, or reliability engineering experience.
• Strong Python skills and experience building reusable automated test frameworks.
• Experience with API, UI, integration, regression, and CI/CD-based testing, including measurable quality gates.
• Experience testing generative AI, RAG, copilots, or AI agents.
• Experience with Playwright, Selenium, LLM evaluation frameworks, and prompt testing.
• Experience developing agent-powered test generation, workflow validation, or intelligent UAT assistants.
• Experience with adversarial testing, Responsible AI testing, and production quality monitoring.
• Ability to translate requirements, business workflows, and AI behaviors into measurable quality criteria.
• English: B2 Upper Intermediate