AI Developer (LLM Evaluation)

IT Craft is looking for an experienced AI Developer to design and implement an evaluation framework for an LLM-based assistant on AWS Bedrock, including automated testing and CI/CD integration.

• Design and implement automated end-to-end and regression testing infrastructure for an LLM-based assistant
• Build an evaluation harness that runs a structured set of prompt scenarios against the deployed service
• Implement automated checks for response correctness, tool usage, and guardrail behavior
• Generate clear pass/fail results based on agreed evaluation baselines
• Collaborate with the internal DevOps team to integrate evaluation runs into CI/CD processes and identify quality and behavior regressions before and after deployments
• Analyze evaluation results and continuously improve test coverage and reliability
• Plan and organize the implementation independently, define follow-up tasks, and coordinate with internal specialists through regular synchronization meetings

• Hands-on experience with LLM evaluation tools such as Promptfoo, Langfuse, Arize, or comparable solutions
• Practical experience designing or implementing automated evaluation and regression testing for LLM-based applications
• Understanding of approaches used to evaluate non-deterministic LLM responses, tool calls, and guardrail behavior
• Solid experience with AWS and CI/CD
• Ability to design a solution from scratch and deliver it independently
• Familiarity with Kotlin and Spring would be a plus
• Strong written and spoken English (Upper-Intermediate+ / Advanced)

• Competitive compensation according to the qualifications
• Flexible working hours, remote work
• Opportunity for career growth
• Compensation for technical courses, educational resources, and sport
• In-house English training
• A friendly team of open-minded people