W2 - QA Engineer with AI- Remote
QA Engineer with AI Duration: 5-6 months. Number of openings:2 Location: Remote JD: About the Project: We’re developing a new customer-facing chatbot integrated into a modern Drupal-based website. The chatbot will use GenAI and RAG (Retrieval-Augmented Generation) to provide automated FAQ-style support, with future potential for agentic automation and integration into enterprise systems. The solution will be built on AWS, primarily in Python, using frameworks like LangGraph or AWS Strands Agents. Role Overview: We’re seeking a hands-on QA Engineer to design and execute testing strategies for a GenAI-based, multi-agent, cloud-native chatbot platform. The focus will be on validating LLM-driven functionality, ensuring data and prompt security, and verifying integrations, performance, and reliability in AWS. Key Responsibilities:
- Develop and execute test plans for chatbot logic, GenAI responses, and agent interactions.
- Validate RAG functionality: data retrieval accuracy, response relevance, and hallucination detection.
- Test API integrations between chatbot, AWS services, and enterprise systems.
- Implement automation for regression, performance, and cost-efficiency testing.
- Ensure LLM prompt security (e.g., guardrails, injection testing).
- Support analytics validation: user engagement, response accuracy, and model performance metrics.