LLM / Generative AI Engineer

In this role, you’ll apply hands-on experience building LLM-powered products to evaluate and improve how language models perform in real-world applications. You’ll assess prompting, retrieval, fine-tuning and generation quality, identify failures in model behaviour, and develop reference answers and evaluation criteria that help AI systems produce more reliable, grounded and useful outputs.

• Evaluate LLM outputs across realistic application scenarios, including RAG question-answering, information extraction, summarisation and instruction following.
• Build edge-case and adversarial test sets that expose weaknesses in grounding, faithfulness, reasoning and instruction adherence.
• Compare model versions, prompts and configurations to identify performance improvements, regressions and behavioural drift.
• Design and apply evaluation rubrics covering factual accuracy, faithfulness to source material, completeness, relevance, tone and safety.
• Develop high-quality reference answers and prompts that demonstrate how an experienced practitioner should handle real-world tasks.
• Identify hallucinations, unsupported claims and flawed inferences, documenting the evidence behind each evaluation decision.
• Analyze evaluation results and translate findings into structured, actionable feedback for AI research and engineering teams.

• Retrieval-augmented generation (RAG) and retrieval quality evaluation
• Prompt engineering and instruction following
• LLM application development and API integration
• Fine-tuning and model adaptation
• Text generation, summarisation and information extraction
• Embeddings, semantic search and vector databases
• LLM evaluation, benchmarking and hallucination detection
• Multilingual and Indic-language language models

• 3+ years of relevant experience in AI engineering, machine learning or applied NLP, including hands-on experience building LLM-powered applications.
• Experience shipping at least one LLM-powered feature or pipeline to real users.
• Strong Python skills and practical experience working with LLM APIs or frameworks such as LangChain or LlamaIndex.
• Working knowledge of transformers, tokenisation, context windows, sampling strategies, embeddings and vector search.
• Understanding of RAG pipelines, source-grounded generation and the factors that influence output quality.
• Ability to distinguish hallucinations from reasonable inference and explain evaluation decisions with clear, evidence-based reasoning.
• Strong analytical and problem-solving skills, with the ability to identify subtle model failures and communicate findings precisely.
• Excellent written and spoken English (C1 or higher), with the ability to work independently and deliver structured feedback against asynchronous deadlines.

• Hands-on experience with fine-tuning techniques such as LoRA or PEFT.
• Experience with LLM evaluation and benchmarking frameworks.
• Familiarity with Hugging Face Transformers and model deployment workflows.
• Experience working with vector databases and production RAG systems.
• Experience evaluating multilingual models or working with Indic-language datasets.
• Prior experience in AI training, model evaluation or expert-led data annotation projects.

• Python, LLM APIs, LangChain, LlamaIndex, Vector Databases, Hugging Face

• Scenario-based assessment: evaluate a set of model-generated answers against provided source documents, identify unsupported claims and omissions, and justify each scoring decision against a defined rubric.