Production LLMs
EmployeeGPT
Building a useful internal knowledge tool while production RAG practices were still emerging.
The problem
Useful information was spread across documents and platforms. Finding an answer often meant searching multiple places or asking another team.
The approach
Our team developed EmployeeGPT during the early wave of RAG applications, when production patterns and tooling were still emerging. We integrated the bot with Slack to help people find answers across internal sources. The project gave us an opportunity to explore a new technology and work through what it needed to be useful in practice.
Making it work in production
We developed source citations, budget guardrails for ingesting new data, processes to keep indexes up to date, and support for handling queries at scale. We also iterated on chunking techniques and embedding models to improve retrieval.
Many of these capabilities have since become standard or available off the shelf. Building them at the time involved experimentation across ingestion, retrieval, and serving, alongside the model itself.
Sharing the work
We shared the work and lessons from building EmployeeGPT at:
- ODSC APAC · August 2024: “Leveraging LLMs for Internal Knowledge Discovery,” which I co-presented with Timothy Low.
- AIAI Berlin, Generative AI Summit · September 2024: “LLMs for internal knowledge discovery: real world challenges and learnings,” which I presented.
Published account
I co-authored the technical write-up with Timothy Low and Sai Bharath Gottam. It explains the ingestion and retrieval setup, practical constraints, and impact reported in 2024.