DLUHC Innovation Squads: developing our HR chatbot from problem to vision to solution

As outlined in our team’s first post, the initial phase of work from our department's first Innovation Squad achieved a couple of things.

First, it proved that a small team of people with no prior experience can be stood up quickly to take a lean UX approach to experimenting. Second, it shone a light on a universal problem — the size of HR enquiry backlogs.

To keep this work going, we met with our People Capability and Change team and HR service desk colleagues to align on how to move forward. We wanted to keep our focus on solving the problem for users while maintaining our pace and momentum.

We used these sessions to develop our problem statement and vision.

Problem: HR's content and structure isn't user friendly and is impacting the efficiency of the HR desk
Vision: For HR to lead the way towards improved staff interactions with departmental service desks, using new technology

Immediately following these sessions, Tom from the HR service desk team, joined us to help uncover and analyse HR data.

This is one of many changes to the team, whose shape has changed considerably in the last few weeks — although Umar remains our constant, as delivery manager.

Ben, our user researcher, joined us to run some user testing on the outputs of our 2 test chatbot models. This research has helped us understand the strengths and limitations of chatbots in helping answer HR enquiries.

As we begin to recognise that a chatbot only offers value if it provides the answers users need, we've also brought in Helena, our content designer.

What we’ve learnt

We had a call with a data scientist, who explained the considerable step change in the capability of publicly available Large Language Models (LLMs) — the technology behind the latest chatbots.

Before LLMs, chatbots were often a source of pre-scripted answers. At most, the chatbot would breakdown user questions, highlight key words and return the scripted answer that was “most similar”.

With an LLM, it will still break the sentence down and analyse key words. But now, rather than fetching an answer from a list, it generates a string of words that it thinks are the most likely response to a question.

However, this means they can give factually inaccurate answers to questions. So the latest development is a method called Retrieval Augmented Generative Large Language Models (RAG-LLMs).

Here, the model is fed business specific information (such as HR policy and guidance) and will only provide responses based on that information source. These models are rather complicated to set up, but will provide tailored answers and references to the source material.

So, in order for this kind of technology to be successful it needs to be built on well designed and high quality content. Quality in = quality out.

Quick wins

Tom, Helena and Ben have been working to analyse the types of enquiries received. This has already led to the identification and implementation of quick wins with regard to better use of Freshdesk (the HR service desk software) for categorisation and tagging of enquiries.

Examples include tagging enquiries that did not need human interaction to resolve.

What’s next?

We'll work with business and performance analyst colleagues to establish how we will measure success. Without this, we won't know if what we're doing is making a positive impact, or demonstrate a 'return on investment'.

We will investigate the top 10 most frequent types of HR enquiry from end to end, so that we can understand how best to address them. Once best practice is established we can scale and replicate first across remaining HR enquiry types and then across other teams within DLUHC.

Drop a comment in the comment box or get in touch with our Innovation Squads Team at innovationsquads@levellingup.gov.uk if you have any questions or would like to continue this conversation.

Subscribe to DLUHC Digital for updates from the team and visit Working at DLUHC for the latest job opportunities.