Building an AI-Powered Website Search Solution – Part One

At Revium, our focus is on the practical application of AI to solve real-world problems, and given we are still in the early stages of the widespread development and adoption of generative AI, the best way to do that is through experimentation.

Recently, one of our internal AI research projects explored the potential to replace traditional website search with an AI-Powered Search Solution. The results have been very positive and as a result we will be rolling out the solution for some of our clients in the very near future.

In this two-part series we will talk through the solution we have built, its advantages and disadvantages and share a little about how we built AI powered website search. As well as some of the challenges we encountered in the process.

However, as with any AI project the first question we start with is whether or not applying AI is going to add value?

Traditional vs. AI-Powered Search

The core functionality of traditional search engines on most websites primarily relies on matching keywords typed in by a user with keywords that appear in the text on pages in that website.

This approach has a few shortcomings - one of the most notable is its inability to understand natural language which means it cannot interpret what the user means. As a result traditional website search relies on the user modifying their behaviour to match the way it operates. As an example, if a human walked into a physical store looking for hiking boots they might ask the attendant about what colours they come in, which ones they think would best suit their needs and what boots fit in their price range.

When we go to a digital storefront for the same thing and use the search bar we modify our behaviour to match what the technology needs - typing in a list of keywords trying to anticipate which ones will have the best chance of delivering us the results we are looking for. In terms of user experience this is clearly not the best approach.

With AI we have the ability to interpret natural language and as a result we can understand the context of user queries, interpret meaning, and deliver more accurate website search results which opens up the door to key opportunities;

• Customisation of the search results so they reference the context of the user’s search terms

• Allowing the user to express the initial query in natural human language rather than a string of keywords

AI-powered search also takes into account factors such as location, previous searches, and user profile information (if available) to provide more relevant and personalised results. In other words it tailors results based on individual user preferences and behaviour.

Overall, traditional search works well for simple, direct queries with clear keywords. However, AI-powered search is more versatile and insightful. It excels at handling complex queries, context awareness, and personalisation. This is likely why Google is moving to use AI search as the basis for the future of Google Search and, based on our experiences with AI-powered search, why over the next few years it will become the preferred approach for search on consumer facing websites as well.

So what does AI search look like?

We have intentionally designed the solution to have a similar look and feel to the traditional website search.  

In terms of the search bar, the only difference is that we provide a tooltip icon that users can interact with that will explain to them that this is an AI powered search tool and that they should feel comfortable asking questions instead of throwing keywords at it. Here is a mockup of the search bar.

The only real disadvantage of AI-powered search cuts in at this point – we couldn’t figure out a way to elegantly handle predictive search suggestions, and in the end we decided to just leave them out. Whilst it is a compromise on existing search experience expectations, on balance we feel it is a small price to pay for the other benefits AI search delivers.

Similarly, the results page is also deliberately very inline with a traditional experience.

The primary difference is that the first result is provided as more of an answer to a question rather than a page link and page description. In our testing we found that more often than not the first result was the one the user was looking for and so giving them an answer immediately rather than sending them off to a page to find it provided a better user experience. Here is a mockup of the search results page.

We have also added some follow up buttons to encourage the user to continue their line of enquiry using a chatbot – if they click on this button they are transferred to a chatbot window where the bot has the context of their search history and is ready to continue a conversation with that pre-existing context factored in.

After that the rest of the results look very much the same as a traditional website search experience. The only notable difference is that the text of each result is AI generated based on the query. In other words, we tailor the responses to inform the user about what content on the page is relevant to their search term.

As an example if you searched our website for “content management system benefits” the search results would include one of our case studies that talks about a project for Belgravia Leisure. Using traditional search the result would take the meta-description for the page and then cut it off after a certain number of characters giving you this result…

“Discover how we centralised Belgravia’s digital assets and infrastructure into one functional system through astute web…”

The AI result however forms up a descriptive sentence based on the page content and the context of the search query which looks like this…

“Learn how Belgravia Leisure's CMS unifies digital assets, boosting efficiency, user experience, and cost control."

So whilst there are subtle variations that add a bunch of additional value to the user, the interface and experience feels very similar to the traditional one. The reason for this is because we know users are creatures of habit. People visiting a website come with a learned experience that informs how they identify key elements of a site (like the menu, buttons, search bar, etc.) and what to expect when they go to interact with them. Disrupting that expectation can be jarring for users and can risk them abandoning their visit if their frustration levels get too high.

That said, the web is always constantly evolving and over time we think that users will become acclimatised to a new way to search – in other words, as AI search becomes more prevalent the traditional search interface will evolve to something new, likely more akin to a chatbot type interface.

What about performance?

In all of our testing on our refined AI website search tool we found that the results were unnervingly good, to the point where in the overwhelming majority of searches the first result gave us the answer we were looking for.

Of course, the quality of the results relies on the quality of the content it has at its fingertips, and in every case where we received sub optimal results it was because of issues with content not being on the website or conflicting content on the website.

How did we build it?

Given we have already developed a Large Language Model (LLM) powered AI chatbot platform that covered a lot of the fundamental capabilities needed for AI search, we were able to save some time and branch off that work as a foundation for the AI website search project.

At their core, both the AI chatbot and AI search rely on a database of knowledge that they create, which can then be searched for relevant information to help the LLM form up a response to a user query. This approach is called Retrieval Augmented Generation (RAG) and it allows us to ensure the responses the AI provides are based on the information we provide it.

In part two of this series we will explain a few of the core elements of a RAG based AI system and how we tackled optimising them to get us the best possible search results.