Three key insights from building enterprise search solutions

Q: What are the key insights you gained while building enterprise search solutions?

A: There three main insights worth highlighting:

1. Enterprise Search is not simply site search. There are a lot of differences, such as the relevance of search results. Users in the consumer search market have different needs than the enterprise search users. Therefore, the tools for measuring relevance of the enterprise search results presented to users should be different. In site search you use popularity to rank the results, while in enterprise search you set it based on business context. The relevance in the consumer world is usually based on traffic, and uses tools like cookies or search history. In the enterprise search sphere it is more about profiling in means of your role so you will see content with higher weights which are relevant to your current role. For example, in our enterprise search platform, we use what we call “Expert Profiles” and “Interest Profiles” which are automatically created for users in real time. Interest profiles are created by tracking what the user actually sees or reacts to. You can use it to target and recommend content to users. Expertise profiles are created by tracking the content that a user creates and extracting a conceptual understanding of it. You can use expertise profiles to trace users who are experts in particular subject areas. So, if you’re looking for content in a specific field, you’ll see a list of experts who worked on similar content, created relevant documents or even answered questions in forums.

The key takeaway is that in the enterprise search world the modelling of data should be more oriented to the user’s business role. Meaning the context should be based on role rather than on behavior.

2. Your Enterprise Data is not enough. After talking to many companies working with us, we have found that many are currently not taking advantage of some of the true opportunities in Big Data. A real value can come from combining data coming from multiple data sources: internal (on-premise) and external (off-premise). This is the key for getting some real valuable insights.

Let’s take an example from the gaming industry. Your goal is to identify customers churn trends and handle it in real time. In order to address that, you can combine data coming from both internal and external sources. You can identify potential churn by tracking tweets and posts (e.g.: with the relevant game’s hashtag) and checking the sentiment of it. When you identify it, you can offer in-app free content or other attractive deals.

The key takeaway is to not concentrate only on data you own, but to pay attention to data outside your organization as well. Secondly, make sure you combine the various sources of data to get a better value or enriched experience.

3. Be Cautious of Data Blindness. There are cases where you really need to find the needle in the haystack, and most times it’s hard. We are in the age of data overload, and we all know that. But another factor that might make our lives harder in that sense is actually coming from an unexpected place. For search optimization we use a lot of different tools such as profiling, context based search, automatic relevance, etc. While these tools are aimed to make our lives easier by supporting the common scenarios, there might be scenarios where I really need to find the un-common data, and actually by over-profiling - it might be very well hidden from us. Let’s take an example to clarify this: I’m a lawyer working in a patents office. Most of my day job involves documenting, filing and tracking patents. So when I search for something, I get results which fit my common use-case – mostly patents relevant to what I searched for. In some cases, I deal with patent infringement, and in rare case I need to perform an investigation of technology related patents. This means that I need to get results from technology related sources, but due to heavy profiling, I’ll probably not see it in the top of the search results. So my advice here is to consider giving an option to disable the features of “profiling” and “search Optimization”. We added an option to our admin search called: “Show unmodified results”. This way the user (in the case the admin) is able to see the raw search results without all the modifications. So while I mentioned profiling is an important tool, beware of over-profiling and over-optimization, or at least offer an option to get out of it.