-->

KMWorld 2024 Is Nov. 18-21 in Washington, DC. Register now for $100 off!

  • November 4, 2024
  • By Marydee Ojala Editor in Chief, KMWorld, Conference Program Director, Information Today, Inc.
  • Features

Lucidworks shares results from a global survey about GenAI adoption, fears, and strategies in 2024

Article Featured Image

The second part is putting controls in place—establishing guardrails—since not everyone should see every piece of data. Employees should see only the information they need to do their jobs. Decisions about which information can be seen by whom needs to be part of the GenAI implementation discussion.

Cost concerns

Finally, there’s the cost part of the puzzle: LLMs are about 10 times as expensive per query as lexical search or semantic vector search. Sometimes that’s worth it. If you’re helping a customer buy a refrigerator or a car or planning a trip to Europe, spending 10 times per search is worth it. But if it’s just a customer shopping for information, not necessarily a product, you don’t want to have your costs go up to answer those questions.

When queries are run on a model that embeds internal client and company data, the cost is based on hosting the model, not the query. Another way of managing the cost is doing query routing. Look at the inquiry that the user is making and decide if the answer can come via a cheaper lexical search. Or, is a more complicated interaction using an LLM necessary?

Implementation costs will probably not decline, but the cost of model usage is going down rapidly. Companies are getting smarter and more efficient in how they index data, how they train their models, and how they run their models. Research into alternatives to GPUs (graphic processing units) is ongoing. Commercial models are getting more efficient, smaller, and more purpose-built for individual use cases. That makes them more cost effective. Companies are using a mix of models. Each use case for a company might have two or three models—a commercial model, an open source model, and a semantic vector search model. By routing to the more cost-effective approach, you can make it a more efficient process.

Data security and trust

Since the models were first introduced, people became more aware of the issues and the risks. Companies did not want their data becoming part of a public dataset. External security is now pretty well in place, but the fear is still there. Concerns about accuracy have tripled; concerns about costs have gone up more than tenfold. Security is still a concern, but nowhere near the concern evinced by other areas.

Trust is important. It’s not simply trusting that data will be secure, it’s also about the hallucination problem. LLMs work on predictions, but they don’t typically have an option not to predict if they don’t know. So they will make something up. This whole trust aspect has become critical. It’s hard to make them foolproof, but there are tools and technologies, such as RAG. You can guide the models so they don’t give you an answer if they don’t know the answer. You can look at binary models, which use another model on top of the existing model. This ensures that hotel guests, for example, won’t be told about a beach that’s 2,000 miles away in response to a query for a local beach. Ironically, it’s almost as much work to have a model not give you an answer, as it is to give you an answer.

Road maps

KM teams should have a well-defined road map for GenAI adoption. You wouldn’t deploy any other computer system or technology without a project plan, a road map, or set milestones or objectives. Therefore, it’s critical at this point to step back and develop a project plan and a road map. The most common first set of initiatives concerns governance. How will this model be used? Who will have access to it? What models will we use? You don’t want every department contracting with or deploying a different technology. Over time, you’ll lose the benefits and drive costs up. Start with governance processes, standardized models, and deciding access standards, then create a leadership council or a steering committee to evaluate the use cases requiring a business case.

KMWorld Covers
Free
for qualified subscribers
Subscribe Now Current Issue Past Issues