Understanding what drives AI Token Cost in Data & Analytics

In the world today we hear a lot about how AI can make things easier and solve a lot of problems, and how you no longer need to spend a lot of effort to create things. But everything is not always as easy as it seems, and from our experience, AI isn't the silver bullet to solve all problems.

Therefore, I wanted to walk through what I've seen about AI and how it solves problems, including all the pitfalls you might experience when you actually use AI, or when you use AI for business intelligence development and for all your reporting needs.

AI as Your Junior Developer

First things first, to set the ground rule: we do consider AI to be extremely good at what it does. We always use the example of the 80/20 rule. We consider AI to be extremely good at delivering the initial 80%, breaking the ice, so to speak. But in our opinion, that last 20% of the work definitely needs human oversight. That's the so-called human-in-the-loop design, where all AI and all results from AI have to be corroborated and tested by humans to make sure things are working well.

You should think of AI as your junior developer in your environment rather than your senior. It should never be that the AI makes the final decisions and does all the code with no oversight given, because you trust it so much. With a junior developer, you typically make sure that what they've delivered is correct, fits all the requirements, and fits well within the environment. Otherwise, if you just left the junior developer to do what they thought was best, you might end up with a lot of difficult problems that are very difficult to solve in the long run.

To summarize: leaving AI ungoverned increases cost in terms of money, maintainability, and accuracy. And those pitfalls are what this discussion is about.

But before we go further, let's look at what it costs to use AI.

What Exactly Drives This Cost?

When using AI, every word costs money. When prompting the AI, every input request has a cost, and every output has a cost as well. That cost is measured in tokens. Token pricing is typically per thousand or per million, and it can scale very fast.

So what exactly drives this cost? It can be broken down into three things:

1. The actual length of the prompt
2. The size of the context required to run that prompt
3. The complexity and length of the response it generates

All things considered, the biggest hit in cost in these scenarios is the context required to run that prompt. I'll get to what context entails when it comes to analytics and reporting further down

The Problem with the Context Window

So what is the problem with the context window? Really, when we think about context, we should think about it as the amount of information the AI can hold in its mind at once. However, context has a hard limit. Once it's exceeded, the AI will start dropping older information to make space for newer information.

As context grows, AI performance degrades. You might start seeing the AI inferring rather than reasoning, you might see it lose track of earlier instructions, and you'll see that results become less reliable and harder to reproduce.

This change in context, the size of the context and how it keeps changing during the interaction, is why we say AI results are non-deterministic.

Being non-deterministic means you're never guaranteed the exact same result for the exact same question, and in the real world, getting the same answer and trusting the results is very important.

This is the problem we face when we leave AI ungoverned. Ungoverned AI tends to generate overly complex artefacts, code, and output. The more complex the output is, the more the context window grows. Similarly, you might find that AI will add extra layers, extra rules, and extra words that were unasked for. We have found that AI tends to overcomplicate all results, and it requires a lot of insight and understanding to figure out which parts of that output are not needed and need to be refined.

This is why you need a human-in-the-loop design → to ensure the output stays lean and optimized, so that context cost is reduced as much as possible.

A Real-World Example: Revenue Per Employee

Now let's look at what this would look like in the real world in terms of business analytics and reporting.

Imagine you've put all your data in a data lake and given the AI access to that data. Now you ask it: "I want to know what my revenue per employee was." So the AI goes off, it scans all the data in the data lake, it infers all the column names, and it tries to understand which columns to use. It might even decide to build its own derivation, its own mapping rules, or its own business logic to perform the task.

After all of that, it generates an answer, and it looks nice in terms of token usage. But you should consider that all of that data scanning, all of the column names, all of the business logic, all of it was used as tokens. And if your data was large, that token cost will similarly be large.

The bigger problem with this approach, though, is that you don't know what business rules were applied. You don't know what that derivation logic looked like. You also don't know what types of relationships it derived between the tables. And that way, you can't really know if the data is accurate. All you know is that it could have created duplications as well. Due to the size of the context, you can't even be guaranteed that next time it will return the same answer.

In this way, we've found that it can take a lot of time to police the AI and reverse-engineer how it got to its result, to understand whether the numbers are accurate and what type of logic it applied.

This was a very simple example. There are a lot more complex examples where a lot more logic would be needed for the AI to understand what you were asking and what it needs to return. The more complex the question, the less you can trust the result.

Now imagine you have your whole team using the AI to ask questions of the data. Every request will scan all that data. All of that logic will be reapplied, and each employee might even get a different answer, due to each employee's context being specific to that request. This is one of the reasons you might start seeing articles about companies finding the use of AI to be really expensive, due to all of its token usage.

It's not all doom and gloom, though. A lot of advancements are being made in how AI handles its context windows and its memory, in terms of reducing context bloat and making sure the AI is more trustworthy and returns similar results more frequently. This, however, does not remove the issue, as AI will always remain non-deterministic, especially with reliable dynamic data. Therefore, we consider human-in-the-loop designs to be the best approach when using AI.

What Human-in-the-Loop Looks Like in Practice

So what does the human-in-the-loop design look like when it comes to reporting and business analytics?

What we've seen work very well is that the user prompts the AI to generate a dashboard for them. The AI comes back with a very nice-looking dashboard. It's highlighted some interesting metrics and then the human goes off and recreates that dashboard using tools such as Power BI, Power Query, or Excel.

In this way, the AI has assisted the user in thinking through the problem, it's broken the ice, and it's done a lot of the work for them. And with the user recreating that dashboard themselves, they've had the opportunity to step through all the business logic the AI applied, and to see where the AI went wrong or what type of business rules it derived, and how to maintain it.

With the dashboard or the Excel being formalized as well, this means it can also be refreshed automatically, rather than re-prompting the AI to generate another dashboard. This ensures determinism, because it's all been formalized as a proper structure. Now your whole team can use that Excel and save a lot on token usage.

This works even better when a data warehouse is implemented, as that ensures relationships stay clean and narrow, and that all the data is properly grouped and all the columns are named properly. So when the user prompts the AI to generate the dashboard, the AI has even more information to use to derive interesting metrics and to understand what visuals to use. And it also ensures that all the business rule mappings are clean and well maintained.

Three Main Takeaways

To wrap up, there are three main takeaways:

1. Token costs are real, and they scale fast — especially when you aren't governing your AI.
2. Non-determinism is a real problem with AI, as it means you cannot continuously trust the results, especially when it comes to reporting.
3. Human-in-the-loop designs don't cause bottlenecks. You should consider this design to be a quality gate control, to make sure your environment stays lean and operates properly.

So just remember: AI should always be considered your junior dev, and it's a very valuable part of a team. But ownership of the results should be taken and reviewed by senior developers, or senior members of your team.