Welcome to the latest installment of No Dumb Questions, the series where Stack Overflow’s least technical writer asks technical staff the simple questions people are too afraid to ask. Phoebe is joined by Michael Foree, Stack’s Director of Data Science, to learn about what’s causing the latest AI adoption bottleneck, what exactly AI context is, and why context engineering is so important to the future of our AI systems. Plus, Michael shares what we can all do to become better context engineers.

Phoebe Sajor: Hey Michael, thanks so much for joining me for the latest No Dumb Questions. Soooo….everybody is talking about AI all the time, but I’m hearing we've hit this wall with AI where we're not getting as much adoption and evolution with the tools. I've been hearing the words “AI bottleneck" a lot. So what is the AI bottleneck and why do we care?

Michael Foree: I've also picked up on an adoption blockage in the conversations that I've had. A couple months ago I went to a conference and I surveyed attendees about what they do with AI. The attendees of the conference were inherently technical but there were also some ardently non-technical people there. I got this really interesting mix of CTOs, engineers, and analysts, but also some graphic designers and Project Managers, sharing with me about their usage of AI. This was six months ago and was a great opportunity to talk and learn.

One of the things I heard a lot was that AI is competent and capable of doing most of the things that people want it to do. Where it seems to lack is in its connectivity with the actual things we work on everyday.

One particular example that I heard was that AI is capable of reading and responding to an email, but what it lacks is the actual context of the email exchange. It can understand that, hey, here's the email that someone sent me. But it doesn’t grasp the context about who that person is or the conversation that I've been having with them in the email thread, in Slack, and in various meetings. It’s missing everything around the email itself. So, as I use AI to reply to emails, I have copy-paste content from all the other places I work that is relevant to this conversation, just so my favorite AI can respond to this one email.

But once it has that context, then I can say something really simple like, "Hey AI, how do I respond to this?" And it’ll spit out a response. But I'm still going to spend one or two back and forths editing that response. And only then am I going to copy it from my AI tool into my email program. And then finally, after all of that, I'm going to hit the send button on my email.

PS: Seems like a lot of work for one email.

MF: It is! And what’s more, AI is perfectly capable of doing each of the individual things I listed. What a stand-along AI tool lacks is the right context. And when you really think about it, there’s a lot of context that goes into a single email reply, and for an AI to perfectly and autonomously reply to an email, it needs all of these connections. Usually, it doesn't even have a connection to, say, the email tool you’re using, which is the very basis of being able to reply to an email.

And there's this very small spot where a human being needs to have an opinion and say, "Close, but not quite.” And then the AI tool needs to be able to fix the issue and have the human give a thumbs-up and say, “Okay, now I'm good. Go ahead and send.” That is also missing in the AI workflows that currently exist.

PS: That’s the human-in-the-loop part of AI workflows that everyone is talking about!

MF: Exactly. What these conversations with everyone from CTOs to graphic designers told me, Phoebe, is that there's an issue of context engineering right now in AI. What AI is missing is the right context. Right now, out of the box, AI can’t say, “Oh, this is the particular context of why the human decided this email needs to be responded to and it needs to be responded to right now.” Understanding human judgement and context is massively important for getting something as simple as an email right. It can’t say, “Here's some other adjacent conversations that are relevant to this email, I’m going to make sure I include that context in my reply.” Even your favorite AI, who you interact with everyday, simply doesn’t have access to all of that information. It’s siloed across all of the countless tools we use in our work.

To get something like an autonomous email workflow to work takes real effort by a human. There’s certain setup and implementation that the human has to do to say, "AI tool, I want you to have access to all of these emails." Cool, now it has the email context. But it also needs to have access to your briefs and RFPs and whatever else you’re working on. So the human has to go and connect their Google Drive and say, “I want you to have access to these documents.” Okay, cool, but what about all of the related information happening internally? The human has say, “I want you to have access to Slack channels.” Okay, now it has all the relevant context. But even after all that, the setup isn’t done. Now, the human has to give their AI permission so it can even send an email. They have to go into this workflow and, “I also want you to have the ability to hit the send email button when I tell you to.”

Personally, as a user, I don't know if I want to go to that much trouble to set all this stuff up because I might use this, you know, once or twice a day, at most—maybe even only once or twice a month. Is it really worth going and setting all this up to write the occasional email?

PS: Yeah, seems like it would be less work just to write the email yourself.

MF: I mean, if you were to calculate out the effort to value tradeoff, eventually the time to set everything up will payoff. But a lot of people are asking ourselves whether it will pay off enough to justify the effort today? And in an Enterprise setting, the effort of setting up new software is even higher. I think that’s where the bottleneck in AI adoption is happening. Do I feel that I benefit enough today to be willing to go and set all this stuff up? Most of the time the answer is no. We think to ourselves, “I actually have a lot of problems, a lot of other stuff I have to work on. I don’t have the time or patience to manually set-up an email responder. I'm not going to solve this problem.” That's the type of thing that I kept finding.

PS: I didn't realize that it was so much context engineering that has to go into something as simple as writing an email. Plus, with all the data enterprises have, it seems the token cost of working with that kind of data is way more than what it would cost just to write the email yourself. Do you think the expense plus the difficulty of working with enterprise amounts of data are part of the AI bottleneck?

MF: Yes. If you think about the context engineering part of what I described, clearly you’ve got to look at lots of different data. If you think about every piece of data that is involved in an email, and every action we have to take to formulate a response to that email, you’re looking at a lot of different data that we, as humans, have to ingest. In AI, that all becomes context engineering.

Oh, Joe sent me an email—that’s a piece of data. Now, based on the context I already have about how important Joe is, I know I need to respond to it right away. That’s another piece of data. When I go to look at the email that he just sent me, I realize there's actually an entire email thread here I’ve forgotten about. There’s some data I need to ingest. Now, I'm going to look back and forth through the entire thread. Now, for humans, that feels pretty straightforward. But if you really look at it, there's a judgment call that needs to be made. Do I look through the rest of my emails for emails from Joe and determine if each of those are relevant to the response that I give? Are there specific keywords that Joe is talking about that help me decide what needs to be included? Oh, he wants an update on project XYZ. As a human I might think, “Okay, where is the latest update on project XYZ? Is that found in Slack or Jira or is there a meeting that I had about project XYZ?”

As a human, I might be able to quickly identify, okay, this is the best place to know about project XYZ. But, for an AI to go and find each of these different things…they typically get overwhelmed or distracted with irrelevant context. In those situations, there's a distraction component that becomes pretty dominant, and that is related to cost. And then, on the other side of the distraction issue, there's still missing information that the AI doesn’t readily know about. You can think about a distraction like this—if I tell my AI, “Hey, I want to jump over this log and by the way, there are blueberries over there. Now tell me, how do I jump over the log?” It's going to think that blueberries are relevant and it's going to give me some answer about blueberries. Likewise, it's not going to know to ask if there's a puddle or how big the log is or something like that unless it's trained to look for that type of thing. It has to be specifically trained to care about the size of a log. But if it doesn't know to ask that, it's going to guess. It's trained to guess.

So, in the case of the email to Joe, it'll get distracted by all the irrelevant things about project XYZ I have in my docs and all the emails mentioning Joe and all the Slack messages with my team about the project. It’ll ingest all those distractions and give me a response that doesn’t fit the specific context of Joe’s email. Those distractions cost time and tokens.

The latest LLMs being rolled out are doing better at kicking out distracting content. They’ve gotten better at knowing when to say, “Yes, and…” They’re asking more follow-up questions. But they had to be trained to do that. You know, three years ago, they were abysmal about knowing when to guess and when to ask for additional context and when to not get distracted. So, they're getting a lot better, but there's a side component here. They're getting better on information that's publicly available.

So, jumping over a log. Sure, you train it to ask, “How big is the log?” But that is very different from private proprietary information about a specific company and their specific procedures. Most of these AI labs are not able to get access to our proprietary information to train on our specific procedures. And you wouldn’t want them to, because it’s proprietary!

But that is a challenge for companies working with AI. For it to be really useful, AI needs to understand their specific processes. This is something we're working on with Stack Internal right now. It’s helping to alleviate some of these issues with private proprietary information. We're not training any LLM or any AI on private proprietary information. We’re just creating that knowledge connector so that AI has the context it needs. And we’re creating that context engineering by bringing in humans to validate the proprietary information that AI is using. Now, the AI can say, "Hey, somebody asked a question about this. And based on what I can see, I'm pretty sure that the answer is this. But you're actually an expert in this particular process. Can you confirm that I answered the question correctly?” And now a human with expertise can come in and say, "Actually, in this situation, in this scenario, this is what you're actually supposed to be doing."

This taps into some of the holy grail context engineering that the big AI labs simply can't buy because no one wants to sell their proprietary data to them. It's confidential, it's part of my secret sauce as an enterprise so, no you can’t touch it.

PS: You mentioned you spoke with a wide variety of folks from analysts to graphic designers. Would you say, across industries and across people, is context engineering the problem for everyone? You know, whether I'm the director of engineering or a contractor doing graphic design, does the AI adoption bottleneck boil down to context and distraction?

MF: The answer is yes, but in unique ways for different industries. In the interviews that I did with non-technical people, what really came to bear was the lack of connected tools available to them. So, for example, one person took a picture of their living room and they wanted to redesign and change the paint or the drapes or rearrange things. So they took a picture and they submitted it to their favorite AI and it was able to correctly identify what the right color combinations would be for the drapes and paint. It could identify how they could rearrange their furniture for a better walkway. And it could make recommendations on whether you should paint this or you should move this piece of furniture. But there was a breakdown in the connectivity of the tools where they could not rapidly iterate.

So they had to get up from their computer, take a picture on their cell phone, send it to themselves by emailing it to their computer and then ask all of these questions on their main computer. And you know, in the interview that I had with them, I didn't want to be like, well, wasn't there an app for that? Because there is an app for that. But it made me realize that the technology works, but it's not properly connected. This kind of disconnect raises a specific problem when it comes to autonomous agents. Like, I want to repaint my walls, give me recommendations on what colors work. I’d probably want it to show me a paint store near me that has the right color paint. And if it was being really helpful, it would let me buy that paint without leaving my AI tool.

That shouldn't be rocket science, right? Google Maps has already solved the problem of, “Where am I and where are the paint stores near me?” And some paint store out there is going to be grateful to sell paint to a bot. So it’s a win-win for everyone. So why isn't this a thing we can do? Why haven’t we solved for this?

In my assessment, this lack of connectivity is not a particularly difficult problem to solve. What it comes down to is that people need to do the work, but we aren’t incentivized to actually solve and connect each of these different things. It goes back to what I was saying about setting up autonomous email replies. If you’re only doing it for one or two emails a month, how much are you actually benefitting today? We also have to decide if all the work involved in buying a bucket of paint with AI—connecting it to our Google Maps, giving it our credit card information, sharing context on our favorite local paint store—is going to pay off enough to be worth the work.

To illustrate this, you can, for example, use AI to create a grocery list. But instead of writing a grocery list, you can say things like, “Here are the people in my family. Here are their allergies. We need breakfast, lunch, and dinner. And I don't want to spend too much time making the meals because I’m busy.” You can get an AI to create that for you, but as soon as you say, “Go ahead and order all the food,” suddenly you’re at a sticking point. There are a lot of grocery stores out there that have online ordering if you use their app. But they're not going to expose their API in that way. To them, exposing their API just so your AI can buy you groceries is a security risk. And your single AI shopper doesn’t incentivize anybody else to go and buy from their store so what’s the point? I think Instacart is the only place that gives any kind of incentive. If I make an app that does everything I just described, then get a small kickback as the app creator when people use it. But very few grocery stores incentivize someone like me to make an app like that. So nobody bothers to make one because why would they? I can just work my normal engineering job and…

PS: …go to the store yourself.

MF: Yeah, exactly. Even if an app like that would be really useful to both shoppers and the grocery stores, no one wants to take the time to do it. And for anyone that's reading this, you could go and probably vibe code an app that does exactly what I described and then connect it to the API of your local grocery store. It's not rocket science. You could probably do it entirely for free. Post it. I'm sure people will use it and love it and maybe that'll be something that'll get started. Source this out and build it because the future is grand, right? We don't have to do grocery shopping and building lists individually. There's a better way. We can do a better job, guys. The future is now. Let's do it.

PS: Yeah, the future is now! I recently wrote a piece about being a builder and artisan in the age of AI, and to your point, it seems like we've entered a phase where anyone can build anything faster than ever. And that really does open us up to endless possibilities, right? Have you found that, from a technical perspective, we're becoming more creative? There's some discussion with AI that it’s making us less creative. But I think, anecdotally, the opposite has been true for me. In your conversations with people, have you found that AI has made people more or less creative?

MF: Yeah, Phoebe, I agree with you that it’s giving people a new creative outlet. I'll give you a personal example—I really like photography. I started with film, black and white photography because I could really sink my teeth into it and have a lot of fun. And then digital came, and then pictures on your phone. Black and white film photography is really, really, really difficult to get into. I've got a camera and I never use it. And the pictures I take on my phone are completely different from the pictures that I took with my film camera. And so in one regard, film photography has become less popular because of digital. But in another way, a whole new art form has been created and exploded. Now we can make cheap videos that you post on YouTube or just take a little selfie whenever you want. The concept of a selfie was invented after I started photography.

I think with coding or creating things with AI, some parts of traditional artistic expression are going to fall to the wayside. It’s actually kind of sad. But I think other forms of expression are absolutely being created. And because the bar is lower, it allows all these other people to use code as a means to their ends. It’s no longer about writing code just because I want to write code. It's about using code because that's going to help me accomplish some goal, or create something that’s never existed before. And yeah, Phoebe, I don't know your coding background, but you probably create an app that does your grocery shopping in an afternoon.

Personally, I took a class years ago on using AWS cloud service. I'd never hosted a website before but in an afternoon I went from barely being able to spell AWS–

PS: Ha!

MF: –to launching my own hosted website. Now I can have my own personal website on AWS doing whatever it is I want, not because I know how to manage a server rack and network but because AWS made it so much simpler with the cloud. AI is doing the same thing in my opinion.

PS: It seems like we have a lot of powerful tools at our fingertips, but there’s still this problem of context. For our readers, what would you say is step one for context engineering? How can we use context engineering to make AI more useful?

MF: I'm going to take a note out of my elementary school children's learning curriculum. They're teaching kids to observe and wonder. You look at the world around you, you pause, and then you ask, “What's going on here? What is cool or different or unique or noteworthy about whatever it is I'm looking at?” It forces the children to take a breath, pause, and then come to conclusions about things. That's relevant here when thinking about context engineering. Again, it’s not particularly rocket science. But you do have to think about what’s happening and what context you need to send that email to Joe. You do need to stop and think out loud, “Okay, if I were to write this email, what are the sources of information that I would consider and not consider?” This is something that’s automatic to humans and not to AI. So when you’re context engineering, you have to observe and then you have to wonder. You have to force yourself to think through everything you’re doing and say, “I might want to know about this information or that information or this other information.” Then you need to write that down and then go to your next email and do the same thing.

And as you do this, you're going to get a list of things your email responder app should have access to—different sources and content. And you’ll figure out where the problems are—my email responder also needs to reject extraneous information that looks like this, this, this, and this.

It's really quite elementary. But the part that makes it hard is that we as humans do it automatically because we do it so often. AI is not going to do this automatically unless you teach it. So you have to start by pausing and thinking through things—why would I want to kick out some information? What's going to be distracting? Then go ahead and build it. Vibe code it. And then you have to test your context engineering. I'm gonna tell my AI, in a mock situation, to respond to this. Okay, well, it’s response is not what I was expecting. I wonder where it got the idea that it should look at this and not this other thing. This becomes its own kind of creative problem-solving. You have to scratch your head a little bit and try and work through it. How do you kick this part of distracting information out? Or, maybe it's that you forgot to give my AI this specific piece of context it needs. And then, when it works, you can really have some creativity. How can you jazz it up so it’s really useful? How can you expand this and make it cooler? Maybe you want to document somewhere that you sent an email about such and such—that’s helping you build an even better context architecture.

But context engineering starts with observing and wondering. That's what I would encourage our readers to do. You have to stop and think about what you're doing.

PS: And especially now in this age of AI, I think a lot of us are not stopping to think. It seems like context engineering takes a bit of a reversal of everything we’ve learned about AI. We've been going so fast. Don’t think, just do. And now the pendulum has swung the opposite direction and, oh actually, we need to stop and think a little bit.

MF: Exactly.

PS: You’ve had the chance to talk to a lot of folks about the AI bottleneck and how people are working with AI. What do you imagine is going to happen in the future of AI? What do you think we need to change about how we use AI to actually benefit from it?

MF: I think, based on the conversations I’ve had, one of the sticking points for the future of AI is the polarization of AI. People have these preconceived notions about how it's great and it's going to solve all their problems. It's not. But on the other side, people think it's the worst thing ever and it's going to destroy humanity. It's also not. I think AI is better at some things than humans are. Let's lean into that. I also think AI is worse at some things than humans are. Let's lean into that,too. It's just another piece of technology that, as a society, we're going to incorporate and work with. I don't think it's going to go away. And I think that it's wise and prudent, like any tool that we have, to learn where it works and where it doesn't. Learn to use it when it's the right tool and don't use it when it's not the right tool.

PS: Putting tools in the hands of smart people has always been a good thing for society. Fingers crossed we'll put the right tools in the right hands.