Transcript#
This transcript was generated automatically and may contain errors.
Good morning, good afternoon, and good evening depending on where in the world you're joining us from. Welcome to our expert panel, Operationalizing Agentic AI, Governance, Observability, and Human Control at Scale, brought to you by our friends at Rubrik, SAP, Posit, and Revify. I'm Jenna Holland, Director of Marketing and Digital at Data Science Connect, the world's largest data and AI ecosystem for enterprise leaders.
Today, we're excited to welcome attendees from all around the world to explore one of the toughest and most important challenges in enterprise AI, operationalizing agentic AI at scale while maintaining governance, observability, and meaningful human control. And the timing couldn't be more relevant. We're seeing agents move rapidly from demos into real enterprise workflows. And recent headlines with models autonomously compromising external organizations' infrastructure are giving us a very real preview of what can happen when capable systems are given goals, tools, and autonomy.
At the same time, major enterprise AI platforms are increasingly treating governance, identity, security, and observability, not as add-ons, but as core infrastructure for deploying agents upscale. The question is no longer simply, can AI act autonomously? It's how do we stay in control when it does?
And here to lead us into that conversation is our distinguished panel of experts today. Dr. Francis Boykin, Director of Advanced Analytics from AT&T, Shubham Trivedi, Digital Solution Advisor at SAP, Sarah Allman, Senior Developer Advocate at Posit, Girish Bhat, SVP at Revify, and Paul St. Ville, AICTO at Rubrik.
Before we begin, a few housekeeping items. To make this feel a little more in-person, we suggest you add a photo to your profile and include your bio, LinkedIn profile, and other information you want your fellow attendees to see. Next, please take advantage of the Q&A and chat features in the right sidebar. We'll be addressing your questions throughout the webinar, so be sure to upload the questions you'd like the panel to address. And if you're wondering where your organization stands compared to others on your AI journey, check out the poll questions in the right sidebar.
Finally, to learn more about how to operationalize agentic AI responsibly with the governance, observability, and human oversight needed for enterprise-scale deployment, check out the calls to action below throughout the webinar from our sponsors. Now, without further ado, let's kick off our panel.
Hi, everybody. Welcome to the stage. Thank you. Francis, I'll hand it over to you. Awesome.
So welcome, everyone, to this awesome opportunity to listen to some great panelists speak about agentic AI. Today's presentation is going to be around operationalizing agentic AI, the governance, the observability, and human control at scale. It's brought to you by Rubrik, SAP, Posit, and Revify. But I do have to give you one little disclaimer that the panelist commentary does not reflect the views and or opinions of their respective organizations.
So why don't I go ahead and introduce our panelists? I'm Dr. Francis Boykin. I'm director of advanced analytics at AT&T. I've been there about 21 years. And our esteemed panelists include Shubham, who is digital solutions advisor at SAP. Sarah, who's a senior developer advocate at Posit. Guresh, who is SVP at Revify. And Paul, who is at Rubrik.
Welcome, panelists. So excited to have you here to talk to us and all of our guests here about agentic AI.
Setting the stage: agentic AI in enterprise
So as you probably know, agentic AI is no longer theoretical. Systems that can reason, plan, take action across workflows are entering production environments, and with them comes a new operational reality. The question is no longer can AI act autonomously, but how do we stay in control when it does? I know you've heard some of the cool things that have been happening in the news. We'll talk more about that a little bit later.
But this webinar focuses on the practical mechanics of deploying agentic AI responsibly. We'll examine how to design governance frameworks that go beyond static guardrails, implement observability that surfaces intent and intermediate reasoning, and maintain meaningful human-in-the-loop oversight without slowing down innovation. From dynamic policy enforcement to real-time monitoring and escalation paths, we'll explore how enterprises can operationalize autonomy without sacrificing accountability.
There are going to be a few great takeaways. Key to those takeaways include governance for autonomous systems, how to design policy layers, permissioning, models, and approval workflows that align agent behavior with organization risk tolerance and regulatory requirements, observability beyond outputs, why logging final responses isn't enough, and how to instrument reasoning traces, tool usage, and memory state to create transparency in agent decision-making.
Number three, human-in-the-loop by design. Practical patterns for escalation, intervention, and override that preserve human authority while enabling agents to operate efficiently across complex workflows. Finally, from pilot to production control, the architectural considerations, modular workflows, sandboxing, audit trails, and feedback loops that make agentic systems safe and sustainable at enterprise scale.
So, with all of that said, I'd like to have each of our panelists introduce themselves, give us a brief introduction about their role, and how the work that they currently do intersects with agentic AI or autonomous systems in production environments. So, I'll turn it over to our panelists.
Panelist introductions
Okay, so it looks like I'm the one. So, my name is Shubham Trivedi. I'm a digital solution advisor at SAP, and I'm focused mostly on SAP Business AI Platform, which brings together business technology platform, BDC, and AI together. And I work closely with our enterprise and corporate customers, helping and evaluating how AI can be put into the business process, and yes, get them started on this journey.
Great. Yeah, hi, I'm Sarah. I work at Posit. So, Posit makes open source and enterprise tools for doing data science in Python and R, including two IDEs. And so, what I mostly work on, I'm on the AI team at Posit, and I work on Posit Assistant, which is an agent that lives in our IDEs. It's a general purpose coding agent and a specialized data analysis agent. And a lot of my work also focuses on, like, how do we make agents reliable for data analysis? How do we, you know, push them in the direction of giving correct and reproducible answers? Yeah, I'm excited to be here.
First of all, Frances, thanks for hosting a webinar with four perfect strangers, at least. Right? It's kudos to you. I'm Guresh with Revify. Revify is a platform for cost and performance optimization and observability for AI agents and data platforms. We have lots of experience, not only building AI agents, but also operationalizing this. In fact, 24 months ago, we released Raiden, our first AI agent. We've learned a lot operationalizing LLM, AI agents in particular, because initial reaction from the market was they were using it, even though it had autonomy, an autonomous capability, a level five capability, people were using that as a chatbot. And over time, it evolved to the extent to now where not only can they identify where are the prompts failing for LLMs, for example, and what are the prompts, how the prompts need to be fixed. Not only we have solved the token economics problem, like months and months, quarters ago, many quarters ago before anyone was talking about it. And we have customers like AMD, Stanley Black and Decker, who are using it at massive scale for data cost optimization and performance.
For me, if I think one last comment I wanted to make about agents is I think of agents as smart teammates. Everyone is focusing on making agents smarter, that's important. But I frankly think it's a little bit easier. But making the agents or adopting or adapting them as your teammates, I think that's a critical operational challenge is what we are seeing from our customer base. I'm glad to be part of this panel.
I think of agents as smart teammates. Everyone is focusing on making agents smarter, that's important. But I frankly think it's a little bit easier. But making the agents or adopting or adapting them as your teammates, I think that's a critical operational challenge is what we are seeing from our customer base.
All right. Hi, everyone. My name is Paul St. Phil. I'm a field CTO for AI here at Rubrik. My main charter is helping organizations design and implement their AI resiliency strategy. So the AI resilience strategy is basically saying I have AI powered business processes that I need to keep trustworthy and operational in scenarios where they're attacked, they fail, or it's just kind of like unexpected behavior. So that's kind of what I do day to day. Happy to be on the line and be a part of the panel.
From automation to agency
Amazing. So we've got some excellent folks to help us learn more about agentic AI. And we'll start off with talking about automation to agency. So many organizations are moving from scripted automation to systems that can reason, plan, and take multi-step actions. Architecturally, what distinguishes a truly agentic system from traditional AI pipelines or workflow automation? Anyone can jump in here. But what would be your thoughts for the guests here about how that actually changes the picture of where we are today from where we were yesterday?
I can jump in. I look at it very simply. So for us, and my viewpoint is that automation is a tool. It's a power tool. And in contrast, I look at an agent as a teammate, my teammate, my colleague's teammate, or our teammate. So the difference really where it comes to is, are you looking at static, predetermined workflows, whether it is ML pipelines, AI pipelines, or static data pipelines, or even sometimes even dynamic pipelines for that matter? Whereas agents have the ability to not only, if empowered, and if they truly have the right level of autonomy or agency, as we call it these days, do they really have the ability to follow a plan or even reason with the plan, adjust the plan, redo the goal as relevant, and make the relevant judgment?
I love the way that you pull out that piece that these agents work alongside you, that we need to leverage them as part of the team. That's so different than the way that we have looked at advanced analytics and AI, in my opinion, up until the last maybe six months to a year. So we've looked at it differently, and now we're really starting to focus on, hey, how do I leverage these agents in a way that makes it seem like I've got another person sitting beside me helping me out day to day? Anyone else have anything they'd like to add?
Yeah, I'll jump in with just, there's the React pattern, and then there's also the dynamic enrichment of context, like enabling skills and tools so that the agents, just like a co-worker, can go out and read your guides and see what's available. I've seen a bunch of concern about the lethal trifecta. I almost think you go backwards where you make it more like an automation when you don't trust it to actually use tools and have access to data in your environment. And then I think in the near-term future, if we're thinking about assistants and digital co-workers like Aresh just described, I think long-term memory is going to also be key to it, where you're going to want persistence across sessions, and then that's going to be a part of defining what feels agentic. It has memory, it has dynamic access to data that's relevant when it needs it. That'll make it agentic in my mind.
Taking those two points forward is because if you see a traditional automation, it follows, like was discussed, it follows a predetermined path. If you do X, Y has to happen. Whereas we are flipping the script with agentic, where you are reasoning it towards a goal, where this agentic AI is planning its own steps, it has the capability to adapt mid-executions.
So it's planning the steps rather than what the developer here has hard-coded where we have in the automation, which is previously done. It has the capability to invoke tools and take real actions even in the mid-execution phase. So it just does not return an answer. It develops this memory, and like Paul said, this memory goes across multiple sessions. It doesn't just stay there.
So the moment an agentic AI system decides to act, this governance becomes a design time conversation. Stopped being a design time, I would say, and becomes a runtime one. So this is what truly distinguishes. One where you have hard-coded the rules and it cannot think out of that box tool. And one where it has the capability to build its own rules and adapt based on the user input and the output it is expected.
Well, you bring up governance. Oh, I'm sorry. Go ahead, Sarah.
Just to say quickly, I definitely agree with everything that people said so far. I tend to think of it as like it's more you're making a goal-directed system instead of an instruction-directed system. So you can be a little bit broader with the aims of the entire agentic system than maybe you could if you were, like was just said, like you're stuck in the sort of design phase, then it needs to implement the plan.
The other thing that I think has really changed is that like the systems we're building have changed, but also the way that people are building them has really fundamentally changed with coding agents. And I think that sort of opens us up to all of these new different interaction pairing where like it's been discussed. You might think of these agents as like a teammate, but they're not people, you know, so we have to sort of think about how the human-agent interaction is going to work, both when you're building the systems and then for the end users of the systems. And I think that opens up a lot of interesting questions and design problems.
Memory, performance, and trade-offs
Thank you, Sarah. So speaking of questions, there is one question in the question section here that says, Paul spoke about the importance of giving the agent the long-term memory. Won't the performance take a hit in that case?
Yeah, I think. So one thing I'll say is that I, that's why I kind of left it as saying maybe it's a future thing. Like today, I judge it just based on access to tools. It may, but I do think like this is the trade-off that we're all playing. Like we're all trying to manage performance. We're all trying to manage what that context window has. So yes, but I think like the loop engineering, graph engineering, context engineering, it's all the same game. Like we're trying to manage how much can we get out of an agent from a productivity perspective versus the impact that it's having on performance and resources.
Yeah, if I can jump in there, Francis, I absolutely agree with Paul. It's a great question, by the way. Because I think if we believe in static systems versus learning and dynamic systems, memory is critical. Not only for tools, but I mean, humans, we have learned over time, right? All the way from before the child has been born itself, the child is learning. And now here we are, we are doing a five perfect strangers thing. We have learned a lot over the time. So I think it's critical to have the trade-off comes in, are you giving the right results? If the results are merit, perhaps a little bit of a hit in performance or what people would call latency, I know Francis, he may address this, I'm all for it. Are you giving the accurate results with the higher degree of confidence versus giving AI slop kind of answer or a hallucinated answer? I would always take a slight performance hit on that.
Are you giving the accurate results with the higher degree of confidence versus giving AI slop kind of answer or a hallucinated answer? I would always take a slight performance hit on that.
Governance frameworks for autonomous agents
Thank you. All right. So we mentioned governance. And so let's talk a little bit about governance for autonomous systems. So as AI agents begin operating across tools, we've talked about that as well, APIs and business processes, what does a practical governance framework look like? How should organizations think about permissioning policies, policy layers and boundaries for agent behavior? So how do we corral and ensure that, as you said, we don't get the crazy hallucinations? What are we looking for as far as ensuring that what we're getting out of our agents is something that is useful for business day to day?
So practically speaking, an agent needs to have three layers. One is it should have the right scoping. Number two, it should be by default, least privileged permissioning per agent, rather than having a shared service account. And most importantly, the policy layer should sit outside the agent's own reasoning. So if it tries to do a bad action, in case the model judgment fails, it stopped there and there. And most importantly, you should have a tier level of approval. Now, how would you say what do you mean by a tier level approval is low risk actions, you can give them full approval. But for anything that is non-reversible, something that has got to do with personal data, monetary thing or external communication, a human in the loop is required. So start by giving the least privileges, make sure the policies outside the agent's understanding. And if possible, for any touch points, have a human in the loop.
So these are the three ways I would look at it. Because these agentic AIs, they may work in background, but they do need a human oversight. And this is mirroring how organizations work. A finance analyst can only approve up to a certain limit. Anything over that limit, you need a finance manager. Same way for an agentic, we have to create those rules, we are giving the autonomy there. But the governance aspect that we have for human beings has to be replicated for the agentic AI. As we discussed in our previous conversation a couple of minutes back, they are like a friend, an assistant, a coach, and an executive assistant working with humans simultaneously. So the same policy and the governance rule ideally should go there.
Dynamic guardrails
So speaking of designing those guardrails, let's talk about dynamic guardrails. So we talked a little bit about the static guardrails, they often break down when agents adopt or plan across multiple steps. So what techniques or architectural patterns allow governance policies to adapt dynamically as agents reason through tasks?
I think it's been a little bit spoken to just now, which was the idea that you want something governing the agent that sits outside of its reasoning. I'm a fan of the LLM as a judge-esque approach, like you use AI to govern AI. The main challenge being that most of these things are non-deterministic. If in the beginning, we're keeping true to what you said, which is what you want your agents to have agency, then that means that what you're watching for is what's the intent of what it's doing. There are some deterministic controls or deterministic watch guards that you want that are DLP-esque. This type of information should never even traverse the prompt. But I do think that when you get into these higher to Souris point goal operated functions, it gets into this place of like, I have to try to understand the intent of what the agents reasoning through and trying to accomplish to then be able to keep pace with it and control it.
Yeah, so I absolutely agree with that. And by the way, it's interlinking those two questions was spot on. And Paul, I appreciate your context there. So why do we need dynamic guardrails, right? Because we talked about automation, which is pretty static in terms of there's no change in policies or rules as things. It's all predetermined like we discussed. What we have learned over time from a Revify perspective, and even my perspective, is there are a bunch of sequences or actions that an agent may take. Individually, if you look at each of those actions, they look pretty reasonable. And you may say, okay, there's low risk, I'll approve. But when you stitch all of them together, and with the ability to adapt and reason, all bets are off. And I think that's where one of the approaches we have done with RAID and our agent is we actually say, what are your plans? How would you go about essentially, we are reasoning with the agent to say, list out what are the steps you're going to take in this scenario? We cannot obviously cover all of them. We start with that approach.
Think of this as engaging a non-deterministic agent to some extent, and trying to give us deterministic answers. So that's something that we repeatedly do. And I know we may talk about human in the loop. So we start with that sort of reasoning with an agent first to understand how extensive does it go. And then we also look at what we have learned is cost and optimization have become big issues. I mean, everyone is talking about token economics or clamping down on spend and things like that. So we look at what's the plan, what's the finops or cost governance implication, and then trying to understand the context. And we kind of write that in. Think of this as writing into your markdown file, for example. You don't want to push a significant update on Friday 5 p.m. before going for a long weekend. So these are the things we engage with the agent upfront so that there are no broken rules to the extent possible. So so far, we and our customers haven't run into issues. There are dynamic guardrails, but we try to get the agent to spell it out in advance to the extent possible.
Yeah, sorry. I didn't mean to talk over you. I think the auto mode story is a really good example of a dynamic guardrail. So auto mode is something they have in cloud code and other coding agents or sort of like what Paul was saying. Part of the approval process is done by an LLM. So you might have your static guardrails written down, but the part of the mode is also another separate model assessing the risk of a tool request, like some code that has been proposed to be run. Something that I found interesting was a couple days ago, I think Anthropic made auto mode the default, partially because they found that in practice, it was actually safer than just showing the human the tool request every time.
So they found that I think humans were able to flag like 15% of risky actions, but the auto mode flagged 90% or something. I think this is a good example of like, it might seem riskier to do something like hand a guardrail to another LLM to enforce. But it's not always enough to just show humans the information and like expect them to be able to verify or review because like in practice, they might not actually be doing that review. I think like human, human in the loop, definitely, we definitely need humans in the loop. It sounds great. But like if in practice, people are just hitting allow every time, which is what they found was happening, the guardrail doesn't end up being enforced. So I think there's, yeah, again, there's like lots of sort of interesting problems in that space.
But it's not always enough to just show humans the information and like expect them to be able to verify or review because like in practice, they might not actually be doing that review.
I think I would take that further. So rather than human in the loop, you could have a policy document. Now when the agent is executing every time it checks against that policy document. So for example, in the case of accounts payable, you may want to have a policy document for checking whether this payment should go or not. So that's how you would probably I agree that human in the loop at every instance will not work, but validating it against a policy document, which can be taken as a set of rules can be one way of ensuring that, you know, and at the same time, it is auditable also, because if you have the agent action telling that I took this decision based on the policy document of the organization, automatically, by default, you're bringing you're bringing the auditability built in. So rather than focusing why this happened, and why did the agent do this stuff, you are giving it a grounding saying that this is my policy. This is how we operate. And so I mean, just taking that answer further.
Observability for multi-agent systems
Awesome. Just want to make sure that we we do have a couple of questions in the chat. The first one when agents work as a team using MCP, what does good observability look like for tracing decisions, handoffs and tool calls end to end?
Yeah, I don't I don't have an answer for it. But I do think like, what I'm seeing become more and more popular now is the idea of agentic identities, like, agent, identity space with agents is like a big one, lots of different views of it. But what I'm hearing a lot of popularity around is the idea that like, give your agents identities. And then when they're doing an operation, most likely it's on behalf of a human or it has some type of approval. flow, where now it's going to be mint, minted a scoped token. So now we know that the agent is doing something, it has restricted privileges for only a specific point in time. And then you want to layer observability on top of that. Because usually what happens is bringing an observability with no sense of control over what's happening, like the juice isn't worth the squeeze. So I think the idea is what's the end to the means here, you're trying to observe so that you can restrict and scope.
Yeah, so thanks for asking the question. So I'll try to not talk about River five, we do offer AI observability and AI FinOps. So what we learned was, I'll tell you a little bit of by how we ended up here. What we noticed was data platform usage. Last year was speaking, picking up, picking up as more and more LLM scope.
And it was a complete disconnect. People were even before token economics became an issue, that data spend on the Google Cloud data, BigQuery, Snowflakes, Databricks were continuously increasing. What we realized was the impact of the compute on running the AI workloads was not quantified at all.
What is an agent? Or is it a human? Which department? Attribution to showback and chargeback models? Is the prompt failing? How much time is it taking? Can you optimize the prompt? And also doing tracing, end-to-end tracing. We talked about function calls earlier, right? Let's say the first call, first prompt took 10 function calls. The next iteration might have taken nine or maybe more. So understanding that, bringing the concept of tracing to observability, particularly in the agentic context, is critical.
So that you are at least trying to take a non-deterministic infrastructure and try to get closer to the determinism which folks in IT data and AI are getting used to now.
So to put what these two gentlemen said in a much more simple language is, observability in an agentic system is like a flight recorder. You are not reconstructing what happened, but you are reconstructing why that decision was made. And that's how I would look at observability.
Because what we are thinking observability is a log file, log file tells you, I called this API agentic, I called this API, I went into this database, pulled this data out, and I did this decision. It doesn't tell you why it did it. But whereas if you look at it from a perspective of flight recorder, when you look at all the data points together, you understand why the pilot made that decision based on what factor. So I would say, think of it as a flight recorder where you need to understand why the decision has been made, so that you can understand what went wrong, what could be done better and what really happened.
So to put what these two gentlemen said in a much more simple language is, observability in an agentic system is like a flight recorder. You are not reconstructing what happened, but you are reconstructing why that decision was made.
So yeah, to put it in very simple terms with what Girish and Paul put forward. Yeah, I'll actually take it to the next level. We do it actionability and remediation too. It's not just recording, but we'll actually fix things for you. Flight recorder, we can't fix what happened on the flight path because it's one and done. But in the reality of as you're looking at tracing and end to end, we can say, you don't need to do this. You can fix this the next time. The actionability and remediation is what we also do.
Observing is interesting. Frankly, it's glorified monitoring. It's now making it actionable and remediating is where the industry is now heading towards.
Human in the loop by design
Great conversation there. I'm going to skip ahead a little bit and jump to human in the loop by design. Because I know we've touched on it a little bit, but I think that is a great area for us to dig into. So where should humans realistically sit in the loop for agentic systems? Approval checkpoints, escalation triggers or post-action review. And how do you design those mechanisms without slowing down the system's ability to operate? Anyone have thoughts there?
So it has to be matched to the risk. In fact, you should have all the three. So you should see on the confidence level the agent has and then match it to the risk. So the goal isn't to keep a human in the loop, it is to make sure the human is a step away from the loop where it matters. So depending on the tackle. So you're talking about, it could be based on what the action the agent is taking. So it depends on that.
Yeah, I really like what Shubham said there. We look at things that potentially, if you frame it from a risk, I didn't really think about it from a risk perspective, is the action reversible or not? If it's irreversible, you must absolutely, potentially, absolutely must approve it, review the trade-off.
And then is it ambiguous, meaning is the outcome clear or is it, first of all, is it high-risky? Is it ambiguous? And then which case you need approval? And then if it's mundane, well, it falls into the category of automation or regular tooling. So that's a post-mortem analysis. So that's how we look at it.
But I think Paul, initially, in the first question itself, he brought this up. The ability, sometimes latency is okay when you look at a human in the loop potentially impacting the time to the right outcome. Because if it's the right outcome, I want to emphasize that. Having sometimes human in the loop is important, paramount, to get the right outcome if that's what is important if you're in a high-risk environment.
Yeah, I'd just like to add, I think I've heard some customers talk to me about the idea of human in the loop where they, not as a blocking function or something can't proceed without a human, but more so as the means to efficient optimization. Like no, I want a human in the loop, but the human in the loop is going to be that the system's going to come back with, here are all of these improvements that I want to make to myself as far as how I'm operating, as far as you mentioned policies, as far as the policies that are governing us and how we should tweak them. That's where the humans should be in the loop.
And I think it also addresses the YOLO mode problem of like human fatigue, where now if a human's focused on partnering with the system to implement optimizations versus being a cop or a security guard, you get the best out of both worlds. You still need human in the loop though, for some blocking scenarios, but this is usually what they're asking. How do we use it for optimization?
I think we can take Paul's point forward, sorry, I cut you there. It's tuition reimbursement. All organizations give some form of tuition reimbursement, but how it is done, an email or a ticket is raised to HR that I want to do an X course from a Y university, is my tuition reimbursement allowed as per company policy?
The HR goes, checks, okay, is the employee in the organization for, let's say, two years or three years minimum before the company can go ahead and reimburse the tuition fee and then checks whether this college and this Y course is approved. Now this is a perfect case where an agent can come in and do that basic work and the approval of whether the tuition fee has to be sent at the end of a semester is where a human in the loop can come.
So information, whether I'm eligible for a tuition reimbursement and whether my college and the course I'm taking in that college, that can be taken care of in an agent AI perspective because that's a low risk action. But we would definitely like to have a human in the loop where whether the college and the transcript checking and whether the payment reimbursement has to be done, I would have a human to check both from an HR perspective and from an account payable perspective. That's how I think a balance can be done for a low risk and a high risk action.
Yeah, sorry, I just got cut off. No, awesome. Thank you so much. Any other comments?
Yeah, I think like I said at the beginning, so I work for Posit, we make mostly tools for doing data science. And so, you know, there's definitely concerns about like approval in terms of like, is this a risky action? But a lot of our concern is around, is this analysis correct? Like, you know, you're analyzing data from a clinical trial, are you getting the right answer out of that? That can be very important. And we want the analysis to be correct.
And for that use case, I think I often sort of struggle with the human in the loop term or conception, because I think people hear that, and they often think, oh, the human is the, it's like a review machine, you know, that the agent does stuff, we need a human to review, and then it's all going to be okay. I think human in the loop can mean many things, but that's often what people's conception is. And I'm pretty skeptical that that works for data analysis specifically.
If one of our earlier coding agents, I think our understanding of what was going to happen is we thought it was a little bit risky for it to be writing this data analysis code, but the code is there and visible for the human. And so the human is going to check the code. And we sort of emphasize, like, it needs human review. And I don't know if people do that. I don't know if people are reviewing this code, because there's a couple problems, like, it's so easy to not review it, like you see with the auto mode stuff, you just hit approve. And then also, if the agent writes a bunch of code, it is another task for the human to then, like, load that into their brain and understand what's going on.
And then if you're just presenting it to them and saying, you know, review for correctness or review for risk, I don't know that, like, they're going to do that, because it's a difficult task. It's missing all of this context that the agent has been constructing. And so then you need to, like, have different systems around how do you help the human get enough information so that they can make good decisions. One of the things that we do is just, like, you don't have the agent, you know, do infinite tool calls, have it for some tasks, have it do a few, so it's easier for the human to keep up or things like that.
You definitely need humans in the system, but I am really sort of skeptical of them being, like, review, yeah, review machines.
You definitely need humans in the system, but I am really sort of skeptical of them being, like, review, yeah, review machines.
And one thing I think Sarah didn't say is humans tend to, how should I gently say, get lazy. I've seen the humans relaxing, they might have been a lot more rigorous when it comes to actual coding, when they're looking at a peer's code review, when it becomes AI, a coding tool-generated thing, I've seen, like, I do the same thing, I use some of these coding tools. I assume I have a, for some sense, I have a higher and give a higher degree of trust to the tool, because smart people have created it, perhaps, but that's a mistake. So I'm getting inherently lazy. Hopefully not everyone is like I am.
Safe tool use and agent governance
So we've talked a little bit about tools. Thanks, Sarah and Girish, for pulling the tool question in as well, because the next question that I had was, how do we, we, our agents often interact with external systems, databases, APIs, internal tools, even other agents. So what strategies help ensure safe tool use and prevent agents from taking actions that exceed their intended scope?
So that's the that's the important thing going outside of the scope that they've been designed for. So any any experience with how do we strategize to help ensure safe tool use? I think that's one word, governance, you have to have a control tower, which actually sees what each agent is doing.
You should not only have a command line view, you should actually have a diagrammatical view to see that my agent started here, was given this input, it went ahead and got access to this data from this database, or it went ahead and interacted with another specialized agent, it together, they went ahead and did this, you need to have a top down view. And it should be, you should be able to go zoom it further and further.
So having a traceable and auditable path is really important and having a control tower of agents. So you can, you know, you can have a software that's whose core job is to just monitor what agents are doing. And you should also have a kill switch. So let's say the agent is executing a task and you realize that this is not the right task and it is doing a wrong, it's against the policy, it's against this, or maybe it's just taking the wrong set of steps. You should have a simple kill switch button which says stop it right then and there, because you can contain the damage.
So the one word for it is having a very clear set of governance. And for that, you need a control tower of agents. That's how I would like to.
I completely agree. I think that a lot of times like these agents get labeled as like overreaching and look what it did. And the idea being that I gave it a goal, but I didn't realize the scope that was open to it in achieving that goal.
Going back to a comment that you made earlier again about policies, in my mind, there's the idea that we built something to manage human cognition and that is business logic, that is regulatory frameworks. These things are naturally ambiguous so that you can operate your business, but you get what you are and aren't supposed to do.
I think it all ties back to the same idea. You need governance, you need governance driven by an AI system that's looking at the intent of what AI is doing in your environment. And you can't be overly deterministic with how it looks at things so that it can understand like, hey, I want my agents to overreach, by the way. I want them to overreach because it may give me something that I may not have seen as a path to accomplishing that goal. I just need to make sure that that overreach never has a negative repercussion on my business. So I completely agree on the governance. It's it's why I think we keep circling the same things of this idea of, you know, observability, understanding of intent analysis, and then the ability to do policy based governance instead of just trying to translate traditional rules and controls.
Great comments there. Anything else on that?
One thing I would add is I think I'll agree with what was shared. It's bringing back my AI team head concept. We give them access to what the role requires. We give them keys to specific rooms, never the master key. And it's kind of what we had previously alluded to, the concept of least privilege and zero trust and enforcing at the tool boundary, not within the reasoning layer at all, because you absolutely want to enforce at the tool boundary, because what if the agent has changed its decision? And when it interacts with other things, that's when you want to enforce the rules and governance. So that's what has been successful for our customers.
From prototype to production
Awesome. So we've talked about how we have human in the loop. We've talked about governance multiple times. We've done all of our — just imagine we've done everything that we need to do or to get to a certain point. Now we've got building out a pilot and we want to push to production. So let's talk about that. Let's talk about how we get from experiencing with agents in a controlled environment to what the biggest operational gaps are that you have seen with teams trying to move from a prototype agent into enterprise grade deployments, because that's the ultimate goal. We — it's fun to do all of the pre-work and really dig in deep and understand and build knowledge. But the goal is to push to production at some point. So can you tell me a little bit and our guests a little bit about what you've seen in your own organizations around getting from prototype to production?
OK, I can go ahead since everyone is — OK, so, well, actually, you need a couple of — so rather than governance, I think we've talked a lot about it. You need a very good data connect data set to operate on an integration set together. I think on agent AI, both they are — if you, you know, garbage in, garbage out. So if you do not have a rich data set and a very well integration set, you need to have that rich set of data and integration set in so that all the scenarios that would probably come in your production, you do not want to learn from your mistakes in your production system. You want to learn from your mistakes in the sandbox. I'm not even talking of development and quality. You need to have that rich customer data and very well integrated.
So how is my sales data, sales table data connected to my customer satisfaction data? These two are interrelated. If I don't sell well, my customer relationship is problem. I'm just putting it at a very simplistic level. But you need to find these data pointers and these data sets have to be initially put in a sandbox environment. And most importantly, how is data is interlinked, is which aspect of my unique key of my table integrates sales and customer relationship and then run that AI agent on top of it and run it in sandbox exhausted. And then you — I think you will see this is the problem because when we are in a sandbox data and let's be honest, sandbox databases and systems are not really well maintained. They are meant where people try, they do their moonshot ideas and then they leave. But I think now has become even more critical where you have these sandbox systems which have anonymized production data, making sure that no critical customer data that can uniquely identify customer uniquely, you anonymize it and put it in the sandbox and train it. And then you bring it out to production. And of course, whatever is the standard process of development, quality, and then production. So that's how I would put at it. So that's the biggest operational gap, making sure you have the right data set and the integration of that data set.
Yeah, definitely agree that like getting the data right is a big part of the process. Something that we see is like often, you know, depending on sort of the maturity of the organization or their data organization, you might have a lot of code and you have data in the right places, but there's a lot of business logic or additional context that lives in the data team's heads or lives somewhere that the agent can't access. And that can be like a major problem if you're trying to make some kind of agent that does something with the data or ingest the data, does analysis on the data. Because the tables themselves or even like, you know, minimal data dictionaries often can't tell the agent enough information for it to do, you know, joins that make sense or analyses that make sense.
You — even things like, you know, you have two columns with the same similar name and your data team knows from five years of experience that you always use this one for this case, but they don't have that written down anywhere. The agent isn't going to know that. And then it's going to give you the wrong answer. So figuring out where that team wants to, you know, encode that information, getting that encoding done successfully. And I think also having a good feedback loop for if you have an agent, it's answering questions about that data. You're encoding the context somewhere. You need to know when questions were answered incorrectly, so that you can improve the context layer with underlying data.
Another thing we also come across is that when you're doing a prototype, it's often — make a prototype. Everyone who has access to the prototype has the same level of data access. When you move it to production, you now have people across the organization who need to use the agent and each person has different permissions. And so you need to think about, you know, user level scope permissions and which might mean that different people are going to get different answers out of your agent. And that can be really important.
Yeah, I think I quickly want to interstitch those two. What we have seen is, to your question, Frances, prototypes are — there's always this innovation lab or team that's tag team that comes together, a proof of concept that everyone is excited there. And then they declare success. The demo set, the data is not representative. And then they want to — the performance is great in this prototype of proof of concept. Then they turn it on. Then who's — who's going to — who's going to — who's running point? No one. Right. So is it a chief AI team, chief AI officer? Then it becomes organized. Literally, what means is ownership and playbook. That's what we see in enterprises that sometimes extends the scope. And also, is the scale that you got and performance that you got in a prototype? Is it representative? Maybe it didn't really represent that accurate actual data. I think Sarah was alluding to this.
And the third and most important thing, which we see is no one thinks of cost. No one thinks about the tokens being burned. Everyone looks at just, hey, I'm spending so much on cloud or chat GPT. They ignore the application. They ignore the training. They ignore the data platform impact as they do. Cost is getting to be, as we all know, in the past six months, the number one criteria for all AI use cases. Right. So I think the cost governance is a criticality. That's a significant gap we are seeing between prototype and production these days.
And the third and most important thing, which we see is no one thinks of cost. No one thinks about the tokens being burned. Cost is getting to be, as we all know, in the past six months, the number one criteria for all AI use cases.
Feedback loops and continuous alignment
Amazing. Great commentary there we have. Just about five minutes left. I want to combine the last couple of questions because I want to make sure that we close out on feedback loops and continuous alignment because that's — hey, after, as you said, after we've gone from pilot to production, there's more to be done. It is not just drop it and run. And then I also want to talk a little bit about how we build at scale. So how do we keep our autonomous AI systems controllable at enterprise scale? So looking at what are our feedback loops and continuous alignment and then how do we build out or scale as well? So let's jump there and see what we can get there in the last few minutes.
Yeah, I guess I'll jump in and try to pull together a couple of concepts here. So I think you want a — ideally a single policy engine that is able to review all of your runtime activity to identify when things are going awry with misaligned intent for whatever reason and implement some type of blocking function or notifying and pulling in a human. And then you'd want a secondary capability in that system to help you with optimization of workflows where applicable. Ideally, you know, that would be AI powered and ideally it'd operate in natural language to remove barriers between teams. But I think it's underlined by the idea of, I need very rich observability. I also need some type of LLM as a judge or AI to govern me. And then in the end to cap it off, I also want some type of optimization engine, something that's looking at it though, and can think outside the box.
I think the most effective feedback here could be where human feedback is concentrated. So let's say agentic AI has done a certain task and it has a confidence level as low or it's uncertain. I think that's the feedback we should be more focused on. So you should be looking at it constantly, you know, like a security — you know, you can constantly keep updating your cyber security policy. So same way as feedback, you have to constantly keep checking. And the beauty of agentic AI is it's able to tell the confidence of its answers. So in my case, it's able to tell the confidence of its answers. So anywhere the confidence is, let's say, as per company policy, that for cases related to accounts payable, confidence has to be minimum, let's say 99%. Anything below that, you need a human check. So I think that's a good way of, you know, making sure those feedback loops work.
So it's alignment of this is not a one time, but you have to constantly keep working on it. So it's not a one time activity because your vendors are changing, your suppliers are changing, your customers needs are changing. So the agentic AI tool has to change accordingly. So keep working on it. That's how I would say. So you just govern the plan and not the output. That's how I would like to put it.
So you just govern the plan and not the output.
Yeah, definitely agree. One other reason why these things are constantly changing is that the models are constantly changing. So unless you want to be stuck with a model from a year ago, you're going to need to update what models the organization allows the agent to use. And then it can be important that, you know, if we switch to a new model, is everything that we know about our current system going to fall apart? And so for that, like having evals that you run, you know, on the working agent, and then with a new model that you're testing out can be really important. And yeah, just having feedback loops, where some — in some way the human is involved and can work to improve.
But yeah, I think a lot of the times these — working on agents can feel like you're trying to hit a moving target, just because you do one thing, and then a new model comes out and the things you thought the model — you know, LLMs are bad at, and suddenly they're good at, but now the behavior is slightly weird in another way. And so it can be really important to have evaluation and feedback loops. The only thing I would add, Frances, is human interaction is critical, because when humans take an action, it's deterministic. That's how we can improve the quality of the overall agentic system.
Closing thoughts and audience questions
All right. That — this has been an amazing opportunity to learn so much about how these systems can — opportunity to learn so much about how these processes work, how we can move forward with building out agentic AI within our organizations. It's been an awesome panel, such great information. I do have a couple of questions, and I'm not sure if we'll be able to get through them all, but the one question that I did want to go back to was a broad question about a faculty member who's online, who's a faculty member teaching statistics and data science, and what kind of AI tools and training do our students need to be able to work with to stay competitive in the job market? I just think that's an important question to get through, so I wanted to throw that out there before we sign off.
Well, I will add this. One of the things that I like from a learning and a student perspective, because I'm a lifelong learner, is any free webinar, any free session that you can go to — I just went to one with LangSmith two days ago, and it was amazing. There was a small part of it that included a workshop, but the great thing is that it was free. They fed us breakfast and lunch. I know students will enjoy that, but the great thing is that it gives you an opportunity to not only see these tools in action, but it also gives you the opportunity to talk with so many. There was at least 100 people there, and we had an opportunity to mix, mingle, talk, and learn from each other, so definitely, I think, use some of the free opportunities to learn some of this as well.
Yeah, and what I would say, absolutely — I just give a shout out to LangChain. They make LangChain. Oh, LangChain, yeah. Yeah, so great, great tool vendor. One thing I would always encourage, pick your tool of choice. It, frankly, doesn't matter, eventually, in the long run, whether it's your open AI cloud, whatever. Think systems. Having a systems thinking is critical in everything we do, whether it is from a data science perspective, SAP, or Rubrik, or others, right? Think end-to-end systems, because if you do things in siloed, it's not as effective.
I always have a systems viewpoint. That's what I would encourage students to think.
Well, thank you so much from our panelists. Thank our guests for showing up to hear such great information be imparted here, and please do look out for more of our opportunities to learn from panels and workshops within the Data Science Connect community. Thank you so much.
Thanks, guys. Have a great day.
Thank you all for joining us today. Great insights from our panelists, and special thank you to Frances for moderating. If you missed any of this webinar or want to re-watch anything, the recording will be available a few minutes after if you refresh this page. And additionally, it will be posted on our YouTube page tomorrow. Thank you again to our partners at Rubrik, SAP Posit, and Revify. We'll see you all next time.