Agents: The Hype is Real, so is the Danger

I have spent the last few years exploring AI applications in one of the least forgiving corners of healthcare: clinical research billing compliance. So when people tell me AI agents are overhyped, I disagree. They are extremely powerful. In the right hands, they can automate tasks that were once a pipe dream. A properly executed AI agent can increase productivity tenfold. However, when they tell me agents are ready to run unsupervised in a hospital revenue cycle, that’s where it’s time to pump the brakes.

Both things are true at once. The hype is real. So is the danger.

The largest risk of AI in general isn’t just that it gets something wrong, it’s that it looks convincing. Let’s use the field of Quantum Mechanics as an example. My only experience with Quantum Mechanics was a course in my undergraduate studies that I barely squeaked through. It is safe to say that ‘at best’ I have a surface-level understanding of the topic. If I ask AI to break down several Quantum Mechanics research papers and draw correlations, the model will certainly make an attempt. The output will probably look convincing, use proper Quantum Mechanics terms, cite different principles and theories...but the reality is, I am not remotely qualified to review this output. The output is convincing on the surface level, but how can I actually tell if the model used outdated research, made assumptions, or solved the problem correctly? I can not truly analyze and quality-check this output, because I do not have enough of a foundation in this field to do so. This is where the danger of AI truly sits. Agents multiply this danger tenfold because instead of simply outputting incorrect answers, the agents will now act on it.

The power: from answering to doing

A chatbot answers a question. An agent takes a goal, breaks it into steps, picks up tools (a PDF parser, a database, a web search, your file system), and keeps working until the job is done. That shift from answering to DOING is the whole story.

Take the Medicare Coverage Analysis, the work I do every day. Done by hand, an analyst:

1. Pulls the schedule of events out of a dense clinical trial protocol.
2. Translates each protocol term (“Targeted Physical Exam”) into billing language and the right CPT code. There are 9 possible codes for an outpatient exam alone.
3. Looks up the National and Local Coverage Determinations tied to each code.
4. Decides, under NCD 310.1, whether each item is routine care billable to Medicare or a research cost the sponsor pays.
5. Writes the justification and builds the budget around it.

That is hours of cross-referencing per protocol. Typing a prompt into a chatbot was never going to do it; this is a multi-step, multidisciplinary task. An agent can accomplish this, because it works the way an analyst works: one step at a time, checking its tools along the way, then handing back a draft with its reasoning attached.

A few years ago I thought this level of automation was impossible. It is not anymore.

The danger: an agent acts on its mistakes

Here is the problem. A chatbot that gets something wrong gives you one bad answer. An agent that gets something wrong ACTS on it, and it does so fast, confidently, and at scale. In most industries that is an inconvenience. In Medicare billing it is a False Claims Act problem (settlements and judgments topped $2.68 billion in fiscal year 2023 alone).

The risks I worry about most:

• Errors at scale. One misclassified imaging test becomes the same misclassification across every visit, every study arm, and every patient enrolled.
• Stale data. CPT codes change every year, and NCDs and LCDs are revised on their own schedules. An agent working from a retired code is wrong from step one, and nothing in its output will look wrong.
• Made-up citations. Generative models can produce a plausible reference to an NCD or LCD that does not exist. NIST calls this “confabulation” and flags it as a specific risk in high-stakes settings.
• Confidentiality and PHI. Every sponsor relationship starts with a confidentiality agreement, and patient data falls under HIPAA. An agent with broad tool access can move a confidential protocol somewhere it was never allowed to go.
• Accountability. “The agent did it” is not an audit defense. The OIG will look at your controls and ask whether AI outputs were validated.
• Automation bias. The better the agent gets, the more tempting it is to rubber stamp it. That is exactly when errors slip through.

What responsible use looks like

None of this means “don’t use agents.” It means design them for the environment you are in. In Trial Pro, the AI-powered coverage analysis platform I advise on, a few rules are non-negotiable:

• The human makes the final call. The AI recommends. A qualified analyst accepts, modifies, or rejects, and signs off before anything is final.
• Show the work. Every recommendation carries its reasoning and a citation to a real, current source, so the reviewer can actually validate it (and the auditor can see that they did).
• Constrain the sources. Generative output is tied to a structured database of current codes and policies, not the model’s memory. This is the gap I’m building Elessar to close: maintained CMS and FDA data (fee schedules, coverage policies, drug pricing) that an agent can query directly, with every number traced back to a named CMS file and release date.
• Scope the permissions. Read access is not write access. An agent that drafts a budget should not be able to submit a claim.
• Wall off confidential documents. Uploaded protocols stay in a controlled environment and are never used to train public models.
• Train the people. Analysts need to know where the agent fails and when to override it. That is the best defense against automation bias.

None of these are new ideas. They line up with what the NIST AI Risk Management Framework already asks for: accountability, explainability, and trained personnel.

The bottom line

IBM’s Watson for Oncology recommended unsafe and incorrect cancer treatments. It became an embarrassment instead of a catastrophe for one reason: physicians reviewed the output and caught it. The human in the loop was the fail-safe.

Agents raise the stakes on that lesson, because they don’t just recommend. They act. Use them; they will change how regulated work gets done. But in healthcare, finance, and research compliance, the agent should do the work and a person should own the decision.

Fast is not the same as right.