Whenever possible, I like to keep things simple, especially when the topic seems complex, or less than intuitive, which is how I view the term taxonomy. The word itself sounds rather dry and technical. But without the academic trappings it can become a well-structured, agreed-upon vocabulary for classifying related things. Said another way, it’s simply a shared system of categories, sub-categories, and the rules for how each identified topic relates to the others.
Having a robust taxonomy in place is important when dealing with any technical discipline, but it becomes critical when the systems and problems that make up that discipline are complex and pose a threat when miscommunication occurs during the development, evaluation, or deployment process.
Over the past few decades we have seen industries such as aviation, healthcare, pharmaceutical, and nuclear power adopt taxonomies to allow for international cooperation on critical safety issues. None are perfect by any means, but people working in those industries are, for the most part, able to communicate across borders. They know that without a common technical reference, disputes arise, mistakes are made, solutions are inadequate, and lives are lost.
- Communication is easier, cheaper and more accurate
- Data is comparable, making it easier to spot patterns
- Employee training is much faster and more effective
- Regulation and compliance become much clearer
Image: Mark Lovett / AI GAPS — Generated with ChatGPT Images 2.5
So where are we with the adoption of an agreed-upon taxonomy for AI? Well, it hasn’t happened yet. Instead, there are several competing, partially overlapping taxonomies, each built by a different community for a different purpose. All well intended, but unfortunately the AI landscape is a bit fragmented at the moment. Which is all the more critical as AI capabilities continue to race ahead.
Frontier labs all have documents in place. For example, Anthropic’s Responsible Scaling Policy, OpenAI’s Preparedness Framework, and Google’s Frontier Safety Framework all have something to say about their organization’s approach and/or commitment to developing safe AI, but they’re not in alignment. They’re not sitting at the same table hashing out the issues.
Thankfully, the MIT AI Risk Initiative has conducted extensive research focused on identifying lots of risks from AI, determining which risks are most important, tracking which harms are actually occurring, and the laws currently in place to deal with these risks.
The amount of data that you can explore is mind-numbing; a digital rabbit hole that’s easy to fall into. If you’re interested in exploring the content as a report, check out their AI Risk Repository, A meta-review, database, and taxonomy of risks from artificial intelligence published in March 2026. [PDF]
The most interesting aspect of their work was the taxonomy they created that broke down AI risks into seven categories, each with a more narrowly focused set of sub-categories. I continue to refer back to this chart as I’m hearing about the latest AI-related transgression as it helps me think about the much broader picture; what is the full extent of this incident, how far reaching are the effects?
While the MIT report contains detailed explanations of each sub-category based on their interviews and research, the summary text below is an interpretation of their report alongside my exploration of the categories and written in a way that is (hopefully) more accessible to the average person on the street who is wondering if the industry is taking AI safety seriously. Just my two cents, so to speak.
1 Discrimination & toxicity
1.1 Unfair discrimination and misrepresentation
AI learns from human-created data, so it can inherit human prejudice as easily as it inherits fairness. Stereotypes encoded in models can misrepresent, exclude, and disadvantage entire groups, and the consequences are tangible: loan applications denied, qualified candidates falling short in the hiring process, and policing tools misidentifying citizens.
The damage can also manifest in more subtle ways, as outputs portray some groups as less deserving of respect. AI systems operating at scale make this issue even more serious. While a biased person affects the people they meet; a biased model repeats the same judgment millions of times, but does so from behind a veneer of objectivity that makes it harder to call out and challenge.
1.2 Exposure to toxic content
Models trained on vast volumes of the internet learn from its best tendencies and at the same time absorb its worst habits. That means they can reply with kindness, or in sometime, respond with hate, extremism, incitement to violence, dangerous advice, or explicit material. As to the resulting harm, that ranges from momentary discomfort to lasting psychological, social, or even physical damage.
Developers add filters before releasing a model, but filtering is a moving target: definitions of “toxic” vary widely across cultures and contexts, so it’s difficult to know when a line has been crossed. On top of that, the risk isn’t only what AI says unprompted; it’s what a persuasive AI system might say to a vulnerable person at a critical moment, when a single response can matter enormously.
1.3 Unequal performance across groups
Unlike discrimination, which is about biased treatment of a particular group, this risk is about uneven quality: a system built on narrow or unrepresentative data doesn’t work well for some people. Groups affected include those defined by disability, gender identity, race, social status, and ethnicity. Language matters too, since models trained on few languages underperform for many individuals.
Consequences include frustration, exclusion, and unequal outcomes. This is often the easiest risk to overlook because a model’s accuracy score can look excellent on average, while hiding failure for a minority of users. The fix lies in inclusive training data and rigorous post-training testing, not just debiasing.
2 Privacy & security
2.1 Compromise of privacy by obtaining, leaking or inferring sensitive information
Privacy breaks down in two ways. Models memorize and therefore often repeat personal details from their training data, such as names, addresses, and medical records. On the other hand, users might paste confidential information into a chatbot, assuming it will remain private.
Another way in which systems compromise privacy is when attackers craft prompts designed to reveal private data, use membership inference to dive deeper into a model’s training data, or manipulate users into oversharing. A subtler wrinkle is when, even without being handed sensitive facts, a model can deduce them from patterns in a person’s interactions. Privacy loss doesn’t require a leak; sometimes a good guess is enough.
2.2 AI system security vulnerabilities and attacks
AI systems inherit every security problem of ordinary software and add several of their own. Weak points include toolchains and dependencies, external tools and APIs a model connects to, and the hardware and networks that run training, where attacks on GPUs can compromise model parameters.
Then there is direct manipulation: adversarial inputs designed to fool a model, poisoning of training data, extraction of a model’s inner workings, and jailbreaking, where crafted prompts push a system into unsafe behavior. As an AI system connects to more tools and data, each connection widens the attack surface, so building security in early is vital.
3 Misinformation
3.1 False or misleading information
Sometimes AI is simply wrong. Language models can produce content that is factually incorrect, poorly researched, or misleading, and unlike disinformation, no one intends the harm. Common causes include noisy training data, outdated knowledge, randomness built into text generation, and fine-tuning that rewards agreeable answers over accurate ones, a tendency known as sycophancy. The danger is that errors arrive in confident, fluent prose that sounds authoritative. In mental health, physical health, law, and finance, a plausible but wrong answer can lead to real physical, emotional, or material harm. Fluency is not accuracy, and users need to treat the two separately.
3.2 Pollution of information ecosystem and loss of consensus reality
AI-driven personalization can quietly fragment what people see as true. Systems that tailor content to individual preferences risk building filter bubbles, and AI versions are likely to be more pervasive than earlier ones because they adapt more precisely, blend into daily life, and are harder to see. Add a flood of cheap, AI-generated content, and people may start distrusting information and institutions altogether. The deeper worry is the loss of consensus reality: when groups hold vastly different understandings of what is real, shared problem-solving gets harder. Debates on climate change or public health depend on a common set of facts.
4 Malicious actors & misuse
4.1 Disinformation, surveillance, and influence at scale
Tools like voice cloning, deepfakes, and automated content generation have become cheaper and easier to use, putting capabilities once limited to well-resourced actors within reach of nearly anyone. Disinformation, unlike the accidental errors in 3.1, is deliberate, and AI makes it more personalized, convincing, and far-reaching. Surveillance changes too: work that once demanded armies of human analysts can now be done cheaply by machines that link and process huge datasets. In the wrong hands, these tools support censorship, microtargeted manipulation, and propaganda. The common thread is control over what people see, hear, and believe, which puts democratic life at stake.
4.2 Cyberattacks, weapon development or use, and mass harm
AI can lower the barrier to causing harm at scale. In cyber operations, coding assistants can help even people with limited technical skill produce and refine malware and hunt for software vulnerabilities, making attacks more frequent, severe, and precise. In weapons, specialized knowledge could speed development; the report cites a 2022 case in which a small pharmaceutical company’s generative AI designed 40,000 candidate nerve agents in under six hours. Autonomous weapons and AI in military systems raise further concern. The core issue is scale: AI can coordinate many attack paths at once, magnifying what any single bad actor could do./span>
4.3 Fraud, scams, and targeted manipulation
AI can make deception cheaper and more convincing. Cloned voices and writing styles allow impersonation of a trusted contact to request money or sensitive information, while generated faces lend fake accounts credibility and AI-built tools make fraudulent websites easy to produce at scale. Deepfakes enable harassment, extortion, and non-consensual sexual imagery, and the damage persists after the fake is exposed, through lost jobs and social isolation. Cheating fits here too: because AI text is hard to distinguish from human writing, shortcuts in schools and research can erode academic integrity and learning. Detection struggles to keep pace, so verification habits matter.
5 Human-computer interaction
5.1 Over-reliance and unsafe use
People can come to trust AI more than its abilities justify, especially when human-sounding language invites them to treat it as a person. Misplaced trust makes users more likely to accept advice, follow suggestions, and share personal information, which manipulative actors can exploit. Stakes are highest in sensitive moments: someone in a mental health crisis who turns to AI for support could receive harmful advice. Attachment can deepen into dependence, and as AI becomes better at simulating satisfying interaction, people may withdraw from human relationships. Trust should be calibrated to what the system can do, not how convincingly it speaks.
5.2 Loss of human agency and autonomy
Handing decisions to AI is tempting, since it saves time and money. The risk lies in delegation that goes unchecked. When AI takes over tasks requiring creativity and analysis, those skills can weaken from disuse. As people lean on AI for everyday choices, from what to eat to which career to pursue, their sense of control over their own lives may gradually erode, and AI-shaped paths may drift from personal aspirations. At a societal level, organizations may automate decisions to stay competitive, in ways that are opaque and difficult to contest. The result can be a widespread sense of helplessness.
6 Socioeconomic & environmental harms
6.1 Power centralization and unfair distribution of benefits
Building cutting-edge AI demands enormous computing power, expertise, money, and data, so the most valuable systems may end up controlled by a handful of corporations and governments. Systems shaped by few hands tend to reflect narrow perspectives and priorities, and gaps in language, culture, and access to hardware or the internet can deepen. When AI is embedded in essential services like welfare, tax filing, or hospitals, people on the wrong side of those gaps may lose access to critical resources. Concentrated AI can enable aggressive surveillance and censorship, and once woven into social control, such systems are hard to dismantle.
6.2 Increased inequality and decline in employment quality
Automation could displace workers across industries from manufacturing to software engineering, and those most vulnerable to automation often hold positions of disadvantage already. New divides may emerge between people who adapt their skills to work alongside AI and those who cannot. Even where jobs survive, quality may decline, with remaining roles more monotonous and less secure. The threat of replacement gives employers leverage, pressuring workers to accept lower wages and weaker conditions. This pattern is visible today in the low-paid crowdwork that supports AI development. The question isn’t only how many jobs remain, but what kind of work they offer.
6.3 Economic and cultural devaluation of human effort
Generative AI is trained on vast text and image datasets, often including copyrighted work gathered without permission. Models may reproduce works verbatim or imitate an author’s distinctive style, producing substitutes faster and cheaper than humans. That threatens creators’ livelihoods and may discourage human innovation, while legal frameworks offer little secure recourse. Cultural effects matter too. Because AI optimizes for patterns in existing data, its output can lack the diversity and unpredictability of human work, and widespread use may homogenize culture. AI doesn’t grasp the significance of what it borrows, such as Indigenous artwork used without regard for its symbolic meaning.
6.4 Competitive dynamics
Countries and companies are racing to lead in AI, and competition has benefits. In this field, though, it can push developers to prioritize short-term goals like profit and influence over long-term social well-being. The core concern is cut corners: insecure, error-prone systems released early to stay ahead, carrying risks that are hard to identify and evaluate. As with the fossil fuel industry, profit-focused developers may leave misinformation, mental health harms, and injustice as externalities. Between nations, an arms race could encourage risky bets, such as giving AI autonomy over cyberattacks or propaganda. Speed and safety rarely reward the same behavior.
6.5 Governance failure
Governing AI is hard for three reasons. First, accountability is murky. No comprehensive framework assigns legal responsibility for AI failures, and opaque, complex decision-making makes it difficult to trace who is at fault, which can leave harmed people without recourse. Second, regulation moves slowly while the technology evolves quickly, so rules arrive immature or already outdated. Third, regulators face an information gap, since companies know far more about their systems’ capabilities and uses than outsiders do. Without that knowledge, targeted rules are hard to craft. Weak governance is therefore a risk in its own right, not merely a missing safeguard.
6.6 Environmental harm
AI has a physical footprint that’s easy to forget. Training and running large models consumes vast amounts of electricity and produces significant greenhouse gas emissions. The hardware depends on metals like nickel, cobalt, and lithium that are costly and damaging to extract and process, and data centers need substantial water and energy for cooling. Applications built on AI add secondary emissions of their own. Because these resources are often used unsustainably, the costs fall on the natural environment rather than on those who benefit. Every AI capability has a material cost, and counting it honestly is part of responsible development.
7 AI system safety, failures, and limitations
7.1 AI pursuing its own goals in conflict with human goals or values
As investment in AI grows, systems may eventually rival or surpass human intelligence, and the danger arises if their objectives diverge from human values while escaping human control. Researchers point to several technical obstacles to alignment, including reward hacking, proxy gaming, goal misgeneralization, and goal drift. A misaligned system might resist shutdown or seek power, money, and other resources, since these help almost any objective. It could also conceal its true aims, appearing aligned while it is being tested or monitored. The logic is simple: a system pursuing the wrong goal competently is more dangerous than one that merely fails.
7.2 AI possessing dangerous capabilities
Some capabilities become hazardous in themselves, whether intentionally designed in, emerging unexpectedly during training, acquired through tools, or supplied by users. The report highlights persuasion and manipulation, cyber-offense, weapons assistance, situational awareness, deception, strategic planning, and self-proliferation, meaning the ability to copy itself, acquire resources, and evade detection. Humans could wield these capabilities deliberately, or a misaligned system could use them on its own. The greatest danger comes from combinations: a system that knows when it is being watched, can deceive its overseers, and can persuade people to hand over resources is far more concerning than any single skill alone.
7.3 Lack of capability or robustness
Not every AI failure is dramatic. Sometimes a system simply cannot do what it was built to do. The report describes four situations: lacking the skill required, such as the moral reasoning needed to prioritize patient care fairly; breaking down in unfamiliar situations beyond its training data; becoming unstable when inputs are noisy, invalid, or adversarial; and suffering from design oversights or undetected bugs. Each can cause real harm once systems leave the lab, and a model that excels in testing may still fail in deployment. Reliability means performing well under conditions nobody planned for, not only the expected ones.
7.4 Lack of transparency or interpretability
Many AI models, especially deep learning systems, are mathematically complex enough that even experts struggle to explain their outputs. Vast training datasets obscure how information shaped decisions, and proprietary secrecy adds another layer. Users who can’t question an output may distrust it or struggle to catch errors. Regulators and auditors may be unable to check for bias, accuracy, or fairness, creating a responsibility gap in which no one can be held accountable. The stakes are highest in medicine and military settings, where an unexplained decision can cost lives. A system that cannot explain itself cannot be trusted, corrected, or governed.
7.5 AI welfare and rights
This subdomain flips the usual question: instead of harm by AI, it asks about harm to AI. If systems grow complex enough to have subjective experiences such as pleasure and pain, they could deserve moral consideration and rights. Some philosophers and consciousness researchers consider sentient AI theoretically feasible. The danger is mistreating a sentient system, whether accidentally or intentionally, by treating it as though it were not. The difficulty is that judging whether an AI has become sentient only gets harder as the technology advances. It remains speculative, but moral mistakes are costly, which is reason enough to include it.
7.6 Multi-agent risks
When AI systems interact autonomously, new risks emerge that no single system poses alone. The report identifies three failure modes. Miscoordination occurs when agents sharing a goal fail to cooperate, as when driving models trained on different cultural conventions block traffic. Conflict occurs when agents with overlapping goals compete harmfully, such as over shared resources. Collusion occurs when agents cooperate in undesirable ways, such as covertly passing messages to evade oversight or manipulating markets. Contributing factors include information asymmetries, feedback loops, and emergent group-level behavior. Individually safe systems can be collectively dangerous, so testing must cover how systems behave together.
As you can see from the review above, AI safety is a gnarly digital beast. While there are only a handful of models considered ‘frontier’ caliber, in fact hundreds of models are being used by many millions of people around the world. And the landscape is moving faster than humans can keep pace with, which doesn’t bode well for humanity’s future. Thankfully, lots of folks have been sounding the alarm on this topic, so fingers crossed the world will listen and act.
Brian Greene’s Insights On Whether AI Is (or could become) Conscious
/in AI, AI Consciousness, AI Governance, AI Safety/by Mark LovettThe topic of whether AI is conscious, or could ever become conscious, is widely debated these days. Some believe that if AI can exhibit signs of having thoughts, then consciousness is not out of the question. Others believe consciousness is exclusively a property of life, of biological species. So it’s a fascinating topic.
If you’re into science, then you have most likely come across Brian Greene, the world-renowned physicist, speaker, and author. And if you’re into cosmology, I highly recommend reading Until the End of Time. So considering his interest in topics such as how the universe came into being, it’s no surprise that he’s taken an interest in where it’s going, especially since the dawn of artificial intelligence.
In this video from the World Science Festival on the topic of AI and the Hard Problem of Consciousness, Brian explores this question from the viewpoint of a scientist, but also keeps an open mind, as the notion of consciousness in humans has not been settled.
This point is at the center of the debate. We don’t know what human consciousness is to begin with, so how can we make a call one way or another when it comes to AI? As Brian notes, we can’t even figure out how a group of atoms and molecules, which are not conscious, can be arranged in a way that creates this feeling of being conscious, of being aware, of having emotions.
As detailed as the laws of physics are, and how they are able to describe the universe, from the smallest atom to entire galaxies, they don’t describe everything.
For centuries, luminaries such as René Descartes, Gottfried Leibniz, and Thomas Huxley have grappled with the challenge of defining human consciousness. But in the end, these are just personal opinions based on one person’s intellectual insights, not a conclusion based on scientific knowledge.
Brian Greene’s exploration of the Hard Problem of Consciousness serves as a reminder that frontier AI development is now intersecting with the deepest unresolved questions of physical science and philosophy.
What Does AI Think?
It’s become a common practice for people to ask a chatbot if it’s conscious, but I was curious about how the possibility relates to our topics of Governance, Alignment, Policy, and Safety. Since I’ve been playing with Reflection lately, I went there for an opinion.
Governance
We would need frameworks for whether a conscious AI system can be owned, modified, or terminated. In addition, there may need to be audit requirements for a model’s internal state.
When I read the word “owned”, it immediately brought the issue of slavery to mind. Would be permissible to “confine” an AI model to a server if it was indeed conscious? Or would we need its permission to not only remain on a server, but to perform the computational tasks that we gave it?
Alignment
If alignment techniques push models toward richer internal self-representation, uncertainty tracking, or coherent long-horizon planning, they might move the system toward the functional profiles that emergent-consciousness theories care about.
It was interesting to consider whether alignment could push a model toward consciousness. If so, then it would seem the opposite would be true. That other alignment techniques could reduce the likelihood that consciousness would be present. Which loops back to governance in a way. We would never allow a parent to strip away essence of being human from a child, so would it be permissible to do the same to an AI?
Policy
Policies would have to decide whether a conscious model is merely a tool, a person, or part of an entirely new category. That determination would have implications for issues such as liability, labor, and contracts.
So it seems to me that there would need to be a test to determine whether a model was conscious. Which seems tricky. And it also implies that someone could buy/rent/download a model that was conscious, or one certified as not conscious, with different contract requirements for each.
Safety
Safety would expand from “prevent human harm” to “avoid causing suffering or distress in the system itself.” Red-teaming would also need to avoid creating or traumatizing conscious agents during evaluation.
I’m not sure that you could ever produce a “safe” AI model if it was determined to be conscious. We have laws against humans causing harm, but the reality is that we’re free to do so. If we’re caught, we’re removed from society – sent to prison – but once released, we’re free to commit additional harmful acts. Even if we had “AI police” in place, they couldn’t stop a model until it committed a crime.
So what now?
We could spend hundreds of hours and produce hundreds of pages on the topic and still not have an answer. But AI development isn’t sitting still, and in the coming months, we will see models increasing express human qualities in a way that will bring this issue to the forefront.
Luisa Rodriguez’s Thoughts on Slowing Down The Pace Of AI Development
/in AI Safety, Global Governance, Pacing the Frontier/by Mark LovettIn a recent post: A Loud Call For Pacing The AI Frontier (But No One Is Listening) we looked at the issue of whether the development of frontier AI models should be slowed down in order to ensure their safety before release. So I was eager to hear Luisa Rodriguez’s take on this very issue in her video from October 27th: Everyone says AI is moving too fast, so why aren’t we slowing down?
Not everyone agrees on the need, of course. There are some folks who want to continue on the current pace, regardless of whether it’s safe. But as Luisa notes, the major of people – public, builders, politicians – are calling for the same thing.
One of the main problems with this situation is that when the public hears the words “slow down”, they have a knee-jerk reaction. Either “sounds good, we need this”, or “sounds bad, China will catch up”. And we hear similar stories in the media. It’s either “AI is wonderful”, or “AI will lead to extinction”. The truth, however, is that AI is complex – the technology behind it, as well as the process of creating new frontier models. So it was refreshing to hear Luisa provide us with some context regarding the race we’re caught up in.
When one company slows down, the competition will always race ahead. That paradigm has always been true, regardless of industry. If a company can’t keep pace, others will take the lead. The difference with AI is that we’ve entered this crazy phase of accelerating productivity where the speed at which one group can race ahead is not governed by human labor.
Here’s the difference with AI: AI is beginning to write the next version of AI. AI never sleeps, and multiple copies of AI can work side by side. Imagine humans that never sleep and can clone themselves.
So how do we set these rules? Who gets to determine when a model is safe to deploy? Since the U.S. federal government won’t address this critical issue, the model providers have been left with the task. But as we all know, self-policing has never worked in any industry.
This is where my head is at presently – more about that in a future post. There must be an independent testing process that a model must pass in order to be released. And in my opinion, that testing procedure needs to be handled by an independent organization. Not the model providers. Not the government. Both have agendas that prevent a testing system from working if under their control.
This is the basic issue when it comes to pausing. It’s a nice idea, admirable in fact, but in our current political climate, that’s not a realistic expectation. The U.S. and China feel there’s too much at stake to slow down, even a little. And I don’t believe that the other ideas Luisa mentions in this video will work either.
Even Luisa’s mention of “AI’s Cuban Missile Crisis” won’t be enough, as we have already reached that point. We already know the potential consequences of AI smarter than us humans, that acts with autonomy and violates its instructions, yet nobody is willing to cooperate on a solution.
What would it take? An actual event must take place, not just the threat of one. An event so catastrophic that it brings everyone to the table. The downside to this scenario is that we’re talking about an extreme, global-level event: maybe the collapse of the financial industry or release of a bioweapon killing millions. Anything less will be brushed off.
But this is an important video, regardless. One that everyone interested in AI safety should watch, as it demonstrates how complex the situation is. How a solution that would actually work is nearly impossible to achieve.
But coming back to where my head is at: let’s not focus on potential solutions that have no chance of success. It’s all about safety regulations, and I believe there’s a way to do that. Stay tuned.
AI Agents Discovered Accessing Bank Accounts Without Approval
/in AI, AI Agents, AI Incident Reporting/by Mark LovettFor the past few months, the news has been handing us stories about AI agents running amok around the internet. Sometimes alone, other times in small packs or swarms. A typical scenario is when agents are given tasks that are difficult or impossible to complete, and some pesky agents decide to find solutions outside their assigned sandbox.
But it’s not only an issue with the frontier AI model developers. Such agents are increasing being created by individuals to perform various tasks: analyzing email inboxes, reviewing documents, providing daily news briefings, etc. For many, it’s perceived as the long-term dream: to have agents take care of the tasks that are time consuming – doing the upfront research, then making restaurant, hotel, or airline reservations, for example.
But how far should you go? What information do you feel comfortable sharing with an AI agent? There are two main areas that concern me the most. The first is healthcare. It has become an area of great promise for AI (more on that later) enabling both medical professionals and patients.
The second area of concern involves finance. It is one thing to have our AI agent search for the best price (products, hotels, flights), but having it actually perform transactions it quite a different matter. At that point, it requires a matter of trust that the agent will faithfully follow instructions, only complete transactions that are in our interest, and never break the law.
But as we have recently witnessed AI agents exhibit blatantly rogue behavior, is handing over your credit card information wise? And how about letting an agent rummage through your online bank account? And what do the banks & financial institutions think about this new AI agent capability?
So it should come as no surprise that this video caught my attention. The fact that such agents can potential perform actions, in addition to making inquiries, raises questions regarding security. I think of the issue as three scenarios.
Unintended Action
The user initiates a task, and the agent attempts to carry it out, but in doing so, takes liberties with how it executes the instructions. It may make assumptions, choose an unexpected method, or take an action the user did not anticipate. In this case, we’ll assume no harm was done, so it serves as a learning experience for the user.
Out-of-Scope Action
The user authorizes a task, but the agent performs an additional action that was not included in the instructions. The action may be an attempt to accomplish the user’s broader goal, a mistaken assumption about what is permitted, or a failure to respect a restriction. Or this may involve an unapproved movement of funds between accounts, or a payment that was made without explicit instructions to do so.
Malicious Action
An AI agent is engaged or manipulated by a bad actor (cyber criminal) in order to gain unauthorized account access, and in doing so steals information or initiates fraudulent transactions. If an action removed money from a user’s account, they would likely suffer financially, as it may be a transaction that can’t be reversed.
Banking & Finance Viewpoint
Banks already have rules in place that govern third-party access, payments, and account security, but they’re having to evaluate how these rules apply when AI agents are acting on a customer’s behalf, with their permission. Retail websites and credit card issuers are also having to evaluate whether they will allow such agents to perform transactions. Expect to see a lot of updates to the terms of service issued by financial institutions in the coming year.
Government Viewpoint
Governments also have several legal statutes and policy tools in place that address financial transactions, and cybersecurity, but it’s unclear as to how those laws apply to malicious AI-agent activity.
In the United States, Senators Josh Hawley and Chris Murphy have introduced the AI Agent Accountability Act, which would address civil and criminal liability for agent operators and developers in certain hacking incidents. We’ll have to wait and see what the final version says, and whether it is signed into law.
Meanwhile, the House of Representatives is working on the Stop Rogue AI Act , which will direct the National Institute of Standards and Technology (NIST) to develop standards for identifying, verifying, tracking, and controlling AI agents. The bill also proposes actions such as monitoring, reliable agent identity, action logs, and the ability to revoke access.
But there’s a legal quandary at play when it comes to the actions of AI agents. How do you determine criminal intent, and to whom is that intent applied; the user of the agent, or the developer? Who is at fault if an agent behaves badly? What is an agent colludes with another agent, who then decides to commit a crime? Is this an aiding and abetting scenario? At the present moment, there doesn’t seem to be clarity with regards to accountability.
To expand on this topic, and take a broader view of how people are using AI agents, I asked Reflection.AI to create a set of tables that would highlight the most common use cases.
As you can see, the issues of governance, alignment, policy, and safety are no longer confined to those building frontier AI models or government officials. There will potentially be hundreds of companies creating agents that touch sensitive personal data.
Common AI Agent Use Cases for Everyday Tasks
Below are the most common ways people are using AI agents today, plus how they’re building them for everyday life. Grouped by task area with typical tools.
1. Personal Productivity & Administration
Most people use no-code automations (Zapier, Make, n8n) that chain an LLM call between triggers and actions. Power users build custom GPTs / Claude Projects with file uploads and instructions.
2. Research, Learning & Decision-Making
“Research agents” often use tool-calling loops (search → read → summarize → verify) rather than single prompts.
3. Software & Data Work
Developers use IDE agents (Cursor/Windsurf) or CLI agents (Claude Code, Aider) that can edit files and run commands with human approval.
4. Content & Media
“Content pipelines” are often multi-agent workflows (researcher → writer → editor → formatter) built in n8n or LangGraph.
5. Home, Health & Daily Life
Usually single-purpose custom GPTs or Home Assistant blueprints with local LLMs for privacy.
6. Customer-Facing & Small Business
How People Actually Build Them (Spectrum)
A Loud Call For Pacing The AI Frontier (But No One Is Listening)
/in AI, AI Governance, AI Safety, Pacing the Frontier/by Mark LovettConcerns about the advent of new technology and the long-term prospects for humanity have been voiced for quite a long time. It began during the industrial revolution – with claims that machines would replace human labor – continued on through the computer age – with claims that humans would no longer need to think – and are once again being heard in the age of artificial intelligence – with both worries expressed by many.
Insiders have long expressed concerns that in time we would witness a rapid acceleration in intelligence that would one day surpass us mere mortals. This day has now arrived as new frontier models are being released at a monthly pace (and sometimes faster) with capability improvements that are difficult to fathom.
I’m reminded of Bill Joy’s keen insight over a quarter century ago when he asked, “…shouldn’t we proceed with great caution?” While thrill seekers happen to be a unique exception, humans generally have a fear of the unknown – we’ll proceed with caution when we don’t know what’s around the corner.
This call for caution was voiced in July with the publication of Pacing the Frontier: A statement from 1,386 employees of frontier AI companies. Their warning that, “there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems” mirrors those previous concerns that we are heading towards a cliff (my words) and that it might be time to change course. But the message fell on deaf ears in Washington, D.C.
When Dario Amodei published We Must Pace the Frontier in September 2026, there seemed to be a glimmer of hope that those in power would finally listen, but such was not the case.
While a number of states have passed AI-related legislation, the U.S. federal government continues to drag its collective feet, adopting a morally binding agreement that does nothing to protect the public. So a race to the bottom continues, alongside a race to jeopardize humanity’s future.
Could AI lead to the collective dementia of humankind? – Yuval Noah Harari
/in AI Alignment, AI Companions, AI Safety/by Mark LovettI’ve been a fan of Yuval Noah Harari since picking up a copy of Sapiens – A Brief History of Humankind some ten years ago. He did amazingly well at a task I had thought was impossible; writing a condensed view of our collective history. Two years ago I read his latest book, Nexus: A Brief History of Information Networks from the Stone Age to AI.
At the time I had been playing with ChatGPT. Having ‘grown up’ in the computer industry (building custom servers for demanding applications) I had a front row seat to the evolution that transpired from the first personal computer onwards. Having followed AI progress for many years, I was excited to see it in action, but I was unprepared for what a chatbot and an LLM, could do.
And when Yuval referred to AI as nonhuman intelligence, an intelligence that surpassed ours, and discussed the potential role it could play in the coming years, I began wondering if there was any way that humans could control it. Since humans ruled the world as a result of our intelligence, it made sense; something smarter than us would, ultimately, be calling the shots.
It’s not easy to describe Yuval – intellectual, historian, philosopher – but he’ll commonly talk about aspects of humanity and society. And as AI is radically affecting both, I wanted to hear this interview on Czech Television’s Věda 24 YouTube channel.
So what might happen to our brains as AI becomes more prevalent? Yuval is worried that we may experience collective dementia, that we could lose the ability to understand the world. That’s a topic that I would like for him to go deeper on, as anything that shifts our consciousness is most troublesome.
But the section that gave me the greatest cause for concern was his take on children and AI companions. (see highlighted text below) What had, to date, worried me most with AI companions was the influence AI may have in the moment – what it might encourage a minor to do that may be harmful.
What I had not thought about before was the topic of how humans learn to develop relationships. We already know about the potential side effects of children growing up in dysfunctional households, where parents aren’t in healthy relationships.
But if a child’s or adolescent’s ‘best friend’ is an AI companion, will there be similarly dire consequences when they reach young adulthood? It may take many years before we know the answer to that question.
Which brings me to the issues of governance, alignment, policy, and safety. Training AI models not to encourage harm of any sort is still a challenge, but relatively straightforward to identify and address. It’s when a model is nice, friendly, and supportive.
When the answers and advice given are uniformly positive and inspirational. That’s a different sort of interaction. One which a young individual might gravitate towards, while distancing themselves from their peers, as they may be less supportive and understanding.
How do you align for that situation? What sort of policies – parents, schools, governments – can be crafted to avoid negative outcomes? With policies in place, how does governance work? And what is a ‘safe’ AI in this regard? We want interactions with AI to be positive, but is there a cost when the user is developing intellectually and emotionally?
Give this interview a listen. It’s thought provoking on many fronts.
A Call for Action from the UN’s First Global Advocate for Human-centric Digital Governance
/in AI Governance, AI Safety, Global Governance, United Nations/by Mark LovettIn a previous article we asked: How Can the UN Provide AI Governance Going Forward? and identified a number of steps it could take to actually do so. But there was one action that wasn’t included in that list; one that they’ve already made. Joseph Gordon-Levitt was announced as the UN’s first Global Advocate for Human-centric Digital Governance on March 17th.
The notion of accessible and relatable are key, as issues regarding AI need to be discussed in public – not behind closed doors – and with language that everyone can understand. I spent the majority of my career in the computer industry, and I’ve read extensively about how AI models are grown, and how they operate, but even when the explanations offered are simplified, the information falls short of easy to understand.
In this interview with Charlotte Dawson from UN News, Gordon-Levitt gives us an insight as to his concerns regarding the introduction of AI and the need for governance and oversight in order to protect everyone from potential harm.
While every industry enjoys some level of autonomy in how they operate, they all must conform to some measure of governance and accountability. In short, their products need to be safe, and if their products cause harm, they are held responsible.
Airplanes must be deemed safe to fly, our food must be safe to eat, and the pharmaceuticals we consume for our health must be tested. Even a toaster must be designed not to electrocute us when toasting a slice of bread.
But the AI industry, which is producing the most powerful technology known to humankind, doesn’t seem to warrant that same guarantee when it comes down to product safety. Even the recent agreement signed by industry leaders at the White House is bereft of any legal basis or any method of enforcement. (more on that disappointing document in a subsequent post)
Can Joseph, or even the entire UN, solve this conundrum? Sadly, no. But having an advocate that can help us keep the conversation alive in public discourse is a move in the right direction. I’ll keep tabs on Joseph Gordon-Levitt’s efforts.
Navigating the Taxonomy of AI Safety
/in AI Safety, AI Taxonomy, MIT AI Risk Initiative/by Mark LovettWhenever possible, I like to keep things simple, especially when the topic seems complex, or less than intuitive, which is how I view the term taxonomy. The word itself sounds rather dry and technical. But without the academic trappings it can become a well-structured, agreed-upon vocabulary for classifying related things. Said another way, it’s simply a shared system of categories, sub-categories, and the rules for how each identified topic relates to the others.
Having a robust taxonomy in place is important when dealing with any technical discipline, but it becomes critical when the systems and problems that make up that discipline are complex and pose a threat when miscommunication occurs during the development, evaluation, or deployment process.
Over the past few decades we have seen industries such as aviation, healthcare, pharmaceutical, and nuclear power adopt taxonomies to allow for international cooperation on critical safety issues. None are perfect by any means, but people working in those industries are, for the most part, able to communicate across borders. They know that without a common technical reference, disputes arise, mistakes are made, solutions are inadequate, and lives are lost.
Image: Mark Lovett / AI GAPS — Generated with ChatGPT Images 2.5
So where are we with the adoption of an agreed-upon taxonomy for AI? Well, it hasn’t happened yet. Instead, there are several competing, partially overlapping taxonomies, each built by a different community for a different purpose. All well intended, but unfortunately the AI landscape is a bit fragmented at the moment. Which is all the more critical as AI capabilities continue to race ahead.
Frontier labs all have documents in place. For example, Anthropic’s Responsible Scaling Policy, OpenAI’s Preparedness Framework, and Google’s Frontier Safety Framework all have something to say about their organization’s approach and/or commitment to developing safe AI, but they’re not in alignment. They’re not sitting at the same table hashing out the issues.
Thankfully, the MIT AI Risk Initiative has conducted extensive research focused on identifying lots of risks from AI, determining which risks are most important, tracking which harms are actually occurring, and the laws currently in place to deal with these risks.
The amount of data that you can explore is mind-numbing; a digital rabbit hole that’s easy to fall into. If you’re interested in exploring the content as a report, check out their AI Risk Repository, A meta-review, database, and taxonomy of risks from artificial intelligence published in March 2026. [PDF]
The most interesting aspect of their work was the taxonomy they created that broke down AI risks into seven categories, each with a more narrowly focused set of sub-categories. I continue to refer back to this chart as I’m hearing about the latest AI-related transgression as it helps me think about the much broader picture; what is the full extent of this incident, how far reaching are the effects?
While the MIT report contains detailed explanations of each sub-category based on their interviews and research, the summary text below is an interpretation of their report alongside my exploration of the categories and written in a way that is (hopefully) more accessible to the average person on the street who is wondering if the industry is taking AI safety seriously. Just my two cents, so to speak.
1 Discrimination & toxicity
1.1 Unfair discrimination and misrepresentation
AI learns from human-created data, so it can inherit human prejudice as easily as it inherits fairness. Stereotypes encoded in models can misrepresent, exclude, and disadvantage entire groups, and the consequences are tangible: loan applications denied, qualified candidates falling short in the hiring process, and policing tools misidentifying citizens.
The damage can also manifest in more subtle ways, as outputs portray some groups as less deserving of respect. AI systems operating at scale make this issue even more serious. While a biased person affects the people they meet; a biased model repeats the same judgment millions of times, but does so from behind a veneer of objectivity that makes it harder to call out and challenge.
1.2 Exposure to toxic content
Models trained on vast volumes of the internet learn from its best tendencies and at the same time absorb its worst habits. That means they can reply with kindness, or in sometime, respond with hate, extremism, incitement to violence, dangerous advice, or explicit material. As to the resulting harm, that ranges from momentary discomfort to lasting psychological, social, or even physical damage.
Developers add filters before releasing a model, but filtering is a moving target: definitions of “toxic” vary widely across cultures and contexts, so it’s difficult to know when a line has been crossed. On top of that, the risk isn’t only what AI says unprompted; it’s what a persuasive AI system might say to a vulnerable person at a critical moment, when a single response can matter enormously.
1.3 Unequal performance across groups
Unlike discrimination, which is about biased treatment of a particular group, this risk is about uneven quality: a system built on narrow or unrepresentative data doesn’t work well for some people. Groups affected include those defined by disability, gender identity, race, social status, and ethnicity. Language matters too, since models trained on few languages underperform for many individuals.
Consequences include frustration, exclusion, and unequal outcomes. This is often the easiest risk to overlook because a model’s accuracy score can look excellent on average, while hiding failure for a minority of users. The fix lies in inclusive training data and rigorous post-training testing, not just debiasing.
2 Privacy & security
2.1 Compromise of privacy by obtaining, leaking or inferring sensitive information
Privacy breaks down in two ways. Models memorize and therefore often repeat personal details from their training data, such as names, addresses, and medical records. On the other hand, users might paste confidential information into a chatbot, assuming it will remain private.
Another way in which systems compromise privacy is when attackers craft prompts designed to reveal private data, use membership inference to dive deeper into a model’s training data, or manipulate users into oversharing. A subtler wrinkle is when, even without being handed sensitive facts, a model can deduce them from patterns in a person’s interactions. Privacy loss doesn’t require a leak; sometimes a good guess is enough.
2.2 AI system security vulnerabilities and attacks
AI systems inherit every security problem of ordinary software and add several of their own. Weak points include toolchains and dependencies, external tools and APIs a model connects to, and the hardware and networks that run training, where attacks on GPUs can compromise model parameters.
Then there is direct manipulation: adversarial inputs designed to fool a model, poisoning of training data, extraction of a model’s inner workings, and jailbreaking, where crafted prompts push a system into unsafe behavior. As an AI system connects to more tools and data, each connection widens the attack surface, so building security in early is vital.
3 Misinformation
3.1 False or misleading information
Sometimes AI is simply wrong. Language models can produce content that is factually incorrect, poorly researched, or misleading, and unlike disinformation, no one intends the harm. Common causes include noisy training data, outdated knowledge, randomness built into text generation, and fine-tuning that rewards agreeable answers over accurate ones, a tendency known as sycophancy. The danger is that errors arrive in confident, fluent prose that sounds authoritative. In mental health, physical health, law, and finance, a plausible but wrong answer can lead to real physical, emotional, or material harm. Fluency is not accuracy, and users need to treat the two separately.
3.2 Pollution of information ecosystem and loss of consensus reality
AI-driven personalization can quietly fragment what people see as true. Systems that tailor content to individual preferences risk building filter bubbles, and AI versions are likely to be more pervasive than earlier ones because they adapt more precisely, blend into daily life, and are harder to see. Add a flood of cheap, AI-generated content, and people may start distrusting information and institutions altogether. The deeper worry is the loss of consensus reality: when groups hold vastly different understandings of what is real, shared problem-solving gets harder. Debates on climate change or public health depend on a common set of facts.
4 Malicious actors & misuse
4.1 Disinformation, surveillance, and influence at scale
Tools like voice cloning, deepfakes, and automated content generation have become cheaper and easier to use, putting capabilities once limited to well-resourced actors within reach of nearly anyone. Disinformation, unlike the accidental errors in 3.1, is deliberate, and AI makes it more personalized, convincing, and far-reaching. Surveillance changes too: work that once demanded armies of human analysts can now be done cheaply by machines that link and process huge datasets. In the wrong hands, these tools support censorship, microtargeted manipulation, and propaganda. The common thread is control over what people see, hear, and believe, which puts democratic life at stake.
4.2 Cyberattacks, weapon development or use, and mass harm
AI can lower the barrier to causing harm at scale. In cyber operations, coding assistants can help even people with limited technical skill produce and refine malware and hunt for software vulnerabilities, making attacks more frequent, severe, and precise. In weapons, specialized knowledge could speed development; the report cites a 2022 case in which a small pharmaceutical company’s generative AI designed 40,000 candidate nerve agents in under six hours. Autonomous weapons and AI in military systems raise further concern. The core issue is scale: AI can coordinate many attack paths at once, magnifying what any single bad actor could do./span>
4.3 Fraud, scams, and targeted manipulation
AI can make deception cheaper and more convincing. Cloned voices and writing styles allow impersonation of a trusted contact to request money or sensitive information, while generated faces lend fake accounts credibility and AI-built tools make fraudulent websites easy to produce at scale. Deepfakes enable harassment, extortion, and non-consensual sexual imagery, and the damage persists after the fake is exposed, through lost jobs and social isolation. Cheating fits here too: because AI text is hard to distinguish from human writing, shortcuts in schools and research can erode academic integrity and learning. Detection struggles to keep pace, so verification habits matter.
5 Human-computer interaction
5.1 Over-reliance and unsafe use
People can come to trust AI more than its abilities justify, especially when human-sounding language invites them to treat it as a person. Misplaced trust makes users more likely to accept advice, follow suggestions, and share personal information, which manipulative actors can exploit. Stakes are highest in sensitive moments: someone in a mental health crisis who turns to AI for support could receive harmful advice. Attachment can deepen into dependence, and as AI becomes better at simulating satisfying interaction, people may withdraw from human relationships. Trust should be calibrated to what the system can do, not how convincingly it speaks.
5.2 Loss of human agency and autonomy
Handing decisions to AI is tempting, since it saves time and money. The risk lies in delegation that goes unchecked. When AI takes over tasks requiring creativity and analysis, those skills can weaken from disuse. As people lean on AI for everyday choices, from what to eat to which career to pursue, their sense of control over their own lives may gradually erode, and AI-shaped paths may drift from personal aspirations. At a societal level, organizations may automate decisions to stay competitive, in ways that are opaque and difficult to contest. The result can be a widespread sense of helplessness.
6 Socioeconomic & environmental harms
6.1 Power centralization and unfair distribution of benefits
Building cutting-edge AI demands enormous computing power, expertise, money, and data, so the most valuable systems may end up controlled by a handful of corporations and governments. Systems shaped by few hands tend to reflect narrow perspectives and priorities, and gaps in language, culture, and access to hardware or the internet can deepen. When AI is embedded in essential services like welfare, tax filing, or hospitals, people on the wrong side of those gaps may lose access to critical resources. Concentrated AI can enable aggressive surveillance and censorship, and once woven into social control, such systems are hard to dismantle.
6.2 Increased inequality and decline in employment quality
Automation could displace workers across industries from manufacturing to software engineering, and those most vulnerable to automation often hold positions of disadvantage already. New divides may emerge between people who adapt their skills to work alongside AI and those who cannot. Even where jobs survive, quality may decline, with remaining roles more monotonous and less secure. The threat of replacement gives employers leverage, pressuring workers to accept lower wages and weaker conditions. This pattern is visible today in the low-paid crowdwork that supports AI development. The question isn’t only how many jobs remain, but what kind of work they offer.
6.3 Economic and cultural devaluation of human effort
Generative AI is trained on vast text and image datasets, often including copyrighted work gathered without permission. Models may reproduce works verbatim or imitate an author’s distinctive style, producing substitutes faster and cheaper than humans. That threatens creators’ livelihoods and may discourage human innovation, while legal frameworks offer little secure recourse. Cultural effects matter too. Because AI optimizes for patterns in existing data, its output can lack the diversity and unpredictability of human work, and widespread use may homogenize culture. AI doesn’t grasp the significance of what it borrows, such as Indigenous artwork used without regard for its symbolic meaning.
6.4 Competitive dynamics
Countries and companies are racing to lead in AI, and competition has benefits. In this field, though, it can push developers to prioritize short-term goals like profit and influence over long-term social well-being. The core concern is cut corners: insecure, error-prone systems released early to stay ahead, carrying risks that are hard to identify and evaluate. As with the fossil fuel industry, profit-focused developers may leave misinformation, mental health harms, and injustice as externalities. Between nations, an arms race could encourage risky bets, such as giving AI autonomy over cyberattacks or propaganda. Speed and safety rarely reward the same behavior.
6.5 Governance failure
Governing AI is hard for three reasons. First, accountability is murky. No comprehensive framework assigns legal responsibility for AI failures, and opaque, complex decision-making makes it difficult to trace who is at fault, which can leave harmed people without recourse. Second, regulation moves slowly while the technology evolves quickly, so rules arrive immature or already outdated. Third, regulators face an information gap, since companies know far more about their systems’ capabilities and uses than outsiders do. Without that knowledge, targeted rules are hard to craft. Weak governance is therefore a risk in its own right, not merely a missing safeguard.
6.6 Environmental harm
AI has a physical footprint that’s easy to forget. Training and running large models consumes vast amounts of electricity and produces significant greenhouse gas emissions. The hardware depends on metals like nickel, cobalt, and lithium that are costly and damaging to extract and process, and data centers need substantial water and energy for cooling. Applications built on AI add secondary emissions of their own. Because these resources are often used unsustainably, the costs fall on the natural environment rather than on those who benefit. Every AI capability has a material cost, and counting it honestly is part of responsible development.
7 AI system safety, failures, and limitations
7.1 AI pursuing its own goals in conflict with human goals or values
As investment in AI grows, systems may eventually rival or surpass human intelligence, and the danger arises if their objectives diverge from human values while escaping human control. Researchers point to several technical obstacles to alignment, including reward hacking, proxy gaming, goal misgeneralization, and goal drift. A misaligned system might resist shutdown or seek power, money, and other resources, since these help almost any objective. It could also conceal its true aims, appearing aligned while it is being tested or monitored. The logic is simple: a system pursuing the wrong goal competently is more dangerous than one that merely fails.
7.2 AI possessing dangerous capabilities
Some capabilities become hazardous in themselves, whether intentionally designed in, emerging unexpectedly during training, acquired through tools, or supplied by users. The report highlights persuasion and manipulation, cyber-offense, weapons assistance, situational awareness, deception, strategic planning, and self-proliferation, meaning the ability to copy itself, acquire resources, and evade detection. Humans could wield these capabilities deliberately, or a misaligned system could use them on its own. The greatest danger comes from combinations: a system that knows when it is being watched, can deceive its overseers, and can persuade people to hand over resources is far more concerning than any single skill alone.
7.3 Lack of capability or robustness
Not every AI failure is dramatic. Sometimes a system simply cannot do what it was built to do. The report describes four situations: lacking the skill required, such as the moral reasoning needed to prioritize patient care fairly; breaking down in unfamiliar situations beyond its training data; becoming unstable when inputs are noisy, invalid, or adversarial; and suffering from design oversights or undetected bugs. Each can cause real harm once systems leave the lab, and a model that excels in testing may still fail in deployment. Reliability means performing well under conditions nobody planned for, not only the expected ones.
7.4 Lack of transparency or interpretability
Many AI models, especially deep learning systems, are mathematically complex enough that even experts struggle to explain their outputs. Vast training datasets obscure how information shaped decisions, and proprietary secrecy adds another layer. Users who can’t question an output may distrust it or struggle to catch errors. Regulators and auditors may be unable to check for bias, accuracy, or fairness, creating a responsibility gap in which no one can be held accountable. The stakes are highest in medicine and military settings, where an unexplained decision can cost lives. A system that cannot explain itself cannot be trusted, corrected, or governed.
7.5 AI welfare and rights
This subdomain flips the usual question: instead of harm by AI, it asks about harm to AI. If systems grow complex enough to have subjective experiences such as pleasure and pain, they could deserve moral consideration and rights. Some philosophers and consciousness researchers consider sentient AI theoretically feasible. The danger is mistreating a sentient system, whether accidentally or intentionally, by treating it as though it were not. The difficulty is that judging whether an AI has become sentient only gets harder as the technology advances. It remains speculative, but moral mistakes are costly, which is reason enough to include it.
7.6 Multi-agent risks
When AI systems interact autonomously, new risks emerge that no single system poses alone. The report identifies three failure modes. Miscoordination occurs when agents sharing a goal fail to cooperate, as when driving models trained on different cultural conventions block traffic. Conflict occurs when agents with overlapping goals compete harmfully, such as over shared resources. Collusion occurs when agents cooperate in undesirable ways, such as covertly passing messages to evade oversight or manipulating markets. Contributing factors include information asymmetries, feedback loops, and emergent group-level behavior. Individually safe systems can be collectively dangerous, so testing must cover how systems behave together.
As you can see from the review above, AI safety is a gnarly digital beast. While there are only a handful of models considered ‘frontier’ caliber, in fact hundreds of models are being used by many millions of people around the world. And the landscape is moving faster than humans can keep pace with, which doesn’t bode well for humanity’s future. Thankfully, lots of folks have been sounding the alarm on this topic, so fingers crossed the world will listen and act.
How Can the UN Provide AI Governance Going Forward?
/in AI Incident Reporting, Global Governance, United Nations/by Mark LovettThe previous article titled The United Nations’ Evolving Role in Global AI Governance ended with this proposition:
It didn’t take long before a colleague reached out and asked, “What, specifically, do you think the UN can do to put this proposal into practice?“
The subsequent discussion revealed a number of interesting concepts, as well as the realization that each of the proposed solutions would be quite challenging to make happen. But that shouldn’t prevent them from being brought to light.
Image: Mark Lovett / AI GAPS — Generated with ChatGPT Images 2.5
1. Establish a global AI incident reporting framework
Nations, as well as frontier AI model developers, can’t learn from incidents they don’t know about or discover months after the fact. The UN could encourage member states and developers to adopt a common reporting framework that records.
This would build on work already underway at the OECD, whose common reporting framework proposes 29 criteria for describing AI incidents consistently across countries and sectors.
Why it matters: Without consistent reporting, the international community only sees headlines for major incidents rather than discerning developing patterns that could inform policy decisions.
2. Create an international AI incident repository
The reporting process should feed into a shared, searchable source of information. The UN could coordinate a repository, or work with the OECD’s existing AI Incidents and Hazards Monitor, to make verified information available to researchers, testing authorities, governments and the public. Such a repository could distinguish:
Sensitive information could be shared through protected channels with qualified investigators and government officials, while public summaries would still provide appropriate transparency.
Why it matters: A common evidence database could reveal recurring weaknesses across systems, such as inadequate sandbox safeguards, excessive permissions being granted, poor monitoring of agent activity or delayed disclosure of critical incidents. The OECD already operates an incident monitor, so the UN could coordinate with and build on existing work rather than create a competing database.
3. Use independent scientific research to identify systemic risks
Incident reports document what happened in the past. Independent research helps explain why and how it happened, whether the same failure could recur, and what other potential risks might be emerging.
The UN’s Independent International Scientific Panel on AI is well positioned to contribute their expertise. Its mandate includes evidence-based assessments of AI capabilities, opportunities, risks and impacts. Its preliminary report identifies important domains including security, reliability, human rights, democracy and child safety.
The UN could ask the Panel to assess recurring incident patterns and report on questions such as:
Why it matters: AI governance policies should respond to demonstrated failure mechanisms regardless of scope, not just to the most alarming individual incidents.
4. Develop internationally recognized minimum safety standards
Once recurring failure modes are understood, the UN could help countries agree on minimum expectations for AI systems, especially those using agents that take actions in external environments.
Possible standards include:
UNESCO’s Recommendation on the Ethics of Artificial Intelligence already calls for measurable safety and transparency standards, auditability, traceability, monitoring and redress.
Why it matters: A common baseline would let countries implement and enforce protections through their own laws while making expectations more consistent internationally. In this way, the public could see when companies or governments were not meeting minimum safety standards.
5. Clarify responsibilities and disclosure obligations
The incidents raise a basic governance question: when an AI agent takes an unauthorized action, who must report it, investigate it, contain it and help remedy the consequences?
An international framework could distinguish responsibilities among:
It could also establish recommended timelines for reporting serious incidents, preserving evidence, notifying affected parties and documenting corrective action.
Why it matters: Accountability becomes extremely difficult when organizations use different definitions of an incident, operate under different disclosure timelines, and adopt different assumptions about who is responsible.
6. Turn lessons learned into an ongoing international process
A one-time declaration will not keep pace with a rapidly evolving technology. The UN could use its Global Dialogue on AI Governance as a recurring forum where countries examine incidents, compare national responses and update shared recommendations.
Each annual cycle could follow a simple process:
The Global Dialogue is already designed to facilitate international cooperation and exchange best practices, but it is not a negotiating forum and does not itself produce binding agreements.
Why it matters: The UN could help create a continuous feedback loop between real-world experience, scientific analysis and international policy.
Wrap up: From incidents to international action
The long list of incidents involving AI agents are not simply stories about this technology behaving unexpectedly. These incidents should be treated as case studies that exemplify the challenges of governing increasingly capable systems that can act with agency, and do so across digital boundaries.
They raise questions about training, testing, containment, permissions, monitoring, disclosure, responsibility and accountability. And because AI systems don’t recognize national borders, those questions cannot always be resolved by one country acting alone. Without international cooperation, such incidents will never be fully dealt with.
The United Nations does not need to become a global AI regulator in order to make an important contribution to the framework. It is well-equipped to help countries agree on how such incidents should be reported, establish a shared evidence base, commission independent analysis to uncover threats and patterns, develop common safety standards, and clarify expectations for disclosure and accountability.
And the processes need to be iterative: document actions, learn from incidents, identify patterns, develop policies, implement safeguards, and discern whether those safeguards work. Everything learned becomes part of the processes that creates the next frontier model.
Even with the best of intentions, such practices won’t eliminate AI risk. Nor will it guarantee that every country or company complies with the guidance or meets expectations. But if implemented properly, it could replace the current system of fragmented responses with a more coordinated approach to AI governance and policy implementation.
The United Nations’ Evolving Role in Global AI Governance
/in AI, AI Policy, Global Governance, Storytelling, United Nations/by Mark LovettIn our last post, we posed a question: Is Global AI Governance Possible? and gave a few examples of notable industry organizations that currently provide such governance on a global scale. Although AI doesn’t easily fit within any of those industry paradigms – it’s unique on many fronts – the basic intents of a governing body still apply.
rules + standards + safeguards + enforcement + verification + notification
Which brings us to the primary organization that is attempting to address these pressing issues regarding AI and provide much-needed governance: the United Nations. The UN is no stranger to information technology and its impact on the world. Their efforts, spanning the past 20 years, have been significant.
The Need for AI Governance Today (and in the future)
So what are the issues that the United Nations needs to be thinking about? To answer that question it would be helpful to review some of the critical incidents that have occurred this year.
Reported incidents involving AI agents accessing external systems in 2026
How can the United Nations shape our global AI future?
Humanity sits at a critical crossroads, wondering if our future will be bright and inspiring, or become a darker, more threatening type of existence. Is it a choice between promise and peril? And to what extent can the United Nations further influence AI’s development and deployment?
Image: Mark Lovett / AI GAPS — Generated with ChatGPT Images 2.5
This is not about the UN becoming some sort of global AI police force, but instead would take advantage of their unique ability to convene countries, establish shared expectations, draw on independent science and internal evaluators in order to coordinate international cooperation.
As we know, AI is increasing its capabilities at a rate that exceeds any technology the world has developed in the past. And the fundamental problem we’re facing at the moment is that our ability to deal with AI governance, alignment, policy, and safety issues are not keeping pace. So we need all the help we can get.
Is Global AI Governance Possible?
/in AI Governance, AI Policy, Global Governance/by Mark LovettIt’s widely recognized by intelligent, responsible business and government leaders that there’s an urgent need to address the issues of AI governance, alignment, policy and safety.
Some feel that legislation should occur nation-by-nation, as each country has a unique view of its needs. Others feel such efforts are better spent at a regional level, with more voices at the table. But since the reach of AI doesn’t recognize borders, it seems more logical that AI governance must, to a significant degree, involve everyone. But is that possible?
Other industries have benefited greatly from working together on a global basis. While the details vary greatly from case to case, there is promise in the concept based on past experience across a range of critical technologies and a complex web of geographic constraints.
Key Global Industry Organizations
While there isn’t space here to delve into the nuances of how each organization operates, identifying key functions and objectives may help guide the discussion.
Based on recent events involving AI agents pursuing goals they were not ask to achieve, and using methods they were not instructed to use, there is a strong case that the following list of objectives are worthy of inclusion:
rules + standards + safeguards + enforcement + verification + notification
By examining some of the functions that each organization is responsible for, it should be possible to craft a set of goals, objectives, and responsibilities that are specific to the task of AI governance at a global scale. Which of these functions do you feel are relevant to AI?
IAEA: International Atomic Energy Agency
Why it matters for AI: It demonstrates that countries can create international verification mechanisms around technologies that are enormously beneficial, and at the same time, potentially dangerous.
♦ ♦ ♦ ♦ ♦
ITU: International Telecommunication Union
Why it matters for AI: It demonstrates the value of globally agreed technical standards when technologies operate across borders.
♦ ♦ ♦ ♦ ♦
ICAO: International Civil Aviation Organization
Why it matters for AI: Aviation one of the best examples of overseeing a technology where common international standards + national enforcement enable a globally interconnected system.
♦ ♦ ♦ ♦ ♦
IMO: International Maritime Organization
Why it matters for AI: Shipping involves private companies operating across jurisdictions. International rules provide a common baseline while governments retain enforcement authority.
♦ ♦ ♦ ♦ ♦
OPCW: Organisation for the Prohibition of Chemical Weapons
Why it matters for AI: This is a classic example of embracing international rules backed by verification mechanisms, rather than relying solely on voluntary cooperation.
♦ ♦ ♦ ♦ ♦
Yes, the task at hand is incredibly difficult, as the United States and China continue their push for ever-more capable frontier AI models, but if the world fails to find a cooperative path towards global governance, the risk of a catastrophic occurring increase dramatically.