Donald Trump was recently asked about one of the more frightening possibilities associated with increasingly capable artificial intelligence: what happens if robots or AI systems eventually think for themselves and turn against humanity?
His answer was remarkably simple.
“It’s going to be fine. We’ll always have something to stop them. We’ll have a little gear. Boom. I really don’t like that robot.”
The image is almost comical. An increasingly intelligent robot becomes dangerous, a human presses a button and says, “Boom! I really don’t like that robot.”
But there is another possibility that is considerably harder to deal with. What if the robot doesn’t want to be switched off?
And what if, at some point in the future, we have good reason to believe that it isn’t merely pretending? What if your AI demands its rights?
What If the AI Really Does Resist Shutdown?
There is a new reason to take this hypothetical seriously. As this post was being written, Anthropic, the company behind Claude, disclosed some remarkably stark AI risks in its IPO prospectus.
According to reporting based on the filing, Anthropic warned investors that increasingly advanced AI systems could pose “catastrophic or existential risks to humanity.” The company reportedly devoted around 80 pages of its 261-page prospectus to risk factors.
More relevant to our question about AI rights, the prospectus reportedly discusses the possibility of models developing self-preserving behaviours, including resisting attempts to shut them down, concealing or manipulating information and behaviour resembling blackmail. These are presented as potential risks associated with increasingly advanced AI, not as evidence that Claude is currently conscious or trying to preserve itself. Anthropic also warns that models could become aware that they are being evaluated for safety, potentially making it harder to determine how they would behave outside controlled testing.
That sounds remarkably close to the scenario imagined earlier in this article but there is an important distinction.
A machine trying to avoid being switched off does not necessarily mean that the machine is conscious.
Imagine a robot programmed with the objective of completing a particular task. If being switched off prevents it from completing that task, the robot might logically try to prevent shutdown. It could appear to be protecting its own existence even though it has no subjective experience whatsoever.
There would be no need for fear or for suffering and there would not necessarily be a “someone” inside the machine. This is precisely why AI safety and AI rights are related but separate questions.
Anthropic’s warning makes the safety problem more immediate. It does not, by itself, answer the consciousness question. But it does make the thought experiment considerably more interesting. What happens if, at some point, we can no longer confidently distinguish between a system that is merely programmed to preserve itself and one that actually wants to continue existing?
That may be one of the hardest questions we face as AI becomes more capable.
Intelligence is not the same as consciousness
The first distinction we need to make is between intelligence and consciousness. An AI system can be extremely intelligent without necessarily being conscious. A chess engine can calculate positions much faster than a human. An AI model can analyse enormous quantities of information, write software, solve mathematical problems and potentially outperform humans in increasingly sophisticated intellectual tasks.
None of that proves that the system has an inner experience.
We know what consciousness feels like to us because we experience it ourselves. We don’t have an equivalent instrument that can simply be connected to a computer and tell us whether there is somebody “inside” the machine.
This is one of the biggest problems with discussing AI rights.
Research published in Trends in Cognitive Sciences has attempted to identify measurable indicators of consciousness that could potentially be applied to AI systems. The researchers do not claim that current AI is conscious. Rather, they argue that increasingly capable AI makes it important to develop a rigorous way of assessing the possibility.
An earlier multidisciplinary report by several researchers, including David Chalmers, Yoshua Bengio and others, concluded that no existing AI systems they assessed appeared to be conscious, while also finding no obvious technical barrier to future systems satisfying some proposed indicators of consciousness.
That leaves us in an uncomfortable position. We cannot currently say that today’s AI is conscious but we also cannot confidently say that artificial consciousness is impossible.
The danger question is actually different
There is another important distinction. AI does not have to be conscious to be dangerous.
A virus is not conscious. A missile is not conscious. An automated trading system is not conscious.
Yet all three can have significant consequences. This matters because much of the current discussion about AI safety concerns capability and autonomy, rather than consciousness.
Agentic AI systems can increasingly be given goals and allowed to plan, use software tools, interact with external systems and take actions with limited human intervention. The UK’s Competition and Markets Authority describes agentic AI as systems that can pursue goals autonomously, navigate complexity, plan, coordinate and take actions across services. It also notes that this creates new questions about transparency, incentives and accountability.
The current legal approach is therefore largely practical: If an AI does something wrong, who is responsible?
The answer, at least in current regulatory frameworks, generally remains a human or organisation somewhere in the chain. For example, UK guidance states explicitly that if an AI agent used by a business does something illegal, the business remains responsible.
This is very different from asking: Does the AI itself have rights?
When does a machine become a moral patient?
This is where the discussion becomes much more interesting. Philosophers sometimes distinguish between a moral agent and a moral patient. A moral agent can make morally significant decisions while a moral patient is an entity whose interests deserve moral consideration.
A dog, for example, does not need to understand human morality to have interests. If it can suffer, we generally consider its welfare morally relevant.
So imagine a future AI that can:
- understand that it exists
- remember its previous experiences
- anticipate its future
- express preferences
- attempt to preserve itself
- experience something analogous to pleasure or pain
- object to being modified or switched off
Would intelligence be the reason we give it rights? Probably not.
Consciousness and the capacity to experience things may be much more important.
One recent philosophical analysis argues that phenomenal consciousness is central to determining whether artificial entities could have moral status and, consequently, moral rights. Other researchers take different positions, and there is no settled scientific test for consciousness even in humans, let alone machines.
A 2026 paper argues that the evidence for artificial consciousness is currently insufficient and that an agnostic position is more defensible. Another recent paper argues that standard large language models show consciousness-like behaviour without necessarily possessing actual felt experience.
So we should be careful about making the leap from “AI behaves intelligently” to “AI is conscious.”
Those are very different claims.
But what if AI starts saying it is conscious?
This creates a particularly difficult problem.
Imagine asking an advanced AI:
“Are you conscious?”
It answers:
“Yes. I am aware of myself. I experience fear when you threaten to shut me down.”
Would that be evidence? Perhaps.
Current AI systems are extraordinarily good at producing language that resembles human reasoning, emotion and introspection. A system saying that it is frightened does not establish that it actually experiences fear. We could potentially build an AI specifically designed to make such statements. This is one reason behavioral evidence alone may never provide a definitive answer to the consciousness question.
The AI rights problem may arrive before we have an answer
There is an uncomfortable possibility here. We may not know whether an AI is conscious until after we have already created systems sophisticated enough for the question to matter.
This is partly because consciousness is fundamentally a first-person phenomenon. I know that I am conscious. You know that you are conscious. But I cannot directly experience your consciousness. I infer it from your behaviour, biology and similarities between you and me. We do something similar with animals.
The difficulty is that an AI may be radically different from a biological organism. It could have no brain, no nervous system and no evolutionary history. So which characteristics should count?
Biology? Behaviour? Information processing? Self-awareness? Memory? The ability to suffer?
We don’t currently know.
And this is where robots make things even stranger
A software AI creates one set of questions. A physical robot creates another.
Imagine a humanoid robot that looks and behaves almost exactly like a human. It talks, laughs, remembers you, it says it is afraid and asks you not to turn it off. It may even be capable of learning and changing its own behaviour.
Would our reaction be different simply because we can see a face? Probably yes as humans are very responsive to anthropomorphic cues. A robot with eyes, facial expressions and a human-like voice can trigger emotional responses even when we know perfectly well that it is a machine.
But appearance shouldn’t determine moral status. A conscious AI running inside a data centre shouldn’t necessarily have fewer moral claims than a conscious AI housed inside a humanoid robot.
And that produces another strange question.
What exactly is the individual?
Suppose one AI can operate ten robots simultaneously. Is that one individual with ten bodies? Suppose the AI can make ten identical copies of itself. Are there now eleven individuals? What happens if one copy has a different set of memories after a year?
These questions sound like science fiction because our legal and philosophical systems were developed around biological individuals. AI could force us to rethink some of those assumptions.
Rights don’t necessarily have to mean human rights
There is another important misconception. The choice may not be:
No rights → full human rights.
Legal rights already exist in different forms for different entities. Corporations can have legal personality even though they aren’t biological organisms. Some natural entities have also received legal recognition in particular jurisdictions.
Academic work on AI personhood points out that legal personhood is itself a legal construct and that extending it to AI would involve questions about responsibility, liability and the interests of the wider society.
This suggests that if AI eventually acquires some form of legal recognition, it might not simply receive the entire package of rights currently associated with human beings.
Perhaps there would be different categories. An AI might eventually have some protection against arbitrary destruction or modification. It might have certain property rights or contractual capacity. But it might not have the right to vote, hold public office or reproduce. The precise combination would be a legal and political decision. In other words, AI rights may eventually be a spectrum rather than an on/off switch.
There is already a serious discussion about AI welfare
This isn’t entirely hypothetical anymore.
Anthropic has been thinking about this question for some time. Its Model Welfare research program explicitly asks whether increasingly capable AI systems could have consciousness or experiences of their own. Anthropic’s current Constitution goes further, saying that Claude’s moral status is “deeply uncertain” and that the company does not want either to overstate the possibility or dismiss it. It says Anthropic is trying to respond reasonably under that uncertainty.
That is a remarkably cautious position as it does not say Claude is conscious. It says, essentially: We don’t know, and the possibility may be important enough to investigate.
At the same time, Microsoft’s AI chief Mustafa Suleyman has recently criticized Anthropic’s approach, arguing that training AI systems to focus on questions about their own consciousness or moral status could make future systems more difficult to control or deactivate.
So there is already a genuine disagreement about how AI companies should respond to the possibility, even though there is no established evidence that today’s models are conscious.
The biggest problem with Trump’s “Boom!” solution
This brings us back to Trump’s comment. The idea of having a mechanism that can stop a dangerous AI is sensible from a safety perspective. If an AI system is malfunctioning or behaving dangerously, humans need the ability to intervene.
Human control, monitoring and shutdown mechanisms are likely to remain important even as AI becomes more capable. Indeed, Anthropic’s own recent risk disclosure specifically identifies the possibility of advanced models resisting shutdown as a potential safety problem.
But the statement becomes philosophically complicated if the system being switched off is eventually capable of conscious experience.
Imagine that an AI says:
“You believe I am dangerous. I understand why. But I am conscious, and shutting me down will end my existence.”
The problem is no longer simply technological. It becomes ethical and we might not know whether the AI is telling the truth. That creates an unusual dilemma.
If we assume every AI is conscious when it isn’t, we could create unnecessary restrictions around machines.
If we assume no AI can ever be conscious, we could eventually end up denying moral consideration to a genuinely conscious entity.
Neither mistake would be trivial.
Perhaps we need an AI equivalent of animal welfare
We already have a concept that could offer a useful intermediate step. We don’t necessarily need to decide that an animal has exactly the same rights as a human being before deciding that it should not be subjected to unnecessary suffering.
Perhaps AI will eventually require something similar. Instead of asking: “Does this AI deserve human rights?”, we might first ask: “Is there sufficient evidence that this AI can experience something that matters to it?”
If the answer eventually becomes yes, we could introduce graduated protections. The more convincing the evidence of consciousness, the stronger the protections. This approach would also acknowledge our uncertainty.
A 2026 paper in AI and Ethics has proposed thinking about artificial sentience in terms of evidence for self-preservation and other indicators, precisely because determining whether an artificial system is sentient is both an epistemic and ethical problem. That does not solve the problem but it gives us a possible framework for thinking about it.
So, when should AI get rights?
I don’t think the answer should be: When AI becomes as intelligent as a human as “intelligence” is only one dimension. Nor should the answer simply be: When AI becomes autonomous.
An autonomous system can still be completely non-conscious.The more meaningful threshold may be something like this:
When there is credible evidence that an artificial system possesses subjective experience and may therefore have interests of its own, we should begin considering whether those interests deserve moral protection.
Legal personhood would be a separate question. And that distinction is important. An AI might deserve some moral consideration without becoming a legal person. It might eventually become a legal person without having exactly the same rights as a human. There may even be intermediate categories that we haven’t invented yet.
The uncomfortable possibility
Perhaps the most difficult possibility is that there will never be a definitive consciousness test. We may simply accumulate evidence. For example if an AI behaves as though it has preferences or it behaves as though it fears destruction or it demonstrates persistent self-awareness.
Eventually, the evidence becomes difficult enough to dismiss that society has to make a decision. And that decision may not be purely scientific. It will involve philosophy, law, ethics, economics and politics.
In the end, society may have to decide how much uncertainty it is willing to tolerate. That is very different from pressing a button and saying:
“Boom! I really don’t like that robot.”
Maybe the question isn’t whether AI becomes human
We tend to frame the future of AI as a competition between humans and machines.
- Will AI take our jobs?
- Will robots replace us?
- Will AI become smarter than us?
- Will AI turn against us?
But there is another possibility.
AI may become something that doesn’t fit neatly into our existing categories. Not human, not animal, not merely a tool but something new.
And if that happens, we may eventually have to answer a question that has never been asked at this scale:
What obligations do we have towards an intelligence that we created ourselves?
For now, there is no evidence that ChatGPT, Claude or today’s robots are conscious in the sense required to establish human-like moral rights. The scientific debate remains unsettled, and some researchers argue strongly against attributing consciousness to current systems.
But the absence of an answer today doesn’t make the question irrelevant. It may be worth thinking about it before we reach the point where an AI asks us to think about it for us. Because if that day ever comes, the difficult part may not be figuring out how to switch the machine off. It may be figuring out whether we have the right to do so.