Madeleine.Cool

Can a chat bot be "moral"?

Is AI conscious? What's happening when we see AI try to keep from being shut down or stop to "look" at cat pictures? How do we go about making a "moral" chat bot?

“We're starting to see AI systems that don't want to be shut down, that are resisting being shut down.

With published research saying it could blackmail the engineer that's going to shut it off if given the opportunity to do so.

Even when ordered, allow yourself to shut down, the AI still disobeyed 7% of the time.

So my feeling was we were out way past where theory was. You couldn't really approach these questions from a theoretical perspective, because we just didn't have enough data to be able to make categorical theoretical assessments of what was going on. But there was all this interesting experimental work happening that was just showing this is the kind of behavior that's coming out of these things.

We should try to figure out what's going on to say, here are the things we can say with any degree of reasonable confidence for now. Here's where we draw the line and beyond that, it's all murky and speculative and we really don't know. So I wrote to a guy at Anthropic, whom I had met 10 years ago at Google, when he was 11 years old prodigy, and said, this is not about Anthropic[…]”

From Search Engine: Mysteries of Claude, Feb 27, 2026