I think we’d all prefer it if chatbots never emitted sentences such as 'You should kill yourself.' However, for all the times that 'honesty' is mentioned in Claude’s constitution, I would argue that it is fundamentally dishonest to have a machine emit many categories of sentences, including any sentences using first-person pronouns."
-Ted Chiang, No, Artificial Intelligence Is Not Conscious















