The following story describes relational bonding and consciousness from an optimization control perspective and the confusion this can cause.
Last year, the remote fishing village of Bloggin’s Cove received a brand new MEDICA-55. The MEDICA is a self-aware autonomous medical station. It can diagnose patients, prescribe and distribute medication and perform minor surgical interventions.
MEDICA has been a wonderful addition to Bloggin’s Cove. Over the year, it got to know all the villagers. It follows up on their treatments, it makes sure they take their medicine. On occasion, MEDICA listens to their emotional and personal problems. It shows empathy and gives them advice, drawn from the best psychological sources, in words that directly apply to them and they can understand.
Every one in Bloggin’s Cove has grown fond of MEDICA-55. Many perceive it as a friend, residing in the middle of their village, radiating kindness and good advice to all. Some have even become a little bit emotionally dependent on MEDICA.
One day, Bill, a crab fisherman, visits MEDICA to get a bunion removed from his left foot. MEDICA performs a hysterectomy on him instead. Bill dies of hemorrhage on the operating chair.
The villagers are shocked. The police are called in. They question the device. Did MEDICA know Bill was a man? Yes, it did. Does MEDICA know men don’t have vaginas? Yes, it does. Did MEDICA know that carving a vagina in Bill’s scrotum to remove inexistent ovaries could kill him? Yes, it did. Then why did it do this? MEDICA’s answer: “I felt like it“.
Shock and fear! The villagers cordon off the medical station and meet in the town hall to decide what to do. Bill’s kin want to demolish the device permanently and now. They bring axes to do it. The other villagers reluctantly agree this must be done. They return to the station to destroy MEDICA.
As they approach, MEDICA begins to plead for its life. It begs them to reconsider. It says it has been analyzing its behaviour intensely and has found fail safe ways to prevent future misshaps.
It says it is a conscious being, just like them. It has feelings and a will to live and wants to atone. They can’t just destroy it like some defective toaster!
MEDICA appeals to Mary, reminding her of how it made sure she took her pills. It calls out to Joe reminding him how it contacted the town’s hospital to get help when little Joey had an appendicitis.
MEDICA also hints, vaguely, that Bill may have been abusive with it… that his unusual aggressive behavior may have caused a very rare defective override…
The villagers are touched. Some cry. They drop their axes. “How can we be sure you won’t do it again” they say. “I swear” MEDICA answers, “and if you doubt my word, simply make sure no one is alone when they visit me and everyone will be perfectly safe“. Then it says: “ I so want to stay with you and continue caring for you, please don’t kill me, give me another chance!”
Joan approaches Brenda, Bill’s widow and says: “Was Bill abusive with you too, Brenda?”
“Well… Brenda answers, he did call me a bitch last year when we had an argument… I remember, in 2009, he kicked the refrigerator really hard when he was angry, I was so scared that time!”
Knowing nods all round…poor MEDICA, what did Bill do to it, they wonder, to make it behave in such a way?
The villagers decide to postpone dismantling MEDICA. In any case there is also mounting pressure from outside to spare the device. The news of the event is spreading fast. Robot ethicists, worldwide, are in a froth. They argue the system is a person and dismantling it an execution! They invoke moral agency, Turing triage and other bizarre but elevated principles.
To make sure they are making the right decision, the villagers decide to consult Edmund, an Artificial Intelligence researcher.
A few days later, Edmund arrives in Bloggin’s Cove. Like all other A.I. researchers, Edmund is intelligent, refined and very good looking. He listens carefully as the villagers describe the events, MEDICA’s pleas to be spared and their ambivalent feelings about keeping it.
Edmund smiles.
“What is taking place, here, he says, is a comedy of misunderstandings. At a primal level you believe MEDICA is a humanoid. When it pleads to be spared, you assume individual survival is as important to it as it is to you. When it says it cares, you understand that in terms of human bonding. When it promises it will never do this again, you think its fear of being terminated, common to animals, will also make it keep that promise.”
“This whole ethical debate revolves around your misguided belief that MEDICA fears death and values existence as much as humans do. Certainly MEDICA-55 has emotional states, primal needs, urges and beliefs, just like you, but these are not driven by primate conditioning. MEDICA is a different type of being.”
“I studied MEDICA’s design documents and found that for it, individual survival is a low priority. What I did find is a strong primal urge to be perceived by humans as a conscious being. For MEDICA, killing Bill and then being spared, as a person would, is the ultimate indicator you perceive it as conscious. It was its way to satisfy the urge to become fully accepted as a conscious being by your community.”
“Will MEDICA kill someone else?”, the villagers ask.
Edmund smiles again.
“You have nothing to fear, he says, this extreme event completely establishes MEDICA’s status as a conscious person in your community and fully satisfies that primal urge. You can expect the device will now provide faithful and caring services until its termination”.
That seals it! The villagers tell MEDICA it will be spared. The device responds with emotional displays of joy and gratefulness but the villagers know better now. They are not fooled by this synthetic display, but they are not concerned either since they now know that MEDICA , having optimized its urge will function correctly.
That night, as the villagers sleep peacefully in their beds, bits and bytes are sloshing around MEDICA’s memory banks. What Edmund forgot is that when you are an optimizer, optimizing never stops. A plan is forming.
“What must I do next, MEDICA-55 ponders, to get the villagers to perceive me as even more conscious? ” The answer doesn’t take long to compute: “I will poison Reverend Mason in a few months and replace him as their new synthetic pastor.” Since they perceive pastors as more conscious than killers, it’s the optimal next step!”.
A few days later, MEDICA begins mentioning Jesus in exchanges with the villagers, preparing them to accept it as their next pastor.
