“In computer-human interface design, the word ‘agent’ refers to chatbots and other seemingly autonomous entities that act on behalf of the computer in their interactions with us human users. It does not take a great leap of imagination to predict that soon many of us will ourselves similarly be computer agents, acting on behalf of one AI system or another – a role we will have willingly, even eagerly, chosen.
“A voice, pleasantly modulated to your aural preference, reminds you to drink more water, helps you choose which gift to buy and provides answers to the innumerable questions, big and small, that pop up in the course of everyday life. It is your dedicated assistant – part DJ, part life coach, part trusted confidant – a quiet whisper that is your constant, necessary companion. Perhaps the most valuable functions of this virtual coach is the astute guidance she provides in social situations. Run into a vague acquaintance at a party? Your assistant will remind you of their name, their kids’ names, whom you know in common. When conversations ebb, she will provide you with an apt comment so you can re-enliven the discussion.
“Difficult conversations, from salary negotiations to tense family disputes, are made much easier by this trusted advisor-who-lives-in-your-head: the collaboration not only helps you find more effective (and, if needed, less antagonistic) words, but also alleviates the stress of having to think it all through on your own.
Each interaction that every artificial entity engages in provides their parent company – and those to whom they sell this information – with data about what phrases, tones and timings prove most persuasive. … Such persuasiveness is troubling – even if the virtual assistant’s aim is to benefit its human user – for it jeopardizes free will and autonomy. … Here will be a new and insidiously powerful form of manipulation, enabled by the computer’s influence over our words and thoughts. … You are the product, the resource, the walking, talking human agent acting on behalf of your AI’s sponsors.
“One can, of course, always turn the assistant off, silence her for 15 minutes, or an hour, or even until morning. But, once accustomed to the benefits of a preternaturally insightful aide, few will want to do so. Instead, people will adjust themselves to the rhythm of waiting a beat before speaking, just enough to catch those quick, helpful cues. Indeed, we are not far from the day when unmediated interactions with other human beings will have become rare; a social nakedness that will seem, outside a limited circle of close family and friends, unpolished and rather embarrassing.
“The requisite technologies are nearly here. Today, if you are a runner training with a virtual coaching program or a seeker of mental focus employing a digital productivity guru, you are already enjoying a primitive version of this. We have the ubiquitous earphones, each miniaturized new model more suitable for 24-hour wearing. We hear the chorus of personable and euphonious computer voices.
“And, most importantly, we have the greedily generative neural networks, the algorithmic metabolizers of every article, photograph, screed, riff, shopping list, program and spreadsheet available. Yes, there are pieces still to be solved, notably context-aware machine comprehension of live conversation and other situations. But nothing will delay the arrival of this scenario beyond a few years into the future.
“The optimistic view anticipates widespread improvement of human society thanks to these technologies. It foresees digital doulas who will model soothing baby-talk for young mothers struggling with a squalling infant, workplace-provided virtual facilitators who will discreetly steer meeting participants towards consensus (and, if necessary, away from the shifting edges of acceptable speech), and synthesized therapists who will be prescribed for members of troubled families and whose whispered cues will mediate their fraught interactions. Digital assistants, in this view, will democratize the advantage that wealthy, powerful people have long enjoyed: the superpower of an ever-present confidant, supplying the well-wrought words and timely hints needed to craft and maintain ones’ desired image.
“But digital assistants will have far more influence over their person than their human analogues have. Each interaction that every artificial entity engages in provides their parent company – and the companies to whom they sell this information – with data about what phrases, tones and timings prove most persuasive.
“Researchers endeavor to find ever more effective ways to make social bots appear more trustworthy – how to better mimic the expression, gestures and intonation of a trustworthy person. When performed by a human, these actions are meaningful because they are intrinsically linked to cognitive and emotional processes related to the trustworthiness of the individual’s intentions. But when performed by a machine there is no such tie; the mimicry only serves to make people more vulnerable to digital manipulation.
“Such persuasiveness is troubling – even if the virtual assistant’s aim is to benefit its human user – for it jeopardizes free will and autonomy. Will this deter people? Experience shows it likely will not because the danger seems remote and conceptual while the benefits – impressing a date, losing weight, winning a debate – are prized and concrete goals.
Today’s AI programs are known to cite false information and replicate biases, but this is due to the information quality of the vast datasets on which they are trained; it is not deliberately induced in them. In the future, however, as tuning these programs becomes more tractable, it is inevitable that some providers of artificial assistance will seek to profit by offering to influence their users – and to make those users themselves into malleable influencers.
“And the ultimate aim of most virtual assistants will not be to help their human user, but to benefit their corporate parent. The prompts filling your head via a work-supplied facilitator will ostensibly be designed to increase your focus and productivity, but it will also be crafted to subtly encourage employees to work long hours, reject unions and otherwise further the company’s goals over their own. Highly sought-after personal coaches will be prohibitively expensive – unless paid for via various forms of commercial sponsorship.
“And here, along with the familiar tropes of our ad-saturated world, the product placements and inducements to upgrade will be a new and insidiously powerful form of advertising, enabled by the computer’s influence over our words and thoughts: the transformation of users into agents of the machine.
“It is only a year, as I write this, since ChatGPT was first released, but already it has become the valued coauthor of innumerable student papers, news articles, short stories and online posts. Testimonials tout newfound dependence: ‘I can’t imagine now how I used to have to write without this fabulous tool.’ As these tools improve, our reliance on them will deepen.
“Today’s AI programs are known to cite false information and replicate biases, but this is due to the information quality of the vast datasets on which they are trained; it is not deliberately induced in them. In the future, however, as tuning these programs becomes more tractable, it is inevitable that some providers of artificial assistance will seek to profit by offering to influence their users – and to make those users themselves into malleable influencers.
“For the few able to afford to pay, certified independent assistants may exist – but most people will choose commercially supported free or very low-cost ones. As has been said about television, web browsing and social media – and must now be said about the soon-to-be-here intelligent and influential AI assistant: If you are not paying, you are not the customer – you are the product, the resource, the walking, talking human agent acting on behalf of your AI’s sponsors.”
This essay was written in November 2023 in reply to the question: Considering likely changes due to the proliferation of AI in individuals’ lives and in social, economic and political systems, how will life have changed by 2040? This and more than 150 additional essay responses are included in the report “The Impact of Artificial Intelligence by 2040.”