ChatSh*t and Other Conversations (That We Should Be Having, But Mostly Are Not)

The history of education, together with the construction of infrastructure and the application of technology to support that education, from schools and libraries for the former, and the printing press and the internet for the latter, has arguably been about the democratization of knowledge, where in this context “democratization” means providing comprehensive access to knowledge that is relevant, accurate, age-appropriate, up-to-date and meaningful (whether this has been a key driver in the history of education is arguably a different matter). The outcomes of education include the transformation of an individual from uninitiated to initiated in democratic citizenship [1], civic participation and living lives together (better) [2], and as human beings exercising the fullness of their capacities (e.g., Aristotle’s eudaemonia).

This process of democratizing knowledge has, of course, been accelerated in the years since the creation of the internet and mobile communications, and their global penetration in terms of time, geography, and population. Indeed, in the 2020s, there is a generation for some of whom the distinction between “online” and “offline” is almost meaningless, and distinctions of space (between physical and digital) and personhood (between body and avatar, or avatars) are increasingly blurred. For such people, the sequence of technological developments that leads from gopher site to search engine to voice-activated virtual assistant to large language model (LLM) is almost entirely natural and inevitable: each is simply providing increasingly convenient and broader access to a near-complete archive of human knowledge and creativity (i.e., complete to the extent of what knowledge has been digitalized up to the time of the model’s training).

In terms of the democratization of knowledge, LLM, as used in generative AI chatbots, can offer a remarkably powerful, effective, and convenient way of accessing knowledge and solving problems that would otherwise have required consulting a “guru,” a Q&A website (such as stackexchange), or, in pre-internet/online manual days, holding four different pages of a programming language manual open with one hand and trying to type code with the other. At the same time, chatbots such as ChatGPT (and its successors) and text-to-image generators (such as DALL-E) have caught popular public attention. It can seem that almost everyone has “had a go” and also has an anecdote about what “humorous” output has been generated by their particular interactive session with what turns out to be, in this usage, a rather naff conversation piece, albeit one that does answer back in a reasonably recognizable form of the user’s own language.

The process of democratizing knowledge has been accelerated in the years since the creation of the internet and mobile communications, and their global penetration in terms of time, geography, and population.

Quite why this large-scale attention-grabbing should have occurred is not entirely clear: perhaps, generative AI will prove to be as iconic to the “information revolution” as the steam engine is to the Industrial Revolution; or, perhaps, the technology reached a tipping point that created a pressing social need to have chatbot experience to participate in social interaction and alleviate the FOMO; or, alternatively, it was the typically overexaggerated existential threats to humankind which, as it turns out, are rather good for share price. But, in the best spirit of Marshall McCluhan, while it is probably for the better that there is some conversation about the medium, there really needs to be a broader and deeper conversation about the message, that is, the impact that technology has on people, society, and processes.1

There are, of course, several insightful commentaries on the message, for example, Malik [3], Benson [4], and Klein [5] to pick just a few, although their voices are seemingly drowned out by more sensationalist commentary on the medium. Klein [5] makes a particularly cogent critique of hallucinations. A curious outcome of generative AI has been called an (artificial) hallucination, whereby the AI/LLM produces a plausible output that, however, does not seem to be warranted by the training data (in which sense it differs from a human hallucination in that it generates a false exterior output rather than an interior experience based on a false perception). Klein [5] then identifies four hallucinations about generative AI (rather than by generative AI), which are that this new technology can solve the climate crisis, deliver wise governance, and liberate people from drudgery, while its creators (BigTech) can be trusted “not to break the world.” Arguably, none of these “hallucinations” are true; but surely the last is the least believable, and most problematic.

The problem is that it is not exactly “breaking the world” (a phrase that carries those apocalyptic undertones which need to be avoided), but “remaking the world,” in particular, remaking the world such that it is even further entrenched in asymmetric power hierarchies based on unearned privilege [6], deep inequality, and no common knowledge to democratize. So, at the risk of producing “yet another article about generative AI,” this contribution to a conversation about remaking the world in undesirable ways has six talking points: fraud, greed, sustainability, theft, and cognitive deskilling (in two dimensions).

The potential for fraud has already led to the verbification of “ChatGPT,” as in “you’ve been chatgpted,” with implied meaning “you’ve been misled by chatbot.” The phrase has been used in the context of a researcher responding to a request for a paper—the requester has almost certainly asked a chatbot for papers by the researcher and the output looks plausible, and even interesting: it is difficult to criticize the “imagination” in the “hallucination”—but it is still a complete confabulation. There are other limitations too, including context, reasoning, experience, and drift. Ask a stupid question, get a stupid answer, it is said; but for chatbots, perhaps, this should now come with a warning caveat interrogans (asker beware): ask a sensible question, maybe get a stupid answer.

This makes it clear that a generative AI chatbot is not functioning in exactly the same way as a search engine or some putative “smart” user manual. Although the recent OpenAI technical report warns that “Great care should be taken when using language model outputs, particularly in high-stakes contexts” [7, p. 10].2 This is not just about the democratization of knowledge: it could also become the democratization of misinformation (where in this context “democratization” simply means access to the capability to spread misinformation). The risk is that should LLM become a widely available technology and the costs of training (considered later) become less prohibitive, that chatbots would be trained on specific datasets and used for the democratization of disinformation, that is, the spread of misinformation (false facts) with the deliberate intention to mislead or deceive.

Klein identifies four hallucinations about generative AI, which are that this new technology can solve the climate crisis, deliver wise governance, and liberate people from drudgery, while its creators (BigTech) can be trusted “not to break the world.”

This is no longer willful ignorance [8]: it is willful bias. Two of the key features of Nowak’s [9] Regulatory Theory of Social Influence (RTSI) are, first, that there is a symmetry in information seeking: not only do sources seek targets to influence, but also that targets seek sources by whom to be influenced. Second, targets delegate information processing to sources for reasons of cognitive efficiency. The combination of deliberate bias in the training datasets, these two features of RTSI, and confirmation bias [10] is particularly disturbing: it could lead to the fragmentation of civil society into disparate epistemic universes, as groups choose to believe whatever “facts” or narratives are generated by their preferred chatbot. In this sense, generative AI could then be the first postmodernist technology, and we need a new critique of “obscurantism” (i.e., if obscurantism is making knowledge inaccessible, then training LLM on biased data is making nonknowledge accessible).

A standard internet-oriented business model is to launch a service for free. Then, if that service becomes so successful and dominant, given how the network effect at the application layer of the internet can create a de facto monopoly, the focus is no longer service provision for the common good, but relentless monetization for revenue maximization. This is often achieved either by the introduction of advertising or by subscriptions to tiered services. In the displacement of the common good by greed, both come with substantial risks.

In the first case, advertising, it is evident that some chatbots do have guardrails so that the scope of questions that are answered do have some limitations, although, to the outsider, it is not clear where the boundary between chatbot and guardrail is architecturally drawn. Product placement and industrial sponsorship are commonplace in the movies and the arts, but there is no reason to suppose that product references cannot be surreptitiously inserted into a generated text or image. Given how Google has turned knowledge search into a bunfight for advertising space that makes a shark feeding frenzy look restrained, it is not completely unlikely for a similar electronic auction to result in each generated response of a chatbot to be preceded by something like “I have brought this answer to you courtesy of …” the auction winner (and note the inevitable and misleading anthropomorphization with the first person pronoun, rather than the indirect but typical “this message has been brought to you by—, official partners of—.” It could be argued that since both advertising and generative AI are pushing hallucinations, it is a perfect match of opportunity and cynicism).

In the second case, subscription to tiered services, there is a distinct possibility of exacerbating inequality, initially caused by the digital divide. For example, there is an increasing demand for mental health services, especially among the young. Given the shortage of both qualified counselors and funding for treatment, and the cynical and woefully inaccurate snark that counseling involves a series of canned responses (“That’s terrible. How does it make you feel?”), unsurprisingly it has been suggested that chatbots should replace counselors. Notwithstanding the dangers to patients from misdiagnosis and mistreatment [11], there are two ways this ends: either the chatbot is better than the human, in which case the chatbot will be made available to those who can afford it (with the residual hope that a medical advance initially only accessible to the wealthy would become cheaper and more accessible over time); or the chatbot is worse than the human, in which case the chatbot will be made available to those who cannot afford the human. Either way, a two-tier mental health service would increase inequality and would not meet the appropriate UN Sustainable Development Goal 3 (ensure healthy lives and promote well-being for all ages).

It might be worth noting that OpenAI, the makers of ChatGPT-2, -3, -3.5, and -4, already charge a subscription service to access the latest version. But, under the virtuous cloak of competition, it is also disconcertingly opaque about what went into ChatGPT-4: “this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar” [7, p. 2]. But effectively beta-testing the earlier versions for free and not knowing what happened to all the user data is somehow redolent of the adage: “if you are not paying for the product, then you are the product.” Although this seems secondary to the question of who is doing the time-consuming and laborious job of annotation for whom and for how much, there is more than a hint of colonial exploitation here [12].

Although it depends on a large number of factors, such as the size of the training data, the number of parameters, the number of iterations learning input–output pairs, and the actual hardware used to run the software, training LLM demands substantial energy consumption and significant computing power, but also—often overlooked—a considerable amount of freshwater (e.g., for cooling). This implies a heavy investment in infrastructure, high operating costs, and substantial environmental impact. The cost of producing any single query is relatively low, although with a user base (already) in the hundreds of millions, this translates to sizable server farms and warehouse-sized computers. However, with revenue predictions in the region of billions of dollars, OpenAI can probably afford it, or to pay Microsoft’s Cloud Computing fees.3 The question is whether the planet can afford it (and as a follow-up, as might equally be asked of the financial services industry, what tangible benefits are the developers of LLM bringing other than to themselves?).

These expenses probably do not put training and using an LLM in quite the same cost category as a cryptocurrency in terms of energy consumption, but the same question could be asked of both technologies: is it really worth it? During a cost-of-living crisis, which may be transient, and the ongoing climate crisis, which is anything but, and a looming global water crisis due to overconsumption, are the benefits to society of this technology worth the direct and indirect social environmental costs? The question has been asked [13], but it is hard to have a “serious” conversation when entrenched privilege is in denial about the climate crisis and unmoved by evidence of its impact on others, and indifferent to the suffering of those less financially fortunate.

The issue of generative AI and intellectual property has been considered from the perspective of futural appropriation [14], but the quotation from Childres [15], the creator of the extremely popular board game Gloomhaven, is worth repeating: “AI art very much feels like theft to me. You’re training these AI with specific artists’ influences and then just not crediting them at all.” There are murmurings of concern at how generative AI is adversely impacting numerous artistic and creative fields, but again, it feels like the “move fast and break things” approach to digital technology, underpinned by the specious argument that regulation suppresses innovation, will once again outpace deliberation over whether this is something that should be broken (especially when “this something” is a social contract), and what the consequences of that breakage might be (i.e., as above, in what form it is remade).

However, while some organizations are subcontracting certain tasks to chatbots, in particular, content generation, some of Graeber’s [16] “bullsh*t jobs” involved in university central administration are unlikely to be at risk. For example, suppose a college of, say, science and technology were to advertise for someone with a degree in creative writing whose job description was “Development Communications Coordinator in the Supporter Engagement team, crafting communications to inspire donors and encourage giving”: if this were a genuine job, then it sounds more like a job for a chatbot, and the money saved could be spent more usefully in alleviating student hardship or encouraging research opportunities. But this is university central administration, so expectations of prudence are wildly misplaced; therefore, such superfluous jobs will likely be protected, and indeed, will probably continue to proliferate as they have in the last 20 years.

The final talking point concerns the impact on education, and how technology can be involved both in upskilling and, ironically, deskilling. So, for example, there is a growing field of enterprise called prompt engineering focused on using LLM to solve complex tasks [17]. On the other hand, as previously mentioned, chatbots can be used to produce code to solve problems of some sophistication, and this can increase the productivity of a programmer—but what type of programmer? The issue is that it could produce programmers who are little more than assembly workers on a production line: they can bolt components together, but they cannot calibrate, repair or repurpose components. It is a fundamental issue: to meaningfully ask a question, the asker must be able to understand at least part of the answer. Otherwise, confronted by a perceived superior intelligence, the asker dumbs down and does not—cannot—critically appraise what he or she has been told (see [18]).

The second dimension of cognitive de-skilling is, perhaps, more subtle. It has been said that the traditional “three Rs” taught as basic skills in English schools, that is, reading, (w)riting, and (a)rithmetic, should be complemented by a fourth R—rhetoric. This requires developing the ability to analyze some data, construct a coherent argument, and then persuade an audience to prefer that argument over others, appealing to Aristotle’s logos (reason), ethos (credibility), and pathos (sympathy of an audience). These are not skills that are refined by delegating an essential human cognitive skill to a chatbot. In the U.K., with its dysfunctional and elitist education system, it is not hard to imagine “public” (i.e., private) school-educated Oxbridge students whose institutions can afford the relative luxury of debating societies significantly outperforming ChatGPT-“educated” students, throughout their careers. This is not just undermining the democratization of knowledge, but also the democratization of know-how.

It also feels like this form of cognitive deskilling would deprive the latter of an essential quality of what it even means to be human [19]. Certainly, it would be extremely disempowering. In the transition to the Digital Society, “politics” should increasingly be an everyday activity and civic participation a norm, so that ordinary citizens within local communities can take responsibility for their local decision-making, collective efforts, immediate environment, and local history [20]. However, participation in this “politics” would be significantly diminished by being dispossessed of the skills to critique data, construct arguments, and successfully advocate for a course of action, if throughout one’s formative educational years, and subsequently in the workplace, the cultivation and refinement of these skills has been delegated instead to a chatbot. It is, perhaps, significant that so many indigenous cultures relied far more on persuasion between peers in preference to hierarchical coercion to manage their social arrangements [21].

The genie may be out of the bottle and connected to the internet, and pious statements of remorse aside, it is not too late to think again. One could imagine the re-emergence of salons in the style and tradition of the 17th and 18th century French literary and philosophical movements, using paper and pencil and the back of old envelopes, and the opposite of digitalization—instead of people rushing to get their work “on the Web,” they will struggle to keep their work from it. One could also imagine the re-emergence of patronage, which served a particular role in the history of art in early modern Europe, whereby the wealthy commissioned artistic work as proof of status (and also specified its form and content), although this one has probably been busted by the way that some notorious scam artists and grifters over-obviously used NFTs. Unfortunately, perhaps one cannot expect meaningful regulation, changes to employment law, or appropriate behavior in academia until the iron triangle formed by the BigTech-academia-congressional complex is broken [22]. Until then, one can perhaps but expect increased inequality (and see IEEE Technology & Society Magazine, Volume 41, Number 2, Special Issue on Modern Indentured Servitude).

Ultimately, though, it bears repeating that it is not the “AI” that, currently, presents the existential threat [23]; and it is not the technology that is “untrustworthy,” that needs to explain itself, or that should be held to account. These terms apply to “the system” itself, and the organizations within it (industrial, academic, and regulatory), that create and allow unconstrained and unchecked technological intrusion. We need to overcome our own fatalism and the belief that their narrative is somehow inevitable and have a conversation about this. In fact, we do not just need a conversation, we need a very vocal protest.