Generative AI To Be Considered Harmful

This opinion blog was originally written in French. I comissioned a trained translator, Noah Vago, to translate it to english. All discrepancies between the French and the English version are my responsability.

ChatGPT’s launch on November 30th, 2022 has had profound consequences. Since OpenAI released this chatbot online, there have been countless laudatory declarations stating “AI” was supposedly able to transform the workplace, make our lives easier and solve an (often never specified) bunch of problems. Scientific top people such as Philippe Aghion, Professor of innovation economics at the Collège de France, hypothesise that, if we adapt our institutions (one can wonder what “adapting” means in this context), AI can be a tool for economic growth. “AI”’s importance turns it into a strategic and unavoidable issue. This could explain the organisation with great ceremony of an “AI” summit last February. Emmanuel Macron invited, among others, Elon Musk, AI’s self-proclaimed big shot, who had a mere two days previous given in to the impulse of doing two Nazi salutes publicly, and who is an outspoken supporter of the AfD, Germany’s far-right party. As top software companies heavily push the integration of “AI” in their programs, making its use unavoidable – as in, there is no alternative. On an anecdotal note, my optician always asks for my opinion on AI and its – actual or supposed – efficiency. Offline role playing narrators – a hobby I enjoy – can quickly obtain plausible illustrations by using Microsoft’s generative AIs.

Hammered in with such intensity, the narrative according to which “AI” is revolutionary makes the technology at the very least omnipresent. This ineluctability pushes AI’s opponents back to the ranks of mere eccentrics, “unrelenting Gauls” hostile to any kind of “progress”, or, in other cases, representatives of working class people who worry about being “replaced by robots”. Since the beginning of 2023, I haven’t used ChatGPT, Gemini, or any other equivalent, not in my professional practice, not for my hobbies. Students ask me with astonishment why I am horrified when I see them copy and paste a ChatGPT generated script shell. Several rewarding – and quite lively – conversations have taken place during the Dagstuhl seminar that took place last February, each intensifying my ambivalence towards this technology.

After being heavily exposed to the technology and the narrative surrounding it for more than two years, I deem it necessary to lay out my personnal opinion on the matter. To sum it up, I think it is useless in the vast majority of cases in which it is deployed, harmful in its modes of operation, its bases and the ideals it spreads, as well as deeply disconnected from what computer science should be, a tool serving people and societies within the context of the 21st century. This post will further detail some of the ideas supporting this position. I find them justification enough to be at least wary of implementing such technology on such a large scale.

This is not an exhaustive exposition of the current science on the topic. Although posted on a website which was until now mainly dedicated to lists of educational publications and resources, this post does not fit into the same epistemic framework as a scientific paper. This is my personal opinion, within the context of my scientific practice – that of an engineer –, and nourished by my political reflection.

What are we talking about, and where are we situated?

First and foremost, I find it important to specify the position from which I speak, as well as the concepts I will refer to throughout this (very long) post.

At the time this paper is published, I am employed by the Software Security and Reliability Lab (LSL) at CEA List. This lab is, among other things, tasked with the scientific and technological development of a platform dedicated to analyzing C programming code, called Frama-C. I myself am not a developer on Frama-C, but I am surrounded by this culture of formal verification of programs in my day-to-day life. I studied and taught classes on SMT calculus, I am currently working on a lecture on how to use formal methods in order to improve machine learning software, which I will be giving at the European Summer School on Artificial Intelligence, I am part of a team developing a software dedicated to write specifications of machine learning software in a principled way.

I am therefore surrounded by an environment with high standards regarding software development. I think it imperative to have formal warranties on a software’s behaviour – such as a strong type system – in order to arm the workers with the proper tools to present a quality software.

I witnessed first-hand deep learning’s fast-growing popularity thanks to my 2015 research internship, two years after AlexNet’s article was published, and I have contributed to several deep learning libraries. Thus I have one foot through the door when it comes to machine learning and formal methods. I reckon a good chunk of criticism towards deep learning also apply to generative AI. In the present post, I try to tackle criticism that is more specific to generative AI, particularly the fact that it is more than development libraries operating a set of scientific knowledge. It must be considered within the framework of the narrative surrounding it and its influence on humans and societies.

I was inspired to write this post after reading Kate Crawford’s Atlas of AI, as well as an excellent article by Florence Maraninchi (in French): why I do not use ChatGPT.

On the ecological impact

GenAI uses ressources very intensively, this much is hard to ignore. Anne-Laure Ligozat and her team took an interest in GenAI’s environmental cost and how to mesure it¹. In Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model, they estimate the cost of training BLOOM to be between 24 and 50 tons of CO2 equivalent, without taking into account the extraction of raw materials nor the discarding of the components necessary to turn on the program. This is more or less the equivalent of ten to twenty return trips by plane from Paris to New York. The study estimates the inference phase to deploy around 19 kilograms of CO2 equivalent each day it runs. A 53-day operation can thus be the equivalent of a full training phase.

GenAI technologies also entail training and inference, which is costly in terms of electrical energy. Depending on the energy mix of the country hosting the datacenters these operations require, the carbon footprint on this front too is far from insignificant. By way of example, datacenters represent up to 20% of Ireland’s energetic consumption, a country with an energy mix mainly made up of fossile fuel². An article from Epsiloon, a science magazine on current discoveries, notably reports an increase in Microsoft and Google’s carbon footprint, for which AI is largely responsible. The lack of transparency from the GAFAM running these datacenters makes it hard to get a precise picture of the situation.

All in all, even though there is no way to accurately and precisely quantify the environmental impact specific to GenAI, my opinion is that there is a sufficient body of clues to conclude GenAI already has a note-worthy influence on our ecosystem. According to French Ecological Transition Agency ADEME and French Digital Regulator ARCEP, digital technologies amount to 4.4% of France’s carbon footprint³, a percentage which will seemingly keep on growing.

GenAI is not solely responsible for the recently observed rise of digital technologies’ material footprint; however, Microsoft’s attemps to reopen the Three Mile Island nuclear power plant and the numerous programs dedicated to the redirection of energy infrastructures towards datacenters used for GenAI are a part of an array of weak signals indicating that these models’ operators are very much aware of these technologies’ impact – as well as the cost society will have to bear⁴.

No matter what, it isn’t possible to dismiss these considerations when examining GenAI’s usefulness all while available resources are tightening, which in turns causes more and more tension – as exemplified by the conflicts surrounding the consequences of datacenters on water accessibility in Chile and Uruguay. What needs do these machines provide answers to? What justifies putting in place a “planet-wide system” in order to sustain AI⁵?

To what ends?

My choices have led to me being more exposed to content that is critical of AI than to arguments about uses I might think are interesting. This means I personally struggle to find any relevant uses for GenAI. Let’s clarify what I mean by relevant use here:

• the benefit of running a GenAI operation must be evident when compared to its cost;
• serves the common good;
• symmetrically, isn’t used to oppress, kill, maim, assist mass surveillance, support war crimes or crimes against humanity;
• cannot be substituted, for a specified goal, by another existing technology or a currently viable political measure;
• does not alienate the humans operating the software.

It’s clear that very few of GenAI’s uses I am aware of do in fact answer to these criteria. From what I have observed, it is mostly used as a support for white-collar workers: help in redacting reports (which testifies to either a lack of respect for the person who will have to read through it, or to the uselessness of said report), reference letters (I find it contradictory to deem a student or a colleague’s work worthy of recommendation while thinking a machine is best suited to express that), cover letters, or syntheses of scientific articles (I think Galactica AI’s fiasco is a worthy indicator of the value of such tools in this context).

ChatGPT is sometimes compared to a spellchecker with the added benefit of improving our writing by making it flow more easily. It seems to me the functions of those tools are not the same. Language is consensus: we agree on a more or less shared sign in order to communicate an intent and to jointly exchange based on one another’s intents. A spellchecker helps communication by leading a piece of writing towards the aforementionned consensus: a prescriptive grammar. ChatGPT writing a letter, on the other hand, interferes with our intent and leads the text towards another consensus: that of an average intent deemed prescriptive. To be clear: if GenAI’s main functions are generating bullshit documents no one reads and content with no truth to it, polluting informational spaces, is it really worthwhile?

What about studying it?

An exception I have yet to make my mind up about is the use of GenAI as an object of study. One could argue against that by underlying that studying GenAI already contributes to making it seem worthy of scientific and technological interest. GenAI being overly financed and the budget dedicated to scientific studies being seemingly unchanging, another argument against this use is the fact that this necessarily results in a loss for other fieds of study.

Does this program really not know how to count?

GenAI’s answers are consistent with an outcome deemed likely within a span of probabilities trained on a given collection of data. These programs are in no way conditional and run solely on statistical samples and linear algebra. They have no branch instructions, no pattern-matching, no typed systems, no failure mode. We could keep on listing improvements made to programming languages throughout the last 60 years, or stop with this simple assessment: GenAI cannot, by its very definition, sustain outcomes that we can with no doubt predict to be correct. It is not a bugged program, but a grossly underspecified one. It looks like it functions according to some metrics and on a superficial level, but then fails catastrophically when applied to real-life scenarios, by nature too dense to be summarized by a given set of variables and formal specifications.

I often think back on a talk given by Xavier Leroy, Professor at the Collège de France, who said about a GenAI model that it is a program which does not know how to count. This is deeply ironic within the context of computer science: the very core of the practice is to make machines that can count in our stead!

An opinion shared by several coworkers is that GenAI in its current state simply is not relevant to our field of study. On one hand, its formal verification is arduous for numerous reasons that I pointed out in my PhD thesis, among which the fact that it is very difficult to formally specify the behaviour expected from these machines.

On the other hand, GenAI’s help for formal verifications seems hard to carry out. To my knowledge, there are very few databases on this critical code that can be reviewed with formal verification tools⁶ . When it comes to inferring from function contracts, the state of the art is up to good standards. Going above this level would require a substantial effort in regards to collecting and aggregating data, all for, once again, uncertain results. If that is the point of studying this technology, I admit it does not really interest me.

On computer science jobs

Recently, a trend launched by Andrew Karpathy has taken root within the sphere of software development: that of vibe coding. The idea is to use integrated development environments augmented with GenAI (such as Cursor) in order to generate most of the code’s basis with well chosen prompts, going through the literature on a given topic, supplying corrective procedures, and so on. I do not write code that is complex enough to feel the need to turn to such a tool. However, in principle, why not! Writing functions dedicated to printing a complex data structure, managing a system’s I/O, writing validation classes for API schemas, these are all repetitive tasks we could do without. Generating automatically a program from a specification is the ultimate dream for a wide number of researchers working on program synthesis.

The dream being that a clear enough specification combined with a wide enough vocabulary language would allow us to get a correct program in the given language which answers to the given specification. Concretely? I have not heard of any study that indicates that software developed with the help of GenAI is any better than « handwritten code ». On the contrary, it has already been pointed out that Copilot, used to resolve issues, does not do a very good job at it. Such « vibe coding » apps can delete production databases without warning.

My opinion is that the quality of generated code is often average, if not mediocre. Nothing surprising there, as GenAI’s output consists of an average calcuted on the basis of all the written code in its training data and there is way more incorrect code than correct code out there. As generating code by the shovelful is advertised as free, the likelihood of abuse is certain, at the risk of producing more and more bases for mediocre code.

When taking a step back, it seems to me that these tools target middle management in software development rather than developers. The promise of getting workers to be more productive sounds increasingly like a pretext to reduce costs. That is to say, firing developers. Moreover, these tools focus on the writing of code. That is but one component in a software’s life cycle. Before being written, a code is conceptualized within a software architecture in order to provide a service. After being written, the code must be proofread, fixed, documented, integrated and sustained; all of these skills can hardly be automated. And, even if they could be, proofreading mediocre code generated with GenAI does not look like a career to look forward to (and I am not the only one who thinks so).

A revolution of the workforce? Yes, but not in a good way

Ambitious CEOs and highly skilled engineers only consist of the tip of the iceberg (the iceberg being GenAI workers). GenAI requires a critical mass of good quality and annotated data in order to function. These data are not collected and annotated on their own. Antonio Casilli’s work (specifically Waiting for Robots) and Paola Tubaro’s show that promises of automation and increase in productivity attributed to GenAI actually rely on the maintaining of an underclass of clickworkers. Annotating database requires an important workforce, one that is easily distributed and scattered on a large scale. The pay for these clickworkers’ compiling of databases is reliant of the goodwill of microwork platforms such as Amazon Mechanical Turk, platforms that tend to be in no hurry to offer decent working conditions.

There have been instances of utterly absurd situations where automation actually hides a flood of underclass workers faking AI-like behaviour.

Another costly process that is specific to GenAI is the use of Reinforcement Learning with Human Feedback (RLHF). Resorting to this method is necessary because of the underlying technical functionning of a GenAI software. When a model is trained on the entirety of the internet, it should theoretically be able to generate content available on the entire internet… including on its darkest corners. Without preemptive checks, a GenAI software could therefore produce racist content, incitation to hatred or pedopornographic content (and, given the poor track record of Microsoft’s racist robot Tay or Meta’s role in the ethnic cleansing of Rohyngias, this is far from an hypothetical issue). We thus resort to a secondary training phase during which a human is subjected to GenAI generated content and decides whether or not it is acceptable. These people rarely have the means to protect themselves from the psychological damage this work submits them to, sometimes leading to traumatic consequences.

On the continuation of colonial logics

As we can see, GenAI transforms the working world’s structure through the massification of underclass jobs. The workforce tasked with these jobs, often located in the Global South, is vast but scattered, which leaves them with little to no leverage to negotiate with their employers. Combined with the escalation of extractivism, which exploits local populations and pollutes their land, these are signs of the continuation of colonial logics.

Rotten ideological roots

This section might seem less important comparatively as it focuses more on ideology than material considerations. I also think it important to make it clear that I find refusing to use a tool or to adhere to an idea on the sole basis of its creators being despicable people to be a rather unproductive approach. Von Neumann was, according to several testimonies, awful and he invented the architecture all modern computers rely on. Sartre and de Beauvoir were involved in child abduction and their work contributes greatly to feminism and philosophy. The making of such a list could take forever. The case of GenAI is a special case as an enormous capital is needed for its very structure. The need for massive data collection and datacenters to fuel its training as well as its inference is synonymous with a material and symbolic power that only strong nations and big corporations have the ability to deploy. The disproportionate involvement of private corporations in the research field of language automation is a good indicator of this state of affairs⁷. By this polarisation of involved actors and because of the lack of legislative contraints, it is easier for its creators to “hold sway” over GenAI according to their own ideological choices. This tool can only be operated by the dominant, never the subordinate.

Aaron Schwartz’s suicide can be tied to the harassment he went through after he downloaded paywall protected newspapers through his university’s VPN in order to freely distribute them. OpenAI, Deepseek, Anthropic, Mistral and their counterparts are cashing in (or at least making more money than what their corporate structure costs them) from the very principle of extracting this new technology, monetising it and spitting it out after having made it the new normal. The intrinsic “value” of GenAI relies on the plundering of the internet. Talk about double standards.

Eugenics for an AGI

The cronyism between GenAI’s top figures is most visible when looking at Elon Musk, who was involved with DOGE and supports the AfD. Going even further, Timnit Gebru and Emile P. Torres highlight an ideological framework common to GenAI’s leaders: they call it the TESCREAL Bundle⁸. This grouping of ideologies’ distinguishing feature is the belief in a “General AI”, a hypothetical machine with cognitive abilities going far beyond anything humankind could imagine.

This machine’s seemingly godly powers abilities would allow humans to transcend themselves and colonise the galaxy – which is seen as a morally good prospect, thus making contributing to its development a moral imperative. If this definition can be amusing, it nonetheless underlies a moral scam. Believing in a future considered infinitely desirable for all of humankind leads to the neglect of present times. The issues relevant nowadays are deemed secondary. This includes the fight against climate change - contentious given the previously mentioned cost of AI on that matter- and civil rights movements. Gebru and Torres raise the point that this quest for a “General AI” leads to the releasing of software that cannot be tested, cannot be specified (and thus verified), and have catastrophic consequences – some of which this post has alluded to.

In conclusion

Let’s recap.

GenAI is a techno-political programme with a high environmental cost. Its promised uses rarely help the empowerment of communities and individuals, when they work at all.

It disfigures the working world at the expense of working class people by disrupting the balance of power through the outsourcing of workers and the distance thus put between them and their employers, all while justifying data collection on a great scale.

It goes against the very principle of the scientific research I contribute to: making sure we conceive softwares that are safe to use.

Because of the overall picture I just painted, I think GenAI must be considered harmful by anyone who cares about the ongoing ecological and political crises, producing and using safe programs, not hurting or endangering individuals and populations. Computer science has too much to deal with and design to get lost in a fantasy of infinite growth.