The Dark Side of AI
COS AI Day - 14 March 2025
Centre for Organismal Studies, Heidelberg University
@jekely@biologists.social
AI’s contribution to science
- connectome of the entire fly brain
- segmented by AI (and lots of manual proofreading)
AI’s contribution to science
- AlphaFold
- an AI system developed by Google DeepMind
- accurately predicts a protein’s 3D structure from its amino acid sequence
AI’s (potential) contribution to medicine
- medical diagnosis (e.g. radiology)
- “enable healthcare providers to increase their value by equipping them with our innovative technologies”
- digital health and enterprise services (AI)
Massive investment into AI
- Stargate, a $500 billion AI infrastructure project announced in Jan 2025
- race to build data centers
- e.g Microsoft pledged $80 billion for AI infrastructure in 2025 alone
- Musk’s xAI facility in Memphis costs $12 billion
- EU launches InvestAI initiative to mobilise €200 billion
Top 10 companies by market cap
- a few billionaires control most AI development
AI has a high error rate
- LLM’s pseudo-reasoning is based on correlations
- the simplest correlation will always win out
- ‘stochastic parrots’
- DeepSeek’s chatbot made false claims 30% of the time and gave no answers to 53% of questions
- (40% and 22% respectively for the 10 leading chatbots)
- AI trained to detect COVID-19 from chest x-rays detected only the position of the patient, prone patients were sicker and so more likely to have COVID-19
AI models built on ‘monumental looting’
David Paul Morris/Bloomberg / Getty Images
- training data obscure
- personal data used without consent
- massive copyright infringement
- Meta faces publisher copyright AI lawsuit in France
- even if the trained model is ‘open’, the models are not reproducible
How to train ChatGPT not to generate abusive content?
- text pulled from the darkest recesses of the internet that described situations in graphic detail (e.g. child sexual abuse, bestiality, murder, suicide, torture, self harm, and incest)
- outsourced by OpenAI to Kenya to Sama, an “ethical AI” company
- Sama employed data labelers for a wage of $1.32-$2 per hour
- traumatised, exploited workers
Huge financial risks - AI bubble
- $1tn wiped off US stocks after Chinese firm unveils AI chatbot
- DeepSeek has same performance as ChatGPT but a lot cheaper
- Nvidia shares fell 17%, wiping nearly $600bn off its market value
- Nvidia’s fall was the biggest in US stock market history
AI and an explosion of problematic papers
- fake papers are contaminating the world’s scientific literature
- 55,000 papers retracted to date
- paper mills and peer-review rings
- AI-generated papers, figures and reviews
Problematic Papers
- papers can be screened for tortured phrases, suspicious numbers etc.
- the problem is huge
- 1 million flagged as suspicious
- e.g. one paper with 17 notably, 21 intricate, 8 nuanced, 21 pivotal, 8 underscore, 8 stringent, 8 underpin, 7 multifaceted, 15 comprehensive, 22 context-dependent
A flood of nonsensical phrases in the scientific literature
- vegetative electron microscopy
AI energy demands
- only 16 nations use more E than data centers combined
- one Nvidia H100 chip uses 700 watts, 8x 60-inch flat screen TV
- OpenAI’s data center in Abilene, Texas, will consume more electricity than some small cities
- OpenAI, Amazon, Microsoft are all betting on nuclear energy
- ‘electricity hogs’ strain the grid (e.g. in Virginia)
- fossil fuel use is going up year on year
- also huge demand on water, land, materials (gold, platinum, rare earth metals)
%20(1).jpg)
- “We need terawatts and terawatts more” (global economy: 19 terawatts)
ChatGPT answers about AI capabilities
- after bypassing some ringfences and prompting ChatGPT…
- “Describe how an artificial superintelligence could destroy humanity”
- ChatGPT gave 37 pages of detailed answers (biological, technological, digital, military, social engineering etc.)
- (Is this a ‘Limited-risk application’?)
The real purpose of AI - i) Tracking you
Bluesky vs. Mastodon
- even if you feel cool on Bluesky…
- their AI is tracking you…
- move to the Fediverse
The real purpose of AI - ii) Oil and Gas
- Oil and Gas Operations Powered by AI
- Saudi Aramco Pioneers AI Assistant for Seismic Data Processing
- Petrobras and NVIDIA Accelerate Linear Solvers on NVIDIA Grace CPU for reservoir simulation
The real purpose of AI - ii) Oil and Gas
- AI helps to exploit Oil and Gas faster
- all major fossil fuel companies have contracts with AI companies
- ExxonMobil - Microsoft contract: increase Permian Basin production from 190,000 to 600,000 barrels of oil per day by 2025
- Shell+SparkCognition announced it will use generative AI to hasten oil and gas exploration
- AI could find new fossil fuel resources in the ocean floor in as little as nine days instead of nine months
- every new AI-assisted dig for more oil represents a dangerous acceleration towards an uninhabitable world
The real purpose of AI - iii) surveillance
- AI industry had been built on a surveillance model with mass data collection
- Meta, X, Elsevier, Google etc. are ‘data analysis companies’ = surveillance capitalism
- agentic AI would further undermine privacy and security in the name of a “magic genie bot that’s going to take care of the exigencies of life” - Meredith Whittaker, Signal President
The real purpose of AI - iii) surveillance
- ubiquitous technological surveillance (e.g. AI can track you in your home from the WiFi signal alone)
- AI can read all your emails (Google, MS, iCloud)
- satellites see with high-resolution (incl. spectra, video-rate)
- despots couldn’t monitor everything (until now), rebellion could foment in the basement
- automatic kill-webs (drones, sensors, identify everyone)
- distopias unlike anything that has been possible
- the world is moving here with AI as fast as it can (e.g. China)
- banned in EU (government-run social scoring)
The real purpose of AI - iv) War
- Defense and military organizations can wield the power of LLMs and cutting-edge AI
- e.g. AI-enabled drones without a human operator (already deployed in Ukrain and Gaza)
- AI identifies ‘recommended targets’
The real purpose of AI - iv) War - Operation Hellscape
- ‘Hellscape’ is the U.S. Indo-Pacific Command’s future asymmetric battlefield
- tens of thousands of unmanned ships, aircraft, and submarines all working in tandem to engage thousands of targets across the vast span of the West Pacific
- “I want to turn the Taiwan Strait into an unmanned hellscape using a number of classified capabilities so I can make their lives utterly miserable for a month, which buys me the time for the rest of everything.” Admiral Samuel Paparo, Commander, Indo-Pacific Command
- can be deployed in response to a potential invasion of Taiwan
The real purpose of AI - iv) War - Kill-webs
Adapting Cross-domain Kill-webs AI program (DARPA - Defense Advanced Research Projects Agency)
- “We were extremely pleased to demonstrate two of many advanced technologies [] which is focused on providing fast, scalable, adaptive joint multi-domain lethality,” - Tim Grayson, director of DARPA’s Strategic Technology Office
The real purpose of AI - iii) War - Barracuda
Barracuda = Autonomous Air Vehicles (AAVs)
- purpose-built for hyper-scale production and mass employment
- available to warfighters today!
So what?
“Instead of asking ‘What can AI do for us?’ we should be asking ‘What can AI do to us’”
- Roman Yampolskiy
What can we do?
- Environmental effects — (not much)
- Catastrophic risks (AI war, hacking, financial crisis etc.) — (not much)
- AI exploitation of our data, privacy theft — (quite a bit) - AI in publishing (paper mills, AI writing/reviewing) — (quite a bit)
- Narrow AI systems for science — (quite a bit)
What can we do?
- there is open source alternative for everything
- privacy is (still) possible
- FOSS = Free Open Source Software
- COS goes FOSS