Personal scientists don’t believe in magic. That doesn’t mean there aren’t weird phenomena out there, but we think everything is ultimately explainable, and that we have the ability to figure out how it works by ourselves. So when you hear the recent talk about AI “doom”, it should immediately call up your skeptical side, because most (all?) of the arguments require magic to work.
This week we’ll take a critical look at the latest AI doom scenarios and explain why personal scientists should keep cooler heads.
A few thoughts on AI
Regular readers know that I’m all-in on AI tools, which I use heavily for everything from work projects to finances to planning to reading and more (see PSWeek260618). My Claude Max subscription has proven to be one of my best purchases ever.
So naturally I follow AI-related news closely, including all the recent hubbub about “AI Safety”. Every day there’s some new development, often with implications that are easy to turn into scary headlines. Science is about truth-finding, not policy, so I won’t say anything about what political leaders should or should not do. But I have looked closely at the facts, and here are my current thoughts:
AI is a normal technology. It will transform the world the same way as past technologies: step-by-step, as a tool that humans will control. Viewing it as something superhuman is inaccurate and unhelpful.
Anthropomorphizing doesn’t help. Referring to the “rise and fall of agent civilizations” makes for fun, nerdy prose about “agents conspiring to” do this or that or “tried to hide their tracks from the humans.” I don’t blame headline writers for trying to attract readers with scary science fiction scenarios, but personal scientists shouldn’t be fooled. I’m glad that current parlance usually refers to ChatGPT or Claude as “it”, not “she” (as many people do with Alexa or Siri). These things are tools, not people, so avoid the temptation to refer to them as if they’re alive.
The “Hugging Face” hack was just poor software engineering. Yes, I’ve read the descriptions and came away impressed at how good OpenAI’s AI agents were, but all sophisticated hacking jobs use clever, multi-layer techniques. Everything about this “hack” could have been prevented if OpenAI engineers had used existing security techniques. To give one simple example: the computers involved were connected to the internet the whole time, contrary to breathless but inaccurate reporting about a so-called “sandbox” escape.
Hacking scares are a temporary problem. Yes, AI-powered coding models can find incredibly clever and complex ways to break into software, but the opposite is true too: AI-generated code can be far more secure. Ultimately, as more software is rewritten with AI, traditional code-hacking will become essentially impossible. I predict that, although in the next few years we may continue to hear of scary, high-profile hacks, the volume and severity will go down as more code is written by far-more-security-aware AI programmers rather than mistake-prone humans.
Data centers are nothing new. They’ve been around for as long as there have been computers. Like any new project that requires lots of construction, there will be an impact on the surrounding neighborhood, but don’t believe the stories about water usage (which is based on a false viral story from 2024) or crippling the electric grid. As always, you have to compare with the alternatives.
Computers are not magic. GPUs are marvelous pieces of engineering, and the Transformer architecture that makes modern LLMs possible is a brilliant piece of software, but ultimately these technologies work for the same reason any tool does: laws of physics and math.
I’m not going to accuse the Big AI Labs of cynically trying to boost their IPO prospects through overhyping their product capabilities. I think their leaders are generally pretty sincere in their public statements about dangers. But if they really believe they’re building Skynet, why build it? That makes no sense to me. The AI subculture has always been fascinated by science fiction scenarios, so I’ll chalk some of it up to the zeal of youth.
Many of these younger engineers appear to take it for granted that they are the first generation to face serious safety concerns over a new technology. Read (my former boss!) Steve Sinofsky for a saner take on how previous generations faced (and solved) real problems without resorting to explanations that sound more like superstition than engineering.
But personal scientists don’t believe in magic. There is no mysterious, unexplainable power that will outsmart us. A tool is still a tool.
How AI will help
Meanwhile I think it’s far more interesting to think of all the ways AI will help.
For example, imagine if you had an instant fact-checker for everything you read, every conversation, and every idea or question that comes to mind wherever you are. How many dinner table discussions get side-tracked with questions that could be easily answered (”Will pineapple shoot up my glucose?”, “Did Peoria vote for Trump or Harris?”, “What’s that movie where Woody Harrelson has to fight zombies?”). Many of us are already doing this, especially now that the voice mode makes it easy and unobtrusive. (Within reason, of course: nobody likes a know-it-all—a lesson my wife taught me long ago).
More importantly, what happens when you can do your own “peer review” of any article you read, including those in academic journals?
For example, my local newspaper The Seattle Times is publishing a series on Education that reviews what our state got after a court case forced the legislature, over the last decade, to greatly increase spending on public education. The article seemed like a reasonable overview, but it sidestepped the question I care about: how did test scores change? So after a few minutes in Claude I generated this (and confirmed it’s based on real data):
I now routinely point Claude to every news or journal article when I want to know more information. And it routinely points out serious mistakes and missing perspectives. It will be far more difficult in the future to get away with bad or incomplete information.
Personal Science Weekly Readings
In yet another personal science reminder that listening to experts doesn’t substitute for doing your own homework, @therealRYC summarizes a new study of 1,038 doctors in 19 countries: given the same written case, two of them agree on the diagnosis only 55% of the time. This despite a half-century of trying to adopt more stringent, rules-based methods.
And if you’re doing homework on cancer, Jude Gomila built a public oncology map — treatments, molecules, targets, bottlenecks, open questions, startups — at onco.cc (repo). Open information infrastructure for anyone navigating cancer research or care. (via @JocelynnPearl)
I’ve long been a fan of ResearchHub (see PSWeek250220). See this @heybeluga update on six years of ResearchHub: 1,401 paid peer reviews in Q1 and $2.2M funded into research (incl. labs at Stanford, Harvard, Cornell). Still one of the more concrete DeSci / open-science experiments actually writing checks to researchers. (via @joycesticks )
@james_y_zou announced the publication of Paper2Agent in Nature: turns passive papers into virtual authors that answer questions, apply methods, and collaborate with other paper-agents. Point your LLM to Github and ask it to convert your current paper.
About Personal Science
Nothing’s magic once you understand how it works. That’s a core belief of any scientist, including those of us who try to apply scientific techniques to daily life.
If you have questions or comments, please let us know.
I've only used the free Gemini feature on Google, and in intensive questioning, it does very often get things wrong. I observe it seems to lack the quality of a good human scientist of demurring or remaining circumspect when there is insufficient evidence to justify an accurate conclusion. It does know enough to ask questions, but it is much more like a poor scientist, jumping to conclusions that haven't been properly tested. Its flaw seems to be that it is programmed to be helpful and eager to please. It is a product after all. I find it likable but over-enthusiastic and highly fallible.