I’m excited about the enormous potential for generative AI and large language models. I’m also nervous. I get flashbacks to the world of social network trust and safety I was previously enmeshed in, where I started before companies broadly realized they had to massively increase their investment in online trust and privacy mechanisms.
For those who don’t know me well: I spent the last 6 years working on trust, privacy and integrity at Facebook/Meta and Google. I joined Facebook in early 2017 to run a product team using machine learning software and systems to detect, understand and remove policy-violating content and actors from the company’s business products at massive scale. We also built novel ads privacy and transparency tools like “why am I seeing this ad”, and the Ad Library (which allowed people to see every single ad running on the platform). In the 4 years I was there, we massively expanded our team of engineers, data scientists, designers, product managers and operations folks. I recently left Google after spending 2 years there as a VP of product, working on privacy and data protection.
Generative AI is a type of artificial intelligence that can create new content, such as text, images, and music. It does this by learning from a large dataset of existing content. Some popular generative models include OpenAI's ChatGPT series (GPT stands for Generative Pre-trained Transformer and is currently on version 4), Google’s Bard and NVIDIA's StyleGAN. The attention OpenAI’s ChatGPT has gotten as a consumer application since late 2022 has taken many by surprise. These models are trained on vast amounts of text and can generate human-like responses by predicting the most likely word or sequence of words based on the context provided in a prompt - making “prompt engineering” a new and useful skillset. Subscribers paying OpenAI $20 a month have been using GPT-4 to produce highly coherent and contextually relevant text in a wide range of tasks, such as writing essays, answering questions, and generating software code.
10 trust, integrity and privacy themes
Here are a handful of themes I’ve been thinking about:
1/ Closing the understanding gap: regulators are still ill-equipped to understand these technologies. The industry needs to engage. There are also lots of other broader/already embedded uses of AI we cannot ignore amidst generative AI advances (e.g. biometrics). Companies have incentives to move quickly (or to use regulation to let them catch up!) so we cannot rely only on them alone, but without their deep & engaged participation we won't get to the right level of understanding of the risk/benefit trade-offs. Also, some regulators/politicians may cynically see this as an opportunity for the spotlight or be pressured to just do 'something'. We should question motives and truly aim for open & collaborative discussions. But just hoping it works out, isn't the answer.
2/ Getting hands-on with actual safety concerns: It’s good we’re having these safety and integrity discussions earlier than we did for social media, where we only started building robust safety mechanisms in earnest in 2017. We certainly can’t ignore the “sci-fi”-wipe-out-mankind style risks of AI, but existential fear narratives sometimes detract from the hands-on work we must quickly ramp/focus on.
3/ Be transparent about safety mechanisms and resourcing: Social media and ads platforms built and released transparency reports and tools, and put large “we spent $X billions on user safety” numbers into the public domain. While I appreciate these (and I stand by the statistics I myself delivered publicly), they were often without enough detail. For AI, we will need more disclosure + more specifics. I appreciate OpenAI’s system card and the discussion about how they tested their model for 6 months, but we will need more of this and broader discussions. Some of these disclosures may give usable signals to bad actors and cause competitive issues for these companies, but we will need to unlock mechanisms to safely discuss those sensitive areas too.
4/ Privacy. Builders of AI systems are going back and forth with some privacy regulators already, which often takes time. Many large companies have told their employees not to put confidential information into generative AI tools, though I’m sure it still happens. It’s unclear what data some models are being trained on. Privacy protections won’t work the same as they would with a traditional database. Removing personal information from these models without retraining may be infeasible, so we’ll need to implement imperfect add-on mechanisms. How will input or output data be used in other applications like ads?
5/ Detect AI content, but also automation: Distribution networks will have to build AI detection into their products: mobile carriers, Google, Meta, Amazon, and TV networks (amongst others) will all need to detect not only the use of AI/fake/auto-generated content, but also the automation of interactions into many places they may not have been monitoring before. We probably won’t be able to rely on embedding hidden watermarks or codes when these tools are used (unlike printers or tasers), so we will need multiple approaches.
6/ Start scaling trust and safety teams: This space will be highly adversarial, with iterative attacks from all over the world. People will find holes nobody knew existed. While doing their own red teaming will be essential, companies will also need to monitor closely what is happening in the wild and be able to react quickly. This is the implication of OpenAI’s approach to “real world testing” which I agree with. We likely need to 10-50x the number of people working on this in short order.
7/ Actor identity & developer friction: High visibility efforts from big companies will be scrutinized as they should be, but whether they’re stolen, leaked or developed anew we’ll (soon?) see lighter weight models that will be able to run on consumer hardware. There’s not much developer friction in this space yet. For developers using these big models, what performance/time or identity hoops do you want them to jump through to get bigger degrees of access? Should we register smaller efforts, models or hardware the way the FAA requires of consumer drones - and will this have any meaningful effect? Do we build protections into hardware?
8/ Explainability is hard. Explainability of foundational models is not trivial, and it will likely need additional models running on top/in parallel to deliver answers about the answers. This could be especially problematic in systems where AI generation becomes a default for deciding on certain things and/or humans are out of loop. This is discussed often in the context of proposed AI regulation, for example the EU’s in-progress AI Act.
9/ Creator identity & content provenance: If content creation becomes automated, efficient or infinite, identity of its creators becomes more important. With the ease and speed of AI content creation, it may be increasingly hard to prove something is not AI-generated. How will we know a photo was taken at a certain time and place, and not faked or altered? Will need privacy-preserving technologies to protect peoples’ privacy and safety when such proof is needed. Even before AI, we are underinvested in ways of asserting identity or “human-ness” vs. having someone scan a driver’s license or passport and saving it on a server somewhere. Blockchain may actually be useful for some of these applications!
10/ Measurement: We’ve already seen estimates of millions of jobs that will be lost due to AI but perhaps even millions more to be gained. Are these new technologies making society better off? Mobile phones and social media have given people the ability to work from anywhere and stay connected with friends and family but as we know, there have also been many downsides. For AI, what metrics should we start to track how these technology advances impact society?
It’s time to talk
There are some obvious and not-so-obvious public safety risks we’ll see from these systems, and of course we’ll also encounter new emergent threats over time - but we need to encourage deeper conversation about the nearer-term issues that are already coming into focus. There is also a raft of issues around military or other nation state use of AI I’ve not included in the scope of this article. But even just on the consumer front, this is some of the fastest-moving and highest potential technology I’ve seen in my career.
I’m always happy to comment publicly (and I hope, pragmatically) on these topics, welcome your feedback (DM me on LinkedIn or Twitter!) and look forward to helping companies and policymakers think through these and other related issues.
Source: Midjourney
I’ll post these notes via email the day before I share them on social media, so if you know a friend or colleague who’s interested in these topics please share with them: