Rob's Notes 9: Beware Deepfake Scams

Investing scams are starting to use more 'celeb-bait' AI deepfakery

Many government bodies tracking financial fraud themselves admit that many scam victims don’t report their losses, leading me to guesstimate that global scam losses could easily be over $150 billion annually**. I’ll write more about this “market” later, but for now I wanted to share some details of a still-very-active investing scam called “Quantum AI” we found while experimenting with AI for trust & safety.

Sketchy offers for a long time have used what I call ‘celeb-bait’: namely, using well-known celebrities as “bait” - often just to encourage users to click (like implying a star’s death or arrest), or pretending the celebrity uses or has endorsed the product or service. In July 2023 there was a rash of reports from the UK of deepfakes involving local celebrities and global figures like Elon Musk (of Tesla, SpaceX and X fame). Despite ongoing bouts of media coverage, these Quantum AI scams appear to have continued, and so I wanted to share some examples to encourage people to do more research of these deceptive claims, and hopefully avoid the products of deceptive promotion. As a result of work done here, we’ve made a variety of online reports leading to many of these taken down across Meta and Google for violating their policies, but it continues to be whack-a-mole and many moles still continue to slip through.

Without concerted ongoing cross-industry effort and collaboration, it is likely these kinds of false claims and scams will continue to proliferate, and people will lose money. Here is an example representative deepfake video we found in a Facebook ad. Note: we’ve added some video and audio noise of various types to make this less re-usable for scammers reading this post, but I fear they can quite easily find/alter/recreate these videos from a variety of sources using widely-available AI tools.

As AI tools get cheaper and better, the likelihood of them being used for scams is only going to increase, especially since the companies providing them often do little in the way of “know your customer” or “know your developer” vetting of users. There are certainly plenty of legitimate and useful use cases for overdubbing footage with matching lip movements (e.g. translation of corporate training videos into other languages). But it will mean we’ll be able to trust what we see far less. As I posted here, we’ll need platforms to do more to label and/or demote altered content (including providing better user reporting or flagging) but also detect automation enabled by AI. More on this some other time: let’s go back to this scam.

The video pitch is typically for something called “Quantum AI”, although there are many variants. It often starts with deepfaked news coverage from local news sources (above, where BBC News is digitally altered) or an introduction from a local financial celebrity like the UK’s Martin Lewis. It may then have Elon Musk speaking with a blurred background (perhaps ripped from a leaked Zoom call) or at a conference, or with a variety of stock “stock market” video footage showing trading imagery. The fake Musk explains things like:

• They spent $1-5 billion over several years developing a quantum computing trading system, with engineers from Apple, Google and others

• Though it doesn’t always work, his system “wins trades with 90%+ efficiency”

• You can start with GBP250, but quickly make over GBP5700 per day

• Many other celebrities are involved and supporting this e.g. Lewis Hamilton, Richard Branson, Chamath Palihapitiya - sometimes using video footage showing them talking about Bitcoin and not this scheme itself

• That British people will no longer have to work, and other (*&^#%^$* claims

We found discussions of this scam dating back to the end of 2021, and encountered older versions of these videos. It is clear that the average quality of these deepfakes has noticeably improved over the last year, both in terms of how much the voice sounds like Musk and how well his lip movements match up. Note: with the exception of a few deepfake videos we saw on TikTok both directly and re-posted/annotated by TikTok users, the majority of the deepfaked Elon Musk videos we encountered were on Facebook.

We found their ads on Facebook/Instagram & Google Search, and an Android App on Play that Google took down

After asking for feedback online, we received dozens of reports of these showing up in FB/IG ads to UK users, and we also searched and found some of them in the Meta Ad Library. We contacted Meta who removed many of the ads we found and sent to them. We saw various versions of the Elon deepfake content on TikTok. One user video we found on TikTok with 150-odd likes complained about a specific Google Play app. The app included the word “elon” in the app name and appeared to have a number of fake reviews. When we looked up the developer, the same entity also had what looked like a fake Grok AI application - Grok being X/Twitter/xAI’s artificial intelligence chatbot app. Google took this app and developer down after I contacted them.

There were also many squirrely-worded come-ons in Google Search ads for the term “quantum ai” for users in the UK. Google also stopped the search ads we’d flagged to them.

In most of the cases we encountered both on Facebook/Instagram and Google, the ads in question appeared to be using cloaking, where the true website destination is camouflaged from some set of end users including the people and machines reviewing ads. Once you follow one of the post-video links or search engine ad links, you will either get sent to a very random (cloaked) page, or the actual payload (see below). Cloaking is pernicious and adversarial, and uses a variety of questionable hacker tools which use have been the subject of lawsuits, including this one I participated in when I was at Meta).

Here are screenshots of some of the search ads which were running on Google in the UK:

Upon clicking these ads, you are either taken to a cloaked landing page (in this case a “clothing and accessories store” that hides the true intent of these ads from the ad reviewers and some subset of nosy users like us)…. :

… or to the intended destination, the uncloaked version, which was as follows for “herd birds[.]com” which redirected us to this form fill page with a very questionable claim up top:

Here are some example ads from the Meta Ad Library; some are videos and some are not (we removed the Library IDs). In addition to these, we encountered a number of ads employing Facebook Watch videos of the deepfakes. The same comment I made earlier about knowing the identity of people using business tools could perhaps apply to some of these advertisers:

After sign up, the user is taken to a somewhat clumsy trading website. While we didn’t deposit any money, the site looks like it has basic tracking functionality that could give one the (presumably fleeting) impression of making trading profits in furtherance of depositing larger sums. The site auto-generated a user number and password (hidden in the screenshot below). It might be that there are a whole variety of variations of this site with similar functionality, but here are some screenshots of one site that we encountered:

Don’t give out your personal information

Beware - you’ll get lots and lots of phone calls if you do give out your personal information, as many users reported to TrustPilot, saying things like:

“100% scam. No legitimacy whatsoever. All they hound you for is to deposit money into supposedly your trading account. They are not knowledgeable about anything other than scamming you and harassing you. Don't inquire about anything from them otherwise they will never leave you alone.”

We submitted two of these forms with UK-based (+44) VoIP phone numbers and received a variety of calls that went to voicemail. In one case we received 21 calls over 10 days and in the other, 12 calls in 14 days. In both cases, our numbers received 6 calls on the first day trying to reach a live person. We haven’t dug much deeper than that on the telco side, but I presume this scheme is being operated by a variety of groups of callers using various techniques to hide their locations and identities.

We are also in contact with Ofcom and other bodies in the UK and plan to give them any information they deem useful.

What some of the platforms said

We didn’t get any response from Cloudflare after using one of the reporting tools to report the domain being used for what appeared to be a fake trading site. Meta and Google both responded to me that they’d taken action. A Google spokesperson said the following:

"User safety is a top priority on Google Play. We greatly appreciate research from the security community that helps keep Android users safe. We take security claims against apps seriously, and if we find that an app has violated our policies, we take appropriate action. The app has been removed from Play and the developer has been banned. Users are also protected by Google Play Protect, which warns users of apps known to exhibit malicious behavior on Android devices with Google Play Services. We also have strict ads policies that govern the types of ads and advertisers we allow on our platforms. These policies prohibit a host of harmful behavior, including ads that scam users and advertiser attempts to circumvent our enforcement systems through practices like cloaking. We’ve reviewed the ads and accounts in question and taken appropriate enforcement action."

Meta didn’t provide any on-the-record statement, but confirmed to me they took action (as mentioned, I directly worked on some of these issues when I was at the company) and that they don’t permit any content that intentionally deceives or seeks to defraud others, and that they are constantly working to remove scams from their platform. They (and I very much as well) encourage anyone who sees content that might violate their policies to report it so they can investigate and take action.

The reality is that even with the copious resources these companies have, scammers are smart and will continue to find holes that let them evade detection and enforcement. So we will need to work together for the good of people - companies and platforms large and small, and the consumers who know sketchy things and report them - so we can help those who are not sure if what they’re seeing is legitimate. I hope to put AI to work with Trust2.ai and put a dent in the billions of dollars of annual consumer losses due to scams, but it’s still early days there. Expect more details from me on that in future posts here, and on our website. Stay safe.

—

** The EU28 reporting rate for fraud was only 21%. Only 13% of scam losses are reported to Australia’s Scamwatch per the ACCC’s research, and over 30% of people do not report scam losses at all