Charting the Light and Dark of Gen AI for Content Creation

Whether it’s turning people into dolls for their own entertainment, to enhancing operations across the biggest businesses in the world, Gen AI can do amazing things.

For the first time ever, Netflix recently deployed Gen AI to create visual effects in one of its original productions to depict a building collapse in its series The Eternaut, an Argentine science fiction series.

The technology, it said, enabled the production team to complete the sequence 10 times faster than traditional special effects methods while reducing costs substantially.

Ted Sarandos, co-CEO of Netflix, says: “The cost of it just wouldn't have been feasible for a show in that budget.

“That sequence actually is the very first [Gen] AI final footage to appear on screen in a Netflix original series or film. So the creators were thrilled with the result.”

Elsewhere in the world of business, Gen AI is driving efficiencies across operations.

Yet despite the endless opportunities Gen AI seems to offer, the technology has a dark side, with sophisticated deepfake scams posing ever-more serious threats.

How Ferrari identified a deepfake attempt through personal verification

The same technology that enables creative breakthroughs has also become a weapon for sophisticated corporate fraud. 

A deepfake scam is when criminals use AI to create fake audio or video that looks and sounds like real people, often tricking businesses into sending money or sharing secrets by pretending to be bosses or colleagues.

Ferrari narrowly avoided a potential financial disaster when one of its executives reportedly encountered an elaborate deepfake scam impersonating its CEO.

The incident began when a Ferrari executive received unexpected WhatsApp messages from an unknown number purporting to be from Benedetto Vigna. 

The messages, which included a profile picture of the CEO standing in front of Ferrari’s prancing-horse logo, mentioned an urgent acquisition requiring immediate assistance.

“Hey, did you hear about the big acquisition we’re planning? I could need your help,” one message read.

Another message urged: “Be ready to sign the Non-Disclosure Agreement our lawyer is set to send you asap. Italy’s market regulator and Milan stock-exchange have been already informed. Stay ready and please utmost discretion.”

The scammer later escalated the attempt with a phone call using AI-generated voice technology that mimicked Vigna’s distinctive southern Italian accent. 

During the conversation, the impersonator claimed to be using a different phone number due to the confidential nature of a deal that required a currency-hedge transaction.

However, the executive noticed subtle mechanical intonations in the caller’s voice that raised suspicions. 

To verify the caller’s identity, the executive asked a personal question about a book Vigna had recently recommended. 

When the caller could not identify “Decalogue of Complexity: Acting, Learning and Adapting in the Incessant Becoming of the World” by Alberto Felice De Toni, the call abruptly ended.

“Sorry, Benedetto, but I need to identify you,” the executive said during the verification attempt.

WPP facing the risks of deepfake’s increasing sophistication

Ferrari’s experience is just one part of a broader pattern of deepfake attacks targeting senior executives. 

Mark Read, CEO of WPP, the world's largest advertising group, became a target when fraudsters created a WhatsApp account using his photograph and deployed voice cloning during a Microsoft Teams meeting.

The attackers impersonated the CEO and another senior executive to solicit money and personal details from an agency leader within the company’s network. 

“We all need to be vigilant to the techniques that go beyond emails to take advantage of virtual meetings, AI and deepfakes,” Mark wrote in an internal email to staff. 

His internal communication outlines specific warning signs for employees to recognise potential attacks, including requests for passport information, money transfers and references to secret acquisitions or transactions unknown to other company personnel.

“Just because the account has my photo doesn't mean it’s me,” he says. 

The financial sector has also experienced significant deepfake fraud attempts. 

An executive at defunct digital media startup Ozy pleaded guilty to fraud and identity theft after reportedly using voice-faking software to impersonate a YouTube executive, attempting to deceive Goldman Sachs into investing US$40m in 2021. 

What humans can do to help companies can fight back against deepfake threats

There’s a simple human trait AI lacks that can bring a deepfake scam to its knees: curiosity.

The Ferrari executive's quick thinking illustrates the most effective defence against deepfakes: human verification protocols. 

As these threats escalate, companies are implementing multi-layered strategies that don't rely solely on technology to detect fakes.

Rachel Tobac, CEO of cybersecurity training company SocialProof Security, notes the acceleration of voice cloning attacks: “We're seeing an increase in criminals attempting to voice clone using AI,” she says.

The most practical approach involves establishing verification protocols for any high-stakes communication. 

This means requiring callback procedures, secondary confirmation channels or in-person verification for critical decisions involving financial transactions or sensitive information sharing.

Employee education also remains crucial. 

Staff training should cover recognition techniques, including awareness of subtle audio quality issues, unusual speech patterns or requests that deviate from normal business procedures. 

The key is teaching workers to pause and verify rather than react immediately to urgent requests, even when they appear to come from trusted sources.

Companies are additionally implementing multi-factor authentication as a critical first line of defence, ensuring that access to sensitive systems or data is not granted based solely on potentially compromised credentials or communications. 

This approach proved effective when CyberArk, an information security company, began training its executives to identify bot-based scams before they suffered any losses.

How to use AI against itself to beat deepfake scams

Just like how humans and AI are most effective when working together, the same approach applies to combating AI scams.

While human verification protocols form the foundation of deepfake defence, companies are also deploying detection technologies. 

AI-driven deepfake detection systems can identify visual fraud, text fraud and behavioural anomalies during verification processes, though experts warn that attackers' technical sophistication typically outpaces detection capabilities.

Liveness detection technology can detect subtle movements, such as blinking or changes in facial expression, that are difficult for deepfakes to replicate convincingly. 

Financial institutions are particularly focused on these technologies during digital onboarding processes, where fraudsters might use synthetic identities supported by AI-generated media.

Behavioural biometrics offer another layer of protection by analysing patterns in user behaviour, such as typing speed, mouse movements, and navigation habits. 

These patterns are unique to each individual and difficult for fraudsters to mimic accurately.

The cost of doing nothing

Deloitte’s Center for Financial Services predicts that fraud enabled by Gen AI could reach US$40bn in losses in the US by 2027. 

Arup, a multinational design and engineering company behind world-famous buildings such as the Sydney Opera House lost HK$200m (US$26m) after scammers used deepfake technology to fabricate representations of company executives during a video call.

These AI-based deepfake sophistication tools are only going to become more accurate.

Where deepfake scams are extra dangerous is in using what makes us human against us, preying on personal vulnerabilities rather than technical systems, with attackers leveraging fabricated videos or audio to evoke emotional responses. 

This makes traditional cybersecurity measures less effective against social engineering tactics that target human psychology.

Yet the common theme throughout these deepfake attacks – is people – from the vulnerabilities exploited to allow an attack, to the curiosity and vigilance needed alongside technology to prevent them. 

The organisations that survive this new threat environment will be those that combine technological solutions with rigorous human verification protocols and employee training.

“Audio and visual cues are very important to us as humans and these technologies are playing on that,” says Rob Greig, Chief Information Officer (CIO) at Arup.

“I think we really do have to start questioning what we see.”