Posted: 16 Jun. 2023 5 min. read

Generative AI and Fraud – What are the risks that firms face?

The development of Generative Artificial Intelligence (“Generative AI”) offers exciting opportunities for companies to develop automated systems and controls to protect themselves and consumers from financial crime and risk. Over the past few months, there has, however, been extensive discussion about the risks that Generative AI poses to industries regarding the potential for malicious actors to deploy Generative AI to commit fraud in many different forms. These developments in Generative AI also present opportunities to detect, prevent and mitigate fraud.

In this blog, we touch on some of the fraud risks that AI presents and offer some considerations for firms to mitigate those risks. 

Introduction to AI enabled Identity Fraud

Much of the controversy around AI has been focused on the advances made in Generative AI, which can be used to create new media, including audio, code, images, text, and videos. There are also certain websites or apps that can generate human-like responses in chat conversations. The leading Generative AI products include the GPT family (including ChatGPT) and BERT, and these AI programs are rapidly expanding their capacity to provide all forms of audio, visual media and complex documentation upon simple request. The ability to create audio, video, images and documents that appear genuine to the human recipient presents fraudsters with new opportunities to commit fraud in ways we discuss below.

Deepfakes & Voice Spoofing

Generative AI programs such as VALL-E, DALL-E and Midjourney can be used to clone vocal patterns, create audio files, and create fabricated photo and video media. Regula, a global developer of forensic devices and identity verification solutions, published in the findings of a survey that 37% of organisations globally have experienced some kind of deepfake voice fraud attempt. The first ever AI facilitated fraud case was reported in 2019 where scammers used an AI generated voice clip of an energy group CEO to direct the CEO of its UK subsidiary to release EUR 230,000 to a fictitious Hungarian supplier. 

The use of vocal recognition as a means of identity verification for accessing bank accounts may no longer prove to be as secure as they have previously been as AI advances in its ability to mimic speech patterns and vocal identification measures. 

Email Phishing

Fraudsters typically use Email Phishing to manipulate individuals into making transactions that reward fraudsters or offer up personal data that facilitates fraud. It involves a fraudster sending an email that purports to be from a genuine source, requesting the recipient provide security data either directly or via clicking a link to a bogus website. Cambridge based cybersecurity firm Darktrace has warned that since the release of ChatGPT, it has seen an increase in the linguistic complexity, the volume of text, punctuation and sentence length used in suspicious emails by those targeting their customers. They have also seen a decrease in the number of manipulative emails which rely on tricking victims into clicking malicious links. As a result of this switching of methods used, Darktrace estimates that Generative AI is being utilised to construct increasingly sophisticated phishing scams and that cybercriminals may be redirecting their focus on crafting more sophisticated social engineering scams that exploit user trust, which may have used ChatGPT or other forms of AI.

The damage resulting from successful phishing scams (e.g. customer data loss) can impact reputation, market position and bring regulatory scrutiny. Developments in phishing sophistication pose greater challenges for those developing defences and staff training to recognise these attacks.

Synthetic Identity Fraud

Synthetic identity fraud is a type of identity theft in which criminals combine both real and fake personal information to create a new, fictitious identity that can then be used for various identity-related schemes, such as obtaining credit or goods.

This is the fastest growing form of financial crime in the United States and now costs US financial institutions billions of dollars annually. It is particularly pernicious as it is hard to detect with traditional fraud detection methods. Firms are increasingly turning to advanced fraud detection and prevention technologies that themselves incorporate AI, to identify patterns and anomalies signifying fraudulent activity.

Document Forgery Fraud

Fraudulently produced documents present a real risk to auditors and third parties looking to verify information about a corporate entity. Generative AI programs can potentially create bank statements, accounting documentation, Board minutes and other documents required during an audit, due diligence exercise or other process requiring evidence, with comparatively little effort when compared to traditional forgery methods and with greater authenticity. This is exacerbated by new “fraud-as-a-service” offerings, where experienced fraudsters with a suite of skills and technologies offer their services to enable others seeking to undertake a particular form of fraud.  

A number of AI systems promote the ability to identify forged documents using traditional methods (freehand simulation, tracing or simple electronic manipulation) and to detect false accounting. It may only be a matter of time before Generative AI becomes sophisticated enough to be able to generate identity, banking and corporate documents almost entirely indistinguishable from genuine ones by human reviewers.

What can firms do to protect themselves?

The forms of fraud that have emerged from the adoption of AI are complex, and this continues to be a quickly evolving area. Considering the types of fraud we have explored above, we set out below some key considerations for firms as AI usage becomes more widespread by fraudsters:

  • Consider if your fraud risk management framework is robust enough to resist novel modes of fraud enabled by Generative AI or whether outside assistance is required.
  • Reflect on your fraud risk management systems and consider if there are areas of vulnerability in which the forms of AI-enabled fraud set out above could occur.
  • Identify areas of financial controls (e.g. payments) vulnerable to Senior Manager override and ensure that secure systems are in place that prevent irregular transactions.
  • Understand if there are particular identity verification requirements in your business model that could be subject to evasion by the production of AI generated content.
  • Assess the level of training and awareness required of staff members to minimise the risk of vulnerability to more sophisticated phishing scams.
  • Review identification verification procedures and consider whether they are sophisticated enough based on the size and nature of customer and supplier base.

Should you wish to discuss this topic further or require support with considering the fraud risks posed by AI and assurance services relating to your Control Framework, please don’t hesitate to get in touch with our Fraud Governance, Risk and Control team here. Should you wish to further assess your future with AI technologies please contact our AI Assurance team here.

Key contacts

Mark Cankett

Mark Cankett

Partner

Mark is a Partner in our Regulatory Assurance team. He is our AI Assurance, Internet Regulation and Global Algorithm Assurance Leader with 20 years of experience across financial services audit and assurance, regulatory compliance, regulatory investigations and disputes. He has led the development of our assurance practice as it relates to our approach to assisting firms gain confidence over their algorithmic and AI systems and processes. He has a particular sub-sector specialism in the area of algorithmic trading with varied experience supporting firms enhance their governance and control environments, as well as investigate and validate such systems. More recently he has supported and led our work across a number of emerging AI assurance related engagements.

Fraser Beveridge

Fraser Beveridge

Partner

Fraser is a Partner within our Regulatory Assurance practice in London where he leads our Markets Assurance team. He has extensive experience in providing assurance services to financial services clients with a particular focus on the latest emerging regulatory issues.

Key contacts

Barry Liddy

Barry Liddy

Director

Barry is a Director at Deloitte UK, where he leads our Algorithm, AI and Internet Regulation Assurance team. He is a recognised Subject Matter Expert (SME) in AI regulation, has a proven track record of guiding clients in strengthening their AI control frameworks to align with industry best practices and regulatory expectations. Barry’s expertise extends to Generative AI where he supports firms safely adopt this technology and navigate the risks associated with these complex foundation models. Barry also leads our Digital Services Act (DSA) & Digital Markets Act (DMA) audit team providing Independent Assurance over designated online platforms' compliance with these Internet Regulations. As part of this role, Barry oversees our firm's assessment of controls encompassing crucial areas such as consumer profiling techniques, recommender systems, and content moderation algorithms. Barry’s team also specialises in algorithmic trading risks and controls and he has led several projects focused on ensuring compliance with relevant regulations in this space.

Luke Stevens

Luke Stevens

Manager

Luke is a Manager in Deloitte’s Fraud Governance, Risk and Controls team, based in London, as part of the Regulatory Assurance practice. He has experience providing assurance services for clients’ fraud and conduct risk and control frameworks, leading investigations and assisting Internal Audit teams. Prior to joining Deloitte, Luke was a Forensic Investigator at the Financial Conduct Authority for 7 years, leading investigations into insider trading, market abuse, fraud and regulatory misconduct.