How to avoid an AI dystopia?
There exists a future in which regulated industries will likely use and rely on AI interfaces to make decisions and speak to consumers, so how can firms ensure they are ready for this change?
Written by a human
In May, I had the pleasure of presenting a keynote speech at the Global RegTech Summit in which I asked the audience to imagine a “dystopian world” against a backdrop of the increasing use of artificial intelligence (AI). I wondered whether, as AI becomes more powerful, more embedded, and more influential, we are moving fast enough to govern it responsibly. Perhaps we are moving so fast, I thought, that we might forget to ask whether it needs governing at all.
Our latest Industry Insights Report, which surveys compliance, surveillance, and risk managers worldwide, found that 37.6% of respondents saw a lack of regulatory clarity as a barrier to adopting AI. 47.5% said it was somewhat of a barrier. After all, why would firms place significant investment and disrupt ‘good-enough’ processes, without the foundational knowledge that they were meeting regulatory expectations in doing so?
In a move that broadly reflects the pace of change in this industry (never a dull moment) – soon after my speech in May – the Financial Conduct Authority (FCA) published its first-ever “Emerging Technology Horizon Scan 2026”. This report, which is neither regulatory guidance nor policy, draws on research of the FCA’s “Emerging Tech & Research team” to set out “three plausible ways emerging technologies could combine to create new outcomes for consumers, firms, and markets”.
The dystopian future I discussed a month prior now feels uncomfortably close to home. And with this pace of innovation, is there any wonder that prescriptive regulatory clarity alludes us?
Key findings from the FCA’s Emerging Technology Horizon Scan 2026
The FCA’s Horizon Scan splits emerging technology into three use cases:
- Personalized intelligence
- Synthetic (in)security
- Programmable finance
Personalized intelligence
For personalized intelligence, in a nutshell, there exists a future in which regulated industries will likely use AI interfaces to speak directly to consumers, and consumers in turn will rely on AI proxies to do much of their financial decision making.
Financial services will become more personalized, with greater access to data (biometric and health data, for instance, which will be a red flag for some), but humans will be further removed from the decision-making process, thereby seeing “a gradual deterioration of both consumer attention and agency”. The report adds that:
“Everyday financial management could be increasingly conducted through machine-to-machine interactions that humans only occasionally review”.
Interestingly, our Industry Insights report found that customer support is currently the area where firms are using AI the least, with over half (54%) saying they don’t use customer-facing AI. With data security and privacy weighing in as a key barrier to AI adoption, and platforms like AI chatbots ingesting potentially sensitive customer data, firms may not be in a rush to adopt “personalized intelligence” just yet, given the greater potential for vulnerabilities and opaque delegation.
Synthetic (in)security – suspicious perfection
Given my role within a compliance technology organization, the most interesting of these use cases for me was the chapter on “synthetic (in)security” and, more specifically, the concept of “suspicious perfection”.
I routinely speak to surveillance and compliance folks who agonize over structured and complete data, traceable audit trails, and reducing gaps in their systems. The theory behind this, of course, being that if you have complete access to structured, orderly communication data, you can see the whole picture when things go wrong (or right).
The FCA, however, presents a scenario in which:
“The AI of tomorrow may so effectively simulate the conditions of credibility under which false narratives are framed as true that they can bypass both human and algorithmic judgment.”
This could mean that misconduct could occur, but AI would be able to “generate complex synthetic evidence trails” which would support or endorse the activities that took place, in an attempt to legitimize it. It could manufacture false information to “obscure malicious intent”, so compliance and surveillance teams would be none the wiser. Worryingly, we’ve seen evidence of something similar previously where an AI model conducted insider trading and a cover up.
“This may lead to misconduct becoming structurally invisible to both firms’ controls and supervisory tools.”
It begs the question: how could a compliance team spot misconduct if everything looked squeaky clean? You have a complete, traceable audit trail. There is no sign of misdemeanour. How can you flag something that – on the face of it – looks fine?
What does surveillance look like when the red flag is that everything looks too clean? Or, to use a well-worn phrase, when it’s “quiet … too quiet?”
And so, “suspicious perfection” is borne – where the primary indicator of criminal activity is no longer anomaly or error, but a clean bill of health.
The FCA’s Horizon Scan also presents the idea of “autonomous criminal organizations” – whereby, with the use of Agentic AI, an individual could “deploy, manage, and scale a global criminal organization” entirely through software. Historically, criminal organizations would need a group of highly trained professionals to carry out an attack. In the future, a single individual could instruct AI agents to carry out widespread attacks, simultaneously, and without the need for other people. This could see global criminal activity take place leaving little-to-no audit trail.
The same is true of social media. Cast your mind back to May 2023, when a social-media fuelled frenzy contributed to a run on Silicon Valley Bank, which ultimately brought it to its knees. In future, AI could be used to conduct “synthetic market manipulation”, whereby AI agents could “create thousands of fake reviews, social media interactions, and support tickets” to manipulate markets. These so-called “swarm” attacks no longer rely on human interaction – and could cause significant market swings with little clarity as to where, or why, such events are occurring until well after lasting damage is done.
How to prepare for the dystopian future of “suspicious perfection”?
So, if the dystopian future isn’t as far as I thought mere months ago, how can firms best prepare?
- Review your legacy systems – they’ll only hold you back. Legacy systems will be an issue. Many firms are still operating on fragmented infrastructure that was never designed to handle the volume, variety, or velocity of data that the present-day demands (let alone the demands of the AI-era). If your foundations aren’t solid, your ability to detect and respond to the kind of sophisticated, AI-generated misconduct the FCA describes will be severely compromised from the outset.
- Choose agile vendors with the resource and infrastructure to constantly evolve. The “arms race” the FCA describes is a fast one. A vendor that can’t update its detection capabilities as quickly as bad actors are updating their attack methods isn’t a long-term partner – it’s a liability. When evaluating technology, compliance teams should be asking not just what a platform does today, but how quickly it can adapt tomorrow.
- Surveillance can spot patterns – the sharper it is now, the better it will fare. The good news is that surveillance systems trained on rich, high-quality data are better positioned to notice when something feels statistically too clean. Investing in surveillance capability now, before the threat fully materializes, isn’t just prudent, it’s the difference between having a calibrated baseline to work from and starting from scratch when it matters most. And, of course, having experienced, well-trained professionals in the loop will be vital in identifying when things “feel” off.
- Understand where the audit trail comes from, and reorient detection logic to provenance, not just content. If AI can fabricate convincing content, the question shifts from “does this look legitimate?” to “can we verify when and how this record was created?”. Immutable, tamper-proof data that is captured at source, complete with metadata – before any manipulation is possible – may become a critical line of defense.
- Build your community and share intelligence industry-wide
A cause close to my heart. But it’s true to say that no single firm will be able to stay ahead of this alone. The FCA’s own conclusion points to cross-sector collaboration between firms, regulators, and technology providers. Knowing what fabricated evidence trails currently look like in the wild, and sharing that intelligence quickly, is what will allow detection models to evolve faster than the threat does.
If you’re interested in joining our community of periodic roundtables and intelligence-sharing events, get in touch. In the meantime, upgrade your surveillance to a solution that works now – and for the future.
Find out more.
