Improvement in AI-generated voice and video is weakening one of the most familiar assumptions in workplace security that employees can recognise a suspicious request by how it looks or sounds.
A survey conducted by tech research firm Gartner among 297 senior cybersecurity leaders found that 41 per cent had experienced at least one social engineering incident involving a deepfake during an employee audio call in the previous 12 months. Another 36 per cent reported an incident involving a video call.
The findings suggest that cybersecurity awareness programmes built largely around teaching employees to identify suspicious emails, unusual language or other visible warning signs may need to evolve as generative AI makes impersonation more convincing.
Achyuth Krishna, Head of IT, Information Security and Procurement at enterprise software company Whatfix, said the problem is increasingly about verifying identity rather than simply identifying whether a piece of communication has been generated by AI.
“Emails, voices and video calls now arrive looking credible, urgent and familiar. Today, trust itself has become a new attack surface,” Krishna said. AI seems to have further complicated the social engineering part of that equation.
Attackers can now combine information collected from public sources with synthetic voice, video and personalised messages to impersonate senior executives, colleagues, suppliers or customers. A request for a payment or access change can therefore arrive through channels employees previously associated with higher levels of trust.
Gartner’s survey found that traditional social engineering remains much more widespread. About 79 per cent of CISOs reported at least one email phishing, spear-phishing or business email compromise incident in the previous year, while 58 per cent reported vishing or smishing incidents. AI is not necessarily replacing these techniques, but can make them easier to personalise and harder to distinguish from legitimate communication.
Krishna said organisations should therefore move away from relying primarily on employees to “spot the scam” and instead build verification into sensitive business processes.
“The right question is not whether an employee can tell if a message is AI-generated, but whether the organisation has designed the workflow to prevent a single deceptive interaction from becoming a security incident,” he said.
That could mean requiring an independent confirmation before approving a payment, changing privileged access or sharing sensitive information, even when the original request appears to come from a known executive or colleague.
Gartner has made a similar recommendation, calling on companies to make secure verification an expected behaviour for consequential requests regardless of whether they arrive through email, voice, video, collaboration platforms or AI applications. It has also recommended trusted verification channels and stronger authentication for processes such as account recovery, privileged access and payment approval.
The shift matters because conventional security awareness has often placed considerable responsibility on the employee receiving a suspicious message. As synthetic content becomes more convincing, determining whether a caller’s voice or a person’s appearance on a video call is genuine may become an increasingly unreliable security control on its own.
Instead, enterprises may need processes designed on the assumption that convincing impersonation is possible.
Krishna said organisations should define situations in which employees are required to stop, independently verify a request and escalate it. Those processes should also be tested through simulations rather than treated simply as annual security training.
The same identity problem is beginning to extend beyond employees to AI agents operating inside enterprise systems.
As organisations give software agents access to applications and business data to carry out tasks autonomously, Krishna said the agents should have their own identities, restricted permissions and audit trails.
This shows a broader concern among security researchers aboutagentic AI. Gartner has identified compromised AI applications, prompt injection and agents operating outside intended boundaries among the risks security teams will need to prepare for as enterprise AI adoption expands.
For businesses, the emerging challenge is therefore not simply detecting a fake voice or video. It is ensuring that no single message, call or autonomous agent has enough unchecked authority to turn successful impersonation into a payment, data leak or privileged access change.
“Deliberate verification will be as important as improving detection technology,” Krishna said.
Tech Observer Desk at TechObserver.in is a team of technology reporters led by a senior editor who brings latest updates and developments from the world of technology.