Enterprise AI security discussions are shifting from “Can the model be bypassed?” to “Can AI persuade people?” We have noticed that this type of risk cannot be solved by a single security product. It tests identity management, access control and employee judgment at the same time.
What this is
Social engineering uses impersonation, manufactured urgency or borrowed trust to induce people to hand over passwords, files or system access. Generative AI can now produce convincing emails, voice messages and conversations, then combine them with public information to increase the success rate of fraud. AI itself may not have greater attack capabilities, but it lowers the barrier to launching social engineering attacks.
This also means that merely filtering malicious prompts cannot eliminate the risk. As long as a request comes from a trusted colleague and is disguised as routine business activity, an attacker may still bypass the defenses. A more complete solution should combine multi-factor authentication, least-privilege access, anomalous behavior monitoring and employee training, rather than assigning responsibility for security to a chatbot.
Industry view
Our judgment is that AI security budgets will shift more from “protecting models” toward “protecting identities and workflows.” Enterprises will strengthen email and voice verification, approval processes for sensitive actions, privileged account management and human review of high-risk requests. Once AI can act on behalf of employees, access failures will be harder to trace than content errors.
The risk is clear. Companies may overdepend on automated detection while overlooking the most critical link: people still make mistakes under pressure, authority and deadline pressure. Social engineering did not disappear with AI; it simply became more credible. Meanwhile, monitoring employee communications without clear boundaries can create privacy and labor-management problems. Assigning all responsibility to employee training is likewise an excuse to avoid fixing flaws in system design.
Impact on regular people
For enterprise IT: Security teams need to reassess accounts, permissions and approval workflows, paying particular attention to whether AI agents have been granted excessive access to files, email or payments, and whether auditable records are retained.
For individual professionals: Employees will encounter synthetic voice messages, targeted emails and requests impersonating colleagues more frequently. Instructions involving transfers, passwords, keys or customer data should not be trusted merely because their tone sounds natural; they must be verified through an independent channel.
For the consumer market: Fraud content will become more convincing and varied, prompting service providers to strengthen call and transaction risk controls. At the same time, identity verification, privacy protection and AI labeling rules may become new trust signals for platforms and digital products.