A new report from Billtrust, a provider of AI powered accounts receivable technology, carries a clear message for finance and credit professionals: do not put blind faith in artificial intelligence.
In a survey of 500 finance leaders and C suite executives, 82% voiced concern about AI’s potential for misuse. Their worry, Billtrust says, is not rooted in fear of new technology. It is shaped by direct exposure to what happens when AI is deployed without the proper guardrails.
The most striking examples come from criminal misuse. Nearly half of respondents have seen AI generated phishing emails capable of fooling seasoned employees. Almost as many have encountered AI created emails that perfectly mimic the tone and style of executives or vendors. Thirty-one percent reported receiving AI generated fake invoices with convincing branding, and 29% have seen voice cloning used to impersonate trusted contacts.
These incidents raise fundamental questions about trust. Yet Billtrust stresses that they are not an argument against adopting AI. Instead, they reinforce the need for responsible implementation that pairs algorithmic speed with human oversight and transparency. AI excels at volume, speed, and pattern recognition. Humans excel at nuance, ethical judgment, and contextual decision making. Billtrust argues that the strongest systems combine both.
Current fraud detection practices already reflect this partnership. Nearly two-thirds of organizations use bank account verification with human review. Another 63% rely on multi step approval workflows. Forty-one percent use database cross referencing to support human decisions, while one third still depend on manual phone verification. Even with these tools, 27% of respondents said their organizations either do not track suspicious invoice activity or do not know if they do, which Billtrust frames as a visibility and accountability gap.
Almost half of finance leaders worry that AI generated fraud will eventually become too realistic for humans to catch on their own. Billtrust argues that this is exactly why teams need better tools rather than less automation. AI can handle the heavy lifting and flag anomalies, freeing staff to focus on the smaller set of suspicious cases that require judgment and experience.
