SMSF auditors should not view artificial intelligence (AI) and workplace automation as equivalent tools as they will produce different outputs, with one practitioner noting the former can generate false and misleading outcomes.
Tactical Super director Jacob Kewley cautioned practitioners from working with AI tools, given they were neither intuitive nor knowledgeable about the specifics of an SMSF audit.
“One of the biggest misconceptions in our industry is treating AI and automation as the same thing when they are not and their differences are quite significant,” Kewley said during a presentation hosted by The Auditors Institute yesterday.
“Generative AI, such as ChatGPT, Claude, Copilot, predict language at the most base level. It’s a probabilistic system and generates the most likely word that’s going to come next in a sequence given what it’s already been shown.
“Understanding these models are probabilistic means recognising it can fabricate references, can misinterpret data and even deliver incorrect answers with total confidence.
“On the other hand, automation is a different animal and centres around predefined logic. If X happens, do Y, such as a document gets uploaded, a work paper is populated and an output is completed.
“Importantly, because automation is deterministic rather than probabilistic, every run behaves the same way, output is repeatable and you can point to the exact logic that produced it.”
He said this was critical to understand as an audit opinion had to be a defensible conclusion backed by documented evidence and was governed by a number of standards with which a probabilistic AI system was likely to be in conflict.
These standards require that another practitioner could follow the process used, the audit evidence was appropriate and professional judgment was not delegated, he pointed out.
“If we run the same prompt twice in an AI and get two different answers, that just means we’re documenting output rather than methodology,” he said.
“With evidence, if we go with ‘the AI said so’, that has the same type of evidentiary weight as the ‘trustee said so’ or ‘my mate down the street said so’, but that is where the work of the audit starts, not where it ends.
“Ultimately both professional scepticism and the signature on the audit report remain ours.
“Having a brilliant system that scopes both risk and compliance doesn’t do away with that and an auditor can’t delegate scepticism to a system that bears none of the consequence of being wrong.”
Jason Spits
SENIOR JOURNALIST
Jason is a senior journalist with Benchmark Media and writes for the selfmanagedsuper website and magazine and smstrusteenews website. He has covered financial services since 1999 and has written about life insurance, superannuation, investment management and financial advice, in both a freelance and in-house capacity, and also did a brief stint as a media manager for an industry association. He has been the editor of trade titles Money Management and Financial Standard, and was most recently senior journalist with life insurance title Riskinfo.
Articles by Jason