What I Learned: Can ChatGPT Actually Do Audit Work?
ChatGPT - Code Interpreter. Initial experiments and implications for auditors.
Hi everyone,
Last week, ChatGPT publicly released the beta version of its new "Code Interpreter" feature. This feature empowers ChatGPT to write, run, and even debug Python code in the background to tackle highly complicated problems that ChatGPT previously struggled with - things like code, math, logic, and so on.
A bit of background information: Python is one of the world's most popular coding languages. It's exceptionally flexible in its capabilities, and its legibility, especially for English speakers, makes it easy to learn. Where it lacks in specialized performance, it compensates for in broad ability and facilitation of collaboration.
An intriguing potential use case for this language, and thus ChatGPT’s new model, is text extraction from a PDF.
Without delving into the minute details, it's possible to write Python code that employs a process called Optical Character Recognition (OCR) to identify and extract text from images or PDFs.
As auditors, we often receive large volumes of PDFs from our clients, which we must laboriously sift through to find the specific information we need. This task is often tedious, requires little intellectual ability or specialized knowledge, and is extremely monotonous.
So, what do big accounting firms do? They outsource this work to other countries!
Generally, it’s cost-effective, time-efficient, and allows our staff to focus on more meaningful activities. However, it still costs a significant amount of money and time.
These issues, combined with ChatGPT’s new capabilities, led me to believe that I could probably automate the entire process with ChatGPT.
The result was astounding.
Process
Here's a summary of my initial experiments with the product, accompanied by images below:
1. I first opened a recent PDF invoice I received for a personal purchase.
2. I asked ChatGPT to “OCR an image for me and provide a txt file,” and then uploaded the file.
3. From the code it produced, it was clear that it recognized the file as a PDF, not a JPEG or another image type, and correctly chose to use the PyPDF2 and pdf2image libraries from Python.
4. You can view the text output below. It also provided a download link for the txt file.
5. I then requested:
“Now use the text and your understanding of it to produce a table I can copy with the information. The headers of the table should be the following:
Invoice number
Description of invoice
Date
Nature or purpose of invoice
Amount (before tax)”
Unsurprisingly at this point, it gave me a perfect table that can easily be copied and pasted into any Excel spreadsheet.
Why This Matters
Now, you might be asking, “So what? Couldn’t any person with Python knowledge do this?” That's a fair question, but here’s why it matters.
Interpretation
You’re right, most of this task could be accomplished by a Python script. But not all of it. The real potential of this new feature is the utilization of ChatGPT’s ability to interpret words.
Pay attention to column 4 of the table above. This is the nature or purpose of the transaction. This information was not included on the invoice I uploaded. ChatGPT took the text from the invoice and correctly interpreted that this was the purchase of a digital product.
It’s not just finding the right values from the PDF and summarizing it in a table. It’s performing association, interpretation, and for lack of a better word, thinking. That’s why its so powerful.
Accessibility
I’m learning Python code, but I couldn't have achieved this. The outcome essentially saved me a significant amount of time learning how to write such code. Anyone with limited Python knowledge and a clearly-defined desired outcome can do this.
It’s like understanding what arithmetic division is and using a calculator versus actually performing the long division by hand.
Speed
This task took me a few minutes. Honestly, doing it by hand might have been faster. Yet, this was only one invoice. What happens when you have 70, along with purchase orders, delivery receipts, etc.?
I have yet to test this with multiple documents, but even if ChatGPT fails at that, I don’t think it will be long until it can handle it with ease.
At that point, the tool could potentially save accounting firms hundreds of hours on this single use case. I foresee many more use cases to come as well.
Market Competition:
I have been deeply and lengthily pondering the impact that highly capable AI models will have on the general economy, as well as corporate and labor markets.
The ability of Code Interpreter to perform hundreds of hours of audit work in minutes raises serious questions.
What happens when small accounting firms (or businesses in general) can produce the same quality and volume of work as large firms in the same amount of time?
What does that mean for big firms, for the competitive landscape, and for the need for entry-level employees?
Make no mistake. This technology will change everything, and we are only at the beginning.
Final Point
I wrote about it at length here, but I need to emphasize again that information security with these systems still requires addressing before they can be employed with sensitive information.
We auditors can’t use this tool yet because it’s fundamentally unsafe and unethical to upload sensitive client information to ChatGPT. However, the potential of the tool is what truly fascinates me. It won’t be long until the security of these systems is resolved, whether by OpenAI itself, or by businesses building their own proprietary models.
Where we go from here is anyone’s best guess, but one thing is certain - things are about to get wild.