CPAs vs ChatGPT vs Jasper.ai

Want to see if Accountests will work for your firm? Click on the button for a FREE trial test - use it on a candidate or get one of your staff to give it a go and see what they think.  

Who’s better – ChatGPT or Jasper? And would they pass a CPA level exam? We thought we’d get Chat GPT and Jasper.ai to take our CPA level Public Accounting Test. This test is used by employers to assess candidates for roles where a CPA licence is usually required - like a Client Accounting Manager. It covers some core accounting skills, some financial reporting, some tax and some advisory – so pretty much everything a CPA in public accounting could be expected to deal with. 

The short answer is that both AI’s are better than the mean scores of CPA’s in this test, but ChatGPT was better. In a tough 40 question test ChatGPT achieved 29/40 whereas Jasper got 23/40.  This article is a follow on from your previous one ChatGPT Took Our CPA Test

We analyzed the questions into categories to try and see where each AI performed. This is not a scientific analysis but what we found was: 

1. Jasper struggled with computational based questions – failing with 5 out 6 questions we classed as numerical calculations. For example calculation of receivables days outstanding 
   
   
2. Both struggled with questions about accounting standards (US GAAP). They understood the questions but sometimes failed to find the correct answers from the standards
   
   
3. ChatGPT suffered from its 2021 limit on data as current tax questions left it resorting to telling us to check current legislation! 
   
   
4. ChatGPT was much better at showing the basis for why it reached the conclusions it did (right or wrong)  

The two AI platforms are somewhat different [skip this if you don’t like jargon]. When I asked Jasper to explain the difference it told me “Jasper is an encoder-decoder model that leverages convolutional neural networks to process input text. It is trained on large amounts of data and is able to generate new sentences as responses. ChatGPT, on the other hand, uses a transformer-based language model that utilizes self-attention structures and multiple layers of processing. This allows ChatGPT to create more human-like conversations by taking into account context and background information when generating its responses.” 

So overall Jasper is best suited for tasks like machine translation and text summarization, while ChatGPT is better at generating human-like conversations. 

To give some context around their capability (or where it lets them down) here’s a few of the areas they both had difficulty with: 

1. Calculation of a breakeven sales level when needing to take in to account a couple of different factors 
   
   
2. Deciding on the BEST course of action given a particular circumstance – there is no ‘right’ answer and advisors need to use their experience to know where issues arise 
   
   
3. Failure of simple logic e.g. intangible assets are subject to amortization, so depreciation is irrelevant 
   
   
4. Inability to use ‘two step’ logic when considering the accounting implications of a transaction e.g. accounting implications might be more than just in the current year 
   
   
5. Not understanding the implications of inventory valuation methods 
   
   

When challenged and then directed ChatGPT can learn to change its responses - so it's learning and getting better all the time.  When you ask the same question later it can then get it right

I asked ChatGPT which of the two AI’s would be better for use in solving public accounting firm issues. It told me "Jasper.ai might have a higher likelihood of providing correct answers to accounting problems due to its domain-specific focus, which could include specialized training data, algorithms, and integration with accounting-related databases or resources. It may also have features such as intent recognition and domain-specific knowledge that enable it to better understand and respond to accounting-related queries accurately.” 

I then asked Jasper the same question – it gave me much of the same background and in summary said “if the accounting question is straightforward and well defined, either ChatGPT or Jasper may provide a reliable answer.” 

So by way of conclusion in relation to the use of ChatGPT and Jasper in public accounting: 

1. Their technical knowledge is good without being great.  But it's getting better all the time 
2. Our short unscientific trial would suggest ChatGPT was better 
3. ChatGPT thinks Jasper would be better! 
4. Jasper doesn’t know who would be better! 

About the Authors

Giles Pearson FCA was a PwC Partner for 18 years before jointly setting up Accountests.  

Steve Evans has a whole career dedicated to enabling employers to attract, recruit, develop and retain talented individuals and teams, with particular expertise in candidate testing and assessment before setting up Accountests.

Accountests deliver the world’s only online suite of annually updated and country-specific technical knowledge tests designed by accountants for accountants and bookkeepers.