ChatGPT wrong on more than half of Google Analytics 4 FAQs 

What happened when fifty-five tried replacing their consultants with ChatGPT 

In a study of 42 FAQs about GA4, ChatGPT provided only 12 correct responses One in five (19%) responses provided misleading answers fifty-five found that ChatGPT is no match for marketing experts – yet! 

Since exploding onto the scene in November 2022, ChatGPT has been one of the most hotly debated new tech trends, predicted to change the world of work. However, new research from MarTech consultancy fifty-five reveals that the bot is a long way from revolutionising the digital marketing landscape, after it answered only three in 10 (29%) FAQs about Google Analytics 4 (GA4) correctly.  

With GA4 migration on the cards for many clients before the end of Universal Analytics in July, fifty-five consultants were interested to test ChatGPT’s capability, but the hottest new chatbot in town fell short of their expectations. Of 42 FAQs about GA4, covering access to raw data, active users and behaviour modelling, ChatGPT was incorrect in more than half (52%) of its answers. 

On one in five (19%) questions, ChatGPT was ‘semi-correct’. In some cases, the bot misinterpreted the question and provided a factually correct answer to an entirely different question, showing issues with its comprehension where a human would have easily understood what was being asked. In others, it provided a small amount of misinformation in an otherwise correct response, which is less likely to be recognised as being incorrect and could lead to significant confusion. 

All AI is limited by its training data, and fifty-five’s study points to the age and quality of the data it collects as a reason that ChatGPT falls short on GA4. Currently, the bot uses data from 2021, which means that it is often out-of-date given how quickly and how frequently updates and bug fixes are made. At the same time, the service makes no judgements about the quality of the data it uses. It does not rank data based on domain authority, which means data does not come from trusted sources, and is more likely to be incorrect. 

Given the importance of analytics for helping businesses to make marketing decisions and evaluating success, the research shows that, currently at least, Generative AI is no replacement for marketing experts, and is unlikely to be for a very long time, if ever.  

Summing up the research Sam Burr, consultant at fifty-five concludes: “The potential of Generative AI, like ChatGPT, is clear for all to see – and marketers will have to adapt to it in future, but for now they should think twice before using it to inform activity. It can point you in the right direction most of the time, but its responses should be followed up with additional research through alternative sources – it is important to remember that it is a chatbot, not an oracle. It might be able to improve productivity, spark creativity or kick off a conversation, but, until this tech is more developed, it certainly wouldn’t pass probation at fifty-five.”  

Source: fifty-five

  ChatGPT, fifty-five, Sam Burr