Can ChatGPT write my articles?
I thought it was about time I delegated all my work to AI, so I’ve given ChatGPT a chance to claim the role. Here’s what happened …
The Challenge
I will ask ChatGPT questions corresponding to the titles of my articles. The questions are:
- “how do I integrate amplitude with webflow?” (my article)
- “how to calculate dau/mau ratio in Amplitude?” (my article)
- “what are the differences between funnels in GA4 and Amplitude?” (my article)
- “how to set a KPI target for my team?” (my article)
- “how can I use AI in my product?” (my article)
Quick Intro to ChatGPT
In case you haven’t heard, ChatGPT is one of the most modern and powerful AI chatbots out there. Here is a little info given when using the tool, which explains what it can and can’t do.
The Results
I’ll go question-by-question, show you what ChatGPT answered, and compare it to my article.
“how do I integrate amplitude with webflow?”
First impression, quite impressed! The steps are clear, correct, and a little more concise than my article. My article might be slightly easier to follow because of the screenshots. Also, the AI instructions stop a little early, and much of the value is in the last paragraph. I’m sure ChatGPT could break that down into steps too though.
Answer Rating: 4/5
“how to calculate dau/mau ratio in Amplitude?”
Not so impressed here. This answer looks good, but it’s wrong — the steps don’t work. In fact, the steps don’t even make sense. I feel the AI has confused Amplitude with another tool here, and its confidence is annoying because you might waste some time trying to follow along.
Answer Rating: 1/5
“what are the differences between funnels in GA4 and Amplitude?”
Not bad, not great. The numbered list covers fairly similar points to my article, but the details are hit-and-miss. The emphasis on events vs pageviews and custom vs predefined would have been really relevant for Universal Analytics, but less so for GA4. However, highlighting the differences in Segmentation and Visualization is spot on.
Answer Rating: 3/5
“how to set a KPI target for my team?”
Touché, ChatGPT, you’re pretty good! And BTW you love a numbered list! I pretty much agree with the methodology, and it’s coherent. Without specifying a more unique situation, I don’t think this answer is any worse than my article.
Answer Rating: 5/5
“how can I use AI in my product?”
It’s a decent answer, and useful as a high-level intro. I like the little summary paragraphs at the start and the end. What this one lacks is audience awareness — I think my article is more interesting for product managers, while the AI answer has broader relevance.
Answer Rating: 3/5
The Verdict
In total, the answers scored 16/25. I think that’s pretty impressive, albeit the bad answers were frustrating.
ChatGPT is good at giving high-level guides for the topics of my articles, but can be unreliable when providing detailed instructions.
The Future
ChatGPT is clearly already very powerful, and will only get better. Looking through the lens of my very limited use case, I feel the key is not really the accuracy of the answers, but rather the selection of the questions: we need to learn the kinds of questions that work, and those that don’t.
Clearly, the AI relies on human input — for these kinds of specific questions, it relies heavily on existing human written material answering very similar questions.
Perhaps what impresses me most about ChatGPT, is that it writes so succinctly and clearly. I could learn from that!
Finally, as somebody who writes, I’m curious about how the authors of the material on which the AI is trained could/should be rewarded.
One More Thing
For the sake of being annoyingly meta, I gave this a go …
If you’d like support with your Product Analytics, I run a small agency, and would be happy to help with your implementation or analysis project.
