Here's a little Sunday story that you might find useful.
I was asked by a student to write a recommendation letter for him. As it happens, since he's an amazing student with outstanding qualities, I accepted. I accepted, because I had very clearly in mind precisely what I wanted to recommend him for.
... but then, it was the week-end and I had a lot of other things on my plate so I succumbed to the "siren's call" ... I hate the ChatGPT style ("it's not this ... it's that!") so I fired up Gemini and listed the precise qualities of my student that I wanted to extoll. Gemini analysed my input and then innocently asked me: "Would you like me to draft the recommendation letter for you?" ...
I was weak, and I clicked "yes" ... so Gemini drafted a recommendation letter. A pretty decently written one too, but definitely lacking my clunky style in English.
So I thought: I would definitely not do my student a service if it was smackingly apparent that my recommendation letter was written by an AI (even if the substance was provided by me), so I began tweaking the style here and there and wanted to check whether typical online tools would detect to what extent the text was touched by AI. I fired up a Google search "detect AI content". The first results that came up were "sponsored" - paid-for advertising.
Sponsored results are junk, don't waste your time with them
I "rigth-click -> open in new tab" on each one of them and, as the scientist and researcher I am I thought I'd start using a "negative control" - a text that I wrote for my Master thesis in EU Law for which I want to verify that the tool returns "0% AI" because the text was 100% written by me with no AI assistant whatsoever.
So here's the result:
- The top result (those that pay most to Google) was Grammarly "AI Text detector". In a nutshell, I didn't manage to check my text at well with it, I was taken from one link to another until I was brought to a screen to ... buy a Grammarly subscription. Clicked the little "X" in the upper right corner and closed the tab
- The second in the list of sponsored results was TextGuard AI. I pasted my text (the "General conclusions" chapter of my thesis). After a bit of cinematics destined to impress me it returned ... 66% AI content. A swift click on the little "X" in the upper right corner of that tab followed.
- The third is simply called AI Detector and did one better by detecting no less than 87% percent of AI-generated content in my painstakingly entirely handcrafted text.
- The fourth and last sponsored tool, CheckMyText came up claiming 78% of my text was AI-generated.
The regular results are refreshingly good
I then went on with the regular, non-sponsored results.
- The first one in the list ZeroGPT is clearly not the best - the site is ugly and choke-full of adds, but at least the "detected AI content" suddenly cratered from 60 - 90% claimed by the sponsored tools to ... 1,4%! Within the "margin of error"
- The second in the list was the relatively well known Quillbot. The free version played hard-to-get (understandably, they want you to subscribe and pay) and has a limit of 1200 words, but at least it returns the expected result ("0% of text is likely AI"). (image-5-image.png)
- The third was the best of the crop: GPTZero.me. It does have a 10000 characters limit (proved higher, at least on my text, than the 1200 words of Quillbot), but it provides the most satisfying result: "We are highly confident this text is entirely human"
What about my recommendation letter?
Well, the tools weren't kind with Gemini ... GPTZero identified the text as ... 100% AI generated. Which is unfair because I've tried to mangle a few expressions, hoping that I'd get down to perhaps 85% ...
Quillbot somewhat surprisingly only detected 44% AI, but it provided some interesting insights
ZeroGPT saw only 28.4% AI GPT (perhaps because I've used Gemini, not ChatGPT, who knows).
Conclusion
I long suspected it, now I have a strong indication: the "sponsored results" of a Google search are worse than useless. You'd do well to simply ignore them and scroll down to the regular, non "paid-for" results.