One day I put in a question into an AI, chatgpt to be precise and that same day I made a decision about my health based on the answer I got from it. I did this without consulting a doctor or getting a second opinion, it was only me and a gpt 3.5 making a decision about my own health. Thankfully I can say that all turned out okay, that time but that's not the point. The point is what if it didn't because we already know about the limitations of AI and the habit of hallucinations and also getting the wrong context.
The main problem with is how sure they sound in their answers. They give their answers to you as though they are fully confident without hesitation, there's no let me look into that or pause between when you asked your question and when they reply, but just because it sounds like someone who knew what they were talking about doesn't necessarily mean it is correct.
I've experienced situations multiple times where two people that have been trained as professionals, doctors each with PHDs have disagreed on what to do with the patient and have had to discuss, consult and look at different aspects of the situation. But an AI can give you a definitive answer in 3 seconds flat.
In general cases I can say you should use it as you see fit. It is helpful when you need assistance in any area of your life. But if it involves a part of your body or someone else’s body, make sure you take the time to verify it through proper channels because a mistake with that could potentially lead to death itself.
There is not much risk that an AI will obviously mislead you but the bigger risk is AI’s ability to mislead you a manner that appears to be completely reasonable. This is a much more complex risk to prepare for.
As a friend who was a doctor told me, part of the training of the doctor is to realize how little they know, we know an AI does not go through this same process. They're called predictive models for a reason. They're predict and they do so being confident. Now I use Gemini more and I keep seeing them bring up, "this should not be taken as medical advice" whenever I ask a personal medical question. That's a good step at least.
