Hey, I remember those wild IM days! Using Dreamweaver to build websites. Yes, the bots will be bots so if you don't make it easy for them, they just ignore you.

Nothing has really changed, just some extra tricks to get on SERPS with all the AI results at the top. There's also E-E-A-T but that's a whole other discussion.