You can trust whatever you want to trust. Often trust is reasonable. Trust just is not scientific.

So you would say modern physics is no science, because we don't observe quantum particles, the cosmos or time dilation first-hand?

For example, if measuring stuff about the earth was accessible to the average person

Some of the things actually are.
In 2010 it was shown, that a mechanical gyroscope actually could be used to detect earth's rotation: http://www.tkt.cs.tut.fi/research/nappo_files/Symposium_Gyro_Technology_2010_web.pdf
So if you were really determined(and had around 400 $ for equipment(it might be even cheaper today than back in 2010)) you could measure earth's rotation. And proving that all flat earth theories are wrong(all I saw say the earth stands still).

You could also do an experiment, that was already done by flat earthers:
They took a laser and pointed straight in one direction and measure the height of the laser at 3 different positions.
And you would find out, like the flat earthers did, that if two of the heights are the same, the third isn't, implying the earth is round.

And when it's not something I could see and measure, it doesn't affect my life anyways.

You cannot see quantum mechanics. You cannot measure quantum mechanics.
But still the device that is processing these words for you to be shown on a screen works only because of quantum mechanics.