Oh no no no... please do not remove, just be sure they are lichens, hahaha. I know olive groves from my neighbours in Spain and they all grow in arid lands. That would mean the soil is acidic and olives grow in such areas where the bed is rocky. It is very interesting to know if the "lichens" found on your olive trees are really lichen because that might mean your area is environmentally sound.