I feel your pain with the rubble problem, I have it too. Flowers will probably cope but I couldn't grow vegetables successfully until I actually just gritted my teeth and sieved the soil properly. Your other option is of course raised beds: I couldn't do that because my climate is too dry but that's also worth considering