Previous I had put out some pieces of bamboo for solitary bees to lay their eggs in. I have since heard from @solominer and other sources that this is not ideal, partly due to buildup of parasites. I found this company who sell alternatives.
These cardboard tubes have an inner that you can remove once the eggs are laid. Those can be stored safely whilst the grubs develop and then they can be extracted by soaking them for safe storage over the winter. The company offers a service where you can send the tubes to them, but I will attempt it myself.
I have installed them in my little bug house. As well as the plastic bottle they are in there is an overhanging roof to keep the rain off.
I will be keeping an eye on this. If they start filling the tubes I will get some more refills to make room for more. I did not notice so many mason bees around last year, but I hope they will like this arrangement. You can buy fancy boxes and holders for the tubes, but I think this will do.
I have seen a few bumblebees in the garden recently. I think they tend to nest underground.
I am trying to build up other habitats around the garden for various bugs. We have the space, so may as well try to help the wildlife.