Can I use a common ground for a continuous power circuit with switching charging batteries?

Question originally asked on Musing.io - Click here to see answers

I am trying to build a circuit that allows for the switching between two batteries. Ideally it would work so that when one battery is charging the other is in use.

To keep the number of components down, I'm asking if it's possible to use the same ground for the whole circuit.

Here is a simplified diagram of what I propose: http://imgur.com/WZ0WoY1