I know exactly what you mean :) I often say that colours are distraction too and generally the black and white shots have more to say.
I just dont have the heart to remove them from some photos...