I've done it in the past but these days I leave them followed. I figure that way if they come back I'll already be following and they aren't filling my feed so I just leave them on the list these days.