Certain leisure pursuits might influence a man's ability to retire soon—or even at all—depending on the activity chosen.