That slogan was never patriotic.
Apparently it was first used to defend the hunt for communist Americans, black listing them and such. Of course, it didn't even work, and many of those that were caught in their sights weren't even communists, and were just forced to name other "communists". It was a horrible part in history.
It's also been used against anti-war protestors at various times.
It's a nationalist slogan that in effect means that if you don't support what the country is doing, you should leave.
We don't live in a nationalist dictatorship. People should be allowed to speak out against things that are done by the government. Disliking these policy choices has nothing to do with loving or hating the country. Tons of people hated the policies of the Obama administration, and things passed during that time, just as they hated things during Bush, etc. That doesn't mean they should have to leave. Speaking out about your beliefs is fundamentally American.
RE: If it's not racist, it's at least un-American!