We can draw the line where it lends understanding. Telling someone to "Go back to Pakistan" adds nothing to a conversation but rudeness and hurt feelings. Reminding them that things work differently in Australia than they do in Pakistan would have said basically the same thing, without making people combative.

It's long been established that you're free to say what you want, but you're not free from the consequences of what you say. Personally, I believe officials should be held accountable for their speech more often.