The fact of the matter is that he was living in the US illegally and to me that is basically the end of the story, is it not?

No, it’s not. This is all about due process. For better or worse, there was a “no removal” judge’s order. If ICE wanted him removed, they should have followed procedure. Gone before a judge and presented evidence.

Bring him back and do it right. If he then gets deported, fine. But he wasn’t even actually deported; he was renditioned in an extrajudicial manner.