Apostle Paul says concerning those who pass judgement on him, it is a small thing that you pass judgement on me. The Lord is my judge. We have no right to judge the lords servant. I am not only talking about the apostle or ministers but also fellow Christians we are all servant of the most high. I will admit it, at times it is very pleasing to judge our fellow brothers and sisters but instead of condemning, it is better to correct him/she in love.
"I care very little if I am judged by you or by any human court; indeed, I do not even judge myself. My conscience is clear, but that does not make me innocent. It is the Lord who judges me. Therefore judge nothing before the appointed time; wait until the Lord comes. He will bring to light what is hidden in darkness and will expose the motives of the heart. At that time each will receive their praise from God." - 1 Corinthians 4:3-5 7 NIV
"Who are you to judge someone else’s servant? To their own master, servants stand or fall. And they will stand, for the Lord is able to make them stand." - Romans 14:4
It is more deadly to talk bad about God's ministers, he is to be rebuke by the Lord who hired him. Therefore speaking against the servant of God amounts to speaking against God that appointed him. An example is the story of Moses.
"Miriam and Aaron began to talk against Moses because of his Cushite wife, for he had married a Cushite. “Has the Lord spoken only through Moses?” they asked. “Hasn’t he also spoken through us?” And the Lord heard this (Now Moses was a very humble man, more humble than anyone else on the face of the earth.) At once the Lord said to Moses, Aaron and Miriam, “Come out to the tent of meeting, all three of you.” So the three of them went out. Then the Lord came down in a pillar of cloud; he stood at the entrance to the tent and summoned Aaron and Miriam. When the two of them stepped forward, 6 he said, “Listen to my words: “When there is a prophet among you, I, the Lord, reveal myself to them in visions, I speak to them in dreams. But this is not true of my servant Moses; he is faithful in all my house. With him I speak face to face, clearly and not in riddles; he sees the form of the Lord. Why then were you not afraid to speak against my servant Moses?” The anger of the Lord burned against them, and he left them." - Numbers 12:1-9 NIV
Even though the issue against Moses looked right, God said why are you not afraid to speak against my servant with whom I speak face to face. It's not as if the men of God don't make mistakes, but their judgement belongs to God. God afflicted Miriam for speaking evil against moses. Whoever touches God's servant invites the wrath of God.
“Do not touch my anointed ones;
do my prophets no harm.” -1 Chronicle 16:22 NIV