It is very easy to avoid the conflict of interest issue by simply stepping aside in that particular decision when a direct conflict arises. Yes people have built things or are involved with things or they wouldn't be known but no one person has built everything. Direct conflicts like this should be relatively rare.

I'm expressing no view on ranking people or anything of the sort. My view is simply that the committee ought to be a bit larger or have alternates so one member or another occasionally stepping aside on a particular decision. as they should, is not disruptive to the overall functioning of the committee (and similarly for occasional absence for other reasons).