Did some more tests. With the GPU in and the 2x PCI power cables connected to the card, the PC won't do anything (no fan spins), totally dead.

When I take the GPU out and replace it for a very basic PCI-E card I still had laying around the computer works fine.