Assuming the distance from the sun at perihelion remains fairly constant

That assumption is wrong. Due to conversation of energy the perihelion needs to be closer. And because the energy is proportional to r² the actual difference in energy is a lot smaller then 20%.

In other words the further the aphelion is the amount of radiation received varies exponentially

What? It is squared, as you said!

This still is around a 20 percent difference. I know that this is apples and oranges but I am at best dealing with guesstimates.

Just did some calculation(T⁴ is proportional to P) and found that 20% difference in energy would lead to a 13°C temperature difference which is more then we observe(so it seems you are not that good.).

So you actually may be right that the cycle can make a big difference. But since the orbit of earthcould still be approximated as a circle, that temperature change should be visible as some sort of sine wave, but it isn't. The graph shows a huge bump every 100000 years and not some harmonic wave.

Cycles do not last long enough for adaptations to occur.

They are to short for a species to adapt to the current part of the cycle. That's why a species needs to adapt to all parts of the cycles to survive until today.

The major die off we have right now is caused by humans:
Many species die because of plastic, acidified oceans(by CO₂), (artificially) destroyed rain forests, …

The Maunder and Dalton minima fit to the temperature graph, that's true, but if you look at the right of the sunspot graph you see another minimum there, right? Why isn't it also visible in the temperature graph?
I might be wrong about the little ice age, but I am not about the current warming.

To show that CO₂ actually has a big effect on the temperature here a few calculations(you can skip to the result if you want. I only left all my steps here as evidence to how I yielded my results.):
If you look at any spectrum of a sun:

you see that there is a roughly 30% absorption caused by CO₂ which relates to a transmittance of 0.7(70% of the light comes through at that wavelength).
The absorbance A is defined as A = -log₁₀(T).
In case of our atmosphere and the peak at λ = 2050 nm, A = 0.155
The Beer–Lambert law is:
A = εcl
Here we can assume that εl gets some other constant c in the case of our atmosphere:
A = c*l
c(the concentration) is proportional to the number of particles per million(n) therefor with a new constant C:
A = C*n
Currently there are roughly 400 ppm CO₂ in the atmosphere. Therefor:
C = 0.155/(400 ppm) = 3.9*10⁻⁴ /(ppm)
The most important peak of CO₂ is at λ = 15µm and spreads from 13-18µm. It is about 3 times higher as the one at 2050nm as you can see here:

So its constant should be 3 times higher:
C₁₅ = 3*C = 3*3.9*10⁻⁴/ppm = 1.17*10⁻³/ppm
Now consider the two concentrations 300 and 400 ppm:
A₃₀₀ = 300*1.17*10⁻³ = 0.35
A₄₀₀ = 400*1.17*10⁻³ = 0.47
The transmittivity is:
A = -log(T) → T = 10^(-A)
↓
T₃₀₀ = 0.45
T₄₀₀ = 0.34
Therefor that increase in CO₂ leads to an additional absorption of 11% in the region of 13µm < λ < 18µm.
The black body radiation curve of earth looks like this(The 283K graph corresponds to 10°C which is the closest graph I found to 15°C):

The region between 13 and 18 µm makes up about 25% of the area under the curve I would guess.
So the total additional absorption is 25%*11% = 2.8% of the total power(energy/time) earth is emmitting.
The temperature(don't confuse the temperature T with the transmission T I used above) to the power of 4 is proportional to the power:
T₃₀₀⁴/P₃₀₀ = T₄₀₀⁴/P₄₀₀
At 400ppm the radiation power of earth can be 2.8% higher because 2.8% of the radiation is absorbed:
P₄₀₀ = (1+2.8%)*P₃₀₀ = 1.028*P₃₀₀
↓
T₃₀₀⁴ = T₄₀₀⁴/1.028 | x⁴ = 1.028 → x = 1.007
↓
T₄₀₀ = 1.007*T₃₀₀
The temperature at 300ppm is something like 14.5°C which is equal to roughly 289 K
↓
T₄₀₀ = 1.007*289 K = 291 K

That's a change of 2°C which is certainly a lot. Yes we don't see 2°C, but only 1°C. That is caused by some of my estimations(I'm not sure about the area under blackbody radiation graph, it could as well be only 12.5% which would yield 1°C) and also by things like for example the reducing sunspot activity you mentioned.
I totally understand that CO₂ isn't the only factor, but it is the major one today, as it can and currently does create a big change in a small time and doesn't reverse as fast as for example a sunspot minimum. It could take the ecosystem millions of years to reduce the CO₂-levels back to normal.