Viewport Performance increadibly low in Solid mode - RTX 3090 / Ryzen 3950X

[Komprimierungsfehler]

So.. I have been struggling with this for months. And no matter how much I google it, nobody is able to give me a solid answer.

My problem is rather simple. When I am previewing my animation in the viewport in viewport, I am getting terrible framerates. BUT THIS ONLY HAPPENS.. as soon as I place a keyframe uppon a model. As long as the scene doesn't have any keyframes on any models, I will get the FPS that the scene is set to. But as soon as there's a keyframe on a rigged model, no matter how low or high-res they are, they framerate will just dip.. HARD.
And to make things even more confusing I have just swapped my GPU and compared the results.

I was getting 47FPS in a scene 60 FPS Scene while playing it back in Solid mode in the viewport with my 1080TI. After uninstalling the 1080TI and replacing it with a RTX 3090, I immediately only got 27 FPS. So not only did the viewport performance not meet the rquired 60FPS, but it even went down by 20 FPS which is quite the huge jump.
During playback my GPU usage is around 15-20%.
My CPU sits at 19-25% during that task.

I have to be missing something.. I really hope that somebody can just show me the magic button that makes blender use my hardware properly.

My specs:
RTX 3090
Ryzen 9 3950x 16Cores (All-core overclocked to 4.3 GHZ)
64GB Ram / 3600MHZ
1200Watt Platinum Power supply

A bit of extra information: Non of the parts are thermal throttling. They are quite chilly during the operation. And both the CPU and the GPU show the desired performance while rendering or displaying the viewport in Rendered-mode. Just not while trying to see what I am actually working with.
I have tested this phenomenon on both my gaming laptop and a few of my friends PC's. All of them seemed to have a really low Viewport performance.

I am aware that I can lower the framerate and use framedropping to still see what I am doing. But this is my job. I invested a lot of money into my system to not have to deal with those issues and it didn't do anything. Last bit of side details: I had a I7 8700K before my Ryzen. When I switched that i7 to the 3950x, I would even end up loosing a few FPS inside of the viewport. Everything else is completely unneffected. It's literally just solid-mode that seems to be my problem. Wire-mode has the exact same FPS. 27.

Thanks in advance! Really hope that I can get rid of this problem..

Edit:
-None of the scenes even come close to maxing out on either V-ram or normal Ram.
-All scenes that I put into my version of blender seem to have that behavior.

[Pte Jack]

TL:DR it all,

Feel your pain, currently have a 5950x and using a bloody RX580 because of the GPU shortages.

What engine are you using? Eevee or Cycles?

If Eevee, did you try reducing the viewport sample rate.

Also, do you have the armature selected while doing the play back, If yes, try deselecting anything related to the model and the animation. Then press Play.

Also, check to see if your are using GPU or CPU computer for the renders. (try GPU computer to take advantage of those CUDA cores you have)

[Komprimierungsfehler]

Thank you for the quick reply.

I am using cycles. And changing it to Eevee doesn't make any difference, as my problem is only there when I am previewing something in Solid or Wire-mode. Preview-rendering in Cycles and Eevee shows the expected speed. Just not in Solid-mode. That is where my System Usage also drastically goes down and it nearly goes into idle-mode. I have to clarify that I am using MSI afterburner to read these stats. So there aren't a lot of inaccuries as there are in the Taskmanager.

Unselecting any armatures doesn't seem to give me any changes to the FPS. it keeps jumping around 45-49.