So I usually get around 300 fps when I'm just walking around and like 150 during mids, but for the first ~minute after I first join a server for the first time after launching the game I get dips to about half of those every ~5 seconds. If I just leave my game running then I will never get them after that first game. Also it appears that no matter how long I leave the game open before joining that first server I will still get the fps drops.
I know it's not really a big deal normally because usually you don't ready up for at least a minute anyway but if my game crashes or something and I have to immediately join a server it's pretty annoying to play with constant drops to 70 fps.
Specs are i5 4670k @ 4.0 GHz + 780 Ti. Using Comanglia's config with shadows enabled. dxlevel 81