Does it happen in other games? If not, it's probably valve breaking things

It happens even when I'm just sitting on my desktop.

Now that I think of it, I'm also running Whatpulse, which is basically a benevolent keylogger that gives you statistics. I.E

http://puu.sh/2HHFB.png

Might be that.

lol

Went through and removed old mouse and keyboard drivers, turning off other programs one at a time to try and find what the problem is.

I might have had this problem before, does the problem intensify when you are using a program with internet?

No, I'm reasonable certain it has nothing to do with internet, I have pretty good speeds. Though, I can try playing with several browsers opening while playing tf2 to see if it happens more often, having 6 installed can sometimes be helpful.

EDIT: Tried opening Firefox, IE, Chrome, Comodo, and Opera at the same time, after they were all booted up I noticed a significant increase in the rapidity of dropping frames. I'll be going along steadily at 300 and then suddenly drop to ~60 for a few seconds and then stabilize back to 300 over time.

Try doing stuff with the internet cable unplugged, and see if it happens. I honestly cannot think of why this problem would occur now, without anything changed.

Are you connected to your wireless and plugged in at the same time? That happened to me.

No, just ethernet, though, the router I'm connected to is complete shit. It still seems to be more of a hardware/software problem to me, just because of how it slows down everything, regardless of being connected to the internet.

I don't know if it's the same problem, but sometimes (kind of rarely) in TF2 my frames will drop to a steady 60 fps from like 150 for around 2 seconds. I think it's some sort of built in GPU vsync that pops in randomly, but I do have a 144hz monitor. I haven't really noticed it in other programs though.

I did some more research, and found this:

What is csrss.exe?

A Microsoft Windows file stored in the c:\windows\system32 or c:\winnt\system32 directory that has the file description: "Client Server Runtime process." This file 

Is this file a spyware, trojan, or virus?

The csrss.exe file included with Microsoft Windows is not spyware, a trojan, or a virus. However, like any file on your computer it can become corrupted by a virus, worm, or trojan. antivirus programs can detect and clean this file if it has become infected. Because this file is part of Microsoft Windows users should never delete or remove this file if they think it is infected, let the antivirus program handle it.

Is it safe to remove csrss.exe from the Task Manager processes?

No. The csrss.exe is a critical system process that cannot be removed from the Task Manager without causing issues with Windows. When attempting to End Process the csrss.exe you will receive the Unable to Terminate Process window with the error "This is a critical system process. Task Manager cannot end this process." It is normal to receive this error.

[u]The csrss.exe file is using 99%, 100%, or other high abnormal percentage of CPU.

This issue is caused when your Microsoft Windows profile is corrupt. To resolve this issue requires that you delete and recreate the profile. To do this follow the below steps.

Backup all the files in My Documents as they will be lost. It's also recommended you backup any other important files you may be concerned about loosing. 
Log out of the account that is causing the problem and into a different account. If you do not have another account you can create a new account through the User Accounts icon in the Control Panel. 
Once in the other account right-click My Computer icon and click Properties. 
In the Properties window click the Advanced tab. 
In Advanced click the Settings button under User Profiles. 
Finally, in the User Profiles window highlight the name of the profile that is encountering this issue and click the Delete button. 
Once the profile has been deleted you can recreate it if you wish to use the same profile name.[/u]
Source(s):
I work for Mcafee.

I tried playing with the task manager in the corner of my screen and noticed csrss taking up a lot of CPU time when it slowed down, so I'll try this.