recording with a cpu intensive encoder (ie. high compression x264) raises frametimes (Note: FRAME TIMES) by 5~10ms. I'm normally getting 8~9ms, so another 5~10ms more and it's stutter hell. I'm assumning because clock cycles spent doing D3D stuff is shared with encoding i/o.
You should OC your CPU. Play Dx8.1 because it has very slightly lower frametimes than DX9, evident with results in the tf2 benchmark thread.
Try recording with motion-jpeg, frametimes will hardly increase and game is perfectly smooth.
*frametimes were measured using msi-afterburner.