Edit
PLUGINS ALL UP IN THIS
http://teamfortress.tv/forum/thread/8221-mumble-overlay-when-using-obs-game-capture#post-12
OP:
Those of you that stream with OBS using Game Capture will have noticed that you can't see Mumble overlays. I have sort of found a solution.
The result is here: http://www.twitch.tv/atmo_/c/2271073 (you can see me editing the capture region boundaries at the start)
http://mumble.sourceforge.net/Skinning
Figured out that Mumble skins give you control over the background of the window as well as the font of the interface, along with other things.
So starting with a Mumble skin I found, I changed some things around* to make it more friendly for chroma keying into a stream.
The modified* skin is here -- http://doublejump.eu/atmo2Galaxy.zip (feel free to mirror)
You need to create a folder called "skins" in "Program Files (x86)/Mumble/" and unzip to there. Then select the .qss file in Mumble settings > User Interface > Skin. Clearing this line and restarting Mumble will bring you back to normal.
Add a Window Capture source to OBS and select the relevant sub-region of your Mumble window.
Enable the tickbox "Use Color Key". Click the color swatch and set it to RGB 51,51,51. I'm using Similarity 10 and Blend 0. You can experiment with these if you want, but it's difficult to improve one thing without making something else worse.
The drawback to this method is of course that you're really just using Screen Capture, and so you have to be conscious of the usual Screen Capture foibles.
Mumble has to stay on top. If something goes on top of Mumble then that will be captured instead. Having a second monitor is pretty helpful here. You can move the Mumble window around your screen and the capture will stay locked to the right region, but you can't minimize it or scroll within the window.
People joining and leaving can change the position of the channel within the Mumble window. Only solution to this is to use a private mumble with one channel.
Additionally, the edges of text look kind of funky depending on the chroma key settings mentioned above. It's only noticeable when you look at something bright like the sky, so it doesn't crop up too frequently if you stick on first person view (spec_mode 4).
Hope somebody gets some use out of it anyway, if the explanation even makes sense. I don't think I'll bother with it on my personal stream, but I hope to try it out on the broder stream this Thursday. Look forward to seeing it on Crackclan's stream, too.
-----------------------------------------
* namely:
removed mention of font-family in QMainWindow (this seems to set your user's font when you are in a channel, but not the other members)
changed QTreeView#qtvUsers to have size 24, white, Arial font, on a dark grey (#333333 / 51,51,51) background.
changed talking_on.svg from a white pulse to a green circle
changed talking_off.svg from grey cirle to empty file