The "out of range" error is referring to either resolution not supported by your monitor or hz not supported by your monitor. The best way to troubleshoot this would be to start the game in lower resolution. I would try to right click the game in Steam and go to "properties" then click on "set launch options" and use this as a model -width*number*-height*number*
So for example I would personally try -width1360-height768 first as that is a pretty good resolution for any graphics card to be able to handle. The game will not look the greatest but once you get the game loaded you can then use in game settings to adjust it. If that does work remember to remove the custom launch options or set them as the resolution you found to work.
As far as DVI not found that usually points to a bad cable, bad graphics card port or bad monitor port. If you have other options on your graphics card try using one of those to eliminate it being the card itself.
Good luck
**Something else you may want to consider is the power supply itself. I realize you said it is new and higher watts but that does not mean it is able to handle your card. Watts do matter but you also need enough amps on your 12v rail. Your 760 requires 30amps on 12v rail. Check your PSU specs to make sure this meets the needed amps.**