This is my issue:

Sometimes after I've joined a server if I try to join another one it gives me the "Connection Failed after 4 retries" error. I have to wait around 3-5 minutes until I can join the server.

This happens very randomly and the kind of server i was/will be in doesn't seem to make a difference (as in from a community server to casual or mm and vice-virsa) but when it happens there are certain servers that let me connect and some that dont somehow as in I can fail to connect to one server and be able to join other servers.This has cost me some pugs and mm games. Doesn't happen to me in other games like csgo. I don't get timed out or anything like that it only happens when switching servers. Running a powerline connection.

Tried all the things I've been able to find on google such as:

Entering +clientport 27006-27014 into my launch options
Messing with my firewall/antivirus, like adding exceptions for tf2
Removing all my connection settings and launch options
entering: ipconfig /release | ipconfig /renew | in the windows cmd

Hope someone can help me out, thanks in advance.