The easiest solution would be to build an elo system.

Let's take an example:

A random user logs into tf2pickup.eu, and this magic service will look at all (configurable?) leagues and see what their average skill level is. This skill level would just be some number, maybe 1 for open and 10 for invite. Calculate the average skill by weighting the most played division by that factor. We would end up with this base line rating to what a skill level would be, and it's factored based on the overall scores of both time played and highest division played in.

You could then go into more detail on games played, where their internal score (essentially elo now) is based on number of games won (or won in a row). You can then deduct a maximum percentage based on the current score for each loss.

Basically, just write an elo system. That's all.

ELO system eventually makes gamers throw their games in order to have lower ELO and, in consequence, easier games.

ban people who throw?

While yes, people do throw games to get lower elo and thus easier games, if done right, the "formula" for elo should bounce it back.

I'm unfamiliar with how pugchamp did their elo system nor do I know much about if this is a real impact for pugchamp, but that's where I'd start: learning what pugchamp did and hear feedback from pugchampers about their games.

if it doesn't take into account what level they play at it would be incredibly unbalanced, with players who stomp tf2centers rated way above players who only scrim in higher divs and can't put out the same stats.

For the 6s version you could bascially just take the elo from every national site, balance it out so the ratings are adjusted to the different sites settings and assign the players the adjusted elo.

Obviously a lot of people would be missing (mainly brits i guess) but you could always just have a team of a few knowledgable people adjusting their elo. Also helps that basically every div is represented on the national sites so you kinda know which elo to assign to the players.

We also have a bot that shows us the winrate on each class for the german pugs so the admins could use that bot as a tool to balance it out incase 5-0s happen constantly.

i like that idea a lot, brits aren't a problem, their skills could be assigned manually

AI would only classify skill/balance properly if the dataset you're using (logs.tf in this case) isn't a complete mess on a player per player basis. You'd have to prepare and clean up that data and afaik there's roughly three million logs so good luck with that. There's people with logs for dumb maps like 2fort upsidedown, mess around pugs, ultiduo bball HL 6s Prolander 4v4 you name it. Some of these will be from scrims, offis, TF2C, pugs or other stuff. It would also need to consider individual classes, other individuals on the team you're playing to really make sense I think. Probably do-able with something like Keras/TensorFlow, but sounds like a pain to train a model for that with all the variables you have.

Not only that, but you'd have to feed it data it can actually understand. Just throwing 30GB of raw text logs at AI isn't going to do much so you'd probably end up making several monster CSV/JSON files first (parsed from those 30GB logs).

Using some sort of elo score or assigning a score based on ETF2L/RGL division sounds a lot easier. If you want to go the extra mile you can check amount of matches played on ETF2L, how many won/lost etc. But if your goal is to just make the life of a handful of people easier, I think that'll go a long way.

If you like what you have with trends.tf you could just scrape the data rather than use an API. You should probably ask the creator if they are fine with that though.

If anyone wants to write an ELO system, let's talk.
arcadia#0001

it's a bit more complicated than you think.
based on our experience from national website and our own playground which is tf2pickup.pl and hl.tf2pickup.pl we've had a lot of trouble with it because it's just not that easy to catch besides obvious cases where someone play some junky shit like direct hit and that person direct misses everything, unwilling to change the weapon therefore throwing for their team

for cases like this on 6s we've banned several weapons, on highlander we've banned for example all demoman shields, i know a lot of 6s sites have done it already, for instance market gardener is banned on tf2pickup.pl, that specific ban is due to a current ETF2L whitelist and so far i think people don't complain much about it in Poland

however this was the easy part of kinda obvious cases, the not obvious cases are where people just play carelessly and it's really hard to assess if someone is trolling, throwing or just bad (like me in 6s),

also mały had in mind people who rather to sandbag than get more experience and play better and again, it's not bad that someone is mediocre in game like me and does not feel keen to get better because obviously it's just a game and not everyone's top life priority, but what really bad is that some people even on the ETF2L try their best to get into the lowest division possible just in order to have easy game and i know mały wants these people out of the site

however by these arguments ELO sounds like a not bad idea, probably worth considering