As someone that has gone back and forth between playing in prem and div2, I feel like ultimately for me and what might be the same for others is that prem level TF2 is just not fun to play. It's not really about being rolled or whatever but it's more that it's such a fucking drag to play the season where the map pool, play style etc. is just so tiring.
I guess at the end of it all ETF2L is now at a point where if you don't care about prestige or the challenge anymore you'll really just want to play with a group of people that you enjoy spending time with. When you make the jump up to prem you'll find yourself playing against teams that have a massive advantage of teamwork/skill which makes the prospect of playing out the season in officials and scrims a lot less enjoyable.