I'm sure most server admins have now heard that 2 days ago, an exploit was released that affected anybody using the widely used server control panel, SolusVM. If you have not heard, basically, SolusVM is a control panel used on many servers throughout the world, and two days ago somebody (currently presumed to be Robert Clarke though I am not accusing him nor was it his fault since he only discovered the exploit and many others used it for destruction) found an exploit with a unused .php file in the control panel that allowed access to all sub-servers/VPS hosted on the larger server.
Many servers were affected in general, though some bigger networks were targetted. Luckily, Mawrr servers were not targetted. However, exploiters still targetted our entire infrastructure and host, therefore causing our systems to go down.
So what happened to us?
Well, we actually run 2/3 of our servers with host #1, and run the other 1/3 with host #2. We recently moved our sold servers from host #2 to host #1 due to long term reliability and server optimization. However, host #1 was wiped by the exploit, while host #2 is (currently) unaffected. I had not said anything since the exploit happened after UGC Highlander matches, but was hoping they would fix the servers before 6s matches.
I spend extra time and money every week for weekly automated backups, and I manually backup every server after official match times to save SourceTVs. Host #1 has all of our backups securely saved on a separate machine. However they are currently working with entirely new VPSes to speed up the process and since backups are very large files that forever to unzip and move over to another machine in large amounts.
We have asked our host to restore machine TF1 and machine TF2, but we are not sure if they will be able to by tonight's official 6s match time.
So what is the plan?
We are asking any of our teams that were going to use the servers for a 6s match tonight to ask the other team if they can use the server.
If not, please contact us and we will immediately install the 6s configs and maps onto one of our servers at host #2 for you.
If you have a scrim and need a server for sure, we can also do the same on our personal servers, however please try to keep the scrims away from 6s match time.
Note: All of our servers are still ONLINE! Our Mumble included. However, we can not access any of the root console through FTP or SSH, so it is hard for us to change anything. Also, the servers could get wiped clean at any time. So feel free to use your server if needed, but I would not recommend hosting a match on it.
Compensation!?!? Though the servers are expected to only have a downtime of under 24 hours, AND in fact none of the servers are actually down, it's just that I can not access them for maintenance. We would like to strongly emphasize that we are not taking any blame for this, though our provider is also pushing blame onto the exploiters.
I do understand that anybody who paid for my servers has absolutely no wrong or blame in this situation either so I will be giving out the follow for compensation:
-Anybody who has a paid server, whether it was down or not: 2 day extension
-Anybody's who's server actually went down and can not be brought back up because I have no access to the root console: [However many days it takes for me to bring your server online] + 1 day extension
-Anybody who's match was affected, either having to use the other team's server or my backup server: Additional full day usage for whenever, even after your payment runs out.
-Teams that are using my Mumble for FREE: Everything is fine right now, though if the server does crash sometime today I will set up a smaller one for free and anybody who needs it for a scrim or match can temporarily.
-Any other specific issues? Contact me and I will be hopefully more than reasonable! :)
P.S. Before we were even attacked, I was going to make an announcement about our new servers and pre-orders and beta orders. Stay tuned for that!
tl;dr[/b] Mawrr servers were attacked though nothing went down, but host is telling us they may do a clean wipe of all servers, so I have backup servers for anybody that is using my services and needs one for tonight. In addition everybody will be getting extensions on their billing cycle even if you had no issue! :D Everything should be fine within 24 hours!
I'm sure most server admins have now heard that 2 days ago, an exploit was released that affected anybody using the widely used server control panel, SolusVM. If you have not heard, basically, SolusVM is a control panel used on many servers throughout the world, and two days ago somebody (currently presumed to be Robert Clarke though I am not accusing him nor was it his fault since he only discovered the exploit and many others used it for destruction) found an exploit with a unused .php file in the control panel that allowed access to all sub-servers/VPS hosted on the larger server.
Many servers were affected in general, though some bigger networks were targetted. Luckily, Mawrr servers were not targetted. However, exploiters still targetted our entire infrastructure and host, therefore causing our systems to go down.
[b]So what happened to us?[/b]
Well, we actually run 2/3 of our servers with host #1, and run the other 1/3 with host #2. We recently moved our sold servers from host #2 to host #1 due to long term reliability and server optimization. However, host #1 was wiped by the exploit, while host #2 is (currently) unaffected. I had not said anything since the exploit happened after UGC Highlander matches, but was hoping they would fix the servers before 6s matches.
I spend extra time and money every week for weekly automated backups, and I manually backup every server after official match times to save SourceTVs. Host #1 has all of our backups securely saved on a separate machine. However they are currently working with entirely new VPSes to speed up the process and since backups are very large files that forever to unzip and move over to another machine in large amounts.
We have asked our host to restore machine TF1 and machine TF2, but we are not sure if they will be able to by tonight's official 6s match time.
[b]So what is the plan?[/b]
We are asking any of our teams that were going to use the servers for a 6s match tonight to ask the other team if they can use the server.
If not, please contact us and we will immediately install the 6s configs and maps onto one of our servers at host #2 for you.
If you have a scrim and need a server for sure, we can also do the same on our personal servers, however please try to keep the scrims away from 6s match time.
[u][/u]Note:[b][/b] All of our servers are still ONLINE! Our Mumble included. However, we can not access any of the root console through FTP or SSH, so it is hard for us to change anything. Also, the servers could get wiped clean at any time. So feel free to use your server if needed, but I would not recommend hosting a match on it.
[b]Compensation!?!?[/b] Though the servers are expected to only have a downtime of under 24 hours, AND in fact none of the servers are actually down, it's just that I can not access them for maintenance. We would like to strongly emphasize that we are not taking any blame for this, though our provider is also pushing blame onto the exploiters.
I do understand that anybody who paid for my servers has absolutely no wrong or blame in this situation either so I will be giving out the follow for compensation:
-Anybody who has a paid server, whether it was down or not: [u]2 day extension[/u]
-Anybody's who's server actually went down and can not be brought back up because I have no access to the root console: [[u]However many days it takes for me to bring your server online] + 1 day extension[/u]
-Anybody who's match was affected, either having to use the other team's server or my backup server: [u]Additional full day usage for whenever[/u], even after your payment runs out.
-Teams that are using my Mumble for FREE: [u]Everything is fine right now[/u], though if the server does crash sometime today [u]I will set up a smaller one for free[/u] and anybody who needs it for a scrim or match can temporarily.
-Any other specific issues? Contact me and I will be hopefully more than reasonable! :)
[b]P.S.[/b] Before we were even attacked, I was going to make an announcement about our new servers and pre-orders and beta orders. Stay tuned for that!
[u][b]tl;dr[/u][/b] Mawrr servers were attacked though nothing went down, but host is telling us they may do a clean wipe of all servers, so I have backup servers for anybody that is using my services and needs one for tonight. In addition everybody will be getting extensions on their billing cycle even if you had no issue! :D Everything should be fine within 24 hours!
Machine TF2 was restored! Machine TF1 is still unaccessible but working.
The control panels are still down as of now so I can not reboot anything or change any DNS settings. Should not be a big deal though I can not update the servers hosted on the TF1 machine.
Machine TF2 was restored! Machine TF1 is still unaccessible but working.
The control panels are still down as of now so I can not reboot anything or change any DNS settings. Should not be a big deal though I can not update the servers hosted on the TF1 machine.