[Attack ended] We are getting DDoS'ed / Site is currently slowly loading

[BrickBob Studpants]

I haven't been able to access the site at all since Friday evening and only managed to log in just now from my workplace laptop. My IP address likely got banned because I kept trying to reload the page

[JLiu15]

Sorry if this has been answered earlier, but for the past few days Eurobricks would be down on and off for me; sometimes I can access the site, other times it loads really slowly then fails to open (or fails to open immediately on my iPhone). Sometimes I'd have Eurobricks open and then when I try to navigate to a different page this happens even if I only had Eurobricks open for just a few minutes. Is this what you guys have been experiencing as well? 

[Sven J]

Still massive problems here.

First access to EB - I usually take a direct link to the Train Forum - works fine.
Afterwards browsing to the frontpage or to any single thread is already much slower.
When I try to browse on to a third sub-page, EB freezes and only resumes working several hours later.

All in all, very annoying as I used to visit EB a couple of times a day, but at the moment am lucky when I succeed at least once.

[Sven J]

Strange...
One hour after I posted my complaint above, suddenly all was fine again.
Let's hope it stays that way.

Thanks for your commitment @aFrInaTi0n!

[Mr Hobbles]

I've been having issues all weekend, and also today. Basically every device in my house hasn't been able to access Eurobricks for the past few days.

The only way I can connect now is through a VPN, and occasionally that stops working too. It's quite frustrating. :(

Hopefully the DDoS related issues are solved soon! Best of luck to the team.

Quote

Don't know if this is related to the issue but since yesterday I constantly loose the connection to the forum for few minutes up to around half an hour. Then it works again for a short time before I loose the connection again. Often my posts don't go through and I have to try posting later again.

This was my experience for most of the weekend, but as of today it's just stopped working entirely, unless I use a VPN.

Edited Monday at 09:55 PM by Mr Hobbles

[aFrInaTi0n]

@Mr Hobbles I tuned my mitigations down after reading your post. Situation is over since Sunday - so I reckon my configurations are giving you so much issues.

Regularly it should not really hit up regular users (not overdoing it with refreshing the site) - but seems that may occationally not be true as your story tells.

[BrickBob Studpants]

Same issue here, I still can’t access EB properly, neither from home or on my smartphone. The site is not even trying to load

[aFrInaTi0n]

@Sven J I think the attack ended on latest Sunday - since then I can not see the high utilization any longer.

The thing which is still in place: My configuration for the rate-limits on the webserver level. It may be those may now be set too aggressive - I just adjusted those values to allow more traffic before those rules are getting active. Sounds to me like my adjustments were for some (unknown) reasons hitting some singular users harder than others.

[Repkovsky]

53 minutes ago, aFrInaTi0n said:

@Mr Hobbles I tuned my mitigations down after reading your post. Situation is over since Sunday - so I reckon my configurations are giving you so much issues.

Regularly it should not really hit up regular users (not overdoing it with refreshing the site) - but seems that may occationally not be true as your story tells.

Hi, I think mu home IP was banned too, forum works well when I access through the mobile internet, and does not respond when I stitch to home network. I tried to enter Eurobricks on Saturday, and it wasn't some crazy refreshing, but apparently it was already too much.

[aFrInaTi0n]

I adjusted the possible "too aggressive" settings some seconds ago to be less aggressive - please test over the day if that helps already. Sorry for the inconveniences I may have creating for you!

@Repkovsky I just removed all currently temp-banned IPs, so yours should be free again as well.

[BrickBob Studpants]

Thank you, it seems to be working again for me!  Fingers crossed whoever’s behind it won’t strike again!

Edited Tuesday at 08:31 AM by BrickBob Studpants

[aFrInaTi0n]

To be honest, I would really like somebody to give me a chance to test my new config - but only from the IT perspective, I can understand users may just want the site working. :)

[Repkovsky]

1 hour ago, aFrInaTi0n said:

I adjusted the possible "too aggressive" settings some seconds ago to be less aggressive - please test over the day if that helps already. Sorry for the inconveniences I may have creating for you!

@Repkovsky I just removed all currently temp-banned IPs, so yours should be free again as well.

Thanks, it works :)

[aFrInaTi0n]

May still be the case that my rules catch any legit user by accident. The chances shall be lesser than before - but this is always the issue with DDoS attacks, by nature they are hard to distinquish from regular user traffic.. Fingers crossed it stays better for all users & if we may get another attack, I hopefully can check if the rules are still helping to keep the site up & running.

I consider this a one-timer from possibly some singular person - if we will get constantly DDoS'ed in the future we will for sure switch to a CDN, as this will scale / work out better from any big providers capacities and their bigger expertice in the field and knowledge of their backend systems for already identified / active botnets, etc pp.

[Mr Hobbles]

Thanks @aFrInaTi0n, my house has started working again! :D

[aFrInaTi0n]

Appreciated - though I always find it hard for myself to accept somebody thanking me for things which started because of my actions.. but this is only my internal struggle please ;)

[AJB2K3]

Hello.
Don't use IP blocking to stop DDOS.
IP Blocking only works on IPV4 and when IP's revolve address innocent people get blocked and guess what, I was one of those who's IPV4 address was blocked.

Only just been able to access the site today for the first time since Friday.

Edited Wednesday at 04:48 AM by AJB2K3

[aFrInaTi0n]

@AJB2K3 I am aware of the possibilities of bans affecting regular users as collateral damage as well. Also I am sad that my efforts hit your IP as well - please take my excuse for that.

Currently I have fine-tuned the rulesets a bit further, for lowering the chances of regular users being affected. As there is no golden rule of thumb available in reality, we need to check if that may hold for such little DDoS attacks, or if this may get an ongoing scenario, a CDN provider may be the better choice, I strongly agree to that. But again this further setup and additional operation costs in the long run. So I can only ask for understanding of us thriving for going with the least-costly approach 1st to be tested.

If one can't reach the whole site for longer than 10 minutes, dont hesitate to write up a quick email handing your account name to unban@eurobricks.com - then I can quickly check for your IP and unban it manually.

[aFrInaTi0n]

I just checked the latest IP being reported from by temp-ban solution - all examples I picked out the logfiles are located in china and have similar networks (first three octets being the same) - I could not find any IP which corrosponds to any internally IP logged of a user visit.

I will check this within the next weeks to observe how well my solution is doing.

[aFrInaTi0n]

Little update from today:

• No followup DDoS since the first occurance
• I had adjusted my mitigation measurements a bit, as they were set a little bit too aggressive and caught some user IPs to get banned.
  • All banned IPs got unbanned after the adjustments.
  • My manual checks for currently caught & banned IPs did not give me any IPs related to user accounts - so I hope that this may not accidentally happen for any regulary browsing user

Again a big excuse from me for the people it affected - I think it has luckily only been a handful. And again, if something is not working out for you since the last weeks, please don't hesitate to report it to us / me here or in a private message.