Found Part of the Issue

[Scott]

Well part of the issue is we had a site direct linking into our downloads. Bjorn is working on making that more secure. I was overloading out Apache connections. We are also working on tweaking Apache so it performs better.

Tell me how the server is acting for you now.


Bjorn3D Support.

[PP Mguire]

Its running really smooth and fast for me right now.

[srpeters18]

So far it has been a lot better the last few hours. I know earlier today, from about 9 am to noon EDT, it was very slow, sometimes taking as long as 30 seconds to load a page.

[AJ.]

It's alright. It's slightly faster than before but I do randomly get hang-ups.

[bobletman]

I havent noticed any problems so far.

[swmeek]

Definitely faster than what it was yesterday.

[zachig]

I'm glad you were able to find and solve part of the problem.

Forum pages seem to load much faster now! :grin:

Thanks!!! :icon_tiphat:

[SwedBear]

Unfortunately this is not something that will go away easily.

Simply put, we are under attack by people who want to disrupt the site. It's not necessarily directed just at us but it that doesn't make it better.

As Scott said we, wit The Planet's help, found out that the problem was that tons of conenctions were made to download one of our hosted files. This filled up the Apache connections and since the downloads were large they were not released (which Apache connections usually are). Any second a regular day we have 20-40 simuntanious connections. As people move around conenctions are being released and new connections made. These hackers gobbled up almomst 300 connections. While I could just increase the connections it won't help as they just will gobble up more and more connections and eat more memory.

At the time I thought it was someone linking to our files directly so I moved the files and everything seemed ok until earlier today when I noticed the conenctions being eaten up again. Same file but from the new location. This time it was an IP from Russia (last time it was Chinanet). And the same IP was downloading all the files.

It wasn't until this morning I made a connection with my work (nobelprize.org). We recently released a new in-site Mediaplayer which streams from a Real Helix server and a Windows Media Streaming server. Pretty quick we noticed that the Streaming servers reported that two IP's from Chinanet were starting the same video over and over again trying to overload the server. And this is excactly what is happening here.

Some maliscious hackers are using the file downloads to try to overload servers so they get slow or stop responding. A poor mans DDOS-attack.

Right now my only defences is to keep a eye on the server and put the IP's that start the huge downloads in our firewall.

/B

[PP Mguire]

A poor mans DDOS-attack.I was gonna try and sound cool and smart by saying this but you took it from me :(

[zachig]

It is sad that poor evil people are spending their time and hacking knowledge on such poor lame hacks. :evil:

I wish there was a way to revenge back on them and put some trojans and viruses in the files they're downloading from the site...

[ghidora]

Bjorn, I thought that there was a way to config Apache to reject connections to resources based on the "referrer" URL? I'm pretty sure there is a way to add that kind of restriction in the .htaccess file in the directory with the file.

Something like the following:

deny from all
allow from .bjorn3d.com

[Dragon_CPU]

Still taking a bit to load pages for me kinda annoying but it did that the other day to for awhile and then all of a sudden it was speedy. As of now I liked it the way u had it before the change this morning, but time will tell.

[bigfoot]

Seem's a lot better the last two day's for me hope it stay's that way

[Santino11447]

It made a big difference for me. Faster than ever. Even moving from topic to topic I seem to stay logged on. Hope it stays that way.

[SwedBear]

ghidora - yeah, that is one thing I am doing to see if it helps. It's never been a problem before so that's why we haven't worked on things like that. I know I plaaed around with something like that in the beginning when setting up files but I didn't get it to work and then forgot about it.

So good you reminded me :). I also want to set up the file downloads inside VBulleting with an add-on instead.

/B

[Sadasius]

SweadBear......Those are not hackers believe it or not. It's bots and lots of them. The internet is loaded with them and the Google bots are particularily vicious as they will try to go through every port you have to try and get whatever information they can get. I have a list of IP's from google on my DDoS attacks list like you would not believe. They try to get in the FTP, they try to download whatever they can, they try and post (not google but others)in forums to place ads, they try to inject admins into the database and the list goes on and on. Most of my problems went away when I installed a security add-on that just bans the IP's of the bots that are particularily vicious or malignant. It's very rare for it to be people now doing it themselves. They will just send a bot through the net to do the dirty work and to see what networks are open they can hijack when needed.

[SwedBear]

Well, I include bots in the wide term of hackers as I do not expect one person sitting and manually do these things :).

What add-on did you install to automatically ban the IP's? I have the APF firewall plus BFD (Brute Force Detection) which automatically bans IP's that try to acess SSH etc. but I haven't found a way to limit how many times a IP's can access a file before getting banned?

Using a .htaccess file and only allow bjorn3d.com didn't work well. The problem is the filedownload-app does sends you to the fil and thus the downloader's domain is used to access the file so limiting it only to bjorn3d.com does not work.

I'm currently testing a few anti-leeching scripts though that might work.

/B

[Sadasius]

Mine is all scripted in a PHP add-on and it works really well. It's a modified phpBB security script. So far it has blocked 504 attempts since last year. Almost 95% of them are those damn 'Google bots'. I have no idea why they program those bots to snoop into everything they way they do. I monitor them from time to time and they will try to power through password protected areas using brute force. They are particularly vicious. At any given time I have about 12 of them snooping my website. But despite as good as this scrip is they have still gotten through a few times but could not really do anything except name themselves a folder which they have done once about 600 times. All numerical and in no particular order.

To cut down on the brute force attacks I have included a script to allow only three tries to login and then it locks the account. If it tries on another account it bans the IP address.

http://img301.imageshack.us/my.php?image=securitysq6.jpg

[srpeters18]

I'm assuming this is why all downloads are disabled now? I've never looked at our downloads section, but just thought I'd click on it for kicks, it tells me I'm not authorized to view it, I'm guessing you've shut it down bjorn?

[zachig]

I'm assuming this is why all downloads are disabled now? I've never looked at our downloads section, but just thought I'd click on it for kicks, it tells me I'm not authorized to view it, I'm guessing you've shut it down bjorn?
I haven't checked the "download" section recently, but I guess that Bjorn shut it down till he'll fix the problem and/or find a solution.

Let's hope he'll find a solution soon...

[srpeters18]

Okay, it got really slow again last night, did anybody else notice this as well? It was slow enough that FF would time out. I tried pinging the server, they all timed out.

[Scott]

Can you tell us what time that was and what time zone you are in. It would help when looking into the logs.

[srpeters18]

Eastern time zone, I'm gonna say around 10 pm or so. Don't remember exactly, I know it was kinda late though.

[Scott]

well if you could not ping it was probably backbone or router to it. The issue we have been having is bots direct linking into our downloads and overloading Apache. That would not affect a ping.

[Kougar]

Can you tell us what time that was and what time zone you are in. It would help when looking into the logs.

Scott, the site is taking several minutes right now just to load forum pages. Been at a crawl for the past ~8 minutes and counting. 4:03pm CST here. Hope that helps.

Edit: Is now 4:20, site is speedy again.

[Miles]

I experienced the same timed slowdown and the same timed return to normal as Kougar except I'm on EDST so it was an hour later here.

[Scott]

Its not the site. Router issues. The site is fine and we have been logging it.

[jellyrole]

I know someone that can help you guys, he likes new hardware too, so you might be able to just hook him up with maybe a gfx card or something and he'll secure the server for ya!

[Goliath182]

Bjorn is loading like really slow its been over 2 hours and i finally have loaded this thread. All started at 9 pm GMT -5:00.

[joz]

11:10 mn time, bjorn forum is quite fast, no slowdowns, pages load within 5-15 seconds (normal for me.)

[Scott]

I have been watching the logs. The server is fine now and has been for the last 2 days. There are router issues at our host and then just regular internet traffic. But the server it self is running awesome.

The old server would run at 2.50 average CPU usage which is in the RED and 70-80% RAM usage and we would use over 5TB a month of bandwidth.


New server runs .16 average CPU usage, 17-20% RAM usage and only 2.5TB a month of bandwidth. (YES, we found alot of people direct linking to our downloads and we think we have that fixed now).

[jellyrole]

Glald to here it! Hopefully it says like that.

[zachig]

I have been watching the logs. The server is fine now and has been for the last 2 days. There are router issues at our host and then just regular internet traffic. But the server it self is running awesome.

The old server would run at 2.50 average CPU usage which is in the RED and 70-80% RAM usage and we would use over 5TB a month of bandwidth.


New server runs .16 average CPU usage, 17-20% RAM usage and only 2.5TB a month of bandwidth. (YES, we found alot of people direct linking to our downloads and we think we have that fixed now).
MUCH BETTER!!! ;-) Glad you managed to sort that out! :grin:

[Goliath182]

I dont know what it was but i could download files at 1 meg/sec at the time so its not my connection.

[bigfoot]

Yes it seems faster but it still keeps loging me out when i am writing a post got any idear why???

[Kougar]

Yes it seems faster but it still keeps loging me out when i am writing a post got any idear why???

If you are using the stay logged in checkbox and this still happens, then likely something is deleting your user cookie. Unless your browser is configured to reject cookies, then anti-spyware, anti-adware, or Anti-virus programs are the most likely things.

[Xero (1)ne]

I love the new server. :smile:

This is a bit :offtopic: but why do servers use so little CPU?

[SonicWRX]

i am experiencing major slowness and it started around 11:49 (when i noticed it) and is still on gong at 12:07 EST

[Scott]

Yeah we are hosting the new 3DMark file and the server can not handle the NIC bandwidth has been eaten up. So I have asked them to take us off the list for a day or two until it calms down some.

I hope the server will be back to normal in a hour or 2.

[ZhengHe]

Yeah we are hosting the new 3DMark file and the server can not handle the NIC bandwidth has been eaten up. So I have asked them to take us off the list for a day or two until it calms down some.

I hope the server will be back to normal in a hour or 2.

Ah, that's the reason it wouldn't load before. Well it is slowly coming back to normal. I would say at this point it is at 80% and should be completely back to normal in an hour or so. Just my feeling, since some of those who were downloading from us probably still don't realize that we were taken off the list and/or are midway through the download on a slow connection.