Resource usage

[gstark]

Those of you who are hammerring the server waiting for Birddog's specials: DON'T!

You are also hammerring our resource usage, and that causes me real grief. I've just banned about 7 IPs for excessive hits on the server. No doubt the affected users will email me in due course, although it's interesting to note that some of the IPs that I banned last time this happenned still haven't been in touch.

As an observation, I'm home most of the day, and we have two users hitting the site from here. Our total usage yesterday was around 550 hits on the site. Bear in mind that we're not just two userrs, but we're both admins!

Noting that number, I am astounded to see that some users are hitting the site in excess of 2000 times during the same period. Those users need to seriously consider getting a life.

Get out and take some bloody photos!

Be warned that anyone hitting the site more than 1500 times in a day is now likely to be be banned, because you are causing me too many headaches. Please permit others fair usage and enjoyment of this site.

The bottom line is that I may have to - reluctantly - move the services back to NoDaddy, because of the popularity of the site, and the fact that DH won't permit us to use the resources fully.

[Alpha_7]

Any in the 202.92.87.x or 220.233.162.x range ? I have no idea how much my usage is.. but I know I don't have much of a life outside of the forums these days

[the foto fanatic]

1500 hits in a (say) 10 hour day is still 150 hits per hour, or 2.5 hits per minute.
Even allowing for a 15 hour stint at the computer, we are talking 1.66 hpm.

I don't think anyone (perhaps with the exception of Birddog himself) could possibly post so frequently, so in my view, this limit is somewhat generous still.
Or am I missing something?

Edit: If you are "refreshing" just to catch a Birddog special, one refresh every 20 minutes guarantees you at least 10 minutes in his 30 minute window - this would only equate to 45 hits over 15 hours.

[owen]

I don't know how many hits I am doing either, although I do know it is more than my bosses would like

[Mj]

Be warned that anyone hitting the site more than 1500 times in a day is now likely to be be banned

Just to clarify... you really meant to say 15 times didn't you
Cause 1500 times a day requires some serious traetment

[losfp]

I hope I'm not being too much of a drain... I tend to have one or two favourite sites in the background all day while I work, and check for new posts every 10-15 minutes.

But 1500 hits means more than 2 hits every minute for TWELVE HOURS. That's some hitting

I also run a Sydney Swans messageboard in "my other life", and we got some pretty intense resource drain during this year's finals series, for obvious reasons. During the Barry Hall tribunal hearings in the week before the grand final, we exceeded the entire monthly bandwidth (12GB) within TWO DAYS, what with people constantly hammering the server to check the latest news (the infamous BarryGate incident of Sept '05).

I guess Birdy's specials are a bit like that Only 2 more days of speculation! From memory, he said there would only be specials until the 21st.

[stubbsy]

Craig. Like you I visit here frequently but no way would it be 2,000 hits in a day. Let's say you're on 16 hours in a day (unlikely, but stay with me) then that's roughly 1 hit (page visit) EVERY 30 seconds of that 16 hours. If you're on 8 hours throughout a day then it's every 15 seconds etc.

No matter how you look at it 2000 is a hell of a lot of hits in a day.

Edit - looks like I was too slow with my maths since a few of you already said the same thing

[radar]

Gary,

why not post the IP addresses that are hammering the site? The "offenders" can then see that they really need a life. It can be self-regulating. If they continue to hammer the site, then go and ban them and see if they suffer from withdrawl

It would also be interesting to see what phpBB make "one" hit. Is it the actual full page or various sections count as a page hit, eg News, topic headlines, etc?

cheers,

radar

[phillipb]

This is probably a silly suggestion, but is it possible to have log of people and their average clicks? similar to their total post counts.

[gstark]

Any in the 202.92.87.x or 220.233.162.x range ? I have no idea how much my usage is.. but I know I don't have much of a life outside of the forums these days

Craig,

The fact that you posted that message means that you're not (yet ) affected.

The ban I've put in place uses the .htaccess file, and totally bars them from accessing dslrusers.net.

[Alpha_7]

Glad I'm ok then.. I use the view posts since last on (ALOT) as in I basically use it to catch up on every post I've missed.. and then start digging deeper.. I wasn't sure if viewing it like that creates more load or generates more hits.

[gstark]

1500 hits in a (say) 10 hour day is still 150 hits per hour, or 2.5 hits per minute.
Even allowing for a 15 hour stint at the computer, we are talking 1.66 hpm.

I don't think anyone (perhaps with the exception of Birddog himself) could possibly post so frequently, so in my view, this limit is somewhat generous still.
Or am I missing something?

Edit: If you are "refreshing" just to catch a Birddog special, one refresh every 20 minutes guarantees you at least 10 minutes in his 30 minute window - this would only equate to 45 hits over 15 hours.

Remember that a "hit" isn't the same as a page serve. One page serve results in a significant number of hits, as the components of a page (icons, images, etc) get farmed out, so the numbers I've quoted, while accurate, aren't the best indicator of a person's activities.

However, any comparison between any other IP and mine is, because of the simple fact that we have two users here, both of whom are admins, both of whom might be here for several hours throughout the day, yet our numbers are significantly below those of some others ...

Go figure.

[MHD]

DoS?

[Nnnnsic]

From who, Scott? Maxwell?

[Alpha_7]

dpreview ?

[gstark]

Gary,

why not post the IP addresses that are hammering the site? The "offenders" can then see that they really need a life. It can be self-regulating. If they continue to hammer the site, then go and ban them and see if they suffer from withdrawl

It would also be interesting to see what phpBB make "one" hit. Is it the actual full page or various sections count as a page hit, eg News, topic headlines, etc?

cheers,

radar

See my comments above regarding what a "hit" actually is.

I'm loathe to publish the IP addresses that I've banned; I'm not sure what publishing them would achieve - they couldn't see that anyway, because they now cannot get here; I'm sure they'll find a way to email me in due course.


Of interest though, might be the following extract from our daily stats ... Note the variation between requests and pages ...

I've also included the CPU minutes used for each of those days. 40 minutes daily is the target DH want us to not exceed (by too much).

While there's no direct relationship between page hits and CPU usage, clearly there must be something there.


Monday ...

Successful requests: 70,009
Average successful requests per day: 71,803
Successful requests for pages: 15,073
Average successful requests for pages per day: 15,458

51 minutes CPU


Sunday ...

Successful requests: 44,082
Average successful requests per day: 42,774
Successful requests for pages: 8,172
Average successful requests for pages per day: 7,928

27 minutes CPU


Saturday ...

Successful requests: 40,686
Average successful requests per day: 42,361
Successful requests for pages: 8,097
Average successful requests for pages per day: 8,429

24 minutes CPU

Friday ...

Successful requests: 65,519
Average successful requests per day: 63,575
Successful requests for pages: 15,066
Average successful requests for pages per day: 14,618

42 minutes CPU

[gstark]

DoS?

Not evident in the logs, Scott.

Last time there was one clear "offender, being a GoogleBot, which accounted for around 50% of the traffic encountered. This time it all looks just like excessive popularity.

We also need to take into account time differentials between the logging (US based timezones) and when we're most active here.

I'm thinking I may need to purge some of the history from the database, thus making query times shorter.

[bago100]

Naughty Googlebot!

Gary - Birdie; Have you thought about a mass email to announce the next sale? That might save a heap of cpu time - might not either, I wouldn't know but just a thought anyway.

Cheers

Graham

[gstark]

Graham,

Short answer is no, but with spam being what it is, I'm rather reluctant to (ab)use the resources we have and risk the ire os those members (such as my good self) who absobloodylutely abhor junk email.

Interesting suggestion though; thanx.

[elffinarts]

perhaps rather than using the phpbb database to send out emails for those sales (which I've never been in time for LOL) - make an opt-in mailing list instead. One that requires verification to be added to the mailing list and therefore no reason to complain of spam.

I can see from all this that my own site will never have issues with bandwidth since the 5gig/day allowance is miles from what this far larger community uses.

why not switch to the same server? http://www.powweb.com

[pippin88]

elffinarts: I don't think it's raw bandwidth that is the problem, but rather CPU and Ram usage?

Gary: how did you get the CPU time figure?

[thaddeus]

A friend and I have a dedicated server which hosts a number of websites. We have plenty of bandwidth and CPU available. The reliability of the server is very high, as Leek, rog and embi will attest.

I am happy to host dslrusers at my cost. That way, all the worrying about hits, CPU usage, banning IPs, etc should go away. (DOS aside!)

[oli]

A friend and I have a dedicated server which hosts a number of websites. We have plenty of bandwidth and CPU available. The reliability of the server is very high, as Leek, rog and embi will attest.

I am happy to host dslrusers at my cost. That way, all the worrying about hits, CPU usage, banning IPs, etc should go away. (DOS aside!)

I was about to ask about this. Is this site hosted on a shared server where the host can say it's using "too many resources" and threaten to cut you off?

[gstark]

Nick,

When you get notified of the "problem" they start providing additional information which includes this data.

Russell, could you please email me details of the server?

The issue, as Nick points out, isn't bandwidth, but DH places an arbitrary limit of 40 CPU minutes per day on each site.

Of course, there's no mention of this in their blurbs as and when you sign up.

And they also provide a 97 day money back guarantee .... thinking .....

[DaveB]

Meanwhile, I'm left wondering if we'll ever get the RSS feed back!