Almost here.

[gstark]

Next year is almost here. Can you believe it?

So too is Christmas, and our Christmas Dinner is this Friday evening.

Last year's dinner was a blast, and if you've not yet booked .... tough luck!

Bookings are now closed!

And if you're in, you'll get to enjoy not just a great meal and the wonderful company of all of the DSLRUsers.com members, but also you'll get to see who will be the winner of our Picture of the Year prize.

Voting for which has now closed.

Too bad about a couple of our late cancellations ... perhaps they might have been winners too, but won't now be able to get their prizes on the night.

We're still looking at the issues regarding the resource usage at Dreamhost, and hope to have the RSS feed reestablished within the next
day or so.

[MHD]

Hmmm... thats odd Gary...

wonder what it could be?

[leek]

Gary,

Can you not tell from your site statistics what proportion of hits come from the RSS feed, and thereby estimate the effect of removing it...

I'd rather not lose RSS...

P.S. I've reduced the frequency of my RSS polling from 5 minutes to 30 minutes... Perhaps all the other RSS users could do the same...

[Nnnnsic]

Statistics usually relate to bandwidth data, not server cpu usage, which is the area they're pulling us up on.

I honestly don't expect the RSS to change much with the cpu usage and personally wonder what sort of cpu their servers actually use.

I also don't know how anyone can really "use" their bandwidth if the cpu usage tops out before bandwidth even gets close to the max limit.

Regardless John... RSS is down at the moment... so your modifications will do nothing to help you.

As far as I know... since the RSS feed is one of my projects... this is a temporary measure.

This is really one of those issues where Telstra could really shift their arses into gear and start pushing ADSL2 through other providers so we can go back to home hosting...

[leek]

Statistics usually relate to bandwidth data, not server cpu usage, which is the area they're pulling us up on.

I know, but you should be able to see what proportion of site hits come from the RSS feed page... You should then get a rough idea what proportion of CPU time is being spent to render the RSS feed. If it's 50%, then it might be the cause of your problem... If it's 5%, then that's unlikely...

[Paul]

Would the FasterFox plugin for Firefox have any effect on this?
There's been a small discussion on the Whirlpool forums about it
Apperently it pre-fetches each page on a website therefore increasing the traffic.
I'm a complete noob on this sort of stuff, just an idea maybe..?

[MCWB]

Not sure how phpBB works, but could it be a search function issue?

[Glen]

Trent, bugger all search, they just ask the same question again

[sheepie]

Not sure how phpBB works, but could it be a search function issue?

[MCWB]

Trent, bugger all search, they just ask the same question again

I'm having extreme issues with the search function!! Edit: WTF, why does my post say boring now???

[Alpha_7]

I tired to search for mate in public, and didn't get the response I was looking for...

[gstark]

In fact, it may be a search issue. I have seen a large number of queries coming in from google, many of which were searches. I've banned the IP subnet, and made sure that the robots.txt file that Google is supposed to respect is correct.

But apart from that, to answer John, there may or may not be a relationship between requests and CPU usage. Consider a rogue script of some sort, that perhaps performs a recursive search in the database. It might require a ton of CPU resources, but only occupy one http request in the log. That's why the access logs are only of limited usage here.

That said - and let me say (again) that I don't believe that the RSS feeds are the issue - reducing the frequency of the RSS queries will clearly have some sort of effect in this regard.

As an additional observation along these lines, in my RSS feed (using Chunderbird) what I'm seeing is the whole thread, and rendered as a forum page, as I select a message. Comparing this with what I had seen before (perhaps on the old server), this is very different, as before I was only seeing the messages as text files.

I'm simply making observations and exploring ideas ....

[Alpha_7]

If I recall correctly about the time I joined, wasn't there someone using a script that trawelled the forum collecting all the linked images... and then allowing the person to download them.. ? Maybe they aren't downloading them but they are repeatively searching for new images ??

[DaveB]

Reducing the frequency of RSS polls in a client is a compromise, given that the server is only including a fixed number of the most recent messages in the RSS/RDF file. If there have been more posts since your last poll than will fit in that file, you silently lose out on messages.

The larger the file on the server, the more work the server has to do in rendering it, but the less-frequently we can poll it.
The smaller the file, the less work per poll but the more polls are required to keep up with the traffic.

I don't think most RSS server implementations are designed with this many messages per minute to cope with. Apart from DSLRUsers, the next most busy feed on my own poll list is the BBC....

[Glen]

Craig, you are right, it was one of the guys in America, his name escapes me now



Edit, here is the link, maybe someone can work back from it, I am a luddite

http://stats.wolfenbier.com/cgi-bin/pics

[Nnnnsic]

Wasn't it NetMagi?

If I'm wrong, NetMagi, I apologise, but I seem to recall that the script was his...

[Glen]

Leigh, I think you are right

[DaveB]

As an additional observation along these lines, in my RSS feed (using Chunderbird) what I'm seeing is the whole thread, and rendered as a forum page, as I select a message. Comparing this with what I had seen before (perhaps on the old server), this is very different, as before I was only seeing the messages as text files.

When managing the RSS subscriptions in Thunderbird, try ticking the "Show the article summary instead of loading the web page" box to get the behaviour you've seen before.

But this shouldn't cause more CPU load on the server. Firstly I would expect that Thunderbird will cache the pages (at least the images/etc), and secondly if Thunderbird did NOT cache it I would expect it to show up as more non-RSS download traffic rather than just increased CPU load.

[gstark]

Ok ... here's something to think about.

I've just done some tabulation from the log entries for Nov 29, US time.

Let's take, as a starting point, the fact that at this household we have one IP address and two people, both admins, accessing the site. We both also have RSS feeds, yet the total number of accesses reported in the logs (almost 60,000 lines) was 278.

There were 51 IP addresses reporting more accesses than we were, and there were 430 unique IP addresses in total.

The highest number of accesses reported for the one IP address was 2148! The IP Adress is reported as being owned by Optus.

The google access that I noted earlier was the third most prolific "user", with 1925 accesses.

The second most proliific user had 2019 accesses, and is owned by Westnet.

#4 has 1655 accesses, and is IDXNet, a Sydney ISP. So a search tells me.

[sheepie]

The highest number of accesses reported for the one IP address was 2148! The IP Adress is reported as being owned by Optus.

Isn't Birddog with Optus?

[kipper]

I have to agree with Dave on this one, I don't think RRS was intended like this and it was funny I was going to mention something about that but stumbled across this thread. RRS afaik was just a means for linking external websites news threads into your website (and these hardly news threads). Which would mean everytime your website page is requested the RRS feed is requested. However what you have now is clients that continuelly poll the http server for RRS feeds and if you have many clients (could be quite a few sockets open) hitting the website then it might start to chug.

Out of interest what was the format of the RRS feed, I never got to see it. Just wondered how many ks of data we're talking about.

[thaddeus]

Actually, I can see where Dreamhost is coming from!

I run phpbb on a dedicated 2.8ghz Xeon box with 1GB of RAM. I just simultaneously opened up 10 copies of the home page of the forum and hit 4% CPU usage. If you do the maths you'll see that this could aggregate to seriously impact DreamHost's business model of selling multiple websites on each server.

Did you sign up for some kind of limit on CPU time?

As for the IP log-hogs, there's a good chance it's RSS clients. I have found that no matter how many times you ask people to check their polling frequency, there's always a few people who are polling the thing every two or three minutes.

[phillipb]

As for the IP log-hogs, there's a good chance it's RSS clients. I have found that no matter how many times you ask people to check their polling frequency, there's always a few people who are polling the thing every two or three minutes.

Excuse my ignorance in these matters, but is this the same as refreshing the browser every 2 or 3 minutes?

[thaddeus]

Setting the RSS client to poll every 2-3 minutes is less load on the server than refreshing the browser every 2-3 minutes because the server doesn't need to format an elaborate page (with graphics, buttons and stuff) for the client. The key difference is that the RSS client will keep polling every 2-3 minutes as long as the person's computer is on... long after they would have give up refreshing the browser!

[kipper]

phillipb as said, you won't have multiple browser clients repeatedly being refreshed like you do an RSS client.

[phillipb]

So is there a better way to do things? at the moment if I'm interested in a thread, I tend to refresh every few minutes to see if there's any replies.

[Glen]

Gary, I am with Optus will find my IP then PM you, though doubt I am that keen

[gstark]

Glen,

211.30.11.239

[MCWB]

Was with Optus, now with iiNet, so if you see that IP change...

[Glen]

Gary, just realised the way you found that was 10 times quicker than the way I found it

[kipper]

Phillipb well I'm not sure what data RSS feeds whether it's the last 10 threads with every post in that thread or what so I can't comment. However I'm sure there is a better way things could be done for this sort of scenario.

I just had a look at somebody elses RRS feed and it sort of confirmed what I thought. These feeds could be done more intelligently I guess and only fetch data if it's changed the problem that I see is it just polls and fetches all data again each time. Gary am I wrong on this?

[gstark]

Gary, just realised the way you found that was 10 times quicker than the way I found it

Glen,

I think that you can find it the same way that I can.

Sor those who are interested, Birddy's hits from his Optus link were in the range of 900 odd; nowhere near the 2100 of the Optus camper that we seem to have.

I'm going to see if there's an easy way I can locate who posts from that IP (once I have a massage I can track the source, but I need to have the mesage/user first. Here I just have an IP, which is a different ball of wax).

Or I may just ban the top three or four IPs and see who screams thje loudest.

...

I now have some slightly more meaningful numbers from DH, but they're about as useful as a bicycle is to a fish.

They quote their numbers in CPU seconds, and say divide them by 60.

Ok .... php-cgi was 7363.7200 ... divide by 60 = 122.7286 minutes, which is not good; this number is only one of the processes, and the sum of all of them should be sub 40 minutes.

But then the repot goes on to say "CPU percentage assumes 24000 cpu seconds per day total" ... 24000 divided by 60 = 400 minutes per day ????

Huh ???

Our percentage is around 30% ... 122 minutes is roughly 2 hours, which is not 30% of a day .... but 122 is about 30% of 400 ...

If I said I was confused by all of this (and it usually takes quite a bit to confuse me) I think I'd be understating the case.

[DaveB]

I just had a look at somebody elses RRS feed and it sort of confirmed what I thought. These feeds could be done more intelligently I guess and only fetch data if it's changed the problem that I see is it just polls and fetches all data again each time. Gary am I wrong on this?

That's a reasonable description of the behaviour. An RSS client downloads the RDF file (a form of XML) which has a list of items in it. Each item has a unique ID (URL) and an "article summary". The RSS feed here has each post as an item with the summary text as the full post, and the list of items is the last 'N' posts (used to be 15, but Leigh increased it for us).

If CPU load of generating the RDF file was a problem, the php code could cache the generated file, and only update it if there has been a new post. It won't cut down on the bandwidth, but it would reduce CPU load. Who knows: maybe it does this already?

Remember, RSS == Really Simple Syndication.

Changing the protocol so you download just the IDs and then download the relevant articles separately would take the "Simple" out of it!
Even if the DSLRUsers feed was changed so that the article summaries embedded in the feed were not the full articles that wouldn't really have an impact on the CPU load. But it would make the RSS feed much less useful!

[gstark]

I just had a look at somebody elses RRS feed and it sort of confirmed what I thought. These feeds could be done more intelligently I guess and only fetch data if it's changed the problem that I see is it just polls and fetches all data again each time. Gary am I wrong on this?

This is Leigh's field of expertise, but he recently changed the scripts to feed more data - 25 messages, rather than 10 - or something like that. Concievably, that could impose a greater workload on the server, but again, I find it difficult to beliueve that it's the RSS that's causing the problem.

Bottom line, and the answer to your question is that I don't know, but the point, for the moment, is moot.

We need to look at the numbers for the next few days and see whether they drop before we can reeally get any meaningful information. For now, I've needed to change how my scripts run so that the reports DH provide me with give me some more meaningful information.

That said, the longest process was 0.77 (there were 3 that took longer than 0.7, another three .6, 25 .5) and 49, 852 processes in total, which looks to be about 10000 les than the hits reported in the prior day's logfile.

Of course, this ignores the fact that one script call might mean ten or more hits in the log file as it retrieves icons, images, and text from various parts of the filesystem.


And on reflection, while there's a few longer processes there, there's nothing that really stands out, which may mean that it simply comes back to hit count * cpu time = usage, and therefore the only means at our disposal for reducing this is to reduce the hit count in some way, which would probably mean the rss feed in the first instance.

[MHD]

REMINDER: you have 1.5 hours to vote for the Picture of the Year

[leek]

which may mean that it simply comes back to hit count * cpu time = usage, and therefore the only means at our disposal for reducing this is to reduce the hit count in some way, which would probably mean the rss feed in the first instance.

Personally, I think that eliminating RSS might actually increase your hit count Gary... In the short time it has been down, I've been forced to look at each post within the browser, re-rendering each page every time someone replies. Normally when using RSS, I only look at the text preview and only goto the web-rendered page if I want to reply or see all the posts in one go.

Asking people to reduce the frequency of their RSS polling and slightly reducing the frequency with which the RSS page is generated would probably have a more positive effect. BTW, Do you have any record of the CPU usage stats from GoDaddy for comparison??? It may be a problem with the way Dreamhost has their PHP set-up.

[gstark]

Personally, I think that eliminating RSS might actually increase your hit count Gary

Good point, John, although it also depends upon what each of those hits might require. The RSS would always supply x hits, whereas a visit to the home page might create x - 10 (or x+10) hits, and a visit to a thread would be different again.

Putting it veryu simply, I just don't (yet) know, and the only way I can see what's going on is to try a few different things, and see what the various outcomes are from each of those actions.

[MHD]

POTY Gallery voting is CLOSED!

[NetMagi]

Wasn't it NetMagi?

If I'm wrong, NetMagi, I apologise, but I seem to recall that the script was his...

yeh I wrote that script. .

When I use it, it visits every thread on the first page in the image review section once. It does pretty much the same thing I do when I check out the images manually.

It likely uses less acesses to the server than i do in a typical browsing session, because it only accesses the index once instead of every time I click back or on the forum name from inside a thread to browse to the next one.

-Rich

[gstark]

Hi Rich,

Thanx for checking in on this issue.

I'm not at all concerned about your script in this matter, because the I{ addresses with the greatest number of hits on the site, apart from Google, were all Oz based.

I've not yet banned those IPs, but on Monday I'll be sorting and looking at about 5 days' logfiles, and comparing that data with my initial observations.

Basically, any IP that has a hit rate significantly greater than Birddog's might get banned in this exercise. because, while the numbers have dropped since we pulled the RSS, they've probably not dropped far enough as yet.

And for John, despite your cautionary note, the drop has been somewhat significant. so perhaps we need to review what we're doing with respect to the rss feed.

[leek]

And for John, despite your cautionary note, the drop has been somewhat significant. so perhaps we need to review what we're doing with respect to the rss feed.

Bummer...

Maybe a larger number of threads less often... That way, we can each adjust the frequency of our RSS polling and not miss anything...

[sirhc55]

I have absolutely no idea at this level of tech geek talk what is happening - but, just recently I turned off receiving emails of posting information as I was receiving a tremendous amount daily. Would this have anything to do with the situation - emails Remember I don’t know what I am talking about

[DaveB]

And for John, despite your cautionary note, the drop has been somewhat significant. so perhaps we need to review what we're doing with respect to the rss feed.

Personally I'm not keeping up with the site traffic the way I'm normally able to using RSS, so this drop in traffic has come with a drop in use (for me at least).

[gstark]

Dave, Chris ...

Neither of your IP addresses seem to be offenders.

[DaveB]

Dave, Chris ...

Neither of your IP addresses seem to be offenders.

I wasn't suggesting that it might be, but I was pointing out that yes dropping the RSS feed will have reduced the load: it's reduced the amount I'm accessing (and keeping up with) the site - I'm just an example (possibly not representative, but...).

[mic]

Hope you all have a good Feast & a great Christmas, maybe next year for me

I want a macro shot of a Sang Shoy Bow, if anybody can be bothered to put down their glass of Red or 2

Mic.