phpBB2Refugees.com Logo
Not affiliated with or endorsed by the phpBB Group

Register •  Login 

Continue the legacy...

Welcome to all phpBB2 Refugees!Wave Smilie

This site is intended to continue support for the legacy 2.x line of the phpBB2 bulletin board package. If you are a fan of phpBB2, please, by all means register, post, and help us out by offering your suggestions. We are primarily a community and support network. Our secondary goal is to provide a phpBB2 MOD Author and Styles area.

Getting too popular for it's own good!
1 members found this topic helpful
Goto page Previous  1, 2, 3, 4  Next
 
Search this topic... | Search phpBB2 Discussion... | Search Box
Register or Login to Post    Index » phpBB2 Discussion  Previous TopicPrint TopicNext Topic
Author Message
Oliver2000
Board Member



Joined: 29 Jan 2009

Posts: 21


flag
PostPosted: Tue Feb 03, 2009 6:31 pm 
Post subject: Re: Getting too popular for it's own good!

hi there,

jeap my searchwords are really full of a lot of grap. word with 1-2,3 letters and lots with 30-40 as well. (i work in portuguese with my forum)

what seems to be a pain in the neck is also lots of words which are not words icon_smile.gif like 000 000230 232321 etc.

can somebody tell me how to fix the bug to get only words with 3-20 letters and ONLY words but dont index at all any numbers and words with numbers?

i guess this would make my 1gb table much lighter icon_smile.gif

thanks
Back to top
drathbun
Board Member



Joined: 24 Jul 2008

Posts: 729
Location: Texas


flag
PostPosted: Tue Feb 03, 2009 10:14 pm 
Post subject: Re: Getting too popular for it's own good!

Oliver, I have changed my regex (and indeed have a number of posts on optimizing the search process on my blog) but the fix doesn't seem to work properly for non-English boards. Still, if you want to read a few notes about how search works...

phpBBDoctor Search Posts

Jim, there are various MODs that are supposed to link your phpBB database with the wiki userbase, but I've not been able to spend the time to see which - if any - work well.

_________________
phpBBDoctor Blog
Back to top
Slackervaara
Board Member



Joined: 01 Jan 2009

Posts: 70



PostPosted: Wed Feb 04, 2009 12:40 am 
Post subject: Re: Getting too popular for it's own good!

I optimize my database each day and I have noted that it releases a lot of space in the database that concerns Search, but I don't know, if it speed up search.

I have earlier removed manually old threads in the forum that are not informative and important to go back to. It took pretty much time to read and then delete.
Back to top
Jim_UK
Board Member



Joined: 19 Nov 2008

Posts: 656
Location: North West UK


flag
PostPosted: Wed Feb 04, 2009 6:54 am 
Post subject: Re: Getting too popular for it's own good!

drathbun wrote:

Jim, there are various MODs that are supposed to link your phpBB database with the wiki userbase, but I've not been able to spend the time to see which - if any - work well.


When the phpbb.com site comes back up I will go have a look at any wiki mods that might be in the database.

Thanks
Jim

Looks as though the Team are having major problems as a result of this hacker.
Back to top
Jim_UK
Board Member



Joined: 19 Nov 2008

Posts: 656
Location: North West UK


flag
PostPosted: Mon Feb 09, 2009 2:22 pm 
Post subject: Re: Getting too popular for it's own good!

Just a note on the Search engine Sphider

At each use it crashes my test board on localhost by completely "filling" my sessions table.

It also takes hours to spider my test site (DB about 30MB) and as there is no option to set it to auto index the site it would need to be done manually every few hours or at least once a day.

I have ruled out the use of Sphider.

Jim
Back to top
dogs and things
Board Member



Joined: 18 Nov 2008

Posts: 628
Location: Spain


flag
PostPosted: Mon Feb 09, 2009 2:56 pm 
Post subject: Re: Getting too popular for it's own good!

Hi Jim,

You need a MOD for Sphider that makes it follow only the urls in a sitemap.xml. That way it won't follow any other links it finds. Also, it will only index or re-index urls in that sitemap.xml depending on last_mod date and last_index date.

But yes, you need to re-index manually.

What I do is index my board once a day, I update my sitemap.xml and then re-index.

A bit of a job but well, Iīm fine with that.

You could put those things on a Cron job, but my server doesnīt seem to allow Cron. icon_neutral.gif

Greetings. icon_wink.gif

_________________
phpBB2 will never die, I hope!
Back to top
Jim_UK
Board Member



Joined: 19 Nov 2008

Posts: 656
Location: North West UK


flag
PostPosted: Tue Feb 10, 2009 12:11 pm 
Post subject: Re: Getting too popular for it's own good!

Thanks for the info.
Did you not have a problem with it filling the sessions table?

Jim
Back to top
dogs and things
Board Member



Joined: 18 Nov 2008

Posts: 628
Location: Spain


flag
PostPosted: Tue Feb 10, 2009 1:02 pm 
Post subject: Re: Getting too popular for it's own good!

No, not that I have noticed, no problem at all.

The problem I had was to make it index only the urls I wanted as Sphider tends to follow every possible link it encounters. On a bb board that's a lot of links.

The way I have it set up now it only indexes every topic once and re-indexes when if finds out that topic has new posts added to it or is new.

Another thing, as the sitemap-generator I use only produces urls for topics that are readable to the public the forums that have restrictive permissions are not indexed and donīt show up in the search results.

The Sphider database is independent from my board's database.

Took me some work to get it all going nicely but I'm glad I went through it.

_________________
phpBB2 will never die, I hope!
Back to top
Jim_UK
Board Member



Joined: 19 Nov 2008

Posts: 656
Location: North West UK


flag
PostPosted: Sun Feb 15, 2009 9:28 am 
Post subject: Re: Getting too popular for it's own good!

I have made the change to includes/functions_search.php that Dave posted and suggested to my son that he do the same.
I find it almost unbelievable that we should suffer from such high server loads considering the nature of the VPS that we have and how little we have on it. http://hostforweb.com/vps/index.php See HFWVPS-4
His model ship site and my model aircraft site are the most active with his site moving just over 300GB of data per month and mine about 30 - 40GB.
There is also a model shipwrights database that moves about 10GB per month plus a few others (about 4 that are static html moving less than 1GB)

I have seen load figures as high as 40 (no not a mistype) and loads have been high enough during the last week to crash the server on several occasions..

Below the Server Spec on the VPS
Quote:
Processor #1 Vendor: GenuineIntel
Processor #1 Name: Intel(R) Xeon(TM) CPU @ 1.66GHz
Processor #1 speed: 374.395 MHz
Processor #1 cache size: 2048 KB

Processor #2 Vendor: GenuineIntel
Processor #2 Name: Intel(R) Xeon(TM) CPU @ 1.66GHz
Processor #2 speed: 374.395 MHz
Processor #2 cache size: 2048 KB

Processor #3 Vendor: GenuineIntel
Processor #3 Name: Intel(R) Xeon(TM) CPU @ 1.66GHz
Processor #3 speed: 374.395 MHz
Processor #3 cache size: 2048 KB

Processor #4 Vendor: GenuineIntel
Processor #4 Name: Intel(R) Xeon(TM) CPU @ 1.66GHz
Processor #4 speed: 374.395 MHz
Processor #4 cache size: 2048 KB



I have raised concerns with the server owners and they monitored the server for a period of time when loads were high.
They confirm as I was sure of anyway that the server nor the scripts had been compromised as there was nothing untoward running in the background and that it was only the know sites generating the load but with the model ship site being responsible for the vast majority of it.
We are not talking of a site with 100's of folks on line at at a time. The most that I have ever seen has been about 40.
However that site and my own are both very image intensive with "Attachment mod", Useruploads and Galleries plus loads of modifications. My sons site has 144 tables in the database (including a lot relating to the Gallery)

Guess I had better get to the question eh! icon_rolleyes.gif

Is there anything we can do to lessen the loads without changing the functionality of the site?
As there are lots of images sometimes on multiple posts within a single page would it help any to reduce the number of posts per page or could that just increase the problem as users read all pages. Maybe most users only read the last post in the last page of a topic that they have been "watching"

At $1200 per year for the VPS this is becoming an expensive hobby. Note that we are very happy with the host and would not want to change as they go out of their way to help in any issues.
How can we retain the functionality but reduce loads and hence increase speeds.

Ideas please.

Jim

Dave - If you think this constitutes a new topic or worth another one because it may affect others then please split this off.
Back to top
drathbun
Board Member



Joined: 24 Jul 2008

Posts: 729
Location: Texas


flag
PostPosted: Sun Feb 15, 2009 11:23 am 
Post subject: Re: Getting too popular for it's own good!

Jim, I will leave it up to you to request a split if / when you want it to be done. Simply use the "report a post" option. icon_smile.gif For now I think it's just fine here.

Have you removed the nav_links functionality from your boards? That was the cause of my server woes for well over a year. It's an extremely simple change. My server loads were up over 600 (as you say, not a typo icon_smile.gif ) at one point. Never seen that level since I made this change.

The next thing I would look at is your apache config. Specifically the "keep alive" setting. I'll write more on that if you need me to, but check the nav_links thing first.

_________________
phpBBDoctor Blog
Back to top
Jim_UK
Board Member



Joined: 19 Nov 2008

Posts: 656
Location: North West UK


flag
PostPosted: Sun Feb 15, 2009 11:43 am 
Post subject: Re: Getting too popular for it's own good!

I have to say Dave looking at the image in that post that that is pretty much what I see when I go take a look at running processes.
I will have a word with my son and see if he is willing to let me make the changes that you show in the mod. I can not see him objecting to this so will report back having monitored the server over a few days.
I think a topic on optimising the code to reduce load may well be a worthwhile one.
When I first built my son's site he put it on to shared hosting and had his account suspended several times in the first month for creating excessive load. That prompted the move to a reseller account with another host. This of course was still shared hosting and things were not much better there so prompting this later move. Very happy with support from host but need to get these load figures down to something less.

I guess that a lot of the load generated by this none required code is as a result of web spider visits as well.

I shall let you know what happens.

Thanks
Jim
Back to top
drathbun
Board Member



Joined: 24 Jul 2008

Posts: 729
Location: Texas


flag
PostPosted: Sun Feb 15, 2009 11:48 am 
Post subject: Re: Getting too popular for it's own good!

It has been "night and day" difference for me. The easiest and fastest way to test without touching any php code is to go into overall_header.tpl and find {NAV_LINKS} and remove it. If that helps, then the next step is to go through several files (index, viewforum, viewtopic, includes/functions, includes/page_header and I think common) and remove the code that sets up the $nav_links variable. No need to take CPU cycles to build something that isn't going to appear anywhere. icon_smile.gif
_________________
phpBBDoctor Blog
Back to top
~Cowboy~
Board Member



Joined: 08 Dec 2008

Posts: 297
Location: Chicago


flag
PostPosted: Sun Feb 15, 2009 12:36 pm 
Post subject: Re: Getting too popular for it's own good!

The first thing I would do is thumbnail the images so that pages would not load sets of full size images that everyone has already seen every time they visit the page..

Next I would move attachments off the VPN to see if that is what's causing the load and, make sure people are not hot linking to them.

Sometimes with sites like these there is a competitors site trying to establish itself at the cost of your bandwidth. And some times competitors will deliberately try to lag your site by repeat downloading of large files simply to try to frustrate your members into using a different site ...(theirs). This may or may not be happening to you but it's best to be sure.

_________________
Image link
We are not refugees we are trail blazers. icon_wink.gif
Back to top
Jim_UK
Board Member



Joined: 19 Nov 2008

Posts: 656
Location: North West UK


flag
PostPosted: Sun Feb 15, 2009 2:01 pm 
Post subject: Re: Getting too popular for it's own good!

I don't think changing the Attachment config so that it is only thumbnails that display inline would go down right well to be honest. They like to see lots of pictures. In fact I urge them to post pictures as I think it makes the site more appealing.
Imagine two posters. One poster full of text and the other text but including a nice picture. If they both convey the same info which would be most likely to be read? I know which my attention would be drawn to.

I know what you mean about image/bandwidth theft. I found a Chinese site that was not only hotlinking images from posts but was hotlinking the images used to build up the page. Template images!!!!

I did stop hotlinking but it had an adverse effect that some none logged in users could not see the images. That being the case it would be that some prospective users may go elsewhere as they would find the site to be just text based.

Just going to implement Daves code changes now on my own site and then get son to do same.

Jim
Back to top
Sylver Cheetah 53
Board Member



Joined: 17 Dec 2008

Posts: 426
Location: Milky Way


flag
PostPosted: Sun Feb 15, 2009 5:11 pm 
Post subject: Re: Getting too popular for it's own good!

drathbun wrote:
Jim, I will leave it up to you to request a split if / when you want it to be done. Simply use the "report a post" option. icon_smile.gif For now I think it's just fine here.

Have you removed the nav_links functionality from your boards? That was the cause of my server woes for well over a year. It's an extremely simple change. My server loads were up over 600 (as you say, not a typo icon_smile.gif ) at one point. Never seen that level since I made this change.

The next thing I would look at is your apache config. Specifically the "keep alive" setting. I'll write more on that if you need me to, but check the nav_links thing first.

Dave, I am having some problems understanding your Remove Navigation Links 1.0.1 MOD. In "Author Notes" you are saying:
Quote:
## Please install and test only ONE new MOD at a time. That may seem like common
## sense, but you would be amazed at how often this is not done. icon_smile.gif
##
## Install Time Estimate
## The MOD install time is assumed to be for a manual install and is based on the
## number of files to edit. If you use an automatic installer the time estimate
## is essentially meaningless. We do not test with EasyMOD but our MODs should
## work based on our usage of the proper template. Note that EasyMOD seems to
## have issues with certain SQL structures that work fine in MySQL directly.

I do not understand the frase "test only one new MOD at a time". Why, this is related with other MODs? What is "Time Estimate"? It is another MOD that I am supposed to install?

P.S.: Sorry for being stupid. icon_redface.gif

_________________
Image link
My Forum || My Blog

phpBB2 forever! icon_smile.gif
Back to top
Display posts from previous:   
Register or Login to Post    Index » phpBB2 Discussion  Previous TopicPrint TopicNext Topic
Page 2 of 4 All times are GMT - 4 Hours
Goto page Previous  1, 2, 3, 4  Next
 
Jump to:  

Index • About • FAQ • Rules • Privacy • Search •  Register •  Login 
Not affiliated with or endorsed by the phpBB Group
Powered by phpBB2 © phpBB Group
Generated in 0.0788 seconds using 16 queries. (SQL 0.0199 Parse 0.0013 Other 0.0576)
phpBB Customizations by the phpBBDoctor.com
Template Design by DeLFlo and MomentsOfLight.com Moments of Light Logo