phpBB2Refugees.com Logo
Not affiliated with or endorsed by the phpBB Group

Register •  Login 

Continue the legacy...

Welcome to all phpBB2 Refugees!Wave Smilie

This site is intended to continue support for the legacy 2.x line of the phpBB2 bulletin board package. If you are a fan of phpBB2, please, by all means register, post, and help us out by offering your suggestions. We are primarily a community and support network. Our secondary goal is to provide a phpBB2 MOD Author and Styles area.

Fast reindex of search


 
Search this topic... | Search General Support... | Search Box
Register or Login to Post    Index » General Support  Previous TopicPrint TopicNext Topic
Author Message
JLA
Board Member



Joined: 30 Apr 2009

Posts: 298
Location: U.S.A


flag
PostPosted: Mon Aug 10, 2009 9:52 pm 
Post subject: Fast reindex of search

Does anyone know a FAST reindex script for rebuilding the search tables.

The one is DB Maintenance is too slow - especially for 22mil + posts

Thanks

_________________
http://www.jlaforums.com
Back to top
Jim_UK
Board Member



Joined: 19 Nov 2008

Posts: 541
Location: North West UK


flag
PostPosted: Tue Aug 11, 2009 9:15 am 
Post subject: Re: Fast reindex of search

Rebuild search tables attached.

Jim



rebuild_search_2.4.0.zip
 Description:

Download
 Filename:  rebuild_search_2.4.0.zip
 Filesize:  66.91 KB
 Downloaded:  589 Time(s)

Back to top
JLA
Board Member



Joined: 30 Apr 2009

Posts: 298
Location: U.S.A


flag
PostPosted: Wed Aug 12, 2009 3:36 pm 
Post subject: Re: Fast reindex of search

Jim_UK wrote:
Rebuild search tables attached.

Jim


Thank You. I'll have a look at it.

_________________
http://www.jlaforums.com
Back to top
Dog Cow
Board Member



Joined: 18 Nov 2008

Posts: 378


flag
PostPosted: Wed Aug 12, 2009 6:10 pm 
Post subject: Re: Fast reindex of search

JLA wrote:
Does anyone know a FAST reindex script for rebuilding the search tables.

The one is DB Maintenance is too slow - especially for 22mil + posts

There shouldn't be anything out there. What you'll find are just the usual scripts which import every post and run it through phpBB's standard post indexer.

However, that's certainly not a FAST reindex script.

If you wanted speed, you'd either:
a.) not index everything
b.) run a distributed process

Option A would imply not indexing topics with less than X amount of replies, or not indexing posts shorter than X characters. Or conversely, not indexing posts made after reply number X in a topic. In addition, you could increase the stop words file.

Option B would imply running multiple indexing processes. You could write a script in C or Perl, since I believe that they can execute faster. Or just take the standard PHP reindexing script. No matter what route you go, you'd just connect multiple machines into the database and then run maybe 2-3 concurrent indexing processes on each machine. Each individual process could handle one million posts.

If you combine methods A and B, then you would really get fast indexing! icon_surprised.gif

_________________
Moof!
Lincoln's Tomb, Oak Ridge Cemetery, Springfield IL Mac 512K Blog Mac GUI
Back to top
JLA
Board Member



Joined: 30 Apr 2009

Posts: 298
Location: U.S.A


flag
PostPosted: Wed Aug 19, 2009 4:04 pm 
Post subject: Re: Fast reindex of search

Thanks. One thing I found with the above script

In file admin_rebuild_search.php

We made the following change to reflect the LANZER SEARCH change we use on the site

Code:

// add the words
      //Removed Lanzer Search Mod
      //add_search_words('single', $row['post_id'], stripslashes($row['post_text']), stripslashes($row['post_subject']));

//New Lanzer Code
      $search = new fulltext_search();
      $result = $search->add($mode, $row['post_id'], stripslashes($row['post_text']), stripslashes($row['post_subject']));
//End New Code


Now while running this function, it will only process 1 post at a time regardless of the amount of posts you specify in the configuration.

Any ideas on this?

_________________
http://www.jlaforums.com
Back to top
Dog Cow
Board Member



Joined: 18 Nov 2008

Posts: 378


flag
PostPosted: Thu Aug 20, 2009 5:18 pm 
Post subject: Re: Fast reindex of search

JLA wrote:

Now while running this function, it will only process 1 post at a time regardless of the amount of posts you specify in the configuration.

Any ideas on this?

Do you mean one post out of every topic?

_________________
Moof!
Lincoln's Tomb, Oak Ridge Cemetery, Springfield IL Mac 512K Blog Mac GUI
Back to top
JLA
Board Member



Joined: 30 Apr 2009

Posts: 298
Location: U.S.A


flag
PostPosted: Thu Aug 20, 2009 6:20 pm 
Post subject: Re: Fast reindex of search

Dog Cow wrote:
JLA wrote:

Now while running this function, it will only process 1 post at a time regardless of the amount of posts you specify in the configuration.

Any ideas on this?

Do you mean one post out of every topic?


You have a place in the settings to say how many posts are processed per loop. The default is 50.

After making that code change, it will only process 1 post per loop regardless of what you specify in the settings.

_________________
http://www.jlaforums.com
Back to top
drathbun
Board Member



Joined: 24 Jul 2008

Posts: 653
Location: Texas


flag
PostPosted: Sat Aug 29, 2009 3:32 am 
Post subject: Re: Fast reindex of search

JLA, do you have a big stopwords.txt file? I cut a substantial amount of time off of my search rebuild by converting that to a database table instead.
_________________
phpBBDoctor Blog
Back to top
JLA
Board Member



Joined: 30 Apr 2009

Posts: 298
Location: U.S.A


flag
PostPosted: Mon Aug 31, 2009 1:54 pm 
Post subject: Re: Fast reindex of search

drathbun wrote:
JLA, do you have a big stopwords.txt file? I cut a substantial amount of time off of my search rebuild by converting that to a database table instead.


Actually no, it is actually pretty small.

_________________
http://www.jlaforums.com
Back to top
Display posts from previous:   
Register or Login to Post    Index » General Support  Previous TopicPrint TopicNext Topic
Page 1 of 1 All times are GMT
 
Jump to:  

Index • About • FAQ • Rules • Privacy • Search •  Register •  Login 
Not affiliated with or endorsed by the phpBB Group
Powered by phpBB2 © phpBB Group
Generated in 0.0211 seconds using 17 queries. (SQL 0.0017 Parse 0.0026 Other 0.0168)
phpBB Customizations by the phpBBDoctor.com
Template Design by DeLFlo and MomentsOfLight.com Moments of Light Logo