phpBB2Refugees.com Logo
Not affiliated with or endorsed by the phpBB Group

Register •  Login 

Continue the legacy...

Welcome to all phpBB2 Refugees!Wave Smilie

This site is intended to continue support for the legacy 2.x line of the phpBB2 bulletin board package. If you are a fan of phpBB2, please, by all means register, post, and help us out by offering your suggestions. We are primarily a community and support network. Our secondary goal is to provide a phpBB2 MOD Author and Styles area.

Recomanded MODs

Goto page Previous  1, 2, 3, 4, 5  Next
 
Search this topic... | Search phpBB2 Discussion... | Search Box
Register or Login to Post    Index » phpBB2 Discussion  Previous TopicPrint TopicNext Topic

What is your favorite MOD?
(This poll does not expire)
Keep Unread Flags
9%
 9%  [ 1 ]
Super Quick Reply
9%
 9%  [ 1 ]
Simple Subforums
9%
 9%  [ 1 ]
Split posts and merge in one step
18%
 18%  [ 2 ]
Visual Confirmation on New Posters
0%
 0%  [ 0 ]
Easy Resize Posted Images
18%
 18%  [ 2 ]
Limited Post Edit Time
0%
 0%  [ 0 ]
Admin Delete User with All Postings
0%
 0%  [ 0 ]
Move post instead of deleting
0%
 0%  [ 0 ]
None of the above (please, specify wich one you prefer)
36%
 36%  [ 4 ]
Total Votes : 11

Author Message
drathbun
Board Member



Joined: 24 Jul 2008

Posts: 653
Location: Texas


flag
PostPosted: Thu Jan 22, 2009 8:10 pm 
Post subject: Re: Recomanded MODs

After a quick glance at the cache posts MOD what it appears to do is remove the bbcode and smilies parsing steps and store the html on disk for each individual post. This avoids the php process of parsing out bbcode and smilies as the post has already been converted to html. Only if a post is edited does the cache for that post have to be updated.

I was assuming that this was more broad in scope, meaning that entire topics would be cached together, and every time a new post came in the topic cache would have to be updated. Then there are issues with paging and whatnot. Since this MOD is at the post level it would be immune from that.

However... as written there is absolutely no benefit for the database. The entire post text is still retrieved from the database every time a topic is viewed, and only if the post "cache" indicator is on does the program then attempt to read from the disk file. Also the post_edit_time is used to compare to the cache time; the post_edit_time does not increment when someone (an admin or moderator) edits the post. The post_edit_time only increments when the post owner edits the post, and even then it doesn't change if the post is the last post in the topic. That may or may not be a bug; I'm just looking at bits of code.

The code also does an individual file read for each cached post. So if you have your "posts per page" set to 25, and a topic is longer than 25 posts, there will be 25 individual file open + read + close operations. And this after already reading from the database. Jim, you might want to look at your disk I/O statistics on your server as they are probably going to go up after installing this MOD.

So yes, you are saving some page generation time since the bbcode and smilies are already parsed. It seems that the same thing could be accomplished by storing the post parsed in the posts_text table instead.

If I were to write something like this, I believe I would use the database rather than files.

_________________
phpBBDoctor Blog
Back to top
Dog Cow
Board Member



Joined: 18 Nov 2008

Posts: 378


flag
PostPosted: Thu Jan 22, 2009 9:37 pm 
Post subject: Re: Recomanded MODs

drathbun wrote:

If I were to write something like this, I believe I would use the database rather than files.

I wrote this in as a feature of SMPL, (Standard Message Parsing Library) which I use as a portable class for handling rich text such as magic links, bbcode, and smileys anywhere on my site. As one of the features I wanted for my new forums was message caching, that is why I did it. The idea came from Gaia (who uses memcached for just about everything), and also a post in the phpBB Tweaks topic.

Here are some notes on it:
- The post must have at least 100 chars AND...
- ... it must have BBCode OR smileys OR some magic links in order to be considered for caching.
- Cached posts text are stored in a field in the same posts_text table
- I've removed the user's ability to selectively disable smileys and/or BBCode. Instead, each post is analyzed to see if there are smileys or BBCode used. The appropriate switches are turned on/off depending on content. The purpose? We get a speed gain even on un-cached posts since SMPL won't be trying to parse smileys/BBCode where there isn't any.
- When fetching a cached post, SMPL will then run the censor() function and some other small stuff on it
- As an aside, vBulletin stores cached posts in a separate table and uses a LEFT JOIN
Back to top
espicom
Board Member



Joined: 24 Nov 2008

Posts: 55
Location: Woodstock, IL


flag
PostPosted: Thu Jan 22, 2009 10:36 pm 
Post subject: Re: Recomanded MODs

drathbun wrote:
After a quick glance at the cache posts MOD what it appears to do is remove the bbcode and smilies parsing steps and store the html on disk for each individual post. This avoids the php process of parsing out bbcode and smilies as the post has already been converted to html. Only if a post is edited does the cache for that post have to be updated.


A different approach would be to add a BLOB field to the posts (or maybe the posts_text) table that has the parsed message text in it, which is updated when the message is entered or edited. If the message text exceeds a certain size or has no BBCode in it, you could elect to NOT put the BLOB in.

No additional file accesses, no need to make a world-writable directory, no extra queries, but a lot more SQL bandwidth, because of the extra BLOB. Unless, of course, your read query was tailored to know the difference between just displaying, and displaying in editable form... in which case, it would retrieve the "correct" text.

It would slow down writes, though, because two copies would have to be saved, and one of them would be after the raw message was parsed into HTML.
Back to top
drathbun
Board Member



Joined: 24 Jul 2008

Posts: 653
Location: Texas


flag
PostPosted: Thu Jan 22, 2009 11:11 pm 
Post subject: Re: Recomanded MODs

espicom wrote:
A different approach would be to add a BLOB field to the posts (or maybe the posts_text) table that has the parsed message text in it

That's what I had in mind, but long-text not blob since it's not binary. icon_smile.gif

Quote:
No additional file accesses, no need to make a world-writable directory, no extra queries, but a lot more SQL bandwidth

Yes, that's true, but most hosts don't measure SQL bandwidth. They only measure bandwidth as being delivered to the browser.

Quote:
Unless, of course, your read query was tailored to know the difference between just displaying, and displaying in editable form... in which case, it would retrieve the "correct" text.

Right, you would not need both versions. For viewtopic, get the parsed version. For posting (edits), get the unparsed version.

Quote:
It would slow down writes, though, because two copies would have to be saved, and one of them would be after the raw message was parsed into HTML.

But it's write-once read-many. icon_smile.gif The small performance hit on the write would be more than balanced out by the many many reads for the topic.

_________________
phpBBDoctor Blog
Back to top
Ptirhiik
Board Member



Joined: 19 Nov 2008

Posts: 114


flag
PostPosted: Fri Jan 23, 2009 7:21 am 
Post subject: Re: Recomanded MODs

notes:
- storing html means all your styles are using the same bbcode.tpl: that defeats the multi-template ability, unless you are generating a cache per style,
- stored html won't change if the parser changes: security patch won't be applied until the cache gets outdated,
- making the database growing is not necessarily the way to speed up things,
- generated html is always the same, what means you don't have to store the unparsed version of the post: you can recreate it for edit purpose,
- having a bunch of files (we are talking about hundred of thousands here) in a single directory is not the better way to reach them quickly, server side
Back to top
drathbun
Board Member



Joined: 24 Jul 2008

Posts: 653
Location: Texas


flag
PostPosted: Fri Jan 23, 2009 8:37 pm 
Post subject: Re: Recomanded MODs

Ptirhiik wrote:
- storing html means all your styles are using the same bbcode.tpl: that defeats the multi-template ability, unless you are generating a cache per style,

Fair point. I never considered this because I would never use more than one style. How often do styles change bold and italic formatting codes though?
Quote:
- stored html won't change if the parser changes: security patch won't be applied until the cache gets outdated,

This is easily done by setting an admin option to invalidate the cache. Posts could be recached on the first view.
Quote:
- making the database growing is not necessarily the way to speed up things,

I don't see this as a big issue. The posts are indexed uniquely by post_id. The post text table already allows a large amount of text. Storing more should not have a huge impact.
Quote:
- generated html is always the same, what means you don't have to store the unparsed version of the post: you can recreate it for edit purpose,

So you suggest doing a backwards parse to recreate bbcode during an edit? That's an interesting idea. Seems like it could work, but a lazy solution of having both available saves that step.
Quote:
- having a bunch of files (we are talking about hundred of thousands here) in a single directory is not the better way to reach them quickly, server side

Agreed. icon_smile.gif The attachment MOD has the same issue today. I am working on a way to split attachments into folders using a check-digit mechanism, but have never finished it. My attachment files folder has tens of thousands of files in it now, and it's not very effective.

_________________
phpBBDoctor Blog
Back to top
Ptirhiik
Board Member



Joined: 19 Nov 2008

Posts: 114


flag
PostPosted: Fri Jan 23, 2009 9:13 pm 
Post subject: Re: Recomanded MODs

drathbun wrote:
Quote:
- making the database growing is not necessarily the way to speed up things,

I don't see this as a big issue. The posts are indexed uniquely by post_id. The post text table already allows a large amount of text. Storing more should not have a huge impact.

Actually it has: I've seen a huge difference - confirmed by other users - when I dropped the search tables to the benefit of the full-text index in categories hierarchy: the overall database size was divide per two, and any access to whatever table was definitively quicker. I'm not sure how to explain that - disk occupation, buffers saturation, blocks dispersion - but the impact is there.
Back to top
espicom
Board Member



Joined: 24 Nov 2008

Posts: 55
Location: Woodstock, IL


flag
PostPosted: Fri Jan 23, 2009 10:52 pm 
Post subject: Re: Recomanded MODs

(I finally realized what I really dislike about most subforum systems I've seen... When a forum is tagged as having new messages, when it's really a subforum I'm not interested in reading that has the new messages. But, anyway...)

Attachment MOD and other things that involve files uploaded to the system are a pet peeve of mine, because of the "world writable directory" issue. Yes, you can solve that by putting it outside the web tree (assuming your host allows that) or by other methods, but the one I wanted to implement was a table-based solution. Store the files as data in the database, and nothing that is executable can be written to your file system. Again, SQL load, and database size issues, but that sort of things is negotiable with the host.

Ptirhiik, don't associate the change in dumping the search tables with the speed change - it was probably the overhead of maintaining the search table itself, rather than its size, that is the difference. When the SQL server is dealing with the indexes, it's running the show in a compiled language with direct access to the data, rather than as an interpreted process through an API...

I regularly work with tables with average row sizes of 38K, and table sizes in the 10-30GB range, with little speed degradation, assuming the indexes are up to date and contain the fields you're searching on. But, when the client decides to do search on a 15,000,000 record table using non-indexed WHERE clauses, that's their own fault. icon_wink.gif
Back to top
Ptirhiik
Board Member



Joined: 19 Nov 2008

Posts: 114


flag
PostPosted: Sat Jan 24, 2009 9:31 am 
Post subject: Re: Recomanded MODs

espicom wrote:
Ptirhiik, don't associate the change in dumping the search tables with the speed change - it was probably the overhead of maintaining the search table itself, rather than its size, that is the difference.
That's what I thought at the first place, so I did make some tests with dropping the full-text index, and the results were the same (except on search of course icon_smile.gif). I think there is an edge, quite low on regular share hoster, that makes this issue pops - that's why I considered buffers saturation as a potential explanation btw, but I'm not that good as dba to pin-point this as the very reason icon_wink.gif.
Back to top
Sylver Cheetah 53
Board Member



Joined: 17 Dec 2008

Posts: 426
Location: Milky Way


flag
PostPosted: Mon Jan 26, 2009 12:02 pm 
Post subject: Re: Recomanded MODs

Now I am confused. I was intending to install Cache Posts, but now... I do not know. icon_confused.gif
_________________
Image link
My Forum || My Blog

phpBB2 forever! icon_smile.gif
Back to top
Jim_UK
Board Member



Joined: 19 Nov 2008

Posts: 544
Location: North West UK


flag
PostPosted: Mon Jan 26, 2009 3:58 pm 
Post subject: Re: Recomanded MODs

After reading these posts I would still install it.
We have a saying "Proof of the pudding is in the eating"
Well we see posts/topics loading faster with the mod installed - what more can I say.

Install it and keep backups. If after a week or so testing you do not see a difference then you can uninstall it by replacing backups in about 1 minute.


Jim
Back to top
Sylver Cheetah 53
Board Member



Joined: 17 Dec 2008

Posts: 426
Location: Milky Way


flag
PostPosted: Mon Jan 26, 2009 7:44 pm 
Post subject: Re: Recomanded MODs

I will wait for a week or two to see what replyes will be, about this. icon_smile.gif Maybe someone will find something just like this, but better written. icon_wink.gif
_________________
Image link
My Forum || My Blog

phpBB2 forever! icon_smile.gif
Back to top
Dog Cow
Board Member



Joined: 18 Nov 2008

Posts: 378


flag
PostPosted: Mon Jan 26, 2009 8:13 pm 
Post subject: Re: Recomanded MODs

Ptirhiik wrote:
notes:
- making the database growing is not necessarily the way to speed up things,

Assuming this is a valid concern, (and I think that it is) table partitioning is always a valid option, and has been since the release of phpBB 2.0.0 wherein the posts table stores referential data, and the posts_text table stored the message.

Now we can take this a step further and make a third table which merely holds cached posts text. If we don't mind duplicating data, we can even throw in some elements from the posts table (such as user_id, topic_id) and adjust the query in viewtopic.php to select only from this table if a switch has been set.

This switch could be stored in the topics table, and could be either of two setups:
A.) Store a 0 if not all posts are cached, or a 1 if all posts are cached.

In this setup, the posts_cached table will be used ONLY if all posts in the topic have been cached.

B.) Stores the number of pages which have all posts cached, starting with page 0.

In this setup, we allow for the option of having some pages with un-cached posts. On these pages, we would do the two-table select. On pages where all posts are cached, we could do a (potentially) quicker select from just the posts_cached table. This option would only allow for one contiguous block of fully-cached pages, though.

Now here is the third option, and it is the one employed by vBulletin: One merely does a LEFT JOIN on the posts_cached table. Null rows are disregarded, we would have to parse the original post, but rows which have a cached post, we can load up straight away and skip the BBCode parser.

But where to store this table? Certainly, it can go anywhere we want! We shouldn't be limited to one database. It could go in its own database, and be further partitioned by topic ID, or even on a wholly separate db server.
Back to top
Sylver Cheetah 53
Board Member



Joined: 17 Dec 2008

Posts: 426
Location: Milky Way


flag
PostPosted: Tue Jan 27, 2009 11:04 am 
Post subject: Re: Recomanded MODs

Jim_UK wrote:
After reading these posts I would still install it.
We have a saying "Proof of the pudding is in the eating"

Well we see posts/topics loading faster with the mod installed - what more can I say.

Install it and keep backups. If after a week or so testing you do not see a difference then you can uninstall it by replacing backups in about 1 minute.


Jim

How much more faster? You'd say that at least two times faster? If it's not at least two times faster, I don't thing it worth it. icon_confused.gif

_________________
Image link
My Forum || My Blog

phpBB2 forever! icon_smile.gif
Back to top
drathbun
Board Member



Joined: 24 Jul 2008

Posts: 653
Location: Texas


flag
PostPosted: Tue Jan 27, 2009 2:40 pm 
Post subject: Re: Recomanded MODs

Rather than using a left join (outer join) I would simply store an empty row in the cached posts table. In some databases the overhead of a left join is simply not worth the code savings you might get.

Jim, here is my guess as to what you are experiencing. The cost savings you are getting is CPU time. You have an increase in disk access, but a decrease in CPU. As they say, there's no such thing as a free lunch. icon_smile.gif At this time the number of posts that you have cached is not overwhelming your disk, and the CPU is having to do less work. There are certain things that I think make sense to cache (the phpbb_config table is one such example, also theme / template information) and those files should quickly end up in a disk cache because everybody uses them. But my guess is that over time the caching of posts to individual files is going to run into overhead of its own.

I like the concept. I just think it could be improved. icon_smile.gif

_________________
phpBBDoctor Blog
Back to top
Display posts from previous:   
Register or Login to Post    Index » phpBB2 Discussion  Previous TopicPrint TopicNext Topic
Page 4 of 5 All times are GMT
Goto page Previous  1, 2, 3, 4, 5  Next
 
Jump to:  

Index • About • FAQ • Rules • Privacy • Search •  Register •  Login 
Not affiliated with or endorsed by the phpBB Group
Powered by phpBB2 © phpBB Group
Generated in 0.0817 seconds using 18 queries. (SQL 0.0017 Parse 0.0627 Other 0.0173)
phpBB Customizations by the phpBBDoctor.com
Template Design by DeLFlo and MomentsOfLight.com Moments of Light Logo