Welcome to all phpBB2 Refugees!This site is intended to continue support for the legacy 2.x line of the phpBB2 bulletin board package. If you are a fan of phpBB2, please, by all means register, post, and help us out by offering your suggestions. We are primarily a community and support network. Our secondary goal is to provide a phpBB2 MOD Author and Styles area. |
|
Author |
Message |
asavage Board Member
Joined: 21 May 2013
Posts: 12
|
Posted: Tue May 21, 2013 3:39 pm Post subject: Degree symbol prevents Preview |
|
|
I've run a phpBB2 board since 2006. Most of that time, it was 2.0.19, Apache 1.3.41, php 4.1.17, and mysql 4
Recently, I revamped everything: new hardware, 2.0.23, Apache 2.2.24, Apache, php 5.4.15, mysql 5.1.68,
For the forum software, I have archived checkpoints at every MOD added, so I can easily roll back the changes. I do not roll back the database, though.
I just found a niggling issue.
The degree symbol (<Alt+248> for PCs): "°"
(Windoze: you may have to create that symbol using, eg Notepad (hold down the <Alt> key, tap "248" using the [u]number pad only/u] (not the upper row of keys), then release the <Alt> key), then cut-n-paste to your browser; or use the Character Map, but I don't know which font etc. to use if you go that way.)
I cannot Preview a post which contains that character. The post text displays empty, both in the upper preview pane and in the lower compose pane.
I can Submit a post that contains "°" but not Preview. It does not matter if I create a new post, or edit an old one, the behavior is the same. I have lots of legacy posts that use the degree symbol, and they cannot be edited and previewed (though we can edit them and Submit, then edit again if necessary).
I've tried looking at the code, but it's just a hair too complex for me to follow.
I've tested with the files from my old pre-upgrade install of 2.0.19 and it's the same. I can only conclude that the issue is with either mysqld or php.
Any thoughts? |
|
Back to top |
|
|
Salvatos Board Member
Joined: 19 Feb 2009
Posts: 449 Location: Québec
|
Posted: Tue May 21, 2013 4:51 pm Post subject: Re: Degree symbol prevents Preview |
|
|
It works on my forum using the same versions except for PHP 5.3.24. I'd check your encodings first, though. Are you using the right encodings for your database and your files? |
|
Back to top |
|
|
asavage Board Member
Joined: 21 May 2013
Posts: 12
|
Posted: Tue May 21, 2013 4:56 pm Post subject: Re: Degree symbol prevents Preview |
|
|
Even with incorrect encodings, I would expect text other than the degree symbol to display. What is happening is that if a degree symbol is anywhere in the post test, then Preview is selected, NO post text is shown.
I do not mean a blank page, I mean that the page displayed after Preview is selected has none of the post text, only the template.
I'm not certain how to check the encodings to which you refer. |
|
Back to top |
|
|
dogs and things Board Member
Joined: 18 Nov 2008
Posts: 628 Location: Spain
|
Posted: Tue May 21, 2013 4:58 pm Post subject: Re: Degree symbol prevents Preview |
|
|
On my board (2.0.23) I don´t have this problem, I can preview a post with the degree symbol °.
In this board here the preview works fine too with the degree symbol included.
Seems to me like your problem is due to some MOD that altered a file related with the preview function. I am not an expert but I guess involved include mainly /includes/topic_review.php and includes/functions_post.php. _________________ phpBB2 will never die, I hope! |
|
Back to top |
|
|
asavage Board Member
Joined: 21 May 2013
Posts: 12
|
Posted: Tue May 21, 2013 5:39 pm Post subject: Re: Degree symbol prevents Preview |
|
|
A MOD fooling around was my first thought, too. But I rolled back all the files in the /phpb2 tree right back to the fresh install of .23 and no change.
I know it used to work on my .19 setup, which I archived. But when I rolled back to even those files, the problem persists.
My thinking was that since the problem persisted even with old forum files, it had to be the newer php or mysql, but if you folks are using those newer bits, it must be something I've missed.
Maybe I didn't roll back the way I thought. I'll hammer at it some more tomorrow. |
|
Back to top |
|
|
asavage Board Member
Joined: 21 May 2013
Posts: 12
|
Posted: Wed May 22, 2013 4:20 pm Post subject: Re: Degree symbol prevents Preview |
|
|
Salvatos wrote: | It works on my forum using the same versions except for PHP 5.3.24. |
PHP 5.3 still allowed register_globals, register_long_arrays. PHP 5.4 has those options removed.
Do you have those options ON? |
|
Back to top |
|
|
asavage Board Member
Joined: 21 May 2013
Posts: 12
|
Posted: Wed May 22, 2013 4:44 pm Post subject: Re: Degree symbol prevents Preview |
|
|
I changed PHP from 5.4.15 to 5.3.20, and the problem disappeared.
[later]
Though PHP 5.3.20 is not stable here, so I've further rolled PHP back to 5.3.4 now. |
|
Back to top |
|
|
Salvatos Board Member
Joined: 19 Feb 2009
Posts: 449 Location: Québec
|
Posted: Wed May 22, 2013 6:33 pm Post subject: Re: Degree symbol prevents Preview |
|
|
asavage wrote: | PHP 5.3 still allowed register_globals, register_long_arrays. PHP 5.4 has those options removed.
Do you have those options ON? |
register_globals is off, register_long_arrays is on.
Edit: Oh, judging by the code we looked at last week for achaab, I can see how that's going to be an issue. I wonder how much of phpBB2 relies on long arrays...
I'm not seeing a relation with the degree symbol, though? |
|
Back to top |
|
|
asavage Board Member
Joined: 21 May 2013
Posts: 12
|
Posted: Wed May 22, 2013 6:50 pm Post subject: Re: Degree symbol prevents Preview |
|
|
Quote: | register_globals is off, register_long_arrays is on.
Oh, judging by the code we looked at last week for achaab, I can see how that's going to be an issue. I wonder how much of phpBB2 relies on long arrays... |
I am now running PHP 5.3.4 (and was, briefly, with .20) with
register_long_arrays = off
register_globals = Off
and ° previews & posts OK. There is some other interaction with PHP 5.4 going on. |
|
Back to top |
|
|
asavage Board Member
Joined: 21 May 2013
Posts: 12
|
Posted: Thu May 23, 2013 8:14 pm Post subject: Re: Degree symbol prevents Preview |
|
|
This problem appears to be caused by a non-backward-compatible default encoding change to htmlspecialchars().
php manual wrote: | Encoding
Defines encoding used in conversion. If omitted, the default value for this argument is ISO-8859-1 in versions of PHP prior to 5.4.0, and UTF-8 from PHP 5.4.0 onwards. |
I learned about this in a comment to the Backward Incompatible Changes from 5.3 to 5.4 page.
Chris wrote: | Missing some chars like german umlauts after use of htmlspecialchars? That's because the third param encoding has changed it's default value in PHP 5.4 from ISO-8859-1 to UTF-8. [ . . . ] |
Test1.php for php 5.3.4: Code: | test1.php
<?php
echo "Line 1: \"" . htmlspecialchars( 'äöü' ) . "\"\r\n";
echo "Line 2: \"" . htmlspecialchars( 'test °' ) . "\"\r\n";
?> |
Quote: | [c:\temp2] f:\apache2\bin\php -v
PHP 5.3.4 (cli) (built: Dec 20 2010 19:17:51)
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2010 Zend Technologies
[c:\temp2] f:\apache2\bin\php test1.php
Line 1: "äöü"
Line 2: "test °" |
Test2.php for php 5.4.15: Code: | <?php
echo "Line 1: \"" . htmlspecialchars( 'äöü' ) . "\"\r\n";
echo "Line 2: \"" . htmlspecialchars( 'test °' ) . "\"\r\n";
echo "Line 3: \"" . htmlspecialchars( 'äöü' , ENT_HTML401, 'ISO-8859-1') . "\"\r\n";
echo "Line 4: \"" . htmlspecialchars( 'test °' , ENT_HTML401, 'ISO-8859-1') . "\"\r\n";
?> |
Quote: | c:\temp2] php -v
PHP 5.4.10 (cli) (built: Dec 23 2012 16:52:15)
Copyright (c) 1997-2012 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2013 Zend Technologies
[c:\temp2] f:\apache2\bin\php test2.php
Line 1: ""
Line 2: ""
Line 3: "äöü"
Line 4: "test °" |
The first two lines in test1.php echo fine OK under 5.3 but nothing under 5.4 . I added modified lines 3 & 4 to test2.php, and htmlspecialchars() gives output again.
Changing every call of htmlspecialchar() and adding the second & third parameters might fix my problem. However, with EasyMOD 0.4.0 installed, that's around 202 instances to change. |
|
Back to top |
|
|
Salvatos Board Member
Joined: 19 Feb 2009
Posts: 449 Location: Québec
|
Posted: Thu May 23, 2013 11:25 pm Post subject: Re: Degree symbol prevents Preview |
|
|
Would it be possible to change your db encoding to UTF-8 instead? |
|
Back to top |
|
|
asavage Board Member
Joined: 21 May 2013
Posts: 12
|
Posted: Fri May 24, 2013 12:36 am Post subject: Re: Degree symbol prevents Preview |
|
|
If you look at my test files above, you can see that the problem is with the htmlspecialchars() function only. The test fails for old use of htmlspecialchars() without the second & third parameters to tell it (the function) to use ISO-8859-1 encoding instead of its new default of UTF-8.
The database that php might send data to later has nothing to do with this particular problem, and changing the charset on a table or entire database will not change that.
[There's a setting in php.ini of
default_charset = "iso-8859-1"
It is commented out in my php.ini . I un-commented it, restarted httpd, and re-ran those tests, with no change. [later] It turns out that that directive only affects the content-type header of output, so it won't help this situation.]
As Chris mentioned in his comment, there are at least two feasible ways of working around this:- Edit the 202 instances (in 31 files) of htmlspecialchars() and add the 2nd & 3rd parameters:
htmlspecialchars(xxxxxxxxxxxx, ENT_COMPAT | ENT_HTML401 , 'ISO-8859-1' ); - Write a wrapper function that adds those two parameters, and then batch-edit each of those 202 htmlspecialchars() calls to change the function call to your custom function.
Is the problem large enough to warrant doing this? For folks that use characters that are affected by this change in that function, and who need to use php 5.4.x, perhaps. |
|
Back to top |
|
|
StarWolf3000 Board Member
Joined: 10 Jun 2010
Posts: 175 Location: Germany
|
Posted: Sat May 25, 2013 7:22 am Post subject: Re: Degree symbol prevents Preview |
|
|
I rewrote some sections of a not-installed and unmodified 2.0.23 package (only for german language and usage at the moment) to fully support UTF-8. By default this consists of modifying:
- admin/page_header_admin.php
- include/page_header.php
- install/schemas/mysql_basic.sql (ANSI to UTF-8 w/o BOM)
- install/schemas/mysql_schema.sql (ANSI to UTF-8 w/o BOM)
- language/lang_*/lang_admin.php (ANSI to UTF-8 w/o BOM)
- language/lang_*/lang_bbcode.php (ANSI to UTF-8 w/o BOM)
- language/lang_*/lang_faq.php (ANSI to UTF-8 w/o BOM)
- language/lang_*/lang_main.php (ANSI to UTF-8 w/o BOM and setting $lang['ENCODING'] to UTF-8)
- language/lang_*/lang_*.php (ANSI to UTF-8 w/o BOM)
This requires MySQL 4.1 or higher and the usage of utf8 as charset and collation = utf8_unicode_ci, utf8_general_ci or utf8_bin.
Sections and files to rewrite:
Code: | #
#-----[ OPEN ]------------------------------------------------
#
admin/page_header_admin.php
#
#-----[ FIND ]------------------------------------------------
#
$templates->assign_vars(array(
#
#-----[ BEFORE, ADD ]-----------------------------------------
#
// Force Encoding from lang-Files
header("Content-type: text/html; charset=" . $lang['ENCODING']);
#
#-----[ OPEN ]------------------------------------------------
#
includes/page_header.php
#
#-----[ FIND ]------------------------------------------------
#
$templates->assign_vars(array(
#
#-----[ BEFORE, ADD ]-----------------------------------------
#
// Force Encoding from lang-Files
header("Content-type: text/html; charset=" . $lang['ENCODING']);
#
#-----[ OPEN ]------------------------------------------------
#
language/lang_*/lang_main.php
#
#-----[ FIND ]------------------------------------------------
# Caution: This depends on the language. Non-latin will not use ISO-8859-1 but SHIFT_JIS,
# other ISO-8859-X or some like BIG5
$lang['ENCODING'] = 'iso-8859-1';
#
#-----[ REPLACE WITH ]------------------------------------------------
#
$lang['ENCODING'] = 'utf-8'; |
The ANSI to UTF-8 w/o BOM conversion requires a suitable editor like Notepad++ ("Encoding" -> "Convert to UTF-8 without BOM"). |
|
Back to top |
|
|
asavage Board Member
Joined: 21 May 2013
Posts: 12
|
Posted: Tue May 28, 2013 11:31 am Post subject: Re: Degree symbol prevents Preview |
|
|
While I admire the effort to prepare for UTF-8, this will not help with the problem of "no text returned by function" when the function encounters a new or existing degree symbol. |
|
Back to top |
|
|
asavage Board Member
Joined: 21 May 2013
Posts: 12
|
Posted: Tue May 28, 2013 7:23 pm Post subject: Re: Degree symbol prevents Preview |
|
|
I think I've got this licked, at least for my own setup.
- I created a wrapper function, and put it in \common.php .
- Then I renamed all (202) instances of calls to function htmlspecialchars() with htmlspecialchars_wrapped(), using sed.
That's it. Nothing else was required, AFAICT.
I added this to the bottom of \common.php:
========================================
//
// 28May2013 Al Savage (asavage@iname.com)
// phpBB2 was written using calls to htmlspecialchars() that omitted the 2nd & 3rd parameters,
// as -- in versions of php <=5.4 -- the default for those were fine (particularly the 3rd:
// charset encoding default: ISO-8859-1 ("Latin").
//
// Beginning with php v5.4.x, the default character encoding changed, from ISO-8859-1 to UTF-8.
// Problem: if any invalid UTF-8 sequence (degree symbol, umlaut, etc.) are fed to
// htmlspecialchars() (now defaulting to UTF-8 charset), it discards the entire string
// (see https://bugs.php.net/bug.php?id=43896).
//
// The wrapper function below circumvents this problem by creating a wrapper function.
// Each call to htmlspecialchars() must be renamed to "htmlspecialchars_wrapped()" to use the
// new function below. The new function inserts the 2nd & 3rd parameters, if they are not present.
//
// In a better universe, these constants would be in the database. For this version,
// change the constants below (CHARSET, ENT_COMPAT | ENT_XHTML) if needed to suit
// your environment.
//
define('CHARSET', 'ISO-8859-1');
define('REPLACE_FLAGS', ENT_COMPAT | ENT_XHTML);
function htmlspecialchars_wrapped($str, $ent=REPLACE_FLAGS, $charset=CHARSET) {
return htmlspecialchars ( $str , $ent, $charset);
}
========================================
Next, recursively edit *.php files and replace occurrences of "htmlspecialchars(" with "htmlspecialchars_wrapped(".
[Because I run Apache under eComStation, I use 4DOS and the tools available on that platform. I ran this from a 4OS2 command prompt in the phpBB2 directory:
for /r %x in (*.php) do sed -i -e "s/htmlspecialchars(/htmlspecialchars_wrapped(/g" %x
/r = recurse directories
-i = change files in place
-e = expression follows (often an optional argument).
Naturally, this would be entirely different for a *nix system. ]
Probably something like this would be appropriate, though many other methods of search & replace are available:
find . -type f -print0 | xargs -0 sed -i -e 's/htmlspecialchars(/htmlspecialchars_wrapped(/g'
I have (20) MODs installed over 2.0.23, (96) *.php files, of which (31) contained "htmlspecialchars(". In under fifteen seconds, they were all modified to suit.
So far, I am pleased with the result. I can edit existing posts that contain a degree symbol ("°") and Preview works as expected. |
|
Back to top |
|
|
|
Not affiliated with or endorsed by the phpBB Group
Powered by phpBB2 © phpBB Group
Generated in 0.0760 seconds using 16 queries. (SQL 0.0161 Parse 0.0014 Other 0.0585) |
phpBB Customizations by the phpBBDoctor.com
Template Design by DeLFlo and MomentsOfLight.com
|
|