phpBB2Refugees.com Logo
Not affiliated with or endorsed by the phpBB Group

Register •  Login 

Continue the legacy...

Welcome to all phpBB2 Refugees!Wave Smilie

This site is intended to continue support for the legacy 2.x line of the phpBB2 bulletin board package. If you are a fan of phpBB2, please, by all means register, post, and help us out by offering your suggestions. We are primarily a community and support network. Our secondary goal is to provide a phpBB2 MOD Author and Styles area.

Degree symbol prevents Preview

Goto page 1, 2  Next
 
Search this topic... | Search phpBB2 Discussion... | Search Box
Register or Login to Post    Index » phpBB2 Discussion  Previous TopicPrint TopicNext Topic
Author Message
asavage
Board Member



Joined: 21 May 2013

Posts: 12



PostPosted: Tue May 21, 2013 7:39 pm 
Post subject: Degree symbol prevents Preview

I've run a phpBB2 board since 2006. Most of that time, it was 2.0.19, Apache 1.3.41, php 4.1.17, and mysql 4

Recently, I revamped everything: new hardware, 2.0.23, Apache 2.2.24, Apache, php 5.4.15, mysql 5.1.68,

For the forum software, I have archived checkpoints at every MOD added, so I can easily roll back the changes. I do not roll back the database, though.

I just found a niggling issue.

The degree symbol (<Alt+248> for PCs): ""
(Windoze: you may have to create that symbol using, eg Notepad (hold down the <Alt> key, tap "248" using the [u]number pad only/u] (not the upper row of keys), then release the <Alt> key), then cut-n-paste to your browser; or use the Character Map, but I don't know which font etc. to use if you go that way.)

I cannot Preview a post which contains that character. The post text displays empty, both in the upper preview pane and in the lower compose pane.

I can Submit a post that contains "" but not Preview. It does not matter if I create a new post, or edit an old one, the behavior is the same. I have lots of legacy posts that use the degree symbol, and they cannot be edited and previewed (though we can edit them and Submit, then edit again if necessary).

I've tried looking at the code, but it's just a hair too complex for me to follow.

I've tested with the files from my old pre-upgrade install of 2.0.19 and it's the same. I can only conclude that the issue is with either mysqld or php.

Any thoughts?
Back to top
Salvatos
Board Member



Joined: 19 Feb 2009

Posts: 418
Location: Qubec


flag
PostPosted: Tue May 21, 2013 8:51 pm 
Post subject: Re: Degree symbol prevents Preview

It works on my forum using the same versions except for PHP 5.3.24. I'd check your encodings first, though. Are you using the right encodings for your database and your files?
Back to top
asavage
Board Member



Joined: 21 May 2013

Posts: 12



PostPosted: Tue May 21, 2013 8:56 pm 
Post subject: Re: Degree symbol prevents Preview

Even with incorrect encodings, I would expect text other than the degree symbol to display. What is happening is that if a degree symbol is anywhere in the post test, then Preview is selected, NO post text is shown.

I do not mean a blank page, I mean that the page displayed after Preview is selected has none of the post text, only the template.

I'm not certain how to check the encodings to which you refer.
Back to top
dogs and things
Board Member



Joined: 18 Nov 2008

Posts: 621
Location: Spain


flag
PostPosted: Tue May 21, 2013 8:58 pm 
Post subject: Re: Degree symbol prevents Preview

On my board (2.0.23) I dont have this problem, I can preview a post with the degree symbol .

In this board here the preview works fine too with the degree symbol included.

Seems to me like your problem is due to some MOD that altered a file related with the preview function. I am not an expert but I guess involved include mainly /includes/topic_review.php and includes/functions_post.php.

_________________
phpBB2 will never die, I hope!
Back to top
asavage
Board Member



Joined: 21 May 2013

Posts: 12



PostPosted: Tue May 21, 2013 9:39 pm 
Post subject: Re: Degree symbol prevents Preview

A MOD fooling around was my first thought, too. But I rolled back all the files in the /phpb2 tree right back to the fresh install of .23 and no change. icon_sad.gif

I know it used to work on my .19 setup, which I archived. But when I rolled back to even those files, the problem persists.

My thinking was that since the problem persisted even with old forum files, it had to be the newer php or mysql, but if you folks are using those newer bits, it must be something I've missed.

Maybe I didn't roll back the way I thought. I'll hammer at it some more tomorrow.
Back to top
asavage
Board Member



Joined: 21 May 2013

Posts: 12



PostPosted: Wed May 22, 2013 8:20 pm 
Post subject: Re: Degree symbol prevents Preview

Salvatos wrote:
It works on my forum using the same versions except for PHP 5.3.24.


PHP 5.3 still allowed register_globals, register_long_arrays. PHP 5.4 has those options removed.

Do you have those options ON?
Back to top
asavage
Board Member



Joined: 21 May 2013

Posts: 12



PostPosted: Wed May 22, 2013 8:44 pm 
Post subject: Re: Degree symbol prevents Preview

I changed PHP from 5.4.15 to 5.3.20, and the problem disappeared.

[later]

Though PHP 5.3.20 is not stable here, so I've further rolled PHP back to 5.3.4 now.
Back to top
Salvatos
Board Member



Joined: 19 Feb 2009

Posts: 418
Location: Qubec


flag
PostPosted: Wed May 22, 2013 10:33 pm 
Post subject: Re: Degree symbol prevents Preview

asavage wrote:
PHP 5.3 still allowed register_globals, register_long_arrays. PHP 5.4 has those options removed.

Do you have those options ON?

register_globals is off, register_long_arrays is on.

Edit: Oh, judging by the code we looked at last week for achaab, I can see how that's going to be an issue. I wonder how much of phpBB2 relies on long arrays...

I'm not seeing a relation with the degree symbol, though?
Back to top
asavage
Board Member



Joined: 21 May 2013

Posts: 12



PostPosted: Wed May 22, 2013 10:50 pm 
Post subject: Re: Degree symbol prevents Preview

Quote:
register_globals is off, register_long_arrays is on.

Oh, judging by the code we looked at last week for achaab, I can see how that's going to be an issue. I wonder how much of phpBB2 relies on long arrays...


I am now running PHP 5.3.4 (and was, briefly, with .20) with

register_long_arrays = off
register_globals = Off

and previews & posts OK. There is some other interaction with PHP 5.4 going on.
Back to top
asavage
Board Member



Joined: 21 May 2013

Posts: 12



PostPosted: Fri May 24, 2013 12:14 am 
Post subject: Re: Degree symbol prevents Preview

This problem appears to be caused by a non-backward-compatible default encoding change to htmlspecialchars().
php manual wrote:
Encoding

Defines encoding used in conversion. If omitted, the default value for this argument is ISO-8859-1 in versions of PHP prior to 5.4.0, and UTF-8 from PHP 5.4.0 onwards.


I learned about this in a comment to the Backward Incompatible Changes from 5.3 to 5.4 page.
Chris wrote:
Missing some chars like german umlauts after use of htmlspecialchars? That's because the third param encoding has changed it's default value in PHP 5.4 from ISO-8859-1 to UTF-8. [ . . . ]


Test1.php for php 5.3.4:
Code:
test1.php
<?php
echo "Line 1: \"" . htmlspecialchars( ''    ) . "\"\r\n";
echo "Line 2: \"" . htmlspecialchars( 'test ' ) . "\"\r\n";
?>

Quote:
[c:\temp2] f:\apache2\bin\php -v
PHP 5.3.4 (cli) (built: Dec 20 2010 19:17:51)
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2010 Zend Technologies

[c:\temp2] f:\apache2\bin\php test1.php
Line 1: ""
Line 2: "test "


Test2.php for php 5.4.15:
Code:
<?php
echo "Line 1: \"" . htmlspecialchars( ''                                ) . "\"\r\n";
echo "Line 2: \"" . htmlspecialchars( 'test '                             )  . "\"\r\n";
echo "Line 3: \"" . htmlspecialchars( ''     , ENT_HTML401, 'ISO-8859-1') . "\"\r\n";
echo "Line 4: \"" . htmlspecialchars( 'test '  , ENT_HTML401, 'ISO-8859-1') . "\"\r\n";
?>

Quote:
c:\temp2] php -v
PHP 5.4.10 (cli) (built: Dec 23 2012 16:52:15)
Copyright (c) 1997-2012 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2013 Zend Technologies

[c:\temp2] f:\apache2\bin\php test2.php
Line 1: ""
Line 2: ""
Line 3: ""
Line 4: "test "


The first two lines in test1.php echo fine OK under 5.3 but nothing under 5.4 . I added modified lines 3 & 4 to test2.php, and htmlspecialchars() gives output again.

Changing every call of htmlspecialchar() and adding the second & third parameters might fix my problem. However, with EasyMOD 0.4.0 installed, that's around 202 instances to change.
Back to top
Salvatos
Board Member



Joined: 19 Feb 2009

Posts: 418
Location: Qubec


flag
PostPosted: Fri May 24, 2013 3:25 am 
Post subject: Re: Degree symbol prevents Preview

Would it be possible to change your db encoding to UTF-8 instead?
Back to top
asavage
Board Member



Joined: 21 May 2013

Posts: 12



PostPosted: Fri May 24, 2013 4:36 am 
Post subject: Re: Degree symbol prevents Preview

If you look at my test files above, you can see that the problem is with the htmlspecialchars() function only. The test fails for old use of htmlspecialchars() without the second & third parameters to tell it (the function) to use ISO-8859-1 encoding instead of its new default of UTF-8.

The database that php might send data to later has nothing to do with this particular problem, and changing the charset on a table or entire database will not change that.

[There's a setting in php.ini of

default_charset = "iso-8859-1"

It is commented out in my php.ini . I un-commented it, restarted httpd, and re-ran those tests, with no change. [later] It turns out that that directive only affects the content-type header of output, so it won't help this situation.]

As Chris mentioned in his comment, there are at least two feasible ways of working around this:
  • Edit the 202 instances (in 31 files) of htmlspecialchars() and add the 2nd & 3rd parameters:
    htmlspecialchars(xxxxxxxxxxxx, ENT_COMPAT | ENT_HTML401 , 'ISO-8859-1' );
  • Write a wrapper function that adds those two parameters, and then batch-edit each of those 202 htmlspecialchars() calls to change the function call to your custom function.
Is the problem large enough to warrant doing this? For folks that use characters that are affected by this change in that function, and who need to use php 5.4.x, perhaps.
Back to top
StarWolf3000
Board Member



Joined: 10 Jun 2010

Posts: 93
Location: Germany


flag
PostPosted: Sat May 25, 2013 11:22 am 
Post subject: Re: Degree symbol prevents Preview

I rewrote some sections of a not-installed and unmodified 2.0.23 package (only for german language and usage at the moment) to fully support UTF-8. By default this consists of modifying:

  • admin/page_header_admin.php
  • include/page_header.php
  • install/schemas/mysql_basic.sql (ANSI to UTF-8 w/o BOM)
  • install/schemas/mysql_schema.sql (ANSI to UTF-8 w/o BOM)
  • language/lang_*/lang_admin.php (ANSI to UTF-8 w/o BOM)
  • language/lang_*/lang_bbcode.php (ANSI to UTF-8 w/o BOM)
  • language/lang_*/lang_faq.php (ANSI to UTF-8 w/o BOM)
  • language/lang_*/lang_main.php (ANSI to UTF-8 w/o BOM and setting $lang['ENCODING'] to UTF-8)
  • language/lang_*/lang_*.php (ANSI to UTF-8 w/o BOM)

This requires MySQL 4.1 or higher and the usage of utf8 as charset and collation = utf8_unicode_ci, utf8_general_ci or utf8_bin.

Sections and files to rewrite:
Code:
#
#-----[ OPEN ]------------------------------------------------
#
admin/page_header_admin.php

#
#-----[ FIND ]------------------------------------------------
#
$templates->assign_vars(array(

#
#-----[ BEFORE, ADD ]-----------------------------------------
#
// Force Encoding from lang-Files
header("Content-type: text/html; charset=" . $lang['ENCODING']);

#
#-----[ OPEN ]------------------------------------------------
#
includes/page_header.php

#
#-----[ FIND ]------------------------------------------------
#
$templates->assign_vars(array(

#
#-----[ BEFORE, ADD ]-----------------------------------------
#
// Force Encoding from lang-Files
header("Content-type: text/html; charset=" . $lang['ENCODING']);

#
#-----[ OPEN ]------------------------------------------------
#
language/lang_*/lang_main.php

#
#-----[ FIND ]------------------------------------------------
# Caution: This depends on the language. Non-latin will not use ISO-8859-1 but SHIFT_JIS,
# other ISO-8859-X or some like BIG5
$lang['ENCODING'] = 'iso-8859-1';

#
#-----[ REPLACE WITH ]------------------------------------------------
#
$lang['ENCODING'] = 'utf-8';


The ANSI to UTF-8 w/o BOM conversion requires a suitable editor like Notepad++ ("Encoding" -> "Convert to UTF-8 without BOM").
Back to top
asavage
Board Member



Joined: 21 May 2013

Posts: 12



PostPosted: Tue May 28, 2013 3:31 pm 
Post subject: Re: Degree symbol prevents Preview

While I admire the effort to prepare for UTF-8, this will not help with the problem of "no text returned by function" when the function encounters a new or existing degree symbol.
Back to top
asavage
Board Member



Joined: 21 May 2013

Posts: 12



PostPosted: Tue May 28, 2013 11:23 pm 
Post subject: Re: Degree symbol prevents Preview

I think I've got this licked, at least for my own setup.
  • I created a wrapper function, and put it in \common.php .
  • Then I renamed all (202) instances of calls to function htmlspecialchars() with htmlspecialchars_wrapped(), using sed.
That's it. Nothing else was required, AFAICT.

I added this to the bottom of \common.php:

========================================
//
// 28May2013 Al Savage (asavage@iname.com)
// phpBB2 was written using calls to htmlspecialchars() that omitted the 2nd & 3rd parameters,
// as -- in versions of php <=5.4 -- the default for those were fine (particularly the 3rd:
// charset encoding default: ISO-8859-1 ("Latin").
//
// Beginning with php v5.4.x, the default character encoding changed, from ISO-8859-1 to UTF-8.
// Problem: if any invalid UTF-8 sequence (degree symbol, umlaut, etc.) are fed to
// htmlspecialchars() (now defaulting to UTF-8 charset), it discards the entire string
// (see https://bugs.php.net/bug.php?id=43896).
//
// The wrapper function below circumvents this problem by creating a wrapper function.
// Each call to htmlspecialchars() must be renamed to "htmlspecialchars_wrapped()" to use the
// new function below. The new function inserts the 2nd & 3rd parameters, if they are not present.
//
// In a better universe, these constants would be in the database. For this version,
// change the constants below (CHARSET, ENT_COMPAT | ENT_XHTML) if needed to suit
// your environment.
//
define('CHARSET', 'ISO-8859-1');
define('REPLACE_FLAGS', ENT_COMPAT | ENT_XHTML);

function htmlspecialchars_wrapped($str, $ent=REPLACE_FLAGS, $charset=CHARSET) {
return htmlspecialchars ( $str , $ent, $charset);
}

========================================

Next, recursively edit *.php files and replace occurrences of "htmlspecialchars(" with "htmlspecialchars_wrapped(".

[Because I run Apache under eComStation, I use 4DOS and the tools available on that platform. I ran this from a 4OS2 command prompt in the phpBB2 directory:

for /r %x in (*.php) do sed -i -e "s/htmlspecialchars(/htmlspecialchars_wrapped(/g" %x

/r = recurse directories
-i = change files in place
-e = expression follows (often an optional argument).

Naturally, this would be entirely different for a *nix system. ]

Probably something like this would be appropriate, though many other methods of search & replace are available:

find . -type f -print0 | xargs -0 sed -i -e 's/htmlspecialchars(/htmlspecialchars_wrapped(/g'

I have (20) MODs installed over 2.0.23, (96) *.php files, of which (31) contained "htmlspecialchars(". In under fifteen seconds, they were all modified to suit.

So far, I am pleased with the result. I can edit existing posts that contain a degree symbol ("") and Preview works as expected.
Back to top
Display posts from previous:   
Register or Login to Post    Index » phpBB2 Discussion  Previous TopicPrint TopicNext Topic
Page 1 of 2 All times are GMT
Goto page 1, 2  Next
 
Jump to:  

Index • About • FAQ • Rules • Privacy • Search •  Register •  Login 
Not affiliated with or endorsed by the phpBB Group
Powered by phpBB2 © phpBB Group
Generated in 0.0313 seconds using 15 queries. (SQL 0.0024 Parse 0.0010 Other 0.0279)
phpBB Customizations by the phpBBDoctor.com
Template Design by DeLFlo and MomentsOfLight.com Moments of Light Logo