[silva-dev] Special Characters in Content from MySQL Database

[Martijn Faassen]

Timo Schuerg wrote:
>> Hello!
>> We're trying to move a Zope site to Silva. One of the pages aquires 
>> its contents from a MySql Database. With plain Zope this page works 
>> perfectly fine, all Umlauts and Special Characters get displayed 
>> properly.
>> If I try to do the same in the Silva framework I get the following 
>> error message:
>> 'ascii' codec can't decode byte 0xfc in position 11: ordinal not in 
>> range(128)
>> What confuses me: I only get this error message if I use
>> folder/get_title_or_id
>> to build my navigation bar. If I use
>> folder/id
>> the content from the SQL Method renders properly.
>> I came up with the following workaround: Instead of using
>> tal:content="publication/authors"
>> I use
>> tal:content="python:unicode(publication.authors,'Latin-1')"
>> With this trick I don't get an error message anymore, but the content 
>> still doesn't render correctly. All Umlauts get changed to "?".
>> Any Ideas?

I assume publication.authors is a string that's coming from our MySQL 
database.

What is going on is that Silva works with unicode strings (including 
what get_title_or_id delivers), and your MySQL database delivers encoded 
content (probably indeed latin-1). When Python tries to combine the two 
it won't be able to do so and gives the error.

The 'unicode()' call you list there *should* make the system work again 
(as long as you turn any output from the database into unicode), but 
it's possible your browser for some reason does not detect that your 
page template is delivering UTF-8, causing it to display question marks 
whenever a non-ascii character comes up. It would be nicer if making the 
database deliver unicode could be done on a lower level but I'm not sure 
whether that's possible with SQL Methods and MySQL...

You might want to try changing the encoding setting to UTF-8 in your 
browser manually and see whether the issue with the question marks goes 
away. If it does, you should do something similar to what Silva does for 
its own templates - call a Python script from your page template that 
does the following:

context.REQUEST.RESPONSE.setHeader('Content-Type', 
'text/html;charset=utf-8')

that will tell the browser that pages are indeed delivered as UTF-8.

If that doesn't work, we'll have to dig deeper though.

Regards,

Martijn