Member Avatar for SoulMazer

Hi, I'm writing a program that goes through all of the music in a user-specified directory and I am having some trouble with Unicode characters. More specifically, 'ascii' codec can't encode character u'\\xe7' . The song names need to both be printed to a HTML page as well as be passed to other HTML/CGI pages via the GET method.

What can I do to be more aware of these evil Unicode characters? I tried to simply convert to Unicode where I could in my program but then there comes the trouble of keeping this consistent among the databases and song names/albums/artists they are tied to. Also, I seemed to have trouble looking up things in a dictionary when they had inconsistent encoding.

What do all you intelligent people of the Python forum recommend?

  • 2 Contributors
  • 4 Replies
  • 136 Views
  • 7 Hours Discussion Span
  • Latest Post Latest Post by SoulMazer

Dani AI

Generated

A few practical followups tied to 's setup (scanning a music folder, rendering HTML, passing titles via GET) and the pointers from .

Keep text as Unicode internally, normalize it, and only encode at the I/O boundary. Filesystem and metadata can use different forms (macOS often uses decomposed forms), so normalizing before using strings as dictionary keys or DB keys avoids mismatch. Example normalization helper:

import unicodedata

def norm(s):
    return unicodedata.normalize('NFC', s)

When producing HTML or CGI output, always declare the charset and emit bytes encoded to that charset. For GET parameters, percent-encode UTF-8 bytes so browsers/servers agree. Typical patterns:

# Python 3
from urllib.parse import quote_plus
param = quote_plus(norm(title))
link = '/play?title=' + param
# Python 2
import urllib
param = urllib.quote_plus(norm(title).encode('utf-8'))
link = '/play?title=' + param

For storage and lookups: store a normalized form (and a case-folded variant if case-insensitive searches are required). Apply the same normalization function on both insert and query. Use parameterized queries so the DB driver receives the correct Python text type rather than relying on implicit conversions. Avoid mixing byte strings and Unicode for dict keys; always normalize the key before access.

Additional notes: inspect sys.getfilesystemencoding() if filenames look wrong, and include an explicit Content-Type header and a UTF-8 meta tag in generated HTML so browsers render accented characters correctly. For further background and pitfalls, see the Python Unicode HOWTO: Python Unicode HOWTO.

Recommended Answers

All 4 Replies

Member Avatar for SoulMazer

That was a great presentation. It took me a second to figure out I had to click to advance the slides.

Okay, then just one last question. Can I write to a database (via sqlite3) in Unicode? Or what should I do?

Thanks.

Member Avatar for SoulMazer

Perfect. Thanks Gribouillis. Problem solved.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.