Member Avatar for gishi

Hi Guys,

I was hoping someone can help me with this. I have a python program that reads and writes files in utf-8(unicode). When I run it in Eclipse, the output is perfectly fine. When I try making an exe file of my program it is not working. I also tried double clicking the python program to run it in windows dos but the same problem occurred. I tried running it in pythonWin still the same problem. Now I have a problem debugging my program since it is working perfectly fine in Eclipse. Is it a problem about encoding? I placed this:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

but it doesn't seem to work. hope someone can help me with this. I am reading and writing files in xml with utf-8/unicode characters

  • 3 Contributors
  • 4 Replies
  • 387 Views
  • 7 Hours Discussion Span
  • Latest Post Latest Post by TrustyTony

Dani AI

Generated

Short summary and practical fixes

Eclipse’s Python console and a Windows command window are using different runtime environments: the IDE usually presents a Unicode-aware console and uses your project/file encoding, while a plain cmd/packaged .exe uses the system console encoding and whatever the frozen runtime exposes. That mismatch explains why the same script looks fine inside Eclipse but breaks when run from the command prompt or as an .exe. (learn.microsoft.com)

Always use explicit encodings for files (XML included). Open and write text files with an explicit encoding instead of relying on defaults; in Python 3 pass an encoding to open(), and in environments where the builtin open lacks the param use io.open or the codecs helpers. Example (safe for both major versions when appropriate modules are used):

import io

with io.open('output.xml', 'w', encoding='utf-8') as f:
    f.write('some unicode text – characters as needed')

This avoids platform-dependent defaults. (docs.python.org)

Make console/stdout behavior explicit when you need to print to the Windows console or when you build an .exe. For short-term debugging set the interpreter’s stdio encoding (PYTHONIOENCODING) or, inside your program, rewrap stdout with a writer that encodes to the bytes your console expects:

import sys, io

sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8', errors='replace')

On Windows it’s also common to encounter different console code pages and fonts; changing the console code page or using a Unicode-capable terminal can help but is not a portable fix for distributed executables. (docs.python.org)

Troubleshooting checklist

  • Add a tiny reproducer that prints/sys.stdout.encoding and locale.getpreferredencoding().
  • Try redirecting output to a file and inspect that file with a Unicode-aware editor.
  • Ensure any XML you write includes a correct encoding declaration and that you open/write the bytes that match that declaration.

Refer to the Python Unicode HOWTO and the io/codecs docs for concrete patterns and for differences between Python 2 and 3. (docs.python.org)

Notes: this expands on ’s observation (works in Eclipse only), and follows the practical directions hinted at by and while focusing on fixes that don’t rely on one-off console settings.

Recommended Answers

All 4 Replies

Member Avatar for TrustyTony

And what code page have you set up for the windows dos? I found CHCP 1252 works best for me, before I thought I had to live with code page 850 or 437, but fortunately it is not true.

I have not tried how well the code page chcp 65001 (UTF-8) works yet though.

Edited by TrustyTony because: n/a
Member Avatar for gishi

Hi! How do you set the codepage for dos? If i do this will my program run when I make an exe file and use it using other's pc? My program is to read and write in xml. I need to look for characters that is not converted to their xml/html entities. It's now difficult to debug my program since it is doing fine in eclipse but not in dos or in exe. Do I need to write something at the top of my program to make windows read it that it has such encoding?


And what code page have you set up for the windows dos? I found CHCP 1252 works best for me, before I thought I had to live with code page 850 or 437, but fortunately it is not true.

I have not tried how well the code page chcp 65001 (UTF-8) works yet though.

Member Avatar for woooee

Eclipse has the fonts installed to print the encoding. The command line obviously does not. You can try to change the default font but that is dependent on which distro you are using as well as which terminal program (xterm is -fa font-name).

Edited by woooee because: n/a
Member Avatar for TrustyTony

I think encoding of your program to match what CHCP says in command line is enough. Code page 1252 makes scandinavian letter work from command line arguments and show correct in file name for DIR command. However, if I use GNU ls command in windows, those letters are missing. I hope that setting the code page in AUTOEXEC.BAT is enough, but I am not sure. I have only set it by hand and prove now to put there to see if the code page becomes default. If not, must see some windows settings and environment variables.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.