Member Avatar for Mike Askew

Quite rare to see me post in the Java forum but have been picking it up as part of a long-distance university course I am doing.

I am currently trying to read in an ASCII coded .txt file and output it but am seeing some funny characters showing up.

The content is loaded using

        BufferedReader inputReader;

        System.out.println("Loading book...");

        try {
            inputReader = new BufferedReader(new InputStreamReader(new FileInputStream(fileToRead), "ASCII"));

Which I then split on the new page character into book "pages".

When I come to print these out to console using

        try {
            PrintStream output = new PrintStream(System.out, true, "ASCII");

            for (String theWord : words) {

                output.println(theWord);
            }
        }

This: Sorrow came—a gentle sorrow—but not at all in the shape of any disagreeable
Comes as: Sorrow came???a gentle sorrow???but not at all in the shape of any disagreeable

NOTE: The dash is an em-dash

Edited by Mike Askew
  • 2 Contributors
  • 8 Replies
  • 187 Views
  • 1 Day Discussion Span
  • Latest Post Latest Post by JamesCherrill

Dani AI

Generated

Short addendum that fills the gap between the symptoms and a reliable fix.

was right to suspect an encoding mismatch, and confirmed the file needed UTF-8. To debug similar problems quickly, inspect the raw bytes so you know whether the file is UTF-8, Windows-1252 (CP1252), or something else. An em-dash (U+2014) is encoded as the three bytes 0xE2 0x80 0x94 in UTF-8 but as a single byte 0x97 in CP1252. On a Unix-like system:

xxd -l 32 file.txt
hexdump -C file.txt | head

If you see E2 80 94, treat the file as UTF-8; if you see 97, treat it as CP1252.

In Java prefer the java.nio API and explicit charsets so you do not rely on platform defaults. For example, read and write with UTF-8 explicitly and strip a BOM if present:

Path p = Paths.get("book.txt");
try (BufferedReader br = Files.newBufferedReader(p, StandardCharsets.UTF_8)) {
    String line;
    while ((line = br.readLine()) != null) {
        if (line.length() > 0 && line.charAt(0) == '\uFEFF') line = line.substring(1);
        System.out.println(line);
    }
}
System.setOut(new PrintStream(System.out, true, StandardCharsets.UTF_8.name()));

Notes and pitfalls: a correct reader/writer pair is only part of the chain — the terminal or IDE console must also support UTF-8. On Windows set the code page to UTF-8 (chcp 65001) or configure your IDE console. If the source encoding is unknown, use a charset detector (juniversalchardet/uchardet) or convert with iconv (for example iconv -f WINDOWS-1252 -t UTF-8 in.txt -o out.txt).

Recommended Answers

All 8 Replies

Member Avatar for Mike Askew

Looking at this I may have to catch and manually convert them?

Member Avatar for JamesCherrill

If you have an em-dash then the file is not ASCII encoded!
ASCII is a 7-bit code that includes only the english alphbet, numbers and a handfull of puctuation (not em-dashes), so any other characters will be unreadable when you specify ASCII as the character set.

Simply leaving out the character set will give you the default CharSet for your machine, which will work 95% of the time unless you are importing files from places with a different language.

Otherwize, you could try ISO-8859-1 (ISO Latin 1) or UTF-8

Edited by JamesCherrill
Member Avatar for Mike Askew

You sir are a genious, I did misread which encoding it was using, UTF-8 being correct.

Running with my machine default char-set works fine.

Member Avatar for JamesCherrill

No, no genius, just been doing it a long time...
(I started using Java when I still lived in Woodham myself - in the last century)

Member Avatar for Mike Askew

Hehe, where did you move too?

Member Avatar for JamesCherrill

France!

Member Avatar for Mike Askew

Weather any better over there?

Member Avatar for JamesCherrill

Everything is better over here, except for Indian take-aways. Been here 15 years and I'm not going back.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.