Member Avatar for merse

How to determine the encoding of a text file?

And are there any tools to handle the content of a text file independent of its encoding?

  • 3 Contributors
  • 2 Replies
  • 139 Views
  • 14 Hours Discussion Span
  • Latest Post Latest Post by Ancient Dragon

Dani AI

Generated

Clarification and a practical approach that builds on the thread: files are byte streams; an encoding is the rule that maps those bytes to characters. That is the point behind ’s simplification, but it misses the practical reality that many encodings produce identical byte sequences for purely ASCII text. ’s question about "Unicode vs ASCII" is on target: ASCII is a 7-bit encoding; Unicode is a character repertoire most commonly stored using UTF-8, UTF-16 or UTF-32 encodings.

Fast checklist for determining encoding (in order of reliability):

  • Look for a BOM (byte-order mark). Common signatures are EF BB BF (UTF-8), FF FE / FE FF (UTF-16 LE/BE), FF FE 00 00 / 00 00 FE FF (UTF-32). See the Unicode BOM FAQ (unicode.org).
  • If no BOM, test whether the bytes are valid UTF-8. Valid UTF-8 strongly suggests UTF-8 (but is not proof).
  • Use a MIME/heuristic detector and any available metadata (HTTP headers, file metadata, user-supplied info). Tools like the file utility, uchardet or Mozilla’s chardet use heuristics and report probabilities; none are 100% certain for short or ASCII-only files. See the file man page (linux.die.net) and chardet (pypi.org).

Example commands (shell):

file --mime-encoding filename
uchardet filename
iconv -f guessed -t utf-8 filename > filename.utf8
hexdump -n 4 -C filename   # quick BOM check

C++ note: read the first few bytes in binary to check for a BOM, then pass the byte stream to a library that converts to Unicode (ICU or boost::locale are recommended; <codecvt> is deprecated). Detection is probabilistic; when in doubt, prefer explicit metadata or ask for the encoding and convert everything early to UTF-8 for processing.

Recommended Answers

All 2 Replies

Member Avatar for WaltP

How to determine the encoding of a text file?

Text files are not encoded. They are text.

And are there any tools to handle the content of a text file independent of its encoding?

Standard I/O commands.

Member Avatar for Ancient Dragon

by encoding do you mean how to tell the difference between UNICODE and ascii text? Here is one link that may help you

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.