Member Avatar for LieAfterLie

For some reason, newline characters seem to be using 2 bytes in text files instead of one. Is this normal? When I open a text file and look at the size, I get one more for every character I add, except for newlines, which add 2 bytes.

  • 3 Contributors
  • 4 Replies
  • 134 Views
  • 9 Hours Discussion Span
  • Latest Post Latest Post by John A

Dani AI

Generated

Short checklist and practical checks to explain why a newline can seem to add two bytes.

As and pointed out, platform conventions do vary — that is a common cause — but there are other reasons to see a two-byte increase. First, check whether the file is stored in a multi-byte encoding (UTF-16) or contains a BOM; that will make every character occupy two bytes, not just the newline. Use file filename, hexdump -C filename, xxd filename or od -c filename to inspect the raw bytes and any BOM.

If you are working in C on Windows, remember the C runtime can perform text-mode translation between the on-disk sequence and the in-memory '\n'. To see raw bytes from a program, open the file in binary mode. Example quick test program:

#include <stdio.h>

int main(void) {
    FILE *f = fopen("test.txt", "rb");
    int c;
    while ((c = fgetc(f)) != EOF)
        printf("%02X ", (unsigned char)c);
    fclose(f);
    return 0;
}

Terminal/display behavior is separate from storage: carriage-return (0x0D) moves the cursor to column zero, line-feed (0x0A) advances the output down one line. Terminal drivers and legacy devices sometimes translate one into the other (termios flags like ICRNL/ONLCR), which is why a '\r' alone may look like a newline in some environments.

Quick fixes and tips: if you want single-byte newlines, convert the file encoding to UTF-8 and normalize EOLs with tools such as dos2unix/unix2dos or iconv -f UTF-16 -t UTF-8. In C use binary mode ("rb"/"wb") to avoid automatic translation when you need exact byte control. Be mindful of version-control and network protocol settings that can alter line endings on checkout or transfer.

Recommended Answers

All 4 Replies

Member Avatar for Ancient Dragon

The new line in the text file is operating-system dependent. *nix uses '\n', MAC uses '\r', and MS-DOS/MS-Windows uses two characters "\r\n". Which is why text files that are transferred between operating systems are often run through a translator to convert from one format to another.

Member Avatar for LieAfterLie

ahh, carriage return... What exactly does \r do in text files? How is that typically interpreted by the font? I think in terminals, it goes back to the beginning of the line without going down to a new one, but I've seen it displayed differently in different terminals. Some will need both, and some automatically do the newline when they display a \r.

Member Avatar for Ancient Dragon

It has little, if anything, to do with terminals. Just a method of terminating a text line in text files. I don't know why MAC and MS-DOS didn't follow *nix's lead and just save '\n' in the text file -- would seem the logical thing to do since most (but not necessarily all) lines read by fgets() terminate with '\n'. MAC and MS-DOS/MS-Windows os just have to do a little extra work to maintain that standard.

Member Avatar for John A

I don't know why MAC and MS-DOS didn't follow *nix's lead and just save '\n' in the text file -- would seem the logical thing to do since most (but not necessarily all) lines read by fgets() terminate with '\n'. MAC and MS-DOS/MS-Windows os just have to do a little extra work to maintain that standard.

I do believe that Mac OS X uses just \n now, as it's using the FreeBSD kernel as the underlying technology for the operating system, and because of that, is POSIX-compliant.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.