Member Avatar for mmouse

Hi All,

I am reading a textfile and ready to import into an access database. The issue that has just arisen of hidden characters in line of text. The file is sent to us by a third party.

e.g.

Problem_Details: all the objects were all very soft and were split and therefore unfit for sale.

As you can see at the end of the line there appears to be a double space. It is actually a space and a ÿ (alt+0255).

When the program reads in the line (variable s) it only shows

Problem_Details: all the objects were all very soft and were split and therefore unfit for

missing the word 'sale'.

Here is my code

procedure TForm1.translateBranch;
var
temptext,outtext : TextFile;
s,ln,bn,pc,ct,pd,cl,pm,wl : string;
i : integer;
begin
assignFile(temptext,FileListBox1.FileName);
assignfile(outtext,FileListBox1.Directory+'\WRComp.txt');
reset(temptext);
rewrite(outtext);
readln(temptext,s);
try
while not eof(temptext) do
begin
Cursor := crHourGlass;
Readln(temptext,s);
i := length(s);
if Copy(s,0,17) = 'PROBLEM_CATEGORY:' then ct :=copy(s,20,8);
if Copy(s,0,16) = 'Problem_Details:' then pm :=copy(s,19);
if Copy(s,0,14) = 'Branch_Number:' then bn := copy(s,17,3);
if Copy(s,0,13) = 'Product_Date:' then pd := copy(s,16,8);
if Copy(s,0,13) = 'Packers_Code:' then pc := copy(s,16,20);
if Copy(s,0,12) = 'CLAIM_VALUE:' then cl :=copy(s,15,6);
if Copy(s,0,12) = 'Line_Number:' then ln := copy(s,15,5);
if Copy(s,0,11) = 'WhenLogged:' then wl := copy(s,14,10);
if Copy(s,0,9) = 'CallType:' then ct := copy(s,12,8);
if (copy(s,0,11) = '$UpdatedBy:') then
begin
if pd = '' then pd := edit2.text;
if (copy(pd,3,1) <> '/') and (copy(pd,6,1) <> '/') then pd := edit2.text;
if pc = '' then pc := 'not recorded';
if cl = '' then cl :='0.00';
edit1.text := ln+'^'+bn+'^'+pc+'^'+ct+'^'+pd+'^'+cl+'^0.00^'+wl+'^'+pm;
if edit1.text <> '^^^^^^^^^' then Writeln(outtext,edit1.text);
ln := '';
bn := '';
pc := '';
ct := '';
pd := '';
cl := '';
pm := '';
wl := '';
end;
s :='';
end;
finally
closefile(temptext);
closefile(outTEXT);
end;
Cursor := crArrow;
Edit1.text := 'Branch Translation Finished';
end;

The length of the string is correct, it just doesn't show / copy anything after the ÿ.

I can not see why having this character ÿ should be a pain, or how to overcome the issue. This character can appear anywhere!

Many thanks

Marcus

  • 2 Contributors
  • 2 Replies
  • 130 Views
  • 1 Week Discussion Span
  • Latest Post Latest Post by abdullah

Dani AI

Generated

— the symptom (text appearing fine in the file but everything after a small/special character disappears when you display or copy it) points to an encoding/control‑character issue rather than a Readln bug. 's idea of reading char-by-char is useful for diagnosis, but a safer workflow is: detect the exact bytes, decide whether the file is actually Unicode (UTF‑16/UTF‑8) or contains embedded non‑printable bytes, then either decode correctly or sanitize the line before using it.

Quick diagnostics

  • Open the file in a hex viewer (or dump bytes) near the bad line and inspect the bytes around the square/ÿ.
  • Look for BOM bytes (EF BB BF for UTF‑8, FF FE or FE FF for UTF‑16) or lots of 0x00 bytes (classic UTF‑16 read as ANSI).
  • If you find NULs or a BOM, treat the file as Unicode and decode it; otherwise treat it as ANSI/OEM and sanitize.

Two practical snippets you can apply immediately: one to detect a UTF‑16 BOM, and one to strip problematic control bytes (including the 0xFF you saw) while preserving normal printable characters and CR/LF/tabs.

function HasUTF16BOM(const FileName: string): Boolean;
var
  fs: TFileStream;
  buf: array[0..1] of Byte;
begin
  Result := False;
  fs := TFileStream.Create(FileName, fmOpenRead or fmShareDenyWrite);
  try
    if fs.Size >= 2 then
    begin
      fs.ReadBuffer(buf, 2);
      Result := (buf[0] = $FF) and (buf[1] = $FE) or (buf[0] = $FE) and (buf[1] = $FF);
    end;
  finally
    fs.Free;
  end;
end;
function CleanLine(const S: AnsiString): AnsiString;
var
  i, j: Integer;
begin
  SetLength(Result, Length(S));
  j := 0;
  for i := 1 to Length(S) do
    if (Ord(S[i]) >= 32) or (S[i] in [#9, #10, #13]) then
    begin
      Inc(j);
      Result[j] := S[i];
    end;
  SetLength(Result, j);
end;

Notes and cautions

  • If the file is UTF‑16, reading it as TextFile will insert NULs and UI controls (or PChar conversions) may stop at the first NUL. Convert using a Unicode-aware reader (TEncoding/TFileStream or Windows API) before parsing.
  • If you need to preserve high‑ASCII characters (accents), don’t blindly strip bytes >127 — first confirm encoding.
  • Also prefer Copy(s,1,...) (1‑based) and explicit trimming/replacement for NBSP (#160) or other known bad codes once you know which bytes are present.

These steps let you prove whether the ÿ/square is an encoding BOM or a stray byte, and then either decode the file properly or remove/replace the offending byte so the rest of the line is preserved.

Member Avatar for mmouse

Hi All,

The line should read

if Copy(s,0,16) = 'Problem_Details:' then pm :=copy(s,19,l-19);

but it still doesn't read past the hidden ÿ.

I have looked at the text file again in ms word and it can also show up as a small [].

Many thanks for looking everyone

Marcus

Member Avatar for abdullah

you could try this.. i dont particularly like using read and eoln, and truthfully i havent tested it (dont have delphi on this machine), but this may just work.

hope it works out :)

function TForm1.readline2 (var textfile1: Textfile; var linestring:string):integer;
var c:char;
begin
	linestring := '';  c:=#32;
	while not (eoln (textfile) or (c in [#255])) do
	begin
	  read (textfile1, c);
	  linestring := linestring + c;
	end;
	
end;
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.