fastest way to create a file of a specified size

[Petrica7]

Hi.
What is the fastest way to create a file of a specific size (preferably without writing in it) on a nonNTFS partition..?
On a NTFS I use this:

var tf: TFileStream;
begin
....
tf := TFileStream.Create(FileName, fmCreate or fmShareExclusive);
tf.Size := [desired value];
tf.Free;

It's very fast.

But, for example, on FAT32 it's writing the entire file when the code reaches at "tf.Free".

Is there a better way...?
I searched the forum before asking but I couldn't find anything useful. But if there is please show me.

Thank you in advance for your help.

Best regards, Petrica

[Petrica7]

In case someone is wondering why I need to create such files: I'm making a small utility to fill USB sticks/keys with dummy file(s) so it wouldn't get infected on other computers.
It works for the most of the viruses.
You can download the application from: http://www.mediafire.com/?8x7d8ic1l157ml5
It has 4 languages (automatic detection): English, French, Deutch, Romanian.

On NTFS it works very fast but on FAT32 is very slow.

[pritaeas]

Are your NTFS and FAT32 tests on the same machine ? Or do they have different OS'es ? It might be the way the OS is buffering files.

[Petrica7]

I tested on 3 FAT32 formatted sticks/keys: one of my own and other two from my 2 friends. So, 3 different machines and 2 diferent OS's (XP x86 and 7 x86).
Each time was very slow.

I tried something: I removed "tf.free". I know it's not good, I just wanted to see what happens. The program finished fast but when I tried "chkdsk e: /f" it started writing and finished in about the same time as before.
It may be the buffering, I don't know...

[Petrica7]

If it is the buffering how can I solve it..?

[pritaeas]

Not sure, if it is the buffering, then it is an OS thing. Not easy to circumvent.

[Dervish1]

The only faster method I know of would be low-level operating system calls to allocate your disk sectors and update the directory without actually writing data to the allocated sectors.

I haven't seen any Delphi or Pascal libraries for this kind of function, but since I'm sure it's possible to write them, /some/body will have already done so. You'll have to search.

[Petrica7]

Yes, it's possible, thank you for the suggestion.

[pritaeas]

Perhaps the third answer with the C example in this thread can help. They're just WinAPI functions, so you should be able to use them in Delphi too.

var
  hf: THandle;
begin
  hf := CreateFile('test.fil', GENERIC_WRITE, 0, nil, CREATE_ALWAYS, 0, 0);
  SetFilePointer(hf, 1000000000, nil, FILE_BEGIN); // 1Gb
  SetEndOfFile(hf);
  CloseHandle(hf);
end;

Works nicely, although I can't test on FAT(32) systems.

[Petrica7]

Thank you.
But it's working fast only on NTFS.
On my FAT32 stick/key I get these results:
your code: 22.5 sec
direct copy: about 23 sec
I think it's writing in all the clusters...

PS: I have tried also the other solutions from that thread, none of them are fast.

[pritaeas]

The only thing remaining would be to put it in a thread, so you won't have to wait for completion.

[Petrica7]

I forgot to mention that the results are for a 100 MB file, not for 1 GB.

Yes, but the problem is not that my program can't do something else during the creation of the file.
The problem is the time in which the file is created.

I think I understand now why this isn't working fast on FAT32. It's because of the way the files are stored.
Each file is stored in clusters. In each cluster there is information about the next cluster (like a chain).
So when you create a file you have to write these informations in each cluster.
Plus you can't read/write just a part of a cluster, it's all or nothing.
So writing in file or setting its size is pretty much the same thing on FAT32.
I modified my code so now it's writing in 5 MB chunks. Between writing each chunk it's updating the position of a progressbar.

Here is the latest version of my program: http://www.mediafire.com/download.php?cdvsumjf9hthd92
I think it's nearly finished...

[Dervish1]

Thank you.
But it's working fast only on NTFS.
On my FAT32 stick/key I get these results:
your code: 22.5 sec
direct copy: about 23 sec
I think it's writing in all the clusters...

PS: I have tried also the other solutions from that thread, none of them are fast.

I'm afraid the standard libraries will all assume you actually want the file you're creating, and generally won't be significantly faster than the copy. I'm actually surprised that NTFS /doesn't/ have that problem. What would probably be quickest and most effective for your stated problem is simply to make the USB stick read-only, but if you really want to create an empty file without touching every sector included, I think you would have to write directly to your FAT file system sectors, and if something goes wrong it could render that disk unusable without a format. Exactly what sort of data you need to write is beyond what I remember, but give a search for FAT32 structure and for any direct sector I/O functions available through Delphi (or Windows). Once you know the sectors to write to and what information to write, I suspect it'll be very easy, although remember that the file directory information used by Windows is stored in more than one place, and that can cause you a lot of grief.

-- And, reading your previous messages more closely, implies that what I was suggesting is still unlikely to work. At least, the file you created could not be opened and examined safely. That's one of the risks of low-level solutions that try to hack around the library's limitations. The limitations may have been there for a reason.

[Petrica7]

I'm actually surprised that NTFS /doesn't/ have that problem.

NTFS is using streams. A "stream" is made by many contiguous clusters. So it's no need to write the information about the next cluster.

What would probably be quickest and most effective for your stated problem is simply to make the USB stick read-only

Yes, but I'm not sure is 100% ok. And when the user will run chkdsk to check the device I think it will do one of two things:
1. It will "reclaim" the free space back;
2. It will start writing each cluster with informations about the next.

but if you really want to create an empty file without touching every sector included, I think you would have to write directly to your FAT file system sectors, and if something goes wrong it could render that disk unusable without a format. Exactly what sort of data you need to write is beyond what I remember, but give a search for FAT32 structure and for any direct sector I/O functions available through Delphi (or Windows). Once you know the sectors to write to and what information to write, I suspect it'll be very easy, although remember that the file directory information used by Windows is stored in more than one place, and that can cause you a lot of grief.

-- And, reading your previous messages more closely, implies that what I was suggesting is still unlikely to work. At least, the file you created could not be opened and examined safely. That's one of the risks of low-level solutions that try to hack around the library's limitations. The limitations may have been there for a reason.

Probably it is possible but I will have to be very careful where and what do the program writes.
If it was only about my USB stick/key it wouldn't have been much of a problem. But I don't wanna wreck/damage other people's data...

[Petrica7]

I noticed a strange thing on NTFS.
I fill the entire free space with a dummy file.
Windows and GetDiskFreeSpaceEx say that there are 0 Bytes free.
But I can still add data to smaller files (few hundred bytes).
Does anyone know a way to write the entire free space on NTFS..?

[pritaeas]

If with 'add data' you mean write to existing files, yes, that is possible. NTFS still allocates in fixed block sizes iirc.

[Petrica7]

Ok.
I know it's not possible to deny writing completely but is there a way to fill the disk in such way that increasing the size of any file would be impossible..?

[pritaeas]

I think your method would not allow this as it is. Only files that are smaller then the reserved size can increase, but no more then upto that reserved size.

[Petrica7]

A few hours ago I said that I noticed that problem: even if free space is reported as 0 bytes, you can still increase file size by writing in small files.
So my method is not working...
You can try it yourself: take the last version of my program from http://www.mediafire.com/download.php?laa7zo7cj82njca and fill a USB stick/key (NTFS formatted).
Check the free space, is reported as 0.
Try to copy a few hundred bytes file or to insert bytes in smaller files (for example write text in a text file and save it).
It shouldn't work but it is.

[Dervish1]

Sorry, no, you can't get past this, whatever method you might want to use. Any files on the disk that don't already completely fill their last block can still be expanded to that point.

[Petrica7]

Sorry, no, you can't get past this, whatever method you might want to use. Any files on the disk that don't already completely fill their last block can still be expanded to that point.

Maybe...
I too thought that partially filled clusters are not reported well...
Anyway now I'm using this:

var tf: TFileStream;
   i: Integer;
   TrySize, p1, p2, ps, v: Int64;
   FileName: string;
begin
....
//main file dummy00.file created
....
         i := 1;
         while i < 100 do
         begin
            FileName := cbDrive.Drive + ':\dummy';
            if i < 10 then
               FileName := FileName + '0';
            FileName := FileName + IntToStr(i) + '.file';
            try
               tf := TFileStream.Create(FileName, fmCreate or fmShareExclusive);
            except
               Break;
            end;
            v := -1;
            TrySize := 1024;
            p1 := 0;
            p2 := 1024;
            while True do  //tries to find the interval where the maximun file size is
            begin
               try
                  tf.Size := TrySize;
                  p1 := TrySize div 2;
                  p2 := TrySize;
               except
                  Break;
               end;
               TrySize := 2 * TrySize;
            end;
            while True do  //finds the maximun file size (binary search)
            begin
               ps := v;
               v := (p1 + p2) div 2;
               if ps = v then Break;
               try
                  tf.Size := v;
                  p1 := v;
               except
                  p2 := v;
               end;
            end;
            try
               tf.Free;
               tf := nil;
            except
               Break;
            end;
            if v = 0 then Break; //if the new created file is size 0 breaks loop
            Inc(i);
         end;
....
end

I know it's not quite "orthodox" but I have no other idea in this moment.
On my sticks/key it creates dummy01...dummy08 then dummy09 can't be created.
There is still free space but lesser.
This way I diminish the possibility that a virus could writes itself in a new or already existent file...