Member Avatar for chaitu11

I have a (2gb)txt file which has thousands of queries, i need to read those file queries and split those queries into n no. of txt files..and save them in a folder... can any one post an example or guide

Edited by pritaeas because: Moved to PHP.
  • 4 Contributors
  • 11 Replies
  • 256 Views
  • 6 Days Discussion Span
  • Latest Post Latest Post by chaitu11

Dani AI

Generated

For a 2GB file it is important not to load the whole file into memory. As noted, stream the file line-by-line (SplFileObject or fgets) so memory usage stays small. If every SQL/query sits on a single line the split is trivial; if queries can span multiple lines, buffer until a delimiter (for example ;) before counting/writing. The examples below show both common patterns and include practical safeguards (zero‑padded output names, CLI-friendly settings, and minimal buffering).

<?php
// line-based split (safe for very large files)
set_time_limit(0);
$in = new SplFileObject('bigfile.txt', 'r');
$in->setFlags(SplFileObject::DROP_NEW_LINE);
$perFile = 50;
$outPrefix = 'part_';
$pad = 4;

$part = 1; $lines = 0;
$out = new SplFileObject($outPrefix . str_pad($part, $pad, '0', STR_PAD_LEFT) . '.txt', 'w');

foreach ($in as $line) {
    $out->fwrite($line . PHP_EOL);
    if (++$lines >= $perFile) {
        unset($out);
        $part++; $lines = 0;
        $out = new SplFileObject($outPrefix . str_pad($part, $pad, '0', STR_PAD_LEFT) . '.txt', 'w');
    }
}
<?php
// delimiter-based split (multi-line SQL queries)
set_time_limit(0);
$fh = fopen('bigfile.sql','r');
$perFile = 100; $outPrefix = 'queries_'; $pad = 4;
$part = 1; $qcount = 0; $buf = '';
$out = fopen($outPrefix . str_pad($part,$pad,'0',STR_PAD_LEFT) . '.sql','w');

while (($line = fgets($fh)) !== false) {
    $buf .= $line;
    if (preg_match('/;\s*$/', rtrim($line))) { // naive end-of-query test
        fwrite($out, $buf . PHP_EOL);
        $buf = ''; if (++$qcount >= $perFile) { fclose($out); $part++; $qcount = 0;
            $out = fopen($outPrefix . str_pad($part,$pad,'0',STR_PAD_LEFT) . '.sql','w');
        }
    }
}
if (trim($buf) !== '') fwrite($out, $buf);
fclose($fh); fclose($out);

Notes and troubleshooting:

  • Run as a CLI script (php split.php) to avoid web-request timeouts.
  • Ensure output directory permissions and enough disk space (splitting does not shrink total bytes).
  • If queries can contain semicolons inside strings or stored procedures, a simple regex is unsafe — use an SQL-aware parser or a more robust state machine.
  • Zero‑pad file numbers for correct lexical ordering.
  • Test on a small sample before processing the full 2GB file.

This builds on ’s single-line idea and ’s SplFileObject recommendation while avoiding full-file reads and keeping memory use low.

Recommended Answers

All 11 Replies

Member Avatar for pritaeas

What language?

Member Avatar for chaitu11

Splitting large File into Parts in PHP

Edited by chaitu11
Member Avatar for chaitu11

i want it in php

Member Avatar for peeyush.budhia

To achive this you need to make sure that the each query is written in a single line; so that we can convert each line into an array index.
Example:
$arrayQuery[0] = 'Here goes the first query'
$arrayQuery[1] = 'Here goes the second query'

Following is the code:

<?php
$arrayQuery = file('filename.txt');


$newFilePrefix = 'Query';
$fileNumber = 0;

foreach($arrayQuery as $oneQuery) {

    $newNumber = $fileNumber++;
    $handler = fopen($newFilePrefix.$newNumber.'.txt', 'w');
    if(fwrite($handler,$oneQuery)) {
        echo 'Query write in file: '.$newFilePrefix.$newNumber.'.txt'.'<br>';
    } else {
        echo 'Error while writing query in: '.$newFilePrefix.$newNumber.'.txt'.'<br>';
    }
    fclose($handler);
}

?>

Please change filename.txt to your file name.

Member Avatar for chaitu11

how can i read every 50 out of 10000 lines from a text file.using php

Edited by chaitu11
Member Avatar for pritaeas

Use a counter like $newNumber in the example above. After every N lines save the file and create a new one.

Member Avatar for peeyush.budhia

Following is the code to split 50 lines from a large file.

<?php
$arrayQuery = file('sample.txt');

$multiChunk = array_chunk($arrayQuery, 50); // 50 line per file

$newFilePrefix = 'Part';
$fileNumber = 0;

foreach ($multiChunk as $chunk) {

    $fileNumber++;
    $newFileName = $newFilePrefix . $fileNumber . '.txt';
    $handler = fopen($newFileName, 'w');
    $status = array();
    foreach ($chunk as $line) {
        if(fwrite($handler, $line)) {
            $status[] = 1;
        } else {
            $status[] = 0;
        }
    }

    if(in_array(0,$status)) {
        echo 'Something goes wrong in '.$newFileName.' file<br>';
    } else {
        echo 'Data successfully write '.$newFileName .' file<br>';
    }

    fclose($handler);
}
?>
Member Avatar for pritaeas

: Spoon-feeding is not recommended. He'll learn nothing.

Edited by pritaeas
Member Avatar for mschroeder

It sounds like you have a file with 10000+ lines in it, that you need to split into 50 line chunks and save to new files...

If the file is well over your php memory limit, reading the whole thing into memory will only cause you headaches.

I'd highly suggest using SplFileObject to iterate over the file line by line. This will give you a much smaller memory footprint.

Create a blank file also using SplFileObject, iterate over 50 lines of the original file, writing each line to the new blank file.
Keep a counter of how many lines you've read through. When you've reached 50, simply increment the blank file you're using.

I'll be happy to share code examples.

Member Avatar for chaitu11

Thanks all

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.