Member Avatar for bgbiplab

Script to Remove duplicate lines from a text file.

  • 4 Contributors
  • 7 Replies
  • 188 Views
  • 1 Day Discussion Span
  • Latest Post Latest Post by KevinADC

Dani AI

Generated

A compact, idiomatic Perl solution is usually the easiest route. As suggested, a hash to track seen lines keeps the first occurrence and is O(n) time; this implements that behavior and works directly on STDIN or files passed on the command line.

#!/usr/bin/perl
use strict;
use warnings;

my %seen;
while (my $line = <>) {
    print $line unless $seen{$line}++;
}

Usage example: perl uniq_lines.pl infile.txt > outfile.txt

If normalizing input is required (ignore leading/trailing whitespace or case), convert each line to a key before checking the hash. The following keeps the original line text but treats lines equal after trimming and lowercasing:

use strict;
use warnings;

my %seen;
while (my $line = <>) {
    my $key = $line;
    $key =~ s/^\s+|\s+$//g;
    $key = lc $key;
    next if $seen{$key}++;
    print $line;
}

To keep the last occurrence instead of the first, record the final index for each line and then print only lines whose index matches the stored final index (this requires reading the file into memory):

use strict;
use warnings;

my @lines = <>;
my %last;
for my $i (0..$#lines) { $last{$lines[$i]} = $i }
for my $i (0..$#lines) { print $lines[$i] if $last{$lines[$i]} == $i }

Notes and troubleshooting: in-place editing with a backup is possible: perl -i.bak -ne 'print unless $seen{$_}++' file. For very large files where the unique-key set won’t fit in memory, tie a hash to disk (DB_File/GDBM) or use external tools that trade memory for disk (but sort -u changes line order). Also consider CRLF normalization when processing Windows text. This provides a self-contained alternative to the snippet pointer mentioned and avoids regex complexity noted by while following ’s advice.

Recommended Answers

All 7 Replies

Member Avatar for KevinADC

The answer is 49.

Member Avatar for Major Major

The answer is 49.

42. When you know the question you will understand the answer :)

OK I'm gonna get sued by Douglas Adams' estate now.

I'll even help out,

first you open the file... then you read it... then you use regular expressions to find and remove duplicates, then close the file and open it for writing and write the new text back, then close the file again.


this has been done many many times over... use <insert favourite search engine here> to find a script and modify it.

Member Avatar for chaosprime

Hey, he doesn't need to use regular expressions. A hash will do fine.

Member Avatar for Major Major

Hey, he doesn't need to use regular expressions. A hash will do fine.

Didn't think about that

Member Avatar for KevinADC

All he has to do is look in the Snippets section of this website, go to the perl section and read the "Script to Remove duplicate lines" snippet.

42.... been a long time since I read the book. ;)

Member Avatar for chaosprime

But that would be sensible and prevent unnecessary effort. This is the Internet, mister! We don't hold with that sort of behavior!

Member Avatar for KevinADC

But that would be sensible and prevent unnecessary effort. This is the Internet, mister! We don't hold with that sort of behavior!

LOL..... :icon_lol:

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.