Member Avatar for kummetha

How to print the duplicates in a text file. Also the line number where the duplicate text has been found needs to be printed. pls help

KZ

  • 4 Contributors
  • 8 Replies
  • 185 Views
  • 1 Day Discussion Span
  • Latest Post Latest Post by KevinADC

Dani AI

Generated

As demonstrated, tracking seen lines while you read the file is the right idea. If you want a clearer report — each duplicated text printed once followed by all line numbers where it appears — the script below collects positions (handles Windows CRLF) and prints only entries that occur multiple times. Save as find_dups.pl and run perl find_dups.pl input.txt.

#!/usr/bin/perl
use strict;
use warnings;

my $file = shift or die "Usage: perl find_dups.pl filename\n";
open my $fh, '<', $file or die "Cannot open $file: $!\n";

my %positions;
my $ln = 0;
while (my $line = <$fh>) {
    $ln++;
    chomp $line;
    $line =~ s/\r$//;              # normalize CRLF on Windows
    my $key = $line;               # adapt: lc($key) for case-insensitive
    push @{ $positions{$key} }, $ln;
}
close $fh;

for my $text (sort keys %positions) {
    my $count = @{ $positions{$text} };
    next if $count <= 1;
    my $list = join ",", @{ $positions{$text} };
    print "Occurs $count times at lines $list:\n$text\n\n";
}

If you also need the line immediately above every blank line (your later example), this tiny script prints the previous non-modified line each time a blank is encountered. Save as prev_of_blank.pl.

#!/usr/bin/perl
use strict;
use warnings;

my $file = shift or die "Usage: perl prev_of_blank.pl filename\n";
open my $fh, '<', $file or die "Cannot open $file: $!\n";

my $prev;
while (my $line = <$fh>) {
    my $copy = $line;
    $copy =~ s/\r?\n$//;           # strip EOL
    if ($copy =~ /^\s*$/) {
        print "$prev\n" if defined $prev;
    }
    $prev = $copy;
}
close $fh;

Tips: to ignore leading/trailing spaces add $key =~ s/^\s+|\s+$//g; before using $key. For very large files avoid storing every line in memory — use external sorting or a disk-backed DB (DB_File/DBM) instead.

Recommended Answers

All 8 Replies

Member Avatar for msvinaykumar

Hi,

If you are working in Unix then, Use following command

<code>
cat -n file-name > file_name2
</code>

Then use
<code>
sort -u -k 2,4 file_name2
</code>

Member Avatar for kummetha

No, I am not working in Unix. I am working with Windows.
KZ

Member Avatar for msvinaykumar

Or use Uniq command

<code>
$ cat happybirthday.txt

Happy Birthday to You!

Happy Birthday to You!

Happy Birthday Dear vinay!

Happy Birthday to You!
</code>

<code>
$ sort happybirthday.txt | uniq -dc
3 Happy Birthday to You!
</code>

Member Avatar for KevinADC

How to print the duplicates in a text file. Also the line number where the duplicate text has been found needs to be printed. pls help

KZ

What have you tried? Where are you stuck?

Member Avatar for kummetha

many a thanks msvinaykumar, i am looking for a perl script that works with windows.
KZ

Member Avatar for Prakash_8111 
open(FILE,"1.txt") || die "$!";  ## 1.txt is a file name 
%seen =();
$line=0 ;
while (<FILE>)
{
        $seen{$_}++;
        $line++;
        print "$line : $_"  if $seen{$_} > 1 ;
}
Edited by Reverend Jim because: Fixed formatting
Member Avatar for kummetha

thanks prakash, it did really work.
also if you could provide help for a script that reads a blank line in a txt file and prints the line above the blank line.
Eg
abcdef

william
marc
sumeet

nair

output:
abcdef
sumeet
thanks in advance

Member Avatar for KevinADC

At some point you should start doing your own school work and the rest of the people participating in this thread you stop doing it for you.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.