removing duplicate lines in unix

[alex05]

Write a script named killdup. The script sorts all the lines in a file supplied to it as the first argument, and removes duplicate lines. The script stores the resulting output in a file whose name is given by the second argument.
I know how to sort it, but how to use the file as argument and then pass it to another file, which is also an argument.
sort -n | uniq -u | sort -n

[mittelgeek]

You were off to a good start.

Here what I did:

./first

This is a test
Still a test
Test continues
Bogus test
Case test may or may not work
Bogus test
The quick brown fox jumped over the lazy dog.
Now is the time for all good men to come to the aid of their country.
The quick brown fox jumped over the lazy dog.
To be or not to be
Test continues
Four score and seven years ago
Still a test

./killdup

#!/bin/sh

sort -n [B]$1 [/B]| uniq --repeat > [B]$2[/B]
sort -n [B]$1 [/B]| uniq --unique >> [B]$2[/B]
sort -n [B]$2[/B]

user@host:~$ ./killdup ./first ./second
user@host:~$ cat ./second

Bogus test
Still a test
Test continues
The quick brown fox jumped over the lazy dog.
Cast test may or may not work
Four score and seven years ago
Now is the time for all good men to come to aid of their country.
This is a test
To be or not to be

Moral of the story: $0-$9 are positional parameters passed to every shell script. $0 being the actual command line for the script, and $1 being the first parameter, $2 being the second, etc.

$* indicates all the positional parameters as one big value, e.g., /usr/bin/cat ./first ./second. /usr/bin/cat, ./first, and ./second being $0, $1, and $2, respectively.

There are some other special reserved shell variables like $@, $_, $-, etc., but I am too tired to remember them right now.

Hope this helps.

[jim mcnamara]

This awk one-liner works pretty well -

awk '!x[$0]++' file.old > file.new

This checks the whole line

This one just checks for part of the line in this case the first 16 characters.

awk '!x[substr($0,1,16)]++' file.old > file.new