Remove Duplicates

[flebber]

Say I have a basic list.

x = [2,3,1,3,5,4,5]

And I want to find the sum result applied against the list without duplicates.
So with duplicates the code could be.

[elem * 2 for elem in x if elem >= 1 ] 
> [2, 4, 6, 6, 8, 10]

But I don't want duplicates. So my thought was I should make a copy of the list filtered.

# so for elements in x not equal to any other elements in the list * 2.
#So first sort the list
#Then remove any items that are equal to the next item in the list. So items not equal to the next should be kept.

x = [2,3,1,3,5,4,5] 
x.sort()
y =[]
y = [elem for elem in x if x != elem[::1]]

I get an error that it is not subscriptable.

TypeError: 'int' object is not subscriptable

Thought this altered version might work but still same error.

y = [elem for elem in x if elem != (elem[::1])]

I found this in the docs for removing duplicates but it seems rather unweildy for such a small thing.

if mylist:
    mylist.sort()
    last = mylist[-1]
    for i in range(len(mylist)-2, -1, -1):
        if last == mylist[i]:
            del mylist[i]
        else:
            last = mylist[i]

There is a few cool ideas at activestate http://code.activestate.com/recipes/5256 for example Raymonnd H with

def uniq(alist)    # Fastest order preserving
    set = {}
    return [set.setdefault(e,e) for e in alist if e not in set]

But I am just darn curious why I get a subscriptable error.

[Gribouillis]

If you don't need to preserve order, the simplest is to use a set

>>> x = [2,3,1,3,5,4,5]
>>> list(set(x))
[1, 2, 3, 4, 5]

If you need to preserve order, there are also 2 functions in the recipes of the itertools module documentation:

from itertools import imap, groupby
from operator import itemgetter

def unique_everseen(iterable, key=None):
    "List unique elements, preserving order. Remember all elements ever seen."
    # unique_everseen('AAAABBBCCDAABBB') --> A B C D
    # unique_everseen('ABBCcAD', str.lower) --> A B C D
    seen = set()
    seen_add = seen.add
    if key is None:
        for element in iterable:
            if element not in seen:
                seen_add(element)
                yield element
    else:
        for element in iterable:
            k = key(element)
            if k not in seen:
                seen_add(k)
                yield element

def unique_justseen(iterable, key=None):
    "List unique elements, preserving order. Remember only the element just seen."
    # unique_justseen('AAAABBBCCDAABBB') --> A B C D A B
    # unique_justseen('ABBCcAD', str.lower) --> A B C A D
    return imap(next, imap(itemgetter(1), groupby(iterable, key)))

The problem with your code is that in for elem in x , elem is an integer, so python complains when you write elem[::1] .

[flebber]

Have found this solution also which is supposed to be the fastest order preserving script.

def f5(seq, idfun=None):  
    # order preserving 
    if idfun is None: 
        def idfun(x): return x 
    seen = {} 
    result = [] 
    for item in seq: 
        marker = idfun(item) 
        # in old Python versions: 
        # if seen.has_key(marker) 
        # but in new ones: 
        if marker in seen: continue 
        seen[marker] = 1 
        result.append(item) 
    return result

All I was trying to say with elem[::1] was step one element ahead. Is there an alternate way of saying that?

[TrustyTony]

What is wrong with set, if order need not be preserved:

>>> x = [2,3,1,3,5,4,5]
>>> print([2*a for a in set(x)])
[2, 4, 6, 8, 10]
>>>