Member Avatar for nouth

this time I have only one textfile and I want to remove all duplicate lines so that each and every single line in it is unique unto itself.

with open("textfile") as w:
 for line in w:
  W = line
  with open("textfile") as d:
   c = 0
   for line in d:
    D = line
    if W.strip() == D.strip(): c = c + 1
    if c == >1:
     #how can I delete D?
     #and will after deleting it, does this mean that I can make sure that my W won't skip it?
Edited by nouth
  • 5 Contributors
  • 7 Replies
  • 377 Views
  • 3 Days Discussion Span
  • Latest Post Latest Post by TrustyTony

Dani AI

Generated

Short summary: removing duplicates cannot be done reliably by trying to delete lines from the same file while iterating it. The safe pattern is to produce a new file (or use in-place helpers) and then replace the original. This is what suggested with a temp file / rename and what the fileinput module supports. Opening the same file twice and deleting from the inner loop is fragile and will lead to skipped lines or truncated data.

Order vs speed: ’s set-based idea is fast but discards the original order. ’s list approach preserves order but uses text = [] and if line not in text — that works but is O(n^2) in the worst case because list membership is O(n). A better balance is an order-preserving mapping (OrderedDict or dict.fromkeys in modern Python) or a fast seen set when order does not matter.

Example (order-preserving, safe replace):

import os, tempfile, shutil
from collections import OrderedDict

def dedupe_file(path, ignore_ws=False):
    seen = OrderedDict()
    with open(path, 'r', encoding='utf-8') as src:
        for line in src:
            key = line.strip() if ignore_ws else line.rstrip('\n')
            if key not in seen:
                seen[key] = line
    fd, tmp = tempfile.mkstemp(dir=os.path.dirname(path))
    with os.fdopen(fd, 'w', encoding='utf-8') as f:
        f.writelines(seen.values())
    # atomic replace on modern Python, fallback to move otherwise
    try:
        os.replace(tmp, path)
    except AttributeError:
        shutil.move(tmp, path)

Notes and corner cases: choose strip() vs rstrip('\n') deliberately (whitespace differences change equality). For very large files that won’t fit memory, use external tools: sort -u file > out (fast but sorts), or preserve-first-occurrence with awk '!seen[$0]++' file > out. Use atomic replacement (os.replace or shutil.move) so the original file is not left half-written.

Recommended Answers

All 7 Replies

Member Avatar for besticm

where to use this coding..

Regards,

packers and movers in delhi

Member Avatar for tony75

my oldtext.txt
"Which language do you like?
I like python programming.
Which language do you like?
I like python programming.
I like python programming."

oldFile = open("oldtext.txt")
newFile = open("newtext.txt", "w")
text = []
for line in oldFile:
    if line not in text:
        text.append(line)
        newFile.write(line)
newFile.close()

my newtext.txt
"Which language do you like?
I like python programming."

>>> ================================ RESTART ================================
>>> 
Which language do you like?

I like python programming.
Member Avatar for TrustyTony

If the order of lines does not matter, you can use

  with open('filelein.txt') as infile,  open('fileout.txt', 'w') as outfile:
           outfile.write(''.join(set(infile)))
Edited by TrustyTony
Member Avatar for nouth

I want to do it in such a way as to so that I can understand exactly what is happening.

with open(file) as f:
    for line in f:
        if i < 1:
            i = i + 1
        else:
            #I want to remove line from f

how can I delete line from f within for line in f?

okay, I got distracted with this

oldFile = open("oldtext.txt")
newFile = open("newtext.txt", "w")
text = []
for line in oldFile:
    if line not in text:
        text.append(line)
        newFile.write(line)
newFile.close()

I cannot understand what text = [] does and how it can work with if line not in text:
but can I change it to this?

if line not in text:
    text.append(line)
    oldFile.write(line)

or maybe I read something somewhere from about saving it to temporary files, so maybe newFile can be temporary and then it can be used with to reoverwrite the oldFile? no not like append

with open("textfile") as w:
    for line in w:
        W = line
        with open("textfile") as d:
            c = 0
            for line in d:
                D = line
                if W.strip() == D.strip(): c = c + 1
                if c == >1:
                    #I want to delete line in d
#but this is definitely the way that I want to do it, notice I am still opening the same "textfile" twice and then making a new variable for it's line
Edited by nouth
Member Avatar for snippsat

so maybe newFile can be temporary and then it can be used with to reoverwrite the oldFile?

Yes this is normal way,you iterate over contend and do what you need,then save contend in a new file.
If you need newtext to be oldtext you rename the files.
os.rename(oldtext','newtext') or shutil.move('oldtext','newtext')

Python has fileinput that can do inplace edit of orginal file.
It's ok,but the method over with rename is just as good.
If order doesn't matter,soultion by pyTony is good.

'''lines.txt-->
line 1
line 2
line 2
line 3
'''

import fileinput
import sys

def remove_dup(fname):
    text = []
    for line in fileinput.input(fname, inplace=True):
        if line not in text:
            text.append(line)
            sys.stdout.write(line)

fname = 'lines.txt'
remove_dup(fname)

'''lines.txt-->
line 1
line 2
line 3
'''
Edited by snippsat
Member Avatar for nouth

I have learnt what text = [] does now
but I still have some trouble understanding this and how it works outfile.write(''.join(set(infile))) from pyTony

I haven't used import yet but can understand that for line in fileinput.input(fname, inplace=True): and sys.stdout.write(line) have something to do with import fileinput and import sys

but my next question I will ask will have to be about this

Member Avatar for TrustyTony

"Write to outfile joined lines of set of items (lines) in file infile"
Set of values has only one instance of same item.

commented: thanks :) +0
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.