Member Avatar for Edwards8

So, I wrote a little script to convert tifs into jpegs. The tifs are about 600mb in size, I reduce the size, and then convert. Here is the code in its entirety:

## First we import the image library, Image object, and garbage
## collector, as well as os
from PIL import Image
import gc, os
## Get the path for the directory to be converted, then change to that path
print "Please enter the path to the files that you wish to convert."
path = raw_input()
os.chdir(path)
## For each file in every subdirectory, see if it's a tif file
for root, dir, files in os.walk(path):
    for name in files:
        if name[-4:] == ".tif":
            print 'Opening ' + name
            os.chdir(path)
            im = Image.open(root + '/' + name)
            x, y = im.size
            ## Resize the tiff tile at a 2/3 scale.  Make a new directory to
            ## mimic the file heiarchy of the original file, only on the C
            ## drive instead of wherever it was to begin with.
            print 'Resizing'
            im2 = im.resize((int(x*.66), int(y*.66)), Image.ANTIALIAS)
            n = 'c' + root[1:]
            savedfile = n +"/jpegs/"
            try:
                os.makedirs(savedfile)
                os.chdir(savedfile)
            except WindowsError:
                os.chdir(savedfile)
            savedfile = name[:-4] + ".jpg"
            ## Save the file as a jpg, with a high quality
            print 'Saving'
            im2.save(savedfile, quality=85)
            del im
            del im2
            ## Force a memory dump.  Otherwise memory will get cluttered
            ## up, very quickly too.
            gc.collect()
            print 'Memory Wiped'

Due to space limitations in the workplace, I need it to copy the directory structure onto the c drive of the computer the script is running off of, hence the funky directory dance. I'm still kinda new to all of this, and I thought the garbage collector would help me out. But, it'll wipe the memory at first, but things will still accumulate. After converting about 30-40 images (and I need to convert literally tens of thousands) the program crashed with a MemoryError. Page File usage was at about 2G. There is about 3 gigs of ram on this PC, which is running XP. Any tips or advice?

  • 4 Contributors
  • 5 Replies
  • 488 Views
  • 4 Days Discussion Span
  • Latest Post Latest Post by woooee

Dani AI

Generated

Short diagnosis and what likely happened (builds on and ): PIL/Image.open is a lazy loader — it may keep the underlying file handle and decode buffers open until the image is explicitly loaded or closed. Resizing creates new image objects while the original image (and its file pointer) can remain resident; plus CPython’s allocator uses arenas so freed Python memory is not always returned to the OS immediately. That combination explains why del + gc.collect() reduced live objects but the process kept growing until MemoryError. (pillow.readthedocs.io)

Quick, practical fixes to try first (follow ’s file-handle idea and ’s per-file function idea):

  • Open the file object explicitly, open the image from that handle, do work, and then close the Image objects (and the file) in a finally block so nothing is left holding references.
  • Put all per-file work in a short-lived function so local variables get released when it returns.

Example pattern (safe for old/new Pillow; different from the code already posted):

from PIL import Image
import gc

def process_one(infile, outfile):
    with open(infile, 'rb') as fh:
        im = Image.open(fh)
        try:
            x, y = im.size
            out = im.resize((int(x*0.66), int(y*0.66)), Image.ANTIALIAS)
            out.save(outfile, quality=85)
        finally:
            try: im.close()
            except: pass
            try: out.close()
            except: pass
            del im, out
            gc.collect()

If PIL still uses too much memory or TIFFs are huge/ tiled, try streaming or an external tool:

  • Streaming decode with PIL’s incremental parser (feed file chunks to ImageFile.Parser, then close to get an Image). This avoids building a full in-memory buffer while reading. (pillow.readthedocs.io)

  • For heavy batch work, call a command-line tool that streams (ImageMagick’s magick/convert) or use libvips/pyvips (designed to process huge images with low memory). Example CLI: magick input.tif -resize 66% -quality 85 output.jpg. For massive TIFF workloads, libvips often uses far less RAM than PIL. (imagemagick.org)

Notes: ’s splitext tip is worth adopting for robust extension handling. For tens of thousands of large files, the most reliable solution is a streaming tool (vips/ImageMagick) or processing in small batches and restarting the worker process periodically to avoid allocator fragmentation.

Recommended Answers

All 5 Replies

Member Avatar for jrcagle

Well, a quick tangential note: your method for extracting the extension

name[-4:]

is buggy. What if the file extension is ".jpeg"?

Better:

filename, ext = os.path.splitext(name)
if ext == ".tif":
   do stuff ...
   savedfile = filename+".jpg"
   do more stuff...
Member Avatar for jrcagle

WRT to the main problem, the memory error, I agree that there's something wrong, and it doesn't appear to be your code.

You might try e-mailing the PIL support at

Jeff

Member Avatar for Edwards8

If it's anything but a tif, it'll be ignored, as I don't need to do anything with it. However, I think that the method you showed me is much more practical than what I've got (I wasn't aware of that tool to grab the extension name). So, I'll be modifying it. Thanks for the tip. And I'll shoot them an email. Thanks for the help, really. Happy to know that I at least had the code right.

Member Avatar for G-Do

Hi Edwards8,

It turns out that PIL will let you read pieces of an image file in sequence without loading the entire thing into memory. Fred Lundh has a post about it on a python mailing list here. I don't know if this will work for TIFs, but you can always give it a shot.

It sounds like some piece of the file is being kept resident in memory, despite your deleting and garbage collection - and from the sizes you quoted, it looks to be about a tenth of each file. Maybe PIL is reading the files in pieces, and hanging onto the last piece of each? Or maybe the file is being kept open in some way, even though you delete the image object? One thing you could try is to create a file handle on the image file, then use Image.open() on that, instead of the file name. Then, when you are finished, close the original file handle. This may convince PIL to let go of whatever data it's holding onto.

Unfortunately, I don't know a whole lot about PIL, so I'm shooting in the dark, here.

Hope this helps!

Member Avatar for woooee

Try running everything after "if name[-4:] == ".tif":" in a separate function. The memory should be garbage collected when the function exits.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.