Member Avatar for bumsfeld

Is there a way to limit size of a list in Python, so it does not use all memory?

  • 4 Contributors
  • 5 Replies
  • 14K Views
  • 1 Year Discussion Span
  • Latest Post Latest Post by vegaseat

Dani AI

Generated

Python lists do not support a hard size limit; they grow until you run out of memory. If you only need to retain the most recent N items, prefer a fixed-length deque that automatically discards the oldest entry. This keeps the logic centralized even if many parts of your code are appending.

from collections import deque

last_n = deque(maxlen=5)  # keep only the latest 5 items

def add(item):
    last_n.append(item)   # oldest is dropped automatically when full

# later...
for i in range(10):
    add(i)

snapshot = list(last_n)   # [5, 6, 7, 8, 9]

If you want backpressure instead of dropping old items (e.g., producer/consumer code), use queue.Queue(maxsize=N). When full, put() will block (or raise on put_nowait()), which prevents unbounded growth rather than silently discarding data. See collections.deque and queue.Queue.

For large numeric streams, consider storage choices as well as size limits:

  • Use array.array for a compact, typed container when you truly need a list-like structure but with less overhead than Python objects (array.array).
  • If data does not need to live entirely in RAM, memory-map it on disk with NumPy’s memmap, which lets you work with arrays larger than memory and control how much is resident at a time (numpy.memmap).

Recommended Answers

All 5 Replies

Member Avatar for G-Do

Hi bumsfeld,

I believe that the array data structure in the Numeric/NumPy package is both typed and non-resizable, just like C arrays (actually, I'm pretty sure that these data structures are C arrays wearing a Python hat). Check out:

Member Avatar for vegaseat

Thanks G-Do for your information, so NumPy eventually replaces Numeric and Numarray. Interesting!

When I was coding in C, I used to sample lab data and put it into a circular or ring buffer to keep the thing from overflowing. This was an easy way to calculate a moving average.

You can do a similar thing in Python using a list and keeping track of the index. When the index reaches the limit you have set, new data starts at index zero again.

A queue makes this simpler, particularly the deque container, because you can add to the end and pop the front. Here is an example, I hope you can see what is happening:

# the equivalent of a circular size limited list
# also known as ring buffer, pops the oldest data item
# to make room for newest data item when max size is reached
# uses the double ended queue available in Python24
 
from collections import deque
 
class RingBuffer(deque):
    """
    inherits deque, pops the oldest data to make room
    for the newest data when size is reached
    """
    def __init__(self, size):
        deque.__init__(self)
        self.size = size
        
    def full_append(self, item):
        deque.append(self, item)
        # full, pop the oldest item, left most item
        self.popleft()
        
    def append(self, item):
        deque.append(self, item)
        # max size reached, append becomes full_append
        if len(self) == self.size:
            self.append = self.full_append
    
    def get(self):
        """returns a list of size items (newest items)"""
        return list(self)

    
# testing
if __name__ == '__main__':
    size = 5
    ring = RingBuffer(size)
    for x in range(9):
        ring.append(x)
        print ring.get()  # test
 
"""
notice that the left most item is popped to make room
result =
[0]
[0, 1]
[0, 1, 2]
[0, 1, 2, 3]
[0, 1, 2, 3, 4]
[1, 2, 3, 4, 5]
[2, 3, 4, 5, 6]
[3, 4, 5, 6, 7]
[4, 5, 6, 7, 8]
"""
Member Avatar for bumsfeld

So the trick is to make size of ring buffer large enough to retain all pertinent data long enough?
I guess the answer is yes.

Member Avatar for wrybread

Why not something like:

if len(mylist) > 5:
...del mylist[0]

Member Avatar for vegaseat

Why not something like:

if len(mylist) > 5:
...del mylist[0]

Good idea, but if mylist is appended to in several different places in your code than it would not be very practical.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.