Almost every legacy Python script I audit does the same thing: a plain list holding a history log or a rolling average, followed by confusion about why the memory footprint keeps growing and why pop(0) is dragging the server down. For sliding windows and history buffers, the Python Deque is what you want instead.
Fourteen years of backend work has taught me that “simple” and “scalable” are not the same thing. A list is a general-purpose hammer. But when you only care about the last 100 entries, the list itself turns into the bottleneck, because of how Python moves memory around underneath.
List vs Python deque: where the time goes
list.pop(0) and list.insert(0, value) both force Python to shift every remaining element in memory. That is an O(n) operation. With 10,000 items in the list, you pay that shift on every new reading, which is CPU time you get nothing for. When performance matters, that is the point to refactor to a Python Deque.
collections.deque, short for double-ended queue, appends and pops from either side in O(1). It holds its contents as a doubly linked list of blocks, so sliding the window does not shift memory at all.
Using maxlen for automatic history buffers
The part worth learning is the maxlen parameter. Set a limit and the deque does the sliding for you: append to a full deque and it drops the oldest item off the opposite end, so you never write the pop() yourself.
from collections import deque
# Initialize a history buffer limited to 3 items
history = deque(maxlen=3)
history.append("Search 1")
history.append("Search 2")
history.append("Search 3")
# This next append automatically drops "Search 1"
history.append("Search 4")
print(list(history))
# Output: ['Search 2', 'Search 3', 'Search 4']If you want the surrounding code to stay predictable as these structures grow, my guide on Python Type Annotations covers the annotation side of it.
Rolling averages over a live data stream
A moving average over a sensor stream is routine work in data science and IoT monitoring. With a Python Deque you keep the sliding window without any index bookkeeping, which keeps the function short and quick.
def get_rolling_average(data_stream, window_size=5):
window = deque(maxlen=window_size)
for val in data_stream:
window.append(val)
if len(window) == window_size:
yield sum(window) / window_size
# Calculating the average of a sensor reading stream
readings = [20, 22, 24, 21, 23, 25, 24]
print(list(get_rolling_average(readings, window_size=3)))When the loop itself becomes the bottleneck, I have written about calling Rust from Python for the heavy lifting.
Thread safety in CPython
Race conditions in multithreaded code catch people out. In CPython, a deque’s append() and popleft() are atomic and thread safe, which makes it a good fit for producer-consumer patterns where one thread feeds data and another drains it.
The official Python documentation has the full API, and there are longer list-versus-deque comparisons on Real Python and DEV Community.
If Python Deque work like this is eating your dev hours, I can take it on. I have been writing WordPress and backend performance code since the 4.x days.
When to reach for a deque
Lists are not a catch-all. If you add or remove from the front of a sequence often, or you need a fixed-size buffer, use a Python Deque. At any real scale it is faster, and the code around it gets shorter.
