Asked By – Jared
I have a few related questions regarding memory usage in the following example.
If I run in the interpreter,
foo = ['bar' for _ in xrange(10000000)]
the real memory used on my machine goes up to
80.9mb. I then,
real memory goes down, but only to
30.4mb. The interpreter uses
4.4mbbaseline so what is the advantage in not releasing
26mbof memory to the OS? Is it because Python is “planning ahead”, thinking that you may use that much memory again?
Why does it release
50.5mbin particular – what is the amount that is released based on?
Is there a way to force Python to release all the memory that was used (if you know you won’t be using that much memory again)?
This question is different from How can I explicitly free memory in Python?
because this question primarily deals with the increase of memory usage from baseline even after the interpreter has freed objects via garbage collection (with use of
gc.collect or not).
Now we will see solution for issue: Releasing memory in Python
Memory allocated on the heap can be subject to high-water marks. This is complicated by Python’s internal optimizations for allocating small objects (
PyObject_Malloc) in 4 KiB pools, classed for allocation sizes at multiples of 8 bytes — up to 256 bytes (512 bytes in 3.3). The pools themselves are in 256 KiB arenas, so if just one block in one pool is used, the entire 256 KiB arena will not be released. In Python 3.3 the small object allocator was switched to using anonymous memory maps instead of the heap, so it should perform better at releasing memory.
Additionally, the built-in types maintain freelists of previously allocated objects that may or may not use the small object allocator. The
int type maintains a freelist with its own allocated memory, and clearing it requires calling
PyInt_ClearFreeList(). This can be called indirectly by doing a full
Try it like this, and tell me what you get. Here’s the link for psutil.Process.memory_info.
import os import gc import psutil proc = psutil.Process(os.getpid()) gc.collect() mem0 = proc.memory_info().rss # create approx. 10**7 int objects and pointers foo = ['abc' for x in range(10**7)] mem1 = proc.memory_info().rss # unreference, including x == 9999999 del foo, x mem2 = proc.memory_info().rss # collect() calls PyInt_ClearFreeList() # or use ctypes: pythonapi.PyInt_ClearFreeList() gc.collect() mem3 = proc.memory_info().rss pd = lambda x2, x1: 100.0 * (x2 - x1) / mem0 print "Allocation: %0.2f%%" % pd(mem1, mem0) print "Unreference: %0.2f%%" % pd(mem2, mem1) print "Collect: %0.2f%%" % pd(mem3, mem2) print "Overall: %0.2f%%" % pd(mem3, mem0)
Allocation: 3034.36% Unreference: -752.39% Collect: -2279.74% Overall: 2.23%
I switched to measuring relative to the process VM size to eliminate the effects of other processes in the system.
The C runtime (e.g. glibc, msvcrt) shrinks the heap when contiguous free space at the top reaches a constant, dynamic, or configurable threshold. With glibc you can tune this with
mallopt (M_TRIM_THRESHOLD). Given this, it isn’t surprising if the heap shrinks by more — even a lot more — than the block that you
range doesn’t create a list, so the test above won’t create 10 million
int objects. Even if it did, the
int type in 3.x is basically a 2.x
long, which doesn’t implement a freelist.
This question is answered By – Eryk Sun
This answer is collected from stackoverflow and reviewed by FixPython community admins, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0