Python: garbage collection fails? -
consider following script:
l = [i in range(int(1e8))] l = [] import gc gc.collect() # 0 gc.get_referrers(l) # [{'__builtins__': <module '__builtin__' (built-in)>, 'l': [], '__package__': none, 'i': 99999999, 'gc': <module 'gc' (built-in)>, '__name__': '__main__', '__doc__': none}] del l gc.collect() # 0
the point is, after these steps memory usage of python process around 30 % on machine (python 2.6.5, more details on request?). here's excerpt of output of top:
pid user pr ni virt res shr s %cpu %mem time+ command 5478 moooeeeep 20 0 2397m 2.3g 3428 s 0 29.8 0:09.15 ipython
resp. ps aux
:
moooeeeep 5478 1.0 29.7 2454720 2413516 pts/2 s+ 12:39 0:09 /usr/bin/python /usr/bin/ipython gctest.py
according the docs gc.collect
:
not items in free lists may freed due particular implementation, in particular
int
,float
.
does mean, if (temporarily) need large number of different int
or float
numbers, need export c/c++ because python gc fails release memory?
update
probably interpreter blame, this article suggests:
it’s you’ve created 5 million integers simultaneously alive, , each int object consumes 12 bytes. “for speed”, python maintains internal free list integer objects. unfortunately, free list both immortal , unbounded in size. floats use immortal & unbounded free list.
the problem remains, cannot avoid amount of data (timestamp/value pairs external source). forced drop python , go c/c++ ?
update 2
probably it's indeed case, python implementation causes problem. found this answer conclusively explaining issue , possible workaround.
i've done few tests, , issue occurs cpython 2.x. issue gone in cpython 3.2.2 (it drops memory usage of fresh interpreter) , pypy 1.8 (python 2.7.2) drops down same level new pypy process.
so no, don't need switch language. however, there's solution won't force switch different python implementation.
Comments
Post a Comment