Python: garbage collection fails? -


consider following script:

l = [i in range(int(1e8))] l = [] import gc gc.collect() # 0 gc.get_referrers(l) # [{'__builtins__': <module '__builtin__' (built-in)>, 'l': [], '__package__': none, 'i': 99999999, 'gc': <module 'gc' (built-in)>, '__name__': '__main__', '__doc__': none}] del l gc.collect() # 0 

the point is, after these steps memory usage of python process around 30 % on machine (python 2.6.5, more details on request?). here's excerpt of output of top:

 pid user      pr  ni  virt  res  shr s %cpu %mem    time+  command   5478 moooeeeep 20   0 2397m 2.3g 3428 s    0 29.8   0:09.15 ipython   

resp. ps aux:

moooeeeep 5478  1.0 29.7 2454720 2413516 pts/2 s+   12:39   0:09 /usr/bin/python /usr/bin/ipython gctest.py 

according the docs gc.collect:

not items in free lists may freed due particular implementation, in particular int , float.

does mean, if (temporarily) need large number of different int or float numbers, need export c/c++ because python gc fails release memory?


update

probably interpreter blame, this article suggests:

it’s you’ve created 5 million integers simultaneously alive, , each int object consumes 12 bytes. “for speed”, python maintains internal free list integer objects. unfortunately, free list both immortal , unbounded in size. floats use immortal & unbounded free list.

the problem remains, cannot avoid amount of data (timestamp/value pairs external source). forced drop python , go c/c++ ?


update 2

probably it's indeed case, python implementation causes problem. found this answer conclusively explaining issue , possible workaround.

i've done few tests, , issue occurs cpython 2.x. issue gone in cpython 3.2.2 (it drops memory usage of fresh interpreter) , pypy 1.8 (python 2.7.2) drops down same level new pypy process.

so no, don't need switch language. however, there's solution won't force switch different python implementation.


Comments

Popular posts from this blog

jasper reports - Fixed header in Excel using JasperReports -

media player - Android: mediaplayer went away with unhandled events -

python - ('The SQL contains 0 parameter markers, but 50 parameters were supplied', 'HY000') or TypeError: 'tuple' object is not callable -