Multiprocessing in Python and garbage collection

Working with multiple threads in Python often leads to high RAM consumption. Unfortunately, automatic garbage collection in child processes isn’t working well. But there are two alternatives:

• When using Pool(), you can specify number of task after which the child will be restarted resulting in memory release.

p = Pool(processes=4, maxtasksperchild=1000)

• If you use Process(), you can simply delete unwanted objects and call gc.collect() inside the child. Note, this may slow down your child process substantially!

Progress of long processes in BASH   Encrypted swapfile