Working with multiple threads in Python often leads to high RAM consumption. Unfortunately, automatic garbage collection in child processes isn’t working well. But there are two alternatives:
- When using Pool(), you can specify no. of task after which the child will be restarted resulting in memory release.
p = Pool(processes=4, maxtasksperchild=1000)
- If you use Process(), you can simply delete unwanted objects call gc.collect() inside the child. Note, this may slow down your child process substantially!