Tracing exceptions in multiprocessing in Python

I had problems with debugging my programme using multiprocessing.Pool.

Traceback (most recent call last):
  File "src/homologies2mysql_multi.py", line 294, in <module>
    main()
  File "src/homologies2mysql_multi.py", line 289, in main
    o.noupload, o.verbose)
  File "src/homologies2mysql_multi.py", line 242, in homologies2mysql
    for i, data in enumerate(p.imap_unordered(worker, pairs), 1):
  File "/usr/lib64/python2.6/multiprocessing/pool.py", line 520, in next
    raise value
ValueError: need more than 1 value to unpack

I could run it without multiprocessing, but then I’d have to wait some days for the program to reach the point where it crashes.
Luckily, Python is equipped with traceback, that allows handy tracing of exceptions.
Then, you can add a decorator to problematic function, that will report nice error message:

import traceback, functools, multiprocessing
 
def trace_unhandled_exceptions(func):
    @functools.wraps(func)
    def wrapped_func(*args, **kwargs):
        try:
            return func(*args, **kwargs)
        except:
            print 'Exception in '+func.__name__
            traceback.print_exc()
    return wrapped_func
 
@trace_unhandled_exceptions
def go():
    print(1)
    raise Exception()
    print(2)
 
p = multiprocessing.Pool(1)
 
p.apply_async(go)
p.close()
p.join()

The error message will look like:

1
Exception in go
Traceback (most recent call last):
  File "<stdin>", line 5, in wrapped_func
  File "<stdin>", line 4, in go
Exception

Solution found on StackOverflow.

GATK error: CommandLineGATK : Unsupported major.minor version 51.0

If you see an error: CommandLineGATK : Unsupported major.minor version 51.0), while running new GATK, this means you most likely need newer java. Either:

  • Install the latest java with apt-get (if you have root access and Ubuntu/Debian)
  • sudo apt-get install oracle-java9-installer
  • Or download JRE from Oracle website and and set up system $PATH
    ie. if you have your JRE unpacked in ~/src/jre add these lines to your ~/.bashrc:
  • #java RE
    export PATH=~/src/jre/bin:$PATH
    

    And open new terminal window.

You can check installed JRE version with

java -version

The location of `java` binaries can be checked with

which java