On handy docker images

Motivated by successful stripping problematic dependencies from Redundans, I have decided to generate smaller Docker image, starting with Alpine Linux image (2Mb / 5Mb after downloading) instead of Ubuntu (49Mb / 122Mb). Previously, I couldn’t really rely on Alpine Linux, because it was impossible to make these problematic dependencies running… But now it’s whole new world of possibilities 😉

There are very few dependencies left, so I have started… (You can find all the commands below).

  1. First, I have check what can be installed from package manager.
    Only Python and Perl.

  2. Then I have checked if any of binaries are working.
    For example, GapCloser is provided as binary. It took me some time to find source code…
    Anyway, none of the binaries worked out of the box. It was expected, as Alpine Linux is super stripped…

  3. I have installed build-base in order to be able to build things.
    Additionally, BWA need zlib-dev.

  4. Alpine Linux doesn’t use standard glibc library, but musl-libc (you can read more about differences between the two), so some programmes (ie. BWA) may be quite reluctant to compile.
    After some hours of trying & thanks to the help of mp15, I have found a solution, not so complicated 🙂

  5. I have realised, that Dockerfile doesn’t like standard BASH brace expansion, that is working otherwise in Docker Alpine console…
    so ls *.{c,h} should be ls *.c *.h

  6. After that, LAST and GapCloser compilation were easy, relatively 😉

Below, you can find the code from Docker file (without RUN commands).

apk add --update --no-cache python perl bash wget build-base zlib-dev
mkdir -p /root/src && cd /root/src && wget http://downloads.sourceforge.net/project/bio-bwa/bwa-0.7.15.tar.bz2 && tar xpfj bwa-0.7.15.tar.bz2 && ln -s bwa-0.7.15 bwa && cd bwa && \
cp kthread.c kthread.c.org && echo "#include <stdint.h>" > kthread.c && cat kthread.c.org >> kthread.c && \
sed -ibak 's/u_int32_t/uint32_t/g' `grep -l u_int32_t *.c *.h` && make && cp bwa /bin/ && \
cd /root/src && wget http://liquidtelecom.dl.sourceforge.net/project/soapdenovo2/GapCloser/src/r6/GapCloser-src-v1.12-r6.tgz && tar xpfz GapCloser-src-v1.12-r6.tgz && ln -s v1.12-r6/ GapCloser && cd GapCloser && make && cp bin/GapCloser /bin/ && \
cd /root/src && wget http://last.cbrc.jp/last-744.zip && unzip last-744.zip && ln -s last-744 last && cd last && make && make install && \
cd /root/src && rm -r last* bwa* GapCloser* v* 

# SSPACE && redundans in /root/srt
cd /root/src && wget -q http://www.baseclear.com/base/download/41SSPACE-STANDARD-3.0_linux-x86_64.tar.gz && tar xpfz 41SSPACE-STANDARD-3.0_linux-x86_64.tar.gz && ln -s SSPACE-STANDARD-3.0_linux-x86_64 SSPACE && wget -O- -q http://cpansearch.perl.org/src/GBARR/perl5.005_03/lib/getopts.pl > SSPACE/dotlib/getopts.pl && \
wget --no-check-certificate -q -O redundans.tgz https://github.com/lpryszcz/redundans/archive/master.tar.gz && tar xpfz redundans.tgz && mv redundans-master redundans && ln -s /root/src/redundans /redundans && rm *gz

apk del wget build-base zlib-dev 
apk add libstdc++

After building & pushing, I have noticed that Alpine-based image is slightly smaller (99Mb), than the one based on Ubuntu (127Mb). Surprisingly, Alpine-based image is larger (273Mb) than Ubuntu-based (244Mb) after downloading. So, I’m afraid all of these hours didn’t really bring any substantial reduction in the image size.

Conclusion?
I was very motivated to build my application on Alpine Linux and expected substantial size reduction. But I’d say that relying on Alpine Linux image doesn’t always pay off in terms of smaller image size, forget about production time… And this I know from my own experience.
But maybe I didn’t something wrong? I’d be really glad for some advices/comments!

Nevertheless, stripping a few dependencies from my application (namely Biopython, numpy & scipy), resulted in much more compact image even using Ubuntu-based image (127Mb vs 191Mb; and 244Mb vs 440Mb after downloading). So I think this is the way to go 🙂

Serving IPython notebook on public domain

I’ve been involved in teaching basic programming in Python. There are several good tutorials and on-line courses (just to mention Python@CodeCademy), but I’ve recognised there is a need for some interactive workplace for the students. I’ve got an idea to setup IPython in public domain, as many of the students don’t have Python installed locally or miss certain dependencies…
The task of installing IPython and serving it in publicly seems very easy… But I’ve encountered numerous difficulties on the way, caused by different versions of IPython (ie. split into Jupyter in v4), Apache configuration and firewall setup, just to mention a few. Anyway, I’ve succeeded and I’ve decided to share my experiences here 🙂
First of all, I strongly recommend setting up separate user for serving IPython, as only this way your personal files will be safe ?

  1. Install IPython notebook and prepare new user
  2. # install python-dev and build essentials
    sudo apt-get install build-essential python-dev
    
    # install ipython; v3 is recommended
    sudo pip install ipython[all]==3.2.1
    
    # create new user
    sudo adduser ipython
    
    # login as new user
    su ipython
    
  3. Configure IPython notebook
  4. # create new profile
    ipython profile create nbsever
    
    # generate pass and checksum
    ipython -c "from IPython.lib import passwd; passwd()"
    # enter your password twice, save it and copy password hash
    ## Out[1]: 'sha1:[your hashed password here]'
    
    # add to ~/.ipython/profile_nbserver/ipython_notebook_config.py after `c = get_config()`
    c.NotebookApp.ip = 'localhost'
    c.NotebookApp.open_browser = False
    c.NotebookApp.port = 8889
    c.NotebookApp.base_url = '/ipython'
    c.NotebookApp.password = u'sha1:[your hashed password here]'
    c.NotebookApp.allow_origin='*'
    
    # create some directory for notebook files ie. ~/Public/ipython
    mkdir -p ~/Public/ipython
    cd ~/Public/ipython
    
    # start notebook server
    ipython notebook --profile=nbserver
    
  5. Configure Apache2
  6. # enable mods
    sudo a2enmod proxy proxy_http proxy_wstunnel
    sudo service apache2 restart
    
    # add ipython proxy config to your enabled site ie. /etc/apache2/sites-available/000-default.conf
        # IPython
        <Location "/ipython" >
            ProxyPass http://localhost:8889/ipython
            ProxyPassReverse http://localhost:8889/ipython
        </Location>
    
        <Location "/ipython/api/kernels/" >
            ProxyPass        ws://localhost:8889/ipython/api/kernels/
            ProxyPassReverse ws://localhost:8889/ipython/api/kernels/
        </Location>
        #END
          
    # restart apache2
    sudo service apache2 restart
    

Your public IPython will be accessible at http://yourdomain.com/ipython .
The longest time it took me to realise that c.NotebookApp.allow_origin='*' line is crucial in IPython notebook configuration, otherwise the kernel is loosing connection with an error ‘Connection failed‘ or ‘WebSocket error‘. Additionally, in one of the servers I’ve been trying, there is proxy setup that block some ports high ports, thus it was impossible to connect to WebSocket even with ApacheProxy setup…
If you want to read more especially about setting SSL-enabled notebook, have a look at jupyter documentation.

Apache2 reading from sshfs share

Today, I have encountered problems trying to read data from sshfs share in apache2. I was getting 403 Forbidden error. It turned out you need to enable other_user in sshfs, so other users than the one mounting the share can access the data, as apache2 is using www-data user.

# uncomment last line of /etc/fuse.conf
# Allow non-root users to specify the allow_other or allow_root mount options.
user_allow_other
 
# enable other_user and read access by non-root
sudo chmod a+r /etc/fuse.conf
 
# remount
sudo umount DESTINATION
sshfs -o allow_other SHARE DESTINATION

Inspired by serverfault and unix.stackexchange.

Change temporary directory in Linux

Sometimes, the size of / (root) mount is limited and in result some processes requiring large /tmp may fail. This can be solved by setting environmental variable TMPDIR:

mkdir -p /home/$USER/tmp
TMPDIR=$(mktemp -d /home/$USER/tmp/XXXX)
TMP=$TMPDIR
TEMP=$TMPDIR
export TMPDIR TMP TEMP

More info on serverfault.

Transfer WordPress to Amazon EC2

After rather successful year of using WordPress, I have decided to move my blog to AWS. I was considering the move for long time, motivated by Free Tier and finally I found some time to do it.

At first, I have created WordPress Stack using CloudFormation, but personally I prefer Ubuntu over Amazon Linux and I will focus on configuration of Ubuntu EC2 instance here.

  1. Export your existing blog
    WP-Admin > Tools > Export

  2. Login to AWS console and Create Key Pair
  3. Launch EC2 instance
    I use Ubuntu HVM. I recommend t2.micro, as it’s free for the first year. You should specify created/uploaded key.

  4. Login to your EC2 instance using Public DNS or IP and your key
    ssh -i .aws/your_key.pem ubuntu@ec2xxxxx.compute.amazonaws.com

    NOTE: you key should be readable only by you. To achieve that, you can do:

    chmod 600 .aws/your_key.pem
  5. Configure Ubuntu
    sudo apt-get update && sudo apt-get upgrade
    sudo apt-get install apache2 php5 php5-mysql libapache2-mod-php5 libapache2-mod-auth-mysql mysql-server
    
  6. Configure MySQL
    sudo mysql_secure_installation
    mysql -uroot -p
    
    CREATE DATABASE wordpress;
    CREATE USER 'wordpress' IDENTIFIED BY 'SOMEPASS';
    GRANT ALL ON wordpress.* TO 'wordpress';
    
  7. Configure wordpress
    sudo -i
    cd /var/www/html/
    wget https://wordpress.org/latest.tar.gz
    tar xpfz latest.tar.gz
    rm latest.tar.gz
    cd wordpress/
    mv wp-config-sample.php wp-config.php
    sudo chown -R www-data:www-data /var/www/html
    
    # edit wp-config.php
    define('DB_NAME', 'wordpress');
    define('DB_USER', 'wordpress');
    define('DB_PASSWORD', 'SOMEPASS');
    define('DB_HOST', 'localhost');
    
  8. Configure Apache
    # edit /etc/apache2/sites-available/wordpress.conf
    
    ServerName ec2xxxxx.compute.amazonaws.com
    ServerAlias YOURDOMAIN.COM
    DocumentRoot /var/www/html/wordpress
    DirectoryIndex index.php
    
    AllowOverride All
    Order Deny,Allow
    Allow from all
    
    # enable wordpress in apache2
    sudo a2ensite wordpress
    sudo service apache2 restart
    
  9. Enable HTTP access to your EC2 instance
    Go to EC2 console > Instances > Select you instance > Description >
    Click on your `Security group` > Select Inbound > Edit > Add rule > HTTP > Save

  10. Point your webrowser to your EC2 instance: http://ec2xxxxx.compute.amazonaws.com/
  11. Setup your wordpress account
  12. Upload dumped wordpress data
    WP-Admin > Tools > Import > WordPress > > Upload file import
    NOTES:
    You will need to install WordPress Importer plugin.

  13. Assign post to correct user.
    Don’t forget to Import Attachments!

  14. Install your favourite plugins and themes
    As for plugins, I strongly recommend: JetPack, SyntaxHighlighter Evolved, Google Analytics Dashboard for WP and BackUpWordPress or ajax-load-more.

  15. Add favicon
    Copy selected favicon.ico to /var/www/html/wordpress

Voilà!
BTW: You may want to increase security of your instance and setup swap just in case memory usage exceeds your EC2 instance size.

EC2 instance safety instructions

  1. Add non-default user and add it to sudo group
    sudo adduser USERNAME
    sudo usermod -a -G sudo USERNAME
    # switch user
    su USERNAME
    
  2. Edit /etc/ssh/sshd_config
    # change port to non-default port ie 3434 
    # & add this port to your instance Security Groups > Inbound
    Port 3434
    
    # enable password authentication
    PasswordAuthentication yes
    
    # restart ssh
    sudo service ssh restart 
    
    ###
    # make sure you can login with 
    # your new username before continuing
    ###
    
    # disable root login without password by commenting: 
    #PermitRootLogin without-password
    
    # restart ssh
    sudo service ssh restart 
    
  3. Secure MySQL isntallation
    sudo mysql_secure_installation
  4. Reboot
    sudo reboot