Why did I switch back to Raspbian on my RPi server?

Posted on December 11, 2017 by lpryszcz

Frankly, while I’m super happy user of Ubuntu on desktop and servers, I’ve been quite unsatisfied with Ubuntu for Raspberry Pi. Mostly, cause it ruined a few of my SD cards. I’ve realised that writing file access info to the card was one of the key factor here. Unfortunately, Ubuntu for RPi won’t but with noatime… while latest Rasbian has it by defualt.

What I like a lot about Raspbian Stretch Lite, is that beside natively supporting all Raspberry Pi features, it’s also cross platform compatible – it works super well on both, RPi2 and RPi3.

And yes, this blog among few other things is server from RPi2 🙂

On handy docker images

Posted on June 16, 2016 by lpryszcz

Motivated by successful stripping problematic dependencies from Redundans, I have decided to generate smaller Docker image, starting with Alpine Linux image (2Mb / 5Mb after downloading) instead of Ubuntu (49Mb / 122Mb). Previously, I couldn’t really rely on Alpine Linux, because it was impossible to make these problematic dependencies running… But now it’s whole new world of possibilities 😉

There are very few dependencies left, so I have started… (You can find all the commands below).

First, I have check what can be installed from package manager.
Only Python and Perl.
Then I have checked if any of binaries are working.
For example, GapCloser is provided as binary. It took me some time to find source code…
Anyway, none of the binaries worked out of the box. It was expected, as Alpine Linux is super stripped…
I have installed build-base in order to be able to build things.
Additionally, BWA need zlib-dev.
Alpine Linux doesn’t use standard glibc library, but musl-libc (you can read more about differences between the two), so some programmes (ie. BWA) may be quite reluctant to compile.
After some hours of trying & thanks to the help of mp15, I have found a solution, not so complicated 🙂
I have realised, that Dockerfile doesn’t like standard BASH brace expansion, that is working otherwise in Docker Alpine console…
so ls *.{c,h} should be ls *.c *.h
After that, LAST and GapCloser compilation were easy, relatively 😉

Below, you can find the code from Docker file (without RUN commands).

apk add --update --no-cache python perl bash wget build-base zlib-dev
mkdir -p /root/src && cd /root/src && wget http://downloads.sourceforge.net/project/bio-bwa/bwa-0.7.15.tar.bz2 && tar xpfj bwa-0.7.15.tar.bz2 && ln -s bwa-0.7.15 bwa && cd bwa && \
cp kthread.c kthread.c.org && echo "#include <stdint.h>" > kthread.c && cat kthread.c.org >> kthread.c && \
sed -ibak 's/u_int32_t/uint32_t/g' `grep -l u_int32_t *.c *.h` && make && cp bwa /bin/ && \
cd /root/src && wget http://liquidtelecom.dl.sourceforge.net/project/soapdenovo2/GapCloser/src/r6/GapCloser-src-v1.12-r6.tgz && tar xpfz GapCloser-src-v1.12-r6.tgz && ln -s v1.12-r6/ GapCloser && cd GapCloser && make && cp bin/GapCloser /bin/ && \
cd /root/src && wget http://last.cbrc.jp/last-744.zip && unzip last-744.zip && ln -s last-744 last && cd last && make && make install && \
cd /root/src && rm -r last* bwa* GapCloser* v* 

# SSPACE && redundans in /root/srt
cd /root/src && wget -q http://www.baseclear.com/base/download/41SSPACE-STANDARD-3.0_linux-x86_64.tar.gz && tar xpfz 41SSPACE-STANDARD-3.0_linux-x86_64.tar.gz && ln -s SSPACE-STANDARD-3.0_linux-x86_64 SSPACE && wget -O- -q http://cpansearch.perl.org/src/GBARR/perl5.005_03/lib/getopts.pl > SSPACE/dotlib/getopts.pl && \
wget --no-check-certificate -q -O redundans.tgz https://github.com/lpryszcz/redundans/archive/master.tar.gz && tar xpfz redundans.tgz && mv redundans-master redundans && ln -s /root/src/redundans /redundans && rm *gz

apk del wget build-base zlib-dev 
apk add libstdc++

After building & pushing, I have noticed that Alpine-based image is slightly smaller (99Mb), than the one based on Ubuntu (127Mb). Surprisingly, Alpine-based image is larger (273Mb) than Ubuntu-based (244Mb) after downloading. So, I’m afraid all of these hours didn’t really bring any substantial reduction in the image size.

Conclusion?
I was very motivated to build my application on Alpine Linux and expected substantial size reduction. But I’d say that relying on Alpine Linux image doesn’t always pay off in terms of smaller image size, forget about production time… And this I know from my own experience.
But maybe I didn’t something wrong? I’d be really glad for some advices/comments!

Nevertheless, stripping a few dependencies from my application (namely Biopython, numpy & scipy), resulted in much more compact image even using Ubuntu-based image (127Mb vs 191Mb; and 244Mb vs 440Mb after downloading). So I think this is the way to go 🙂

Serving IPython notebook on public domain

Posted on October 25, 2015 by lpryszcz

I’ve been involved in teaching basic programming in Python. There are several good tutorials and on-line courses (just to mention Python@CodeCademy), but I’ve recognised there is a need for some interactive workplace for the students. I’ve got an idea to setup IPython in public domain, as many of the students don’t have Python installed locally or miss certain dependencies…
The task of installing IPython and serving it in publicly seems very easy… But I’ve encountered numerous difficulties on the way, caused by different versions of IPython (ie. split into Jupyter in v4), Apache configuration and firewall setup, just to mention a few. Anyway, I’ve succeeded and I’ve decided to share my experiences here 🙂
First of all, I strongly recommend setting up separate user for serving IPython, as only this way your personal files will be safe ?

Install IPython notebook and prepare new user

# install python-dev and build essentials
sudo apt-get install build-essential python-dev

# install ipython; v3 is recommended
sudo pip install ipython[all]==3.2.1

# create new user
sudo adduser ipython

# login as new user
su ipython

Configure IPython notebook

# create new profile
ipython profile create nbsever

# generate pass and checksum
ipython -c "from IPython.lib import passwd; passwd()"
# enter your password twice, save it and copy password hash
## Out[1]: 'sha1:[your hashed password here]'

# add to ~/.ipython/profile_nbserver/ipython_notebook_config.py after `c = get_config()`
c.NotebookApp.ip = 'localhost'
c.NotebookApp.open_browser = False
c.NotebookApp.port = 8889
c.NotebookApp.base_url = '/ipython'
c.NotebookApp.password = u'sha1:[your hashed password here]'
c.NotebookApp.allow_origin='*'

# create some directory for notebook files ie. ~/Public/ipython
mkdir -p ~/Public/ipython
cd ~/Public/ipython

# start notebook server
ipython notebook --profile=nbserver

Configure Apache2

# enable mods
sudo a2enmod proxy proxy_http proxy_wstunnel
sudo service apache2 restart

# add ipython proxy config to your enabled site ie. /etc/apache2/sites-available/000-default.conf
    # IPython
    <Location "/ipython" >
        ProxyPass http://localhost:8889/ipython
        ProxyPassReverse http://localhost:8889/ipython
    </Location>

    <Location "/ipython/api/kernels/" >
        ProxyPass        ws://localhost:8889/ipython/api/kernels/
        ProxyPassReverse ws://localhost:8889/ipython/api/kernels/
    </Location>
    #END
      
# restart apache2
sudo service apache2 restart

Your public IPython will be accessible at http://yourdomain.com/ipython .
The longest time it took me to realise that c.NotebookApp.allow_origin='*' line is crucial in IPython notebook configuration, otherwise the kernel is loosing connection with an error ‘Connection failed‘ or ‘WebSocket error‘. Additionally, in one of the servers I’ve been trying, there is proxy setup that block some ports high ports, thus it was impossible to connect to WebSocket even with ApacheProxy setup…
If you want to read more especially about setting SSL-enabled notebook, have a look at jupyter documentation.

Apache2 reading from sshfs share

Posted on May 25, 2015 by lpryszcz

Today, I have encountered problems trying to read data from sshfs share in apache2. I was getting 403 Forbidden error. It turned out you need to enable other_user in sshfs, so other users than the one mounting the share can access the data, as apache2 is using www-data user.

# uncomment last line of /etc/fuse.conf
# Allow non-root users to specify the allow_other or allow_root mount options.
user_allow_other
 
# enable other_user and read access by non-root
sudo chmod a+r /etc/fuse.conf
 
# remount
sudo umount DESTINATION
sshfs -o allow_other SHARE DESTINATION

Inspired by serverfault and unix.stackexchange.

Change temporary directory in Linux

Posted on May 25, 2015 by lpryszcz

Sometimes, the size of / (root) mount is limited and in result some processes requiring large /tmp may fail. This can be solved by setting environmental variable TMPDIR:

mkdir -p /home/$USER/tmp
TMPDIR=$(mktemp -d /home/$USER/tmp/XXXX)
TMP=$TMPDIR
TEMP=$TMPDIR
export TMPDIR TMP TEMP

More info on serverfault.

Transfer WordPress to Amazon EC2

Posted on March 26, 2015 by lpryszcz

After rather successful year of using WordPress, I have decided to move my blog to AWS. I was considering the move for long time, motivated by Free Tier and finally I found some time to do it.

At first, I have created WordPress Stack using CloudFormation, but personally I prefer Ubuntu over Amazon Linux and I will focus on configuration of Ubuntu EC2 instance here.

Export your existing blog
WP-Admin > Tools > Export
Login to AWS console and Create Key Pair
Launch EC2 instance
I use Ubuntu HVM. I recommend t2.micro, as it’s free for the first year. You should specify created/uploaded key.
Login to your EC2 instance using Public DNS or IP and your key
```
ssh -i .aws/your_key.pem ubuntu@ec2xxxxx.compute.amazonaws.com
```
NOTE: you key should be readable only by you. To achieve that, you can do:
```
chmod 600 .aws/your_key.pem
```

Configure Ubuntu

sudo apt-get update && sudo apt-get upgrade
sudo apt-get install apache2 php5 php5-mysql libapache2-mod-php5 libapache2-mod-auth-mysql mysql-server

Configure MySQL

sudo mysql_secure_installation
mysql -uroot -p

CREATE DATABASE wordpress;
CREATE USER 'wordpress' IDENTIFIED BY 'SOMEPASS';
GRANT ALL ON wordpress.* TO 'wordpress';

Configure wordpress

sudo -i
cd /var/www/html/
wget https://wordpress.org/latest.tar.gz
tar xpfz latest.tar.gz
rm latest.tar.gz
cd wordpress/
mv wp-config-sample.php wp-config.php
sudo chown -R www-data:www-data /var/www/html

# edit wp-config.php
define('DB_NAME', 'wordpress');
define('DB_USER', 'wordpress');
define('DB_PASSWORD', 'SOMEPASS');
define('DB_HOST', 'localhost');

Configure Apache

# edit /etc/apache2/sites-available/wordpress.conf

ServerName ec2xxxxx.compute.amazonaws.com
ServerAlias YOURDOMAIN.COM
DocumentRoot /var/www/html/wordpress
DirectoryIndex index.php

AllowOverride All
Order Deny,Allow
Allow from all

# enable wordpress in apache2
sudo a2ensite wordpress
sudo service apache2 restart

Enable HTTP access to your EC2 instance
Go to EC2 console > Instances > Select you instance > Description >
Click on your `Security group` > Select Inbound > Edit > Add rule > HTTP > Save
Point your webrowser to your EC2 instance: http://ec2xxxxx.compute.amazonaws.com/
Setup your wordpress account
Upload dumped wordpress data
WP-Admin > Tools > Import > WordPress > > Upload file import
NOTES:
You will need to install WordPress Importer plugin.
Assign post to correct user.
Don’t forget to Import Attachments!
Install your favourite plugins and themes
As for plugins, I strongly recommend: JetPack, SyntaxHighlighter Evolved, Google Analytics Dashboard for WP and BackUpWordPress or ajax-load-more.
Add favicon
Copy selected favicon.ico to /var/www/html/wordpress

Voilà!
BTW: You may want to increase security of your instance and setup swap just in case memory usage exceeds your EC2 instance size.

BioInfoExpert

Bioinformatics day-by-day

Category Archives: AWS

Why did I switch back to Raspbian on my RPi server?

On handy docker images

Serving IPython notebook on public domain

Change temporary directory in Linux

Transfer WordPress to Amazon EC2