How to setup NVIDIA / CUDA for accelerated computation in Ubuntu 18.04 / Gnome / X11 workstation?

I’ve experienced a bit of difficulties when I’ve tried to enable CUDA in my workstation. Those were mostly related to system lags while I’ve been performing CUDA computations. That was because Gnome/Xserver were using NVIDIA card. I’ve realised you’d be much better of using your discrete graphic card for the system and leaving NVIDIA GPU only for serious tasks 🙂 Note, this will disable NVIDIA GPU for GNOME / X11 and also for gaming, so be aware…

Below I’ll describe briefly how I’ve installed NVIDIA drivers and configured Ubuntu 18.04 with Gnome3 and Xserver for comfortable CUDA computations.

The best if you install CUDA toolking and drivers before you plug the card, as just plugging the card may cause issues with running Ubuntu otherwise (it did in my case). In order to install NVIDIA drivers, just follow official Nvidia guide. 

Then after reboot plug the card to your computer and in the BIOS select integrated card as your main card. In my BIOS it was under Advanced > Built-in Device Options > Select Boot card > CPU integrated or Nvidia GPU.

If you experience any problems, uncomment WaylandEnable=false in /etc/gdm3/custom.conf to use X11 for GDM and Gnome. Don’t do that, if you plan to use Wayland!

Now make sure you have Nvidia plugged in and working.

# show available graphic cards
lspci -k | grep -A 2 -i "VGA"

If you installed the drivers from NVIDIA website, you may need to restore java

sudo rm /etc/alternatives/java
jpath=/opt/java/jre1.8.0_211/bin
sudo ln -s $jpath/java /etc/alternatives/java

Make sure to switch to integrated graphics card using either

  • nvidia-settings  > PRIME Profiles and select Intel (Power Saving Mode) (this should work for both, X11 and Wayland)
  • or by editing /etc/X11/xorg.conf to something like that (if you use Wayland, this won’t work!)
Section "Device"
         Identifier "Intel"
         Driver "intel"
 Option "AccelMethod" "uxa"
 EndSection

Reboot your system and make sure Gnome isn’t using NVIDIA GPU (there should be no processes running on your GPU after reboot).

# check processed running on GPU
nvidia-smi

Now, when you run any CUDA computation, your system shouldn’t be affected by high NVIDIA GPU usage.

Create book of abstracts from spreadsheet / google forms

Lately a friend of mine complained about interoperability of abstract submissions from numerous applicants. Having the Book of Abstracts is crucial and we faced similar problem organising #NGSchool events.

Note, you’ll need to be somewhat familiar with LaTeX in order to edit the main.tex file to your liking. If you are not afraid of that, the way to proceed is as follows:

  1. Create google form to collect necessary info, such at this one
  2. Create a new spreadsheet to accumulate responses: Responses > Create new spreadsheet
  3. Download responses spreadsheet as Abstracts.xlsx
  4. Clone abstracts repository
  5. git clone https://github.com/lpryszcz/abstracts.git
    cd abstracts
    # install dependencies
    sudo apt install texlive-base texlive-latex-recommended texlive-fonts-recommended texlive-latex-extra make
    
  6. Edit main.tex to your liking
  7. Copy Abstracts.xlsx to the repository
  8. Create pdf
  9. # prepare abstracts.tex
    ./xls2tex.py
    
    # create main.pdf
    make all
    
    # in the case of problems, just run again this point, but first remove the clutter
    rm main.{aux,blg,log,out,toc,pdf}
    

    You’ll find the abstract book in main.pdf.

Reading documents on Kindle

Recently, I wanted to read some documents (i.e. from Google Docs) on my Kindle. I was interested in exporting .mobi, as reading PDFs on Kindle is really annoying.
I found converting .doc into .mobi not a trivial task. Luckily, nearly everything is possible in Ubuntu with a few steps:

  1. Download your document as .doc / .docx
  2. Open .doc / docx in LibreOffice / OpenOffice and export to ePub (you’ll need to install Writer2ePub before)
  3. Import ePub into Calibre and export it to your favourite eBook reader.

Inspired by Quora.

Transfer WordPress to Amazon EC2

After rather successful year of using WordPress, I have decided to move my blog to AWS. I was considering the move for long time, motivated by Free Tier and finally I found some time to do it.

At first, I have created WordPress Stack using CloudFormation, but personally I prefer Ubuntu over Amazon Linux and I will focus on configuration of Ubuntu EC2 instance here.

  1. Export your existing blog
    WP-Admin > Tools > Export

  2. Login to AWS console and Create Key Pair
  3. Launch EC2 instance
    I use Ubuntu HVM. I recommend t2.micro, as it’s free for the first year. You should specify created/uploaded key.

  4. Login to your EC2 instance using Public DNS or IP and your key
    ssh -i .aws/your_key.pem ubuntu@ec2xxxxx.compute.amazonaws.com

    NOTE: you key should be readable only by you. To achieve that, you can do:

    chmod 600 .aws/your_key.pem
  5. Configure Ubuntu
    sudo apt-get update && sudo apt-get upgrade
    sudo apt-get install apache2 php5 php5-mysql libapache2-mod-php5 libapache2-mod-auth-mysql mysql-server
    
  6. Configure MySQL
    sudo mysql_secure_installation
    mysql -uroot -p
    
    CREATE DATABASE wordpress;
    CREATE USER 'wordpress' IDENTIFIED BY 'SOMEPASS';
    GRANT ALL ON wordpress.* TO 'wordpress';
    
  7. Configure wordpress
    sudo -i
    cd /var/www/html/
    wget https://wordpress.org/latest.tar.gz
    tar xpfz latest.tar.gz
    rm latest.tar.gz
    cd wordpress/
    mv wp-config-sample.php wp-config.php
    sudo chown -R www-data:www-data /var/www/html
    
    # edit wp-config.php
    define('DB_NAME', 'wordpress');
    define('DB_USER', 'wordpress');
    define('DB_PASSWORD', 'SOMEPASS');
    define('DB_HOST', 'localhost');
    
  8. Configure Apache
    # edit /etc/apache2/sites-available/wordpress.conf
    
    ServerName ec2xxxxx.compute.amazonaws.com
    ServerAlias YOURDOMAIN.COM
    DocumentRoot /var/www/html/wordpress
    DirectoryIndex index.php
    
    AllowOverride All
    Order Deny,Allow
    Allow from all
    
    # enable wordpress in apache2
    sudo a2ensite wordpress
    sudo service apache2 restart
    
  9. Enable HTTP access to your EC2 instance
    Go to EC2 console > Instances > Select you instance > Description >
    Click on your `Security group` > Select Inbound > Edit > Add rule > HTTP > Save

  10. Point your webrowser to your EC2 instance: http://ec2xxxxx.compute.amazonaws.com/
  11. Setup your wordpress account
  12. Upload dumped wordpress data
    WP-Admin > Tools > Import > WordPress > > Upload file import
    NOTES:
    You will need to install WordPress Importer plugin.

  13. Assign post to correct user.
    Don’t forget to Import Attachments!

  14. Install your favourite plugins and themes
    As for plugins, I strongly recommend: JetPack, SyntaxHighlighter Evolved, Google Analytics Dashboard for WP and BackUpWordPress or ajax-load-more.

  15. Add favicon
    Copy selected favicon.ico to /var/www/html/wordpress

VoilĂ !
BTW: You may want to increase security of your instance and setup swap just in case memory usage exceeds your EC2 instance size.

Speed of USB 3.0 external hard drives

I’m using several external drives for backup purposes. I was interested whether there are some differences in terms of read/write/access time between these. I have tested two drives connected directly to USB3.0 port and through USB3.0 hub:

  1. HITACHI Touro 2.5″ 500GB USB 3.0
  2. WD My Passport 2TB USB 3.0

Touro 500GB USB3.0
Touro 500GB USB3.0

Touro 500GB USB3.0 HUB
Touro 500GB USB3.0 HUB

WD My Passport 2TB USB3.0
WD My Passport 2TB USB3.0

WD My Passport 2TB USB3.0 HUB
WD My Passport 2TB USB3.0 HUB

There are two interesting results. First of all, there is no speed difference between both drives (read/write/access: 83MB/49MB/17ms). Thus buying super-performance (read super-expensive) external drive make no sense, as the drive speed will be limited most likely by USB3.0 port connection anyway.
Secondly, there is no speed difference if drive is attached directly to the computer USB3.0 or through USB3.0 hub. But, I have noticed mouse lags when mouse was connected through the same USB hub. Thus if you expect to perform extensive disk reads/writes it’s better to connect it directly to the computer or at least avoid many important devices to use the same hub.

Shrink a dynamically growing disk from VirtualBox

I’m preparing Ubuntu image for the course. After installation of required software and datasets, the system image grew a lot. I have decided to shrink its size by disabling swap and removing unused data. But the system image didn’t shrink automatically after removal of the files. You need to do it manually:

  1. Run your system from LiveCD
  2. Install zerofree and free unused space
  3. sudo apt-get install zerofree
    # 10min
    sudo zerofree -v /dev/sda1
    
  4. In the host system, release and remove the image from VirtualBox (but keep it in the filesystem!)
  5. shrink1

    shrink2

  6. And compact image size
  7. VBoxManage modifyhd Ubuntu1404/Ubuntu1404.vdi --compact</code>
  8. Finally, you will have to add .vdi image back to VirtualBox machine.
  9. shrink3

In my case, the .vdi image shrunk from 7.2G to 6.2G (exactly the released size under Ubuntu), so I think it’s worth the effort.

Inspired by AskUbuntu.

Sending emails from command-line using Gmail & SSMTP

I wanted to send email reports from my computers. For that, the easiest is to create additional Gmail account (so your private email is safe) and set-up SSMTP:

sudo apt-get install ssmtp mailutils

Configure SSMTP: comment all lines in /etc/ssmtp/ssmtp.conf and add at the end (replacing HOSTNAME/USER/PASSWORD with your info):

root=postmaster
hostname=HOSTNAME
mailhub=smtp.gmail.com:465
AuthUser=USER@gmail.com
AuthPass=PASSWORD
FromLineOverride=YES
UseTLS=YES

Then, allow access from less secure apps to created Gmail account and enable one-time sign-in for new app without captcha. It’s strongly recommended not to use your private account for that purpose!

And send emails:

echo "email body" | mail -s "Test email from "`hostname` SOMEUSER@gmail.com

VoilĂ !

Note, this works perfectly fine in Ubuntu, as well as in Raspberry Pi.

NX server on Raspberry Pi

Finally I had some time to get my hand on new Raspberry Pi 2.
I think it’s useful to access the RPi2 by ssh (work out of the box) and NX. Installation of NX server is quite time-consuming, but I think it’s worth to give it a try! Here is what I did:

# install dependencies
sudo apt-get install xutils-dev expect xorg-dev libjpeg8-dev libpng12-dev cups xfonts-base x11-xserver-utils autoconf

sudo mkdir -p /usr/NX/{bin,etc}

# install freenx
wget http://pkgs.fedoraproject.org/repo/pkgs/freenx-server/freenx-server-0.7.3.tar.gz/856f597e139018f7ed62713c9d6c9ed5/freenx-server-0.7.3.tar.gz
tar xpfvz freenx-server-0.7.3.tar.gz
cd freenx-server-0.7.3
patch < gentoo-nomachine.diff
make
sudo make install

# copy node.conf
# and replace #COMMAND_MD5SUM="openssl md5" with COMMAND_MD5SUM="md5sum"
# and #COMMAND_START_GNOME=gnome-session with COMMAND_START_GNOME="/usr/bin/startlxde"
sudo cp node.conf.sample /usr/NX/etc/node.conf

# get nx source
wget http://code.x2go.org/releases/source/nx-libs/nx-libs_3.5.0.13-full.tar.gz
tar xpfvz nx-libs_3.5.0.13-full.tar.gz
cd nx-libs_3.5.0.13

# compile & relax for ~1 hour...
make build-full
sudo make install

# copy libs and binaries
sudo ln -s /usr/lib/NX3/lib/nx /usr/NX/lib
sudo cp -a nx-X11/lib/X11/libNX_X11.so* nx-X11/lib/Xext/libNX_Xext.so* nx-X11/lib/Xrender/libNX_Xrender.so* nxcomp/libXcomp.so* nxcompext/libXcompext.so* nxcompshad/libXcompshad.so* /usr/NX/lib
sudo cp -a nx-X11/programs/nxauth/nxauth nx-X11/programs/Xserver/nxagent nxproxy/nxproxy /usr/NX/bin

# link binaries & install
for f in /usr/bin/nx*; do echo $f; sudo ln -s $f /usr/NX/bin; done
sudo nxsetup --install

After these steps, you should be able to connect with NX client using fake GNOME session.
Raspberry Pi 2 over NX

Post is based on TiaoWiki with some updates/improvements.