Bio-Linux Desktop as a Service
As a user of the EOS Cloud, you get your own Bio-Linux instance that runs continuously on the JASMIN servers. The instance is a Virtual Machine (VM) so it is to all intents and purposes you have your own Bio-Linux workstation, but sharing the computing resources on the cloud system. You can log in at any time to access the full set of Bio-Linux tools and to transfer data. You can administer the machine as if it was your own desktop PC, compiling and installing new software, adding user accounts, running services and changing settings. The above screenshot shows Bio-Linux being accessed by the recommended remote desktop client, x2go, which is free and runs on Windows, Linux or Mac. Existing users of x2go will know that this is remarkably fast and also enables easy file and printer sharing. However, there is no restriction on what software you use for access and regular SSH clients plus SFTP are fully supported. You can also access applications like Galaxy that provide a web server interface hosted on the virtual machine (see below).
We can also provide an alternative VM designed for running Docker containers. Other VM types are not supported at this time.
The EOS Cloud is a community cloud, so as new users come on board we hope that you will share information, scripts, databases, etc with fellow users. There is storage space within the cloud to put shared data and applications, but of course your own VM remains totally private and secure to you.
What You Need
To access the system you will need:
- The connection address and port number for your VM
- A user name and password for https://eoscloud.nerc.ac.uk/login
- An SSH key
- Suitable client software installed on your own PC
The first two of these will be provided to you when you first get an account. For details of 3 and 4 see below.
Requesting an Account
In the first instance, please e-mail Tim Booth and provide the following pieces of information.
- Your name and address
- What NERC project or projects you are working on (grant number)
- Your preferred login name for the web portal
- What you primarily intend to do with the system and in particular what reference databases you need access to (eg. Greengenes, Uniprot, …)
- The expected duration of your current project
- How many users you expect to have on the system (you are free to create accounts on the VM as you wish, but we’d like to have an idea of actual user numbers).
- Whether you want to use Bio-Linux, Docker or both.
- Your SSH public key file for initial access (see notes below)
This registration process is still being formalised. For now we want to discuss with all new users to see what people need from the system. We have capacity for around 20 users and hope to fill this within the next couple of months.
Once your account is set up you’ll be sent the details listed in 1 and 2 above.
An SSH key provides a mechanism for logging into a Linux machine that is more secure than a simple password, because it cannot be guessed or intercepted over an insecure network. Therefore the use of keys rather than passwords is mandatory on the EOS Cloud.
A key consists of two small files one of which is the private portion and the other the public portion. For the protection of your own data and the security of other system users it is important that you keep the private portion of your SSH key safe and protect it with a passphrase. This is used to encrypt the key so that even if somebody malicious obtains a copy of the file they will be unable to use it. We can’t generate the key for you as this defeats the whole object of the exercise! The public portion is what you send to us, and once we put it on to your new VM you will be able to log in using the private key file. If you are in doubt about this please ask for assistance.
In Windows generate the key using PuTTY:
- Download and install PuTTY from: http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html
- Run: puttygen.exe
- In the Parameters box choose SSH2 DSA and press Generate
- Move your mouse randomly to generate key pairs.
- Enter a key comment to identify the key.
- Type in your passphrase and confirm.
- Click Save private key to save the private key, call this id_dsa.
- Also export the private key in OpenSSH format as you’ll need it for X2GO.
- Click Save public key to save the public key, call this id_dsa.pub.
Send just the public key in an e-mail to the administrator: email@example.com
In Linux/Mac use ssh-keygen:
- First check the directory listings to see if you already have a public key: ls -al ~/.ssh
- If you don’t have a public key then you will need to generate one:
- ssh-keygen -t dsa -C “firstname.lastname@example.org”
- When you are prompted to Enter a file in which to save the key, just press Enter to continue.
- Next you will be asked to enter a passphrase.
- The public key will now be generated in the : .ssh/ directory under your home folder.
Send just the public key: ~/.ssh/id_dsa.pub in an e-mail to the administrator: email@example.com
If you are going to be accessing your VM from multiple computers, the best practise is to generate a new key on each computer and add all the public keys to the VM. You will be able to do this yourself once you have initial access. Do not enable password-only logins to the VM – this is a condition of using the system!
Connecting to the Cloud – Client Software
You can connect to your VM either via command-line SSH, or via a remote desktop session. We’ll assume you want to do the latter. Whether you are using Linux, Mac or Windows,we recommend using x2go for your remote desktop.
Using x2go client in Linux:
- In Bio-Linux 8, you have it already.
- To install x2goclient In Ubuntu or Debian or Mint: apt-get install x2goclient
- To install x2goclient in Fedora (you will need Fedora 20) and Redhat: yum install x2goclient
Once installed, start up x2goclient and then start a new session by clicking on the icon in the top left:
Enter the hostname and SSH port you have been provided with. The hostname will normally be bigdata2.nerc.ac.uk but the port is specific to your own VM. The user name for the initial account is always manager. Also check the try auto login box to use the key stored in ~/.ssh.
Using x2go in Windows:
Download x2goclient-setup.exe from: http://code.x2go.org/releases/binary-win32/x2goclient/releases/
Enter the hostname and SSH port you have been provided with. The hostname will normally be bigdata2.nerc.ac.uk but the port is specific to your own VM. The user name for the initial account is always manager.
Also point x2go to your private key file. If you generated your key file with PuTTY you’ll need to ensure you exported the private key in OpenSSH format as detailed above.
Using x2go on Mac:
Install XQuartz if you don’t have it already.
Download and install the latest x2goclient.dmg from: http://code.x2go.org/releases/binary-macosx/x2goclient/releases/
Once installed, start up x2goclient and then start a new session by clicking on the icon in the top left:
Enter the hostname and SSH port you have been provided with. The hostname will normally be bigdata2.nerc.ac.uk but the port is specific to your own VM. The user name for the initial account is always manager. Check the try auto login box to use the key stored in ~/.ssh or else specify the key file explicitly.
On Mac there are various known issues with the x2go client: the fullscreen mode in X is troublesome and should probably be turned off in the XQuartz settings. Also turn off file and printer sharing in the x2go options as this can cause lockups, and there are also occasional known issues with the keyboard, though these have hopefully been fixed in the latest client release.
General x2go info
You can leave all other settings at default, including selecting the KDE desktop, even though what you’ll actually get is the MATE desktop environment. Upon connection you should see a desktop environment with icons and menus. The desktop should resize properly when you resize the window and the keyboard should work as expected.
When you finish using your VM just log out, or else close the session window and you will be able to resume your session later. If the VM is shut down or in the unlikely event that it crashes you will be able to stop it and restart it via the web portal, but in general you should just leave it running.
Transferring files to your VM
The shared folder feature in x2go works but is not very good and transfers are slow. You should install an SFTP client and use that to transfer files to and from your VM. Suggested clients are:
- On Windows use: WinSCP
- On Linux use: a regular file browser (choose: File then: Connect to server) sftp://manager@VM_IP_ADDRESS:VM_PORT_NUMBER/home/manager
- On Mac use: CyberDuck
All of these should accept the same SSH key file as you use to login through x2go.
Learning more about Bio-Linux
If you are not familiar with Bio-Linux, look at the tutorial link on the Desktop to get some pointers.
Note – the tutorial is based on running the Unity desktop off a USB stick, so things look rather different but all the applications are the same.
Docker and oswitch
Docker provides a way of running Linux applications that for some reason do not work under Bio-Linux. You can access Docker and oswitch on Bio-Linux, or you request a cut-down VM that is designed just for running Docker containers.
- Docker images are reusable. Once someone creates a docker image, you can reuse it as it is without going through a complex setup process. It doesn’t matter whether the package running in Docker expects to be in old CentOS or latest Ubuntu.
- Docker images are isolated. You can use different version of the same software for different projects. You can even run them parallely!
- Docker images are frozen in time. Theoretically, you can use the same image a decade years down the line to reproduce an analysis!
Docker runs images in a container. A container is like a virtual machine (VM) but much lighter and featureful. Docker containers are isoated not only from each other but also from the host operating system. Containers run their own operating system, have their own file system, and you are different user in the container than on the host operating system. This makes access to gigabytes of data on your host system (pc, hpc node) a bit difficult from within the container.
oswitch simplifies this by automagically making available directories from your host operating system within container. Read / write permissions are maintained. oswitch even maintains the same current working directory, home directory and your shell! within the container, thus ensuring seamless use of docker container interactively or non-interactively (in scripts). See https://github.com/yeban/oswitch#example-usage for how to use oswitch.
Boosting Your VM – Accessing Extra Computing Power
Access the control interface at https://eoscloud.nerc.ac.uk using the username and password you were given when you created your account. When you log in you will see your own VM (or possibly VMs) listed. Click the Boost button to request extra resources, or click the server name to see more details.
Your default server configuration has a single processor and a relatively small amount of RAM. You have options for adding increasing amounts of CPU and RAM by spending credits. When you boost a machine it will restart with the increased CPU and RAM, but when the time expires it will restart again and revert to the baseline specification.
If you are unsure of how long you need the VM to be boosted for, simply select a long period then end the boost early to receive a pro-rata credit refund. Please don’t hoard your credits – when the system is running under-capacity it just wastes power, so give it some work to do. If you spend all your credits we’ll just give you more! Later we’ll work out an appropriate charging model.
Note that the hard disk size is fixed (normally at 1TB) and cannot be boosted up by spending credits. For options on acquiring extra storage contact the Cloud administrators.