[aspect-devel] Interest in providing and using software containers

[Bill Broadley]

Greetings,

I noticed some interesting container related discussion.  I've been tinkering
with docker quite a bit recently.  Figured I'd mention some of my observations.

I've been involved in making various virtual images for CIG tutorials,
workshops, and related and I found it somewhat frustrating process.  Mainly
because virtual machines are generally toys.  They have quite a few problems:
* A GUI running inside a GUI, grabbing the mouse focus, cut/paste, and having
  two docks is annoying.
* A virtual machine is a black box, and is somewhat painful to monitor what's
  going on inside from the comfort of your normal environment.
* You have to carve a certain amount of disk, ram, and CPUs out of your machine
  regardless of actual resource needs.  It reminds me of the old Apple OS 9
  days where the OS was so stupid you had to manually click on apps and change
  their memory allocation in attempts to keep it from crashing.
* Serious RAM overhead, running two kernels (host and guest), two window
  systems, and handling worst case application memory needs is tough.  Some
  people even run two browsers (one inside and one not).
* Virtual machines are heavy weight, you'd need a pretty serious machine to run
  a couple virtual machines.  Virtual machines are also slow to boot.
* Virtual machines seem fairly toy like, developers prefer not to develop in a
  virtual machine, and researchers tend not to research in them.  I've seen
  developers run linux fullscreen on apple laptops, but even then developers
  try to minimize switching between environments as much as possible.
* I suspect most users if the tutorial goes well and they want to use an
  applications for their research their first step will be to install the
  application natively.
* The images are large, often 4-8GB for even fairly simple instances. Burning
  100 to flash drives was quite painful.  I think it was several person days of
  work.

Containers on the other hand:
 * can share 0-100% of your RAM, CPU, and disk as needed
 * start in less than a second from the command line or window (no virtual box
   needed.)
 * are efficient enough to run quite a few of them on older machines.  I ran a
   test or running a single simple command in 4 different OSs in containers and
   it ran in a second or two
 * They can share directories with you in a simple and straight forward manner.
 * They aren't toys, can be used by developers, buildbots, and researchers
   without significant resource penalty.  Even large corporations like google
   have a majority of their workloads running in containers.
 * during this email I created 20 or so containers to tinker.  Creating a
   virtual image is a rather painful process.  Mine built in a minute or two
   each, and even quicker once the building blocks are cached.
 * while Microsoft is somewhat behind, they are trying really hard to close the
   gap.  Microsoft has added Docker/Kubernetes support to Azure, is
   contributing to docker, and released the Ubuntu compatibility layer.
   Microsoft is working hard on being a 1st class container host.

I built a container for Firefox (2 minutes or so).  I can resize it, only have
one window manager, no cursor grabbing, and cut/paste works just like normal.
Some OSs are even starting to put higher risk applications (like a browser) in a
container by default.  I can run top and see the containers resource usage:

 PID  USER PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+COMMAND
11149 bill 20   0  974288 301544  80008 S   1.5  0.9   0:12.97 firefox

It feels native, and even shows up in my dock.

The container is pretty small:
REPOSITORY          TAG                 IMAGE ID      CREATED         SIZE
firefox             latest              ebba8808f912  17 minutes ago  399.6 MB

Granted that's a bit tougher if your OS doesn't support X11 natively.  On
windows with the current popular package manager of the year (chocolatey) you
can fix that with:
$ choco install xming

Sure, it's one more thing to install.  But the result enables many useful tools
to work, not just toys.  You could of course just support VNC in the container,
and then connect from the host OS with vncviewer, but then you have the ugly
aspects of a desktop GUI running inside a desktop GUI.  No worse than virtualbox
though.

For sharing files it's really easy, there's a bit of complexity creating the
container (invisible to the user).  So from inside a container I can use /data:
user at a011b1a043c9:/data$ touch foo bar baz
user at a011b1a043c9:/data$

But outside the container:
bill at left:~$ ls -al ~/aspect/data
total 8
drwsrwsr-x 2 bill bill 4096 Feb 11 00:40 .
drwxrwxr-x 3 bill bill 4096 Feb 10 23:44 ..
-rw-r--r-- 1 bill bill    0 Feb 11 00:39 bar
-rw-r--r-- 1 bill bill    0 Feb 11 00:39 baz
-rw-r--r-- 1 bill bill    0 Feb 11 00:39 foo

I didn't hard code the UIDs or do anything else not portable across machines.
I've not tested OSX or Windows though.

So it's possible to hide the complicated parts from users.  They don't have to
think about containers, images, volumes, what disappears, and what doesn't.  As
long as the person building it makes sensible decisions (like storing files a
user would want to keep in their home directory) the user should mostly get a
point and click experience.  An experience that's likely to be much more native
feeling than a virtual machine with a second mouse pointer, second set of icons,
and living in a small statically sized box with significant memory overhead.

Some of the potential benefits to the CIG community would be enabled by making
two containers for each code.  One Dockerfile intended for developers that
builds from source to make a release.  A second Dockerfile container for end
users that includes the complete runtime, tutorials, documentation, and anything
else to help people get started.  The beautiful thing about the dockerfiles is:
* They are human readable
* They are tiny and easy to download
* They can do wget, git clone, and even talk to our software database to track
  the newest versions.
* Could be completely up to date when the user builds them, instead of manually
  creating virtual images, putting them on thumb drives, and handing them to
  people.

For the developer images:
* useful for Jenkins, TravisCI or other CI systems to run against.
* useful for potential contributors so they can have exactly the same
  environment and even test changes before making a pull request.
* useful for end users who want to customize their build and want to
  compare to a working build.  As a systems admin I'd much rather look at a
  dockerfile for downloading and installing 10 dependencies than a documentation
  page trying to explain it.  Doubly so if the developers updates the dockerfile
  to keep the CI system happy.

For the user images:
* can get running with just a few clicks.  Two commands and two minutes and I
  had my first container going.
* provides a native experience, one mouse cursor, one desktop, one dock, etc.
* don't have to try to decide ahead of time how much CPU/ram to give an
  instance.
* Reasonable for use for any problem a laptop/desktop is capable of running.
  Performance overhead is low, and ram overhead is close to zero.

In fact I've seen quite a bit of interest in the HPC community.  Singularity
avoids the docker security problems.  I've seen enabling containers for jobs has
been discussed in various places like the slurm mailing list (the most popular
HPC batch scheduler).  I'm investigating containers for use with our HPC
clusters, most likely manage by Kubernetes.

The typically response from people suggest "running in the cloud" with full
virtualization is a scoff and a rolling of the eyes.  Even if the CPU overhead
isn't terrible the I/O and in particular communication latencies are generally
unacceptable.

I'm happy to share any of the technical details mentioned above, but might be
best off list since this isn't particularly aspect-devel relevant discussion.