It is 6am on Tuesday the 25th of June, 2013 — at least, it is in US/Pacific — which means we at Joyent are finally lifting the covers off of our new product: Manta. Manta is a brand new system that spans the twin pillars of Object Storage and Compute to provide a revolutionary new way of operating on data in the cloud. You can read more about it in a write-up from Mark Cavage on the Joyent blog.
In this post I'm going to cover one of the pieces I recently added to Manta — namely: the Manta Interactive Session Engine. While Manta was primarily designed to run batch-style jobs across a large number of input objects, this subsystem allows you to run an interactive UNIX program in the environment of a Manta compute job and control it from your terminal.
The client utility that allows you to run interactive jobs is mlogin(1). To crib, briefly, from its manual page:
mlogin allows you to spawn an interactive job in Manta. Once running, your terminal will be attached to the remote process running in the job via a shell session tunneled through HTTPS, similar in concept to SSH. Interactive sessions are a great way to debug a new job script in-situ, or to experience and explore the compute zone environment hands on. It can also become part of a workflow using interactive terminal utilities on large Manta objects without the need to download or transfer the data -- e.g. the use of mdb (the Modular Debugger) on crash dumps and core files.
To get started with
mlogin, you'll need to sign up for a Joyent Public Cloud
account, which gets you access to the Manta Storage service as well as our IaaS cloud offering.
Once you sign in to the portal with your new account, you'll find personalised instructions for
adding your SSH key and downloading and installing the Manta command-line tools, the source for
which is available at joyent/node-manta on Github)
Once you have the command-line tools installed and you've been able to use
upload a file into Manta, you can take a swing at
mlogin. The simplest possible
interactive job is what we call a Reduce task with no input objects. You can spawn
one of these by just typing
mlogin! Let's do that now, and while we're
there we can take a look at the environment inside a Manta compute job.
As you can see, the default
mlogin invocation attaches you to a
shell running in the Manta compute environment — a regular SmartOS zone. Some
environment variables (starting with
MANTA_) are provided that you can use in
the job programs you write.
Let's try something a bit more complicated: an interactive job that uses an input
object, also known as a Map task. Here at Joyent we write a lot of
As part of debugging this software we often take core dumps
of the running program, which we are then able to analyse post-mortem
the illumos Modular Debugger. We have recently taken to uploading those core dumps
into Manta, where we can now (via the magic of
in-situ on the dump without copying it to another system!
In this screenshot I have started a Manta job on an uploaded core file,
mdb(1) on it. In a Map task, the
MANTA_INPUT_FILE environment variable contains the path
to where the system has mounted (read-only) your input object.
I have then used
::findjsobjects, as described in
Cantrill's blog post on finding node.js memory leaks.
One of the more gratifying parts of building
mlogin(1) was the realisation that
apart from the WebSockets engine for forwarding the live shell traffic to and from
the client, the rest of the subsystem was implemented in terms of existing Manta primitives.
I was able to pass configuration into the interactive job through the use of a regular
Manta object, the job itself is a regular Manta compute job, and the forwarding agent that
runs in the job is delivered into the Manta compute zone as a regular Asset. You can see
more details of the implementation in the source,
and I have high hopes that our user community will start producing their own innovative tools
on top of Manta that I haven't even begun to imagine!
We at Joyent Engineering encourage you to check out Manta today, and we look forward to hearing your feedback!