This is a crosspost from VanessaSaurus, dinosaurs, programming, and parsnips. See the original post here.
I’ve recently been interested in developer workflows. Aside from being a developer, I feel like the tooling for our community, and especially for HPC or hybrid environments, is lacking. As a simple example, let’s ask a basic question:
How do I start developing here and move it over there?
For the most part, creating a development container is fairly straight forward, and we can even bind source code to the host to work on in one editor terminal and then build and run or test in another. However, for the moving part, it gets shoddy. Our best bet is to rebuild the container with the most updated source code, push to a registry, and then pull down somewhere else. For a container that is a binary and not layers provided by a registry, we could even scp it. If we do this right, we will have an automated build and deploy that triggers when we merge new code into main, but do you see the problem? What about the code that we want to test that isn’t ready to merge? This is why we typically would need to manually push to a registry with some kind of “work in progress” tag and then pull somewhere else. Minimally we’d need to build fresh again, and then reproduce all the steps to set up our environment.
Interactive Development Containers
Now I don’t have all the answers, but recently @alecbcs and I have been dreaming about what kinds of development environments we want. functionality such as:
- Saving the container state without leaving it.
- Loading or saving or otherwise interacting with named environments.
- Inspecting or interacting with container metadata, also without leaving the container.
- Moving files or sizing the container without the same.
And actually I won’t even get to answering the first question in this post about moving something from one place to another, but rest assured it is an important one. This post is about some prototype or fun testing work that we’ve started around these ideas. The playground for some of these early ideas has been Paks.
Paks is a Python library that I’m calling a developer wrapper for containers. Mind you, it’s more of a playground right now to experiment with ideas. But I’ve had so much fun even this early on that I want to share what I’ve learned.
Wrapper
Because Paks is a wrapper, you will run containers using the paks command. Here are a few quick examples.
$ paks run ubuntu
$ paks run --shell /bin/sh busybox
$ paks run --container-tech podman busybox
What is happening on the backend that took me a bit to figure out is that we will need to run a subprocess, but create a pseudo terminal to better watch and interact with it. This is going to happen in the “interactive_terminal” command below. But unless you want your terminal to get wonky, we need to use termios to grab the current tty and make sure it gets restored no matter what at the end. That looks like this:
def interactive_command(self, cmd):
"""
Ensure we always restore original TTY otherwise terminal gets messed up
"""
# Controller to get history
self.hist = self.commands.history
# save original tty setting then set it to raw mode
old_tty = termios.tcgetattr(sys.stdin)
old_pty = termios.tcgetattr(sys.stdout)
try:
self._interactive_command(cmd)
finally:
termios.tcsetattr(sys.stdin, termios.TCSADRAIN, old_tty)
termios.tcsetattr(sys.stdout, termios.TCSADRAIN, old_pty)
What happens if you don’t do that? Your terminal gets weird and wonky. And then in the interactive command function, this is where we launch a subprocess with a new pseudo terminal:
tty.setraw(sys.stdin.fileno())
# open pseudo-terminal to interact with subprocess
openpty, opentty = pty.openpty()
# use os.setsid() make it run in a new process group, or bash job control will not be enabled
p = subprocess.Popen(
cmd,
preexec_fn=os.setsid,
stdin=opentty,
stdout=opentty,
stderr=opentty,
universal_newlines=True,
)
# Welcome to Paks!
self.welcome(openpty)
The setsid as a pre-exec function is ensuring the child process is a new session and won’t exit, sort of akin to a daemon. So at face value, yes it is doing exactly what you think - we are shelling into the container and watching the command line and looking for paks-known commands. And I didn’t use a Python keylogger because I found that keyboard requires sudo (like really?!) and pynput is really scary because it doesn’t just get keys from the terminal - it’s watching anything you type anywhere! That gave me the heebie jeebies. I hope there is some scanner for pypi that is looking for that package and checking it’s not being malicious.
All of the above said, and all the time spent, I’m not convinced that this exact method is the best way to be running commands from inside the container. There are other ideas that need to be tested!
Structure
We could have talked about this first, but let me show you the basic structure of paks so you get an understanding of the components.
paks
# Backends are different wrappers, so logically we start with podman and docker
├── backends
│ ├── base.py
│ ├── docker.py
│ ├── __init__.py
│ └── podman.py
# The client is what you interact with on the command line. This shows the various commands available.
├── cli
│ ├── config.py
│ ├── env.py
│ ├── __init__.py
│ └── run.py
# This is a central controller for things
├── client.py
# Here's all the built-in, interactive commands paks supports!
├── commands
│ ├── command.py
│ ├── cp.py
│ ├── env.py
│ ├── history.py
│ ├── __init__.py
│ ├── inspect.py
│ └── state.py
├── defaults.py
├── env.py
├── logger.py
# Coming soon - load your own commands!
├── plugins.py
├── schemas.py
├── settings.py
├── settings.yml
├── templates.py
├── utils
└── version.py
So that should give you the gist - we have container wrappers (backends) and then commands that we can issue while we are inside the container. Let’s talk about them next.
Saving State
The first thing I wanted to try with Paks was to save a container state, but not needing to open a separate terminal and save from the outside. The use case for this is that given I’m in an interactive container and I’ve made some changes, I don’t want to exit and rebuild. All y’all reproducibility folks can stop wincing, and realize that we also need more temporary or throwaway development environments like this. Reproducibilty is important, but mostly for the final production thing, and only up to a level of not giving us pain. So how might I do this?
For paks, while you are inside the container (let’s say ubuntu) you simply ask to #save:
$ paks run ubuntu
# touch PANCAKES
# #save
Saving container...
sha256:d82aaa268feb59344cf31a757ce7f5c0caa6a6bbd10b8d0af1d55cdbc50b609b
[+] Building 0.2s (5/5) FINISHED
...
=> => writing image sha256:f58ae524d8644400b33c078f19612cba7849ef8f3ea158e2291ac697a4129080
=> => naming to docker.io/library/busybox-saved
Untagged: dockerio-busybox-joyous-hippo-3922-gloopy-peanut-9044:latest
Deleted: sha256:d82aaa268feb59344cf31a757ce7f5c0caa6a6bbd10b8d0af1d55cdbc50b609b
Deleted: sha256:f58ae524d8644400b33c078f19612cba7849ef8f3ea158e2291ac697a4129080
Successfully saved container! ⭐️
And then you can see that there is an ubuntu-saved container!
$ docker images | grep ubuntu
ubuntu-saved latest 93e336d994de 2 minutes ago 72.8MB
ubuntu latest 54c9d81cbb44 7 days ago 72.8MB
So this has saved me some tiny bit of energy to open up another terminal, remember how to docker commit, and then also rebuild with a squash to minimize the layers (as there is a maximum number we don’t want to hit). What Paks could then eventually do is make it easy to move this entire container between places, e.g., from your local machine to HPC without a hitch. I haven’t started to work on that yet because this is a fun side project.
Environments
One thing I do a lot is use GitHub tokens to do fun stuff with the API. I usually need to keep this in some hidden file, then find it, open it, copy paste it, and export it in the container. And then I do that a million times when I have to run a new container. But with Paks, we can create a named environment on the host (a file to source with exports):
$ paks env edit github
You can also quickly show an environment:
$ paks env show github
GITHUB_TOKEN=xxxxxxxxxxx
And then in our container, as many times as we need, load it seamlessly!
root@9ec6c3d43591:/# #envload github
Loading environment...
Successfully loaded environment github
root@9ec6c3d43591:/# export GITHUB_TOKEN=xxxxxxxxx
root@9ec6c3d43591:/# export GITHUB_USER=dinosaur
If only my GitHub username was dinosaur! 😁️ Is it loaded?
root@9ec6c3d43591:/# env | grep GITHUB
GITHUB_USER=dinosaur
GITHUB_TOKEN=xxxxxxxxx
Okay, so to be fair, there are a bunch of other commands for inspection and size, and I’m not going to go through them all! You can see them in the Paks user guide. And I don’t mean to say you should use this - you probably shouldn’t. But you might be interested to try it out.
Parsing Keystrokes
So the most interesting part of this project has been learning about input from the terminal, and actually the reason I wanted to write this post to share what I learned. Let’s go back to the interactive function where we ran subprocess and created a pseudo terminal. There actually is a pretty simple way to watch what is being typed:
# This is the subprocess return code, keep going until we are done (e.g. have a return code)
while p.poll() is None:
# Wait for io completion (e.g., see man select)
r, w, e = select.select([sys.stdin, openpty], [], [])
# Was it a new input?
if sys.stdin in r:
terminal_input = os.read(sys.stdin.fileno(), 10240)
new_char = terminal_input.decode("utf-8")
# Do something with what you see here
# Was it a new output?
elif openpty in r:
o = os.read(openpty, 10240)
if o:
os.write(sys.stdout.fileno(), o)
I learned a lot from this! Let’s talk about it.
Debugging
So the first thing I learned is that my typical “import IPython” and “IPython.embed()” isn’t going to work as easily as normal, because (at least superficially) I didn’t see a way to have it sort of injected into the process. Anything that is interactive in that loop is still (conceptually) running on my host. So when I use IPython it does some weird stuff with carriage returns, but it’s still possible to interact with a little bit. So what I wound up doing so I could easily see every keypress was to write to file in append mode:
with open('/tmp/file.txt', 'a') as fd:
fd.write(new_char)
This was kind of neat because I could be typing in one terminal, and then have a file open (watching it) that updates with changes, and I’d get a sense of what is going on. I could append anything to this file to debug. And this is also really different from how we normally use subprocess, where maybe we will parse entire lines at once:
p = subprocess.Popen(['python','thing.py'], stdout=subprocess.PIPE)
while True:
line = p.stdout.readline()
if not line:
break
because we are reading on character at a time! So what we essentially need to do is keep a string that we continue appending to unless there is a newline, up or down, or left or right to indicate moving the cursor.
Ascii Characters
I started to quickly see characters that my editor didn’t know - e.g., likely escape sequences and other ascii that showed up in the little question mark box. I quickly realized that I was seeing ascii code (and some characters that couldn’t be parsed) so the solution was to look at the ord of the character and compare to a number. For example, for a backspace the number is 127. So to act on it I might do:
# if we have a backspace (ord 127)
if len(new_char) == 1 and ord(new_char) == 127:
# This is our in progress line. If we have content, backspace!
if len(string_input) > 0:
string_input = string_input[:-1]
# But if we don't, just write the character for the person to see and
# keep collecting new characters (continue in the loop)
if not string_input:
os.write(openpty, terminal_input)
continue
# Otherwise (not a backspace) add to our growing line to parse further!
else:
string_input = string_input + new_char
The above is basically looking for a backspace, and if we find one, we remove one character from the line we are assembling. Otherwise we just add the new character to the line.
xterm sequences
And a similar thing happens for pressing up/down and right/left, except the terminal parses them as “[A”, “[B”, “[C”, and “[D”, respectively, and often with an escape sequence first. There are some nice tables here for the interested reader! And this was also the point that I realized how challenging parsing input is! Along with needing to account for every character, you also need to account for platform differences. That’s also why I view this library as mostly for development and thinking, or at least for mostly Linux and bash shells, because I’m not sure I could ever handle them all. So for the purposes of my library, for now I decided I’m not going to handle moving left and right, nor do I want to deal with weird extra ascii characters that are added, so I just clean them up.
# Get rid of left/right
string_input = string_input.replace("[D", "").replace("[C", "")
# Replace weird characters and escape sequences
string_input = self.clean(string_input)
Yes, that probably means some of your ninja shortcuts won’t work perfectly when running paks, and if you absolutely want one to be parsed please let me know and we can add it.
Newlines
So the gold nugget of content that Paks is interested in is when you press enter. This means you’ve finished typing something and there is some version of a newline or carriage return. This is also a pretty variable thing depending on the platform you are on - newlines can come in very different forms! I tried to honor the two that I see most often:
- \r\n: Windows
- \n: UNIX (e.g., Mac OSX)
- \r: Mac (pre OSX)
has_newline = "\n" in string_input or "\r" in string_input
At this point, we can start acting on what we see. E.g., if the user has asked for any kind of exit, I honor it.
# Universal exit command
if "exit" in string_input and has_newline:
print("\n\rContainer exited.\n\r")
return self.uri.extended_name
The return of the name at the end is to handle cleaning up the image, which was allocated a temporary name.
History
One of the more interesting parts of this project was realizing that people use history, a lot. At least I do. This is going to appear as an up or down press, and only when a newline is found is some item in history re-executed. So first let’s look for exploring history with up/down. There are two cases - pressing up/down without a newline:
# Pressing up or down, but not enter
if ("[A" in string_input or "[B" in string_input) and not has_newline:
string_input = self.get_history(string_input, openpty)
os.write(openpty, terminal_input)
continue
And with one:
# Pressing up or down with enter
if ("[A" in string_input or "[B" in string_input) and has_newline:
string_input = self.get_history(string_input, openpty)
os.write(openpty, terminal_input)
If we don’t have a newline, we add a continue to keep parsing characters the user is typing. If we do have a newline, we let the loop keep running to keep parsing the line of history we retrieved. But let’s step back and talk about that history. We basically want to retrieve whatever line of history that the user is asking for, because to us it looks like up and down errors. You could imagine restoring the previous line, and then editing it. This actually proved to be quite challenging, because I realized (by default) when we start running a container (well, ubuntu and centos) the history is stored in memory and not written to ~/.bash_history. This led to this thread and some people coming in to quickly help and others coming in just to say “Why are you doing this with containers it makes no sense stop.” Yeah, right. If I listened to every person that has ever told me to stop working on something because “REASONS!” I wouldn’t ultimately work on much at all.
The short answer was that I needed a function to be able to get a line of history, and based on the number of times pressing up or down. For my first attempt I said “nevermind this, I’ll just save my own history!” but that got hugely complicated very fast because it turns out, we don’t just stupidly type commands over and over, we are constantly using more characters on the keyboard than letters and numbers, retrieving old things to edit, updating again, and in practice I found that I could keep up with simple parsing, but it would get out of sync for a longer session. There also is the issue that people can tweak the amount of history saved, or how it’s saved, and there are a set of environment variables and commands to do that. So most containers will start running and save history to memory and not file (and this makes sense in case there is sensitive information) but it was problematic for me because I couldn’t parse it. For example, when someone presses up and down a bunch of times, I might see:
[A[A[A[A[A[B[A
This is a reference to some previous command that I can only find in history given I’m parsing the input/output as I am. So my second attempt (well, maybe second through tenth) I was trying different variations of trying to be able to parse the history. If you looked at the tweet you’ll see we need to run:
$ history -a
to start writing what’s in memory to file. I didn’t want to do this on every command, because along with the user seeing it and the UI being awful, it was just too much. Instead, I realized that I had a small opportunity when the user first shells into the container (and is expecting a jump in their UI) to run whatever I need and then clear the terminal. So I ran it there, right before a clear and welcome message.
def welcome(self, openpty):
"""
Welcome the user and clear terminal
"""
# Don't add commands executed to history
os.write(openpty, self.encode(" export PROMPT_COMMAND='history -a'\r"))
os.write(openpty, self.encode(" clear\r"))
os.write(openpty, self.encode(" ### Welcome to PAKS! ###\r"))
And with this method you aren’t aware of the extra commands at all! And did you notice the spaces above? That’s also another trick! Any command that you type with a leading space won’t be saved to history, and this is thanks to HISTCONTROL that has an ignorespace option. I think most people / containers set it to ignore space and to ignore duplicates:
root@1c268386714a:/# echo $HISTCONTROL
ignoredups:ignorespace
That said, I don’t explicitly try to reset this in the container, so that could be a bug if there is a container base that doesn’t do that. And I’m pretty sure centos doesn’t come with clear! I’ll likely need to work on this a bit more.
For now, please consider this only working for debian/ubuntu bases and we can inspect the other ones later!
Okay, so now let’s look at the function to get history (self.hist.run). For now, just ignore the command to get the history, that’s actually done via a Paks command that we will talk about after. Here is what is going on:
def get_history(self, line, openpty):
"""
Given an input with some number of up/down and newline, derive command.
"""
# Calculate the absolute change of ups/downs
up = line.count("[A")
down = line.count("[B")
change = up - down
# pushed down below history (maybe they are angry?)
if change <= 0:
return ""
# Retrieve history, actually via a command run from the outside to get the file
history = self.hist.run(
container_name=self.uri.extended_name,
out=openpty,
history_file=self.settings.history_file,
user=self.settings.user,
)
history = [x for x in history.split("\n") if x]
# No history, nothing to return
if not history:
return ""
# The change is outside the length of history
if change > len(history):
return ""
# here we are looking back up into history (negative index)
newline = history[-1 * change]
# Add back any characters typed AFTER the up/down presses
newline += re.split("(\[A|\[B)", line, 1)[-1]
return newline
The above might not be perfect, but it worked the best for everything that I tried! This allows us to issue a command that paks knows, press up to get it again, and then edit it and have the command work correctly. Speaking of commands…
Commands
The core meat of paks is the commands that it recognizes. Every command has a base class that is going to handle parsing a line (with a main command and optional args or kwargs, depending on the command), ensuring all required variables are passed (this is largely internal to the library and even a developer user doesn’t need to think about it unless they want to change what is passed), and then providing functions for basic kinds of execution. So let’s step back and first look at how we find a command (or executor). Basically, once we have a newline and we’ve parsed it per the above (looking up history and such) we can sniff it to see if it matches a known command pattern:
# If we have a newline (and possibly a command)
if has_newline:
self.run_executor(string_input, openpty)
# Add derived line to the history
os.write(openpty, terminal_input)
string_input = ""
The function “run_executor” is going to make this call if there is a Paks command and handle it. And no matter what, we reset our string input to be empty given that the user pressed enter, because they are going to start typing fresh. But before that, this function “run_executor” is going to see if there are any known commands, and if so, to run them! That function looks like this:
def run_executor(self, string_input, openpty):
"""
Given a string input, run executor
"""
# Get out early if it's not a Paks command (always starts with #)
string_input = string_input.replace("[A", "").replace("[B", "")
if not string_input.startswith("#"):
return
# Do we have a matching executor?
executor = self.commands.get_executor(string_input, out=openpty)
if executor is not None:
# Print any message it wants to the terminal before run...
if executor.pre_message:
print("\n\r" + executor.pre_message)
# Run it!
result = executor.run(
name=self.image,
container_name=self.uri.extended_name,
original=string_input,
)
# And any message it wants to print after
if result.message:
print("\r" + result.message)
The result object holds what you would expect - a return code, some message, and the basic outputs of the call. It’s up to the executor (command) to decide what to show the user. Some might not show anything beyond commands that are run with the executor. So what does that function “get_executor” look like? This is where we delive into the commands module, where there is a simple lookup of the starting prefixes of commands matched to Command classes:
# lookup of named commands and settings
docker_commands = {
"#save": SaveContainer,
"#inspect": InspectContainer,
"#envload": EnvLoad,
"#envhost": EnvHost,
"#envsave": EnvSave,
"#cp": Copy,
"#size": Size,
}
When I add a load functionality, all it will need to do is update this dictionary. And the reason those are “docker commands” is that you can imagine we eventually support other container technologies, and the commands you run are going to vary. Each Command actually has a class attribute for the container types that are supported. Here is a snippet of the DockerCommands class attached to the client that we are calling “get_executor” on:
class DockerCommands:
# Required kwargs for any docker/podman command to run
required = ["container_name", "name"]
def __init__(self, container_tech):
self.command = container_tech
self.lookup = docker_commands
def parse_name(self, cmd):
parts = cmd.split(" ")
return parts.pop(0).replace("\n", "").replace("\r", "").strip()
def has_command(self, name):
name, _ = self.parse_name(name)
return name in self.lookup
@property
def history(self):
return History(self.command)
def get_executor(self, name, out=None):
"""
Backend is required to update history
"""
name = self.parse_name(name)
if name in self.lookup:
return self.lookup[name](self.command, required=self.required, out=out)
To focus on the last function, you basically see that we parse the line (name), and then see if it’s in our lookup. If so, we return the initialized executor, and we need to add the output source in case it needs to interact with the current terminal. The self.command refers to the container technology (e.g., docker or podman in this case).
Then we can look at a particular command (e.g., inspect) and see it’s pretty simple! We have defined the supported container technologies along with optional messages, and a main run function. Here is the command to inspect, which will dump out the json manifest and optionally take a section:
class InspectContainer(Command):
supported_for = ["docker", "podman"]
pre_message = "Inspecting Container..."
def run(self, **kwargs):
"""
Inspect a container fully, or specific sections
"""
# Always run this first to make sure container tech is valid
self.check(**kwargs)
# These are both required for docker/podman
container_name = self.kwargs["container_name"]
# inspect particular attributes provided as args
if self.args:
for section in self.args:
result = self.run_command(
[
self.tech,
"inspect",
"--format",
"" % section.capitalize(),
container_name,
]
)
# Otherwise just dump the whole thing
else:
result = self.run_command([self.tech, "inspect", container_name])
if result:
return result
return self.return_success()
You’ll now know the main Paks trick - because we are still running on the host, we can issue commands to the host while we are in the container! In the above, we can just type:
#inspect
#inspect config
And see the output in the terminal! This is how a lot of the interactions with the host work. It’s kind of simple and silly, but also really cool when you see it work on the container! So the run function above, just as a reminder, is called by this part:
result = executor.run(
name=self.image,
container_name=self.uri.extended_name,
original=string_input,
)
And honestly, that’s the majority of Paks! 🎉️
Discussion
Paks has honestly been so fun to work on, despite long hours of trying to figure things out during evenings and weekends. I’m so excited about the ideas, and I want to share them with others because I think developer tools for containers are kind of lacking. Heck, I stayed up until like 4am writing this post. No, I don’t normally do that, I had some things on my mind, but it was an excellent use of the time, despite the fact that I woke up 4 hours later and I’m going to crash tonight (err tomorrow night… err now that I’m tweaking up the finishing touches to this post)!
Next Steps
I’m working on a “paks load” command that will let someone develop a Python module with some set of commands for their custom use case. The first thing I wanted to try was to generate sboms for spack (e.g., “Generate sboms for this spack install in the container and save them to my host so I can upload alongside the container to a registry). I had some previous work to use spack scripting, but ultimately this weekend did a pull request to add sbom generation to spack proper. And then I’ll be able to work on the load commands. I also want to address some of the anticipated bugs I mentioned above, like properly setting “HISTCONTROL” to ensure we don’t save commands issued by the client to history, and possibly having a cleanup step on save that removes the file. I haven’t added this yet is because if I’m developing in the container and want to say, move it from my local machine to HPC, I kind of want to have my history so I can lazily use it.
But Really…
We have some magic up our sleeves for what we are actually working on to inspire these ideas! I guess you’ll just have to wait for the future, because @alecbcs and I are both have vision and are a great tag team! 🎉️
Security
So there are obviously security issues around a library like this - and I added notes to the documentation that I’ll re-iterate here. Paks is intended for use by a developer that is in their own trusted environment, whether local or on HPC. Because there is an interaction with the host, you wouldn’t use this in production someone to give users an ability to load environments or save. You also wouldn’t want to save a development container with something private in history and push it. I’m still an advocate for, after development is done, pushing changed code to GitHub and having an automated build build, test, and deploy. Could we eventually have a production grade library to enable interactions inside the container? Possibly, but it’s not Paks in Python in its current state. I think that’s okay - we have to start small with ideas and go from there.
Didn’t I see paks before?
Yes, you did! A previous version was intended for making spack build caches on GitHub, but that didn’t work because you couldn’t build a spack package within a container and then pull the same container and install it and hit the cache. I think this might work someday, hence why I haven’t completely deleted the code, but I couldn’t let a cute logo and colorscheme go to waste! So for now it’s on a separate branch but largely I am not working on it. If you want to see this branch, it’s still here!
Thanks for reading friends! I hope this has been interesting and you might be inspired to also work on better tooling for developers, even if that just means exploring the ideas.