Running Linux Tools in Lambda Containers
AWS released the ability for Lambda Functions to use containers in December 2020. This opened up a whole set of new use cases for lambda functions, since now we developers have better control of the runtime environment. I’ve found they are a great fit for performing tasks that require elevated privileges, especially if AWS IAM can control access. Running tasks with elevated privileges in an ephemeral manner is great for security, since there’s nothing constantly running with that access. However, lambda containers are still lambdas and this post shares some gotchas I encountered trying to run Linux command line tools inside a lambda function container.
Lambda Containers do not set $HOME
Since containers include their own operating system, the underlying lambda function runtime doesn’t know anything about what that is. If you are running a tool that expects certain environment variables to be set, like $HOME, you need to set them on the lambda function. For $HOME in particular, I found /tmp a good substitute, since often tools will need to both read and write to that directory.
Only $PATH can be executed
The $PATH environment variable has always been set by the underlying lambda runtime, and with containers that is no different. If you need to install a tool, make sure it’s in $PATH — otherwise nothing you do will let it be executed, even a chmod 777. Since the WORKDIR of the container needs to include the function, that is a good default place to put these executables.
Only /tmp is writable
A lambda container is still a lambda function, so the restriction to only write to /tmp is enforced when running containers. If something needs to be written to that is included in the container, move it to /tmp in the lambda function initialization as a workaround. An example of this is the known_hosts file for ssh, where including one can help speed up execution, but the ssh command also will write to it occasionally.
Container images must be region and account local to a lambda function
As of today, in order to use a container with a lambda function, it must be in an Elastic Container Registry (ECR) in the same region and account as the lambda function. Last year AWS did announce cross-region replication for containers.
Re-using container tags does not update a lambda function
If you set your lambda function to pull from a tag like latest and then move that tag to a new image, the lambda function will not automatically update to the newest tagged container. That change still needs to be done explicitly with a call to update-function-code. I ended up scripting this so that from my local laptop there’s one command I call that does both docker push and aws lambda update-function-code.
Conclusion
Many of these gotchas come from the fact that lambda containers are first and foremost lambda functions. If something is behaving oddly, chances are it’s because of something lambda function related vs something linux or container related. Figuring that out was key to getting some tools working. I hope this post is a useful reference for any lambda container work you do.