Creating a tiny Docker image of a Rust project

Andy Balaam from Andy Balaam's Blog

I am building a toy project in Rust to help me learn how to deploy things in AWS. I’m considering using Elastic Beanstalk (AWS’s platform-as-a-service) and also Kubernetes. Both of these support deploying via Docker containers, so I am learning how to package a Rust executable as a Docker image.

My program is a small web site that uses Redis as a back end database. It consists of some Rust code and a couple of static files.

Because Rust has good support for building executables with very few dependencies, we can actually build a Docker image with almost nothing in it, except my program and the static files.

Thanks to Alexander Brand’s blog post How to Package Rust Applications Into Minimal Docker Containers I was able to build a Docker image that:

  1. Is very small
  2. Does not take too long to build

The main concern for making the build faster is that we don’t download and build all the dependencies every time. To achieve that we make sure there is a layer in the Docker build process that includes all the dependencies being built, and is not re-built when we only change our source code.

Here is the Dockerfile I ended up with:

# 1: Build the exe
FROM rust:1.42 as builder
WORKDIR /usr/src
Creating a tiny Docker image of a Rust project
# 1a: Prepare for static linking
RUN apt-get update && \
    apt-get dist-upgrade -y && \
    apt-get install -y musl-tools && \
    rustup target add x86_64-unknown-linux-musl

# 1b: Download and compile Rust dependencies (and store as a separate Docker layer)
RUN USER=root cargo new myprogram
WORKDIR /usr/src/myprogram
COPY Cargo.toml Cargo.lock ./
RUN cargo install --target x86_64-unknown-linux-musl --path .

# 1c: Build the exe using the actual source code
COPY src ./src
RUN cargo install --target x86_64-unknown-linux-musl --path .

# 2: Copy the exe and extra files ("static") to an empty Docker image
FROM scratch
COPY --from=builder /usr/local/cargo/bin/myprogram .
COPY static .
USER 1000
CMD ["./myprogram"]

The FROM rust:1.42 as build line uses the newish Docker feature multi-stage builds – we create one Docker image (“builder”) just to build the code, and then copy the resulting executable into the final Docker image.

In order to allow us to build a stand-alone executable that does not depend on the standard libraries in the operating system, we use the “musl” target, which is designed to statically linked.

The final Docker image produced is pretty much the same size as the release build of myprogram, and the build is fast, so long as I don’t change the dependencies in Cargo.toml.

A couple more tips to make the build faster:

1. Use a .dockerignore file. Here is mine:

/target/
/.git/

2. Use Docker BuildKit, by running the build like this:

DOCKER_BUILDKIT=1 docker build  .

Keeping track of podcast times with a simple bash script on Linux

Andy Balaam from Andy Balaam's Blog

I was recording some podcast audio tonight and wanted to be able to press a single key when I reached a significant moment, so I could add the times to the show notes.

I couldn’t find anything that already did this, so I wrote a tiny bash script. I ran this script pressed Enter whenever I wanted a time recorded:

T=0
echo
while sleep 1; do
    echo -n -e "\e[1A"
    echo $(($T / 60))m $(($T % 60))s
    T=$(($T + 1))
done

Custom Bash tab completion for my program

Andy Balaam from Andy Balaam's Blog

I love Bash tab completion, and I want it for the command I am writing, so it can automatically complete parts of the command line when I run my program.

Code

Here is the script (install-bash-completion) I wrote to set it up (no need to be root – it installs in ~/.local):

#!/bin/bash

set -u
set -e

DIR=${BASH_COMPLETION_USER_DIR:-${XDG_DATA_HOME:-$HOME/.local/share}/bash-completion}/completions

mkdir -p ${DIR}

cp _myprogram ${DIR}

The actual completion script (_myprogram) it installs looks like this:

_myprogram_commands()
{
    local cur prev opts
    COMPREPLY=()
    cur="${COMP_WORDS[COMP_CWORD]}"
    prev="${COMP_WORDS[COMP_CWORD-1]}"
    opts=$(bash -c "./myprogram --commands")

    COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) )
    return 0
}
complete -F _myprogram_commands ./myprogram

Installing

To install it, run:

./install-bash-completion

Then log out and log in again.

Now when you type ./myprogram and press TAB a couple of times, you should see the possible completions listed.

Notes

The completion script must be named to match the program name, with a leading underscore.

If I wanted it to work when it was installed in my PATH, I would need to change ./myprogram to just myprogram in 2 places.

Notice the line opts=$(bash -c "./myprogram --commands") – it actually runs my program to get the list of completions. This means my program needs to accept a --commands option which prints the valid commands. Alternatively, I could have hard-coded it by replacing that line with just:

opts="cmd1 cmd2 --help --etc"

More info:

Struggling with Rust to figure out the right types for a function signature

Andy Balaam from Andy Balaam's Blog

I am loving writing code in Rust. So many things about the language and its ecosystem feel so right*.

* For example: ownership of objects, expressive type system, compile to native, offline API docs, immutability, high quality libraries.

One of the things I like about it is that I don’t feel like I need to use an IDE, so I can happily code in Vim with no clever plugins.

One thing an IDE might give me would be an “extract function” refactoring. In most languages I am happy to do that manually, because I can let the compile errors guide me on what my function should look like.

However, in Rust I sometimes find it’s hard to find the right signature for a function I want to extract, and I am struggling to persuade the compiler to help me.

Here is an example from my new listsync project, in listsync-client-rust.rs:

use actix_web::{middleware, App, HttpServer};
use listsync_client_rust;
// ...
#[actix_rt::main]
async fn main() -> std::io::Result<()> {
//...
    HttpServer::new(|| {
        App::new()
            .wrap(listsync_client_rust::cookie_session::new_session())
            .wrap(middleware::Logger::default())
            .configure(listsync_client_rust::config)
    })
//...

I would like to extract the code highlighted above, the creation of an App, into a separate function, like this:

fn new_app() -> ??? {
    App::new()
        .wrap(listsync_client_rust::cookie_session::new_session())
        .wrap(middleware::Logger::default())
        .configure(listsync_client_rust::config)
}
//...
    HttpServer::new(|| {
        new_app()
    })

Simple, right? To find out what the return type of the function should be, I can just make a bad guess, and get the compiler to tell me what I did wrong. In this case, I will guess by changing the question marks above into i32, and run cargo test. I get quite a few errors, one of which is:

error[E0277]: the trait bound `i32: actix_service::IntoServiceFactory<_>` is not satisfied
  --> src/bin/listsync-client-rust.rs:27:5
   |
27 | /     HttpServer::new(|| {
28 | |         new_app()
29 | |     })
   | |______^ the trait `actix_service::IntoServiceFactory<_>` is not implemented for `i32`
   |
   = note: required by `actix_web::server::HttpServer`

So the first problem I see is that the error message I am seeing is about the later code, and there are no errors about my new function.

I obviously went a little too fast. Let’s change the HttpServer::new code back to how it was before, and only make a new function new_app. Now I get an error that should help me:

error[E0308]: mismatched types
  --> src/bin/listsync-client-rust.rs:12:5
   |
11 |   fn new_app() -> i32 {
   |                   --- expected `i32` because of return type
12 | /     App::new()
13 | |         .wrap(listsync_client_rust::cookie_session::new_session())
14 | |         .wrap(middleware::Logger::default())
15 | |         .configure(listsync_client_rust::config)
   | |________________________________________________^ expected i32, found struct `actix_web::app::App`
   |
   = note: expected type `i32`
              found type `actix_web::app::App<impl actix_service::ServiceFactory, actix_web::middleware::logger::StreamLog<actix_http::body::Body>>`

So the compiler has told us what type we are returning! Let’s copy that into the type signature of the function:

use actix_service::ServiceFactory;
use actix_http::body::Body;
// ...
fn new_app() -> App<impl ServiceFactory, middleware::logger::StreamLog<Body>> {
// ...

The first error I get from the compiler is a distraction:

error[E0432]: unresolved import `actix_service`
 --> src/bin/listsync-client-rust.rs:1:5
  |
1 | use actix_service::ServiceFactory;
  |     ^^^^^^^^^^^^^ use of undeclared type or module `actix_service`

I can fix it by adding actix-service = "1.0.5" to Cargo.toml. (I found the version by looking in Cargo.lock, since this dependency was already implicitly used – I just need to make it explicit if I am going to use it directly.)

Once I do that I get the next error:

error[E0603]: module `logger` is private
  --> src/bin/listsync-client-rust.rs:13:54
   |
13 | fn new_app() -> App<impl ServiceFactory, middleware::logger::StreamLog<Body>> {
   |                                                      ^^^^^^

This leaves me a bit stuck: I can’t use StreamLog because it’s in a private module.

More importantly, it makes the point that I don’t actually want to be as specific as I am being: I don’t care what the exact type parameters for App are – I just want to return an App of some kind and have the compiler fill in the blanks. Ideally, if I change the body of new_app later, for example to add another wrap call that changes the type of App we are returning, I’d like to leave the return type the same and have it just work.

With that in mind, I took at look at the type that HttpServer::new takes in. Here is HttpServer:

impl<F, I, S, B> HttpServer<F, I, S, B> where
    F: Fn() -> I + Send + Clone + 'static,
    I: IntoServiceFactory<S>,
    S: ServiceFactory<Config = AppConfig, Request = Request>,
    S::Error: Into<Error> + 'static,
    S::InitError: Debug,
    S::Response: Into<Response<B>> + 'static,
    <S::Service as Service>::Future: 'static,
    B: MessageBody + 'static, 

and HttpServer::new looks like:

pub fn new(factory: F) -> Self

So it takes in a function which actually makes the App, and the type of that function is F, which is a Fn which returns a I + Send + Clone + 'static. From the declaration of HttpServer we can see that the type of I depends on S and B, which have quite complex types. Let’s paste the whole thing in:

use actix_http::{Error, Request, Response};
use actix_service::IntoServiceFactory;
use actix_web::body::MessageBody;
use actix_web::dev::{AppConfig, Service};
use core::fmt::Debug;
// ...
fn new_app<I, S, B>() -> I
where
    I: IntoServiceFactory<S> + Send + Clone + 'static,
    S: ServiceFactory<Config = AppConfig, Request = Request>,
    S::Error: Into<Error> + 'static,
    S::InitError: Debug,
    S::Response: Into<Response<B>> + 'static,
    <S::Service as Service>::Future: 'static,
    B: MessageBody + 'static,
{
    App::new()
        .wrap(listsync_client_rust::cookie_session::new_session())
        .wrap(middleware::Logger::default())
        .configure(listsync_client_rust::config)
}

Note that I had to modify I to include the extra requirements on the return type of F from the definition of HttpServer. (I think I did the right thing, but I’m not sure. If I just remove the + Send + Clone + 'static it seems to behave similarly.)

Now I get this error from the compiler:

error[E0308]: mismatched types
  --> src/bin/listsync-client-rust.rs:27:5
   |
17 |   fn new_app<I, S, B>() -> I
   |                            - expected `I` because of return type
...
27 | /     App::new()
28 | |         .wrap(listsync_client_rust::cookie_session::new_session())
29 | |         .wrap(middleware::Logger::default())
30 | |         .configure(listsync_client_rust::config)
   | |________________________________________________^ expected type parameter, found struct `actix_web::app::App`
   |
   = note: expected type `I`
              found type `actix_web::app::App<impl actix_service::ServiceFactory, actix_web::middleware::logger::StreamLog<actix_http::body::Body>>`
   = help: type parameters must be constrained to match other types
   = note: for more information, visit https://doc.rust-lang.org/book/ch10-02-traits.html#traits-as-parameters

The compiler really tries to help here, suggesting I read a chapter of the Rust Book, but even after reading it I could not figure out how to do what I am trying to do.

Can anyone help me?

Wouldn’t it be amazing if there were some way the compiler could give me easier-to-understand help to figure this out?

Converting HTML slides to a PDF with Firefox

Andy Balaam from Andy Balaam&#039;s Blog

I have not found an automated way to generate a nice PDF from some slides written in HTML – if you know of one please add a comment!

In the meantime, if you create slides using my HTML Slides template, then you can make a decent-ish-looking PDF like this:

  1. View your slides in Firefox and open the Print dialog (press Ctrl-p).
  2. Select Print to File and choose a filename to save to.
  3. Under Options deselect “Ignore Scaling…” and select “–blank–” for all the headers and footers. Ensure “Print Background Colours” and “Print Background Images” are selected.
  4. Under Page Setup set Scale to 70.0 and Orientation to Landscape.
  5. Select the dropdown next to Paper size and choose Manage Custom Sizes…
  6. Create a new custom size called “Screen 16:9” with Width 157.5 mm and Height 280.0 mm. Set all the Paper Margins to 5.0 mm. You can modify the name of the custom size by clicking on it in the list on the left.
    Click Close.
  7. Back in Page Setup, make sure your new custom size is selected next to Paper Size.
  8. Click Print.

If that does not work well for you, try experimenting with different Scale settings.

Support the Software Freedom Conservancy

Andy Balaam from Andy Balaam&#039;s Blog

The Software Freedom Conservancy helps Free/Open Source software projects by providing infrastructure, financial structures, and legal help. It is a not-for-profit organisation that is dedicated to software freedom, something that I think is an important prerequisite for a decent world in the future.

Conservancy looks after lots of projects, including these ones that I personally use: Boost, BusyBox, Etherpad, Git, Godot Engine, Inkscape, phpMyAdmin, QEMU, Racket, Samba, Selenium, Squeak and Wine.

It provides an easy way for projects to accept donations, hold assets and negotiate contracts, as well as mentoring and legal advice. It also leads the difficult and thankless work to persuade (and force) companies to comply with the GPL (a Free Software license). Without this work a great deal more Free Software would be distributed without the required access to source code, meaning it is not free at all.

Further, it is a key supporter of Outreachy, which does important work to address the under-representation, systemic bias, and discrimination in the technology industry.

I encourage you to join me and support the Software Freedom Conservancy.

You might also like to listen to the Free As In Freedom podcast.

Coding workshop example worksheets

Andy Balaam from Andy Balaam&#039;s Blog

This week we did a coding workshop exercise: 2 teams implemented the different sides of the SMPP protocol (without speaking to each other) and this morning we tried out connecting them together.

We successfully sent a message and received an acknowledgement!

It was a lot of fun and I we learned a surprising amount about SMPP (and quite how … interesting … the standard is).

In case they’re useful to anyone, here are the worksheets I made up: Team 1 ODT, Team 1 PDF, Team 2 ODT, Team 2 PDF.

Idea for a team who are less interested in SMPP (!) – try a similar exercise implementing FTP, which is a nice simple text-based protocol. I did this once and found it extremely rewarding.