Limiting the number of open sockets in a tokio-based TCP listener

Andy Balaam from Andy Balaam's Blog

I learned quite a bit today about how to think about concurrency in Rust. I was trying to use a Semaphore to limit how many open sockets my TCP listener allowed, and I had real trouble making it work. It either didn’t actually work, allowing any number of clients to connect, or the compiler told me I couldn’t do what I wanted to do, because the lifetime of my Semaphore was not 'static. Here’s the journey I took towards working code that I think is correct (feedback welcome).

Motivation

In the tokio tutorial there is a short section entitled “Backpressure and bounded channels” (partway down the Channels page). It contains this statement:

…take care to ensure total amount of concurrency is bounded. For example, when writing a TCP accept loop, ensure that the total number of open sockets is bounded.

Obviously, when I started work on a TCP accept loop, I wanted to follow this advice.

Like many things in my journey with Rust, it was harder than I expected, and eventually enlightening.

The code

Here is a short Rust program that listens on a TCP port and accepts incoming connections.

Cargo.toml:

[package]
name = "tcp-listener-example"
version = "1.0.0"
edition = "2018"
include = ["src/"]

[dependencies]
tokio = { version = ">=1.0.1", features = ["full"] }

src/main.rs:

use tokio::io::AsyncReadExt;
use tokio::net::TcpListener;

#[tokio::main]
async fn main() {
    let listener = TcpListener::bind("0.0.0.0:8080").await.unwrap();

    loop {
        let (mut tcp_stream, _) = listener.accept().await.unwrap();
        tokio::spawn(async move {
            let mut buf: [u8; 1024] = [0; 1024];
            loop {
                let n = tcp_stream.read(&mut buf).await.unwrap();
                if n == 0 {
                    return;
                }
                print!("{}", String::from_utf8_lossy(&buf[0..n]));
            }
        });
    }
}

This program listens on port 8080, and every time a client connects, it spawns an asynchronous task to deal with it.

If I run it with:

cargo run

It starts, and I can connect to it from multiple other processes like this:

telnet 0.0.0.0 8080

Anything I type into the telnet terminal window gets printed out in the terminal where I ran cargo run. The program works: it listens on TCP port 8080 and prints out all the messages it receives.

So what’s the problem?

The problem is that this program can be overwhelmed: if lots of processes connect to it, it will accept all the connections, and eventually run out of sockets. This might prevent other things working right on the computer, or it might crash our program, or something else. We need some kind of sensible limit, as the tokio tutorial mentions.

So how do we limit the number of people allowed to connect at the same time?

Just use a semaphore, dummy

A semaphore does exactly what we need here – it keeps a count of how many people are doing something, and prevents that number getting too big. So all we need to do is restrict the number of clients that we allow to connect using a semaphore.

Here was my first attempt:

use tokio::io::AsyncReadExt;
use tokio::net::TcpListener;
use tokio::sync::Semaphore;

#[tokio::main]
async fn main() {
    let listener = TcpListener::bind("0.0.0.0:8080").await.unwrap();
    let sem = Semaphore::new(2);

    loop {
        let (mut tcp_stream, _) = listener.accept().await.unwrap();
        // Don't copy this code: it doesn't work
        let aq = sem.try_acquire();
        if let Ok(_guard) = aq {
            tokio::spawn(async move {
                let mut buf: [u8; 1024] = [0; 1024];
                loop {
                    let n = tcp_stream.read(&mut buf).await.unwrap();
                    if n == 0 {
                        return;
                    }
                    print!("{}", String::from_utf8_lossy(&buf[0..n]));
                }
            });
        } else {
            println!("Rejecting client: too many open sockets");
        }
    }
}

This compiles fine, but it doesn’t do anything! Even though we called Semaphore::new with an argument of 2, intending to allow only 2 clients to connect, in fact I can still connect more times than that. It looks like our code changes had no effect at all.

What we were hoping to happen was that every time a client connected, we created _guard, which is a SemaphoreGuard, that occupies one of the slots in the semaphore. We were expecting that guard to live until the client disconnects, at which point the slot will be released.

Why doesn’t it work? It’s easy to understand when you think about what tokio::spawn does. It creates a task and asks for it to be executed in the future, but it doesn’t actually run it. So tokio::spawn returns immediately, and _guard is dropped, before the code that handles the request is executed. So, obviously, our change doesn’t actually restrict how many requests are being handled because the semaphore slot is freed up before the request is processed.

Just hold the guard for longer, dummy

So, let’s hold on to the SemaphoreGuard for longer:

use tokio::io::AsyncReadExt;
use tokio::net::TcpListener;
use tokio::sync::Semaphore;

#[tokio::main]
async fn main() {
    let listener = TcpListener::bind("0.0.0.0:8080").await.unwrap();
    let sem = Semaphore::new(2);

    loop {
        let (mut tcp_stream, _) = listener.accept().await.unwrap();
        let aq = sem.try_acquire();
        if let Ok(guard) = aq {
            tokio::spawn(async move {
                let mut buf: [u8; 1024] = [0; 1024];
                loop {
                    let n = tcp_stream.read(&mut buf).await.unwrap();
                    if n == 0 {
                        drop(guard);
                        return;
                    }
                    print!("{}", String::from_utf8_lossy(&buf[0..n]));
                }
            });
        } else {
            println!("Rejecting client: too many open sockets");
        }
    }
}

The idea is to pass the SemaphoreGuard object into the code that actually deals with the client request. The way I’ve attempted that is by referring to guard somewhere within the async move closure. What I’ve actually done is tell it to drop guard when we are finished with the request, but actually any mention of that variable within the closure would have been enough to tell the compiler we want to move it in, and only drop it when we are done.

It all sounds reasonable, but actually this code doesn’t compile. Here’s the error I get:

error[E0597]: `sem` does not live long enough
  --> src/main.rs:12:18
   |
12 |         let aq = sem.try_acquire();
   |                  ^^^--------------
   |                  |
   |                  borrowed value does not live long enough
   |                  argument requires that `sem` is borrowed for `'static`
...
29 | }
   | - `sem` dropped here while still borrowed

What the compiler is saying is that our SemaphoreGuard is referring to sem (the Semaphore object), but that the guard might live longer than the semaphore.

Why? Surely sem is held within a scope that includes the whole of the client-handling code, so it should live long enough?

No. Actually, the async move closure that we are passing to tokio::spawn is being added to a list of tasks to run in the future, so it could live much longer. The fact that we are inside an infinite loop confused me further here, but the principle still remains: whenever we make a closure like this and pass something into it, the closure must own it, or if we are borrowing it, it must live forever (which is what a 'static lifetime means).

The code above passes ownership of guard to the closure, but guard itself is referring to (borrowing) sem. This is why the compiler says that “sem is borrowed for 'static“.

Wrong things I tried

Because I didn’t understand what I was doing, I tried various other things like making sem an Arc, making guard an Arc, creating guard inside the closure, and even trying to make sem actually have 'static storage by making it a constant. (That last one didn’t work because only very simple types like numbers and strings can be constants.)

Solution: Share the Semaphore in an Arc

After what felt like too much thrashing around, I found what I think is the right answer:

use std::sync::Arc;
use tokio::io::AsyncReadExt;
use tokio::net::TcpListener;
use tokio::sync::Semaphore;

#[tokio::main]
async fn main() {
    let listener = TcpListener::bind("0.0.0.0:8080").await.unwrap();
    let sem = Arc::new(Semaphore::new(2));

    loop {
        let (mut tcp_stream, _) = listener.accept().await.unwrap();
        let sem_clone = Arc::clone(&sem);
        tokio::spawn(async move {
            let aq = sem_clone.try_acquire();
            if let Ok(_guard) = aq {
                let mut buf: [u8; 1024] = [0; 1024];
                loop {
                    let n = tcp_stream.read(&mut buf).await.unwrap();
                    if n == 0 {
                        return;
                    }
                    print!("{}", String::from_utf8_lossy(&buf[0..n]));
                }
            } else {
                println!("Rejecting client: too many open sockets");
            }
        });
    }
}

This code:

  • Creates a Semaphore and stores it inside an Arc, which is a reference-counting pointer that can be shared between tasks. This means it will live as long as someone holds a reference to it.
  • Clones the Arc so we have a copy that can be safely moved into the async move closure. We can’t move sem in to the closure because it’s going to get used again the next time around the loop. We can move sem_clone in to the closure because it’s not used anywhere else. sem and sem_clone both refer to the same Semaphore object, so they agree on the count of clients that are connected, but they are different Arc instances, so one can be moved into the closure.
  • Only aquires the SemaphoreGuard once we’re inside the closure. This way we’re not doing something difficult like borrowing a reference to something that lives outside the closure. Instead, we’re borrowing a reference via sem_clone, which is owned by the closure which we are inside, so we know it will live long enough.

It actually works! After two clients are connected, listener.accept actually opens a socket to any new client, but because we return almost immediately from the closure, we only hold it open very briefly before dropping it. This seemed preferable to refusing to open it at all, which I thought would probably leave clients hanging, waiting for a connection that might never come.

Lifetimes are cool, and tricky

Once again, I have learned a lot about what my code is really doing from the Rust compiler. I find this stuff really confusing, but hopefully by writing down my understanding in this post I have helped my current and future selves, and maybe even you, be clearer about how to share a semaphore between multiple asynchronous tasks.

It’s really fun and empowering to write code that I am reasonably confident is correct, and also works. The sense that “the compiler has my back” is strong, and I like it.

Is your program a function or a service?

Andy Balaam from Andy Balaam's Blog

Maybe everyone knows this already, but for my own clarity, I think there are really two types of computer program:

  • A function: something that you run, and get back a result. Example: a command-line tool like ls
  • A service: something that sits around waiting for things to happen, and responds to them. Example: a web server

How functions work

Programs that are essentially functions should:

  • Validate their input and stop if it is wrong
  • Stop when they have finished their job*
  • Let you know whether they succeeded or failed

*The Halting Problem shows that you can’t prove they stop, so I won’t ask you to do that.

Writing functions is relatively easy.

How services work

Programs that are services should:

  • Start when you tell them to start, even when things are not right
  • Keep running until you tell them to stop, even when bad things happen
  • Tell the user about problems via some communication mechanism

Writing services seems a little harder than writing functions.

What about UIs?

I suggest that programs with UIs are just a special case of services. Do you agree?

What about let-it-crash?

I think that let-it-crash is a good way to build services, but when you build a service that way, I consider the whole system to be the real service: this means the code we are writing, plus the runtime. In this case, the runtime is responsible for keeping the service running (by restarting it), and telling the user about problems.

In effect, let-it-crash allows us to write programs that look like functions (which I claim is easier), and still have them behave like services, because the runtime does the extra work for us. Erlang seems like a good example of this.

What are the implications?

If you are writing a service, your program should start when asked, and keep running until it is asked to stop, even if things are bad.

For example:

  • a service that relies on a data source should keep running when that data source is unavailable, and emit errors saying that it is unable to work. It should start working when the data source becomes available. (Again, if you implement this behaviour by using a runtime that allows you to write in a let-it-crash style, good for you.)
  • a service that relies on the existence of a directory should probably create that directory if it doesn’t exist.
  • a service that needs config might want to start up with sane defaults if the config is not supplied. Or maybe it should complain loudly and poll for the file to be created?

Why not stop when things are wrong?

  • Using this approach, it doesn’t matter the order of starting services. The more services we have, the more painful it is to have an order we must follow.
  • It’s nice when things are predictable. We expect services to keep running under normal circumstances. Using this approach, our expectations are not wrong when things go wrong.

What are the down sides?

  • You must pay attention to the error reporting coming from running services – they may not be working.
  • Services will still stop, due to bugs, or at least due to hardware failures, so you still have to pay attention to whether services are running.

More: 12 Fractured Apps

Shutdown order consistency: how Rust helps

Andy Balaam from Andy Balaam's Blog

Some Java code with bugs

Here’s my main method (in Java). Can you guess the bug?

Db db = new Db();
Monitoring monitoring = new Monitoring();
Monitoring mon2 = new Monitoring();
Billing billing = new Billing(db, monitoring);
monitoring.setDb(db);

runMainLoop(billing, mon2);

db.stop();
billing.stop();
monitoring.stop();

If you would like to hunt down the 2 bugs manually, try reading the full code here: ShutdownOrder.java

But maybe you have an idea already? Maybe you’ve seen code like this before? If you have, you probably have an instinct that there’s some kind of bug, even if you can’t say for sure what it is. Code like this almost always has bugs!

This code compiles fine, but it contains two bugs.

First, we forgot to setDb() on mon2. This causes a NullPointerException, because Monitoring expects always to have a working Db.

Second, and in general harder to spot, we shut down our services in the wrong order. It turns out that Monitoring uses its Db during shutdown, so we get an exception. Even worse, if some other code needed to run after monitoring.stop(), it won’t, because the exception prevents us getting any further.

Of course, this is toy code, but this kind of problem is common (and much harder to spot) in real-life code. In fact, my team dealt with a similar bug this week.

It’s fundamentally hard to figure out your shutdown order. It’s complicated further if classes have start() methods too, which I have seen in lots of Java code.

Given that this is just a hard problem, maybe there’s no point looking for tools to make it easier?

Some Rust code without those bugs

Let’s try writing this code in Rust. Here’s the main method:

let db = Db::new();
let monitoring = Monitoring::new(&db);
let mon2 = Monitoring::new(&db);
let billing = Billing::new(&db, &monitoring);

run_main_loop(&billing, &mon2);

// drop() is called automatically on all objects here

Here’s the full code: shutdown_order.rs

This code shuts down all the services automatically at the end, and any mistakes we make in the order are compile errors, not things we find later when our code is running.

The code to shut down each service looks like this:

impl Drop for Monitoring<'_> {
    fn drop(&mut self) {
        // [Disconnect from monitoring API]
        self.db.add_record("MonitorShutDown");
    }
}

This is us implementing the Drop trait for the struct Monitoring (traits are a bit like Java Interfaces). The Drop trait is special: it indicates what to do when an instance of this struct is dropped. In Rust, this is guaranteed to happen when the instance goes out of scope, which is why our comment at the end of the main method sounds so confident.

Furthermore, Rust’s compiler shuts down everything in the reverse order in which it was created, and guarantees that nothing gets used after it has been dropped.

Rust’s lovely world gives us two relevant treats: no unexpected nulls, and lifetimes.

Treat number 1: no unexpected nulls

First, in Rust, like in other modern languages like Kotlin, we have to be explicit about items that could be missing. In our example, we were able to re-arrange the code so that db can never be missing (or null), and the compiler encouraged us to do so. If we really needed it to be missing some of the time, we could have used the Option type, and the compiler would have forced us to handle the case when it was missing, instead of unexpectedly getting a NullPointerException like we did in Java. (In fact, if we’d structured our code to use final in as many places as possible, we could have been encouraged towards basically the same solution in Java too.)

Treat number 2: lifetimes

Second, if you look a bit more closely at the full code of shutdown_order.rs you’ll see lots of confusing-looking annotations like <'a> and &'a:

struct Monitoring<'a> {
    db: &'a Db,
}

The approximate meaning of those annotations is: a Monitoring holds a reference to a Db, and that Db must last longer than the Monitoring.

This “lasts longer than” wording is what Rust Lifetimes are for. Lifetimes are a way of saying how long something lasts.

Lifetimes are really confusing when you start with Rust, and have caused me a lot of pain. Code like this is where they are both most painful and most helpful. As I mentioned earlier, the problem of shutdown order is fundamentally hard. Rust gives you that pain at the beginning, and until you understand what’s going on, the pain is very confusing and acute. But, once your code compiles, it is correct, at least as far as problems like this are concerned.

I love the sense of security it gives me to write Rust code and know the compiler has checked my code for this kind of problem, meaning it can’t crop up at 3am on Christmas Day…

Final note/caveat

This Rust code is probably over-simplified, because all the references are immutable (you can’t change the objects they point to). In practice, we may well have mutable references, and if we do we’re going have to deal with the further difficulty that Rust won’t allow two different objects to hold references to an object if any of those references are mutable. So it would object to Billing and Monitoring using the Db object at the same time. We’d need to make it immutable (as we have here), or find a different way of structuring the code: for example, we could hold the Db instance only within the run_main_loop code, and pass it in temporarily to the Billing and Monitoring objects when we called their methods. A large part of the art, fun and pain of learning Rust is finding new patterns for your code that do what you need to do and also keep the compiler happy. When you manage it, you get amazing benefits!

Edge computing providers

Andy Balaam from Andy Balaam&#039;s Blog

I’m looking into Edge computing at work. By Edge computing I mean running WASM programs in lots and lots of smallish computers in places near to actual people (rather than in huge cloud data centres). I think it’s cool because I love Rust, and Rust is the leading language to compile to WASM.

Here are some companies providing Edge computing services:

  • Fastly – good links with WASM community (hired Mozilla devs), and early adopters – custom WASM engine wasmtime.
  • Cloudflare – huge, and early adopters – WASM engine is Google V8.
  • AWS Lambda@Edge – docs are light on detail, but it looks like a real offering, probably.

Also-rans:

Who did I miss?

Schema upgrades should be reversible (also other transformations, actually)

Andy Balaam from Andy Balaam&#039;s Blog

Are you writing schema upgrade code? Then I humbly suggest you take the time to write schema downgrade code too.

“Why would I do that?” you might well ask, “I won’t ever need to downgrade.”

Now, I imagine you’re expecting me to say you actually will need to downgrade, but that isn’t what I’m saying.

Can you please get on with what you are actually saying?

Whevener you write code to transform something, be it a schema upgrade, some serialisation, or something else, I would highly recommend that you write code to transform it in both directions.

Reasons:

  • It makes testing easier. The best kinds of tests for things like this are round-trips, where you transform something in both directions and check it hasn’t changed. It’s really hard to mess up tests like that.
  • It often uncovers bugs, because it enforces clear thinking about what the transformation actually means.
  • It may improve your code, because it gets annoying writing similar-but-different code to transform in both directions, so you are pushed towards some kind of abstraction.

Also:

  • You almost certainly are going to need it. Sometimes things go wrong and you need to back up.
  • It will be incredibly useful for testing other parts of your code.

Bidirectional scheme up/downgrades are not easy in SQL, but probably worth it. If you’re writing transformation code in a normal programming language, it’s really not that difficult, and I predict it will be worth it.

Announcing Smolpxl Scores – a high score table for your game

Andy Balaam from Andy Balaam&#039;s Blog

It’s a very early beta for now, but I’m ready to announce Smolpxl Scores, which provides high-score tables for Free and Open Source games.

Each game can have multiple high-score tables – for example, you might want one for each level.

At the moment it’s deployed in my own web hosting and therefore written using the technologies that are most convenient for me to deploy there, which is PHP+MySQL. If it becomes more widely used and the performance suffers I guess I’ll ask for donations to host it somewhere else, and use more fashionable technologies.

To add a score you make a POST request like this:

curl https://scores.artificialworlds.net/api/v1/myappname/mytablename/ -d \
    '{"appId":"myappid","name":"Megan Tria", "score": 13.5, "notes": ""}'

and to look at some existing scores you can request them by pages:

curl 'https://scores.artificialworlds.net/api/v1/myappname/mytablename/?startRank=11&num=20'

or by name:

curl 'https://scores.artificialworlds.net/api/v1/myappname/mytablename/?startName=David%20Lloyd%20Geo&offset=-5&num=10'

The results are ordered by players’ scores, and are provided as JSON.

Each table stores only one score per player.

Of course, the API will evolve over time, but I hope that what I have now will be good enough to support some real-life games, and provide enough feedback to make it better.

As soon as people are actually using it, I will ensure the current API version (v1) remains stable, and release any incompatible updates as later versions.

If you’d like to use Smolpxl Scores to add a high-score table to your game, please create an issue at gitlab.com/smolpxl/smolpxl-scores/-/issues.

This service is only available to Free and Open Source games. Also, if someone abuses it (accidentally or on purpose) I will talk to them, and may eventually have to remove their access if we can’t fix the problem.

Profile a Java unit test (very quickly, with no external tools)

Andy Balaam from Andy Balaam&#039;s Blog

I have a unit test that is running slowly, and I want a quick view of what is happening.

I can get a nice overview of where the code spends its time by adding this to the JVM arguments:

-agentlib:hprof=cpu=samples,lineno=y,depth=3,file=hprof.samples.txt

and running the test as normal.

Now I can look at the file that was created, hprof.samples.txt, and looking at the bottom section I can see how much time is spent in each method.

This worked for me within IntelliJ IDEA community edition by clicking “Run” then “Edit Configurations” and adding the above code to “VM options” for my test.

It should also work in Gradle by editing gradle.properties and adding something like this:

org.gradle.jvmargs=-agentlib:hprof=cpu=samples,lineno=y,depth=3,file=hprof.samples.txt

and should also work in Maven. In fact, I found this information in this stackoverflow question: How do you run maven unit tests with hprof?.

Why a Free Software web games site?

Andy Balaam from Andy Balaam&#039;s Blog

Recently I’ve been having a lot of fun working on Smolpxl, which is a web site featuring some little retro web games that are all Free and Open Source Software.

Here’s a sneak preview of the game I am working on:

A pixellated spaceship avoids some walls, then crashes into them

Why do this?

Apart from the fact that it’s fun, I also think there is a need for a site like this: a safe place for kids to play little games without creepy advertising looking over their shoulder, and perverse incentives for the site creators.

Little web games can be a diversion during train journeys, helpful distractions for parents and teachers to provide for kids, and even be a little educational around mouse and keyboard use. I’ve seen the sites that already exist be helpful in all those contexts, but I’ve always felt uncomfortable that these sites are supported by advertising, which always comes with concerns about privacy, and also leads game creators to focus on “engagement”, creating mechanisms like site-wide currencies and gambling-style rewards that drive addictive behaviours.

Wouldn’t it be nice if we in the Free and Open Source community could write some fun games that are free from those unhealthy influences?

Wouldn’t it be even nicer if we took the opportunity to encourage kids to learn how to make games as well as play them?

Well, that’s the idea. Have a look at smolpxl.artificialworlds.net, play a few games, and think about writing a few more…

Also, if you know of existing Free and Open Source web games that might work well on the site, let me know and I’ll have a chat with their creators: I definitely plan to include games by more people than just me.

Code your first game: Snake in JavaScript (on Raspberry Pi)

Andy Balaam from Andy Balaam&#039;s Blog

Welcome! We are going to code a whole snake game. It’s going to look like this:

A finished snake game being played

It doesn’t matter if you have never written any code before: I am going to try and explain everything from scratch.

I’m going to assume you are using a Raspberry Pi. but you don’t need one – all you need is Git, a text editor (like Notepad on Windows) and a web browser (like Firefox or Chrome or Safari or Edge). All laptops and desktop PCs should have the last two. On a tablet or phone it will be more tricky: try to use a Raspberry Pi or other computer if you can. If you are not using a Raspberry Pi, the screenshots will look a bit different, so you’ll have to do a little detective work to follow along.

For the first part where we download some code, your Raspberry Pi needs to be connected to the Internet, but once the “git clone” part is done, you can do everything else disconnected.

Before we start

If you’ve got a Raspberry Pi with Raspberry Pi OS on it, you are ready to go!

If you’re on another computer, make sure you’ve got Git installed. This should be easy on Linux, and possible on Mac or Windows.

Setting up

The first thing we need to do is download the code we are going to use to help us make a game. We’ll get it from my games web site, Smolpxl games.

Start your web browser:

Click the browser launch button

type in the address: smolpxl.artificialworlds.net

Typing the address into the browser address bar

and press Enter.

Scroll to the bottom of the page and click the link “make your own games at gitlab.com/andybalaam/smolpxl“.

Choosing the gitlab link at the bottom of the page

On the GitLab page that loads, click on Clone on the right:

Clicking the Clone button

and, under “Clone with HTTPS” click the “Copy URL” button:

Clicking the Copy URL button under Clone with HTTPS

Now start the Terminal:

Starting the Terminal by clicking its icon

and type in “git clone” followed by a space, but don’t press Enter yet. Right-click in the Terminal and choose “Paste” to paste in the URL that we copied before.

Your Terminal should look like this:

The Terminal showing text: git clone https://gitlab.com/andybalaam/smolpxl.git

Press Enter, and it should change to look like this:

Results of a successful git clone command

Go to your file manager:

Opening file manager by clicking its icon

and double-click on “smolpxl”:

Double-clicking smolpxl folder

and then double-click on “public”:

Double-clicking public folder

We are going to copy the “snake” folder. Right-click on it:

Right-clicking the snake folder

and choose “Copy”:

Choosing "Copy"

then right-click somewhere in the space below, and click “Paste”:

Choosing Paste

The Pi doesn’t want to paste the new one over the old one, so type a new name for it: “minisnake”, and click “Rename”:

Renaming pasted folder to minisnake

Now go into your new minisnake folder by double-clicking on it:

Double-clicking the new minisnake folder

We’re going to edit the file called game.js. To do this, double-click on it:

Double-clicking game.js

This should open the code in a text editor program called Mousepad. (If you’re not on a Raspberry Pi, you might need to right-click the file and choose “Edit”, or similar, to open it in a text editor.)

We want to delete everything in here and start again. Click “Edit”, then “Select all”:

Choosing Edit, then Select All, and hello to you if you are reading this!

Now press the “Delete” key, and all the code should be gone:

Mousepad with all the code gone

We are ready to start!

A game that does nothing

First, we’re going to type some code that gets us ready, but doesn’t actually do anything useful.

Type in this code:

const game = new Smolpxl.Game();
game.setSize(20, 20);
game.setTitle("Minisnake");

function update(runningGame, model) {
}

function view(screen, model) {
}

game.start("minisnake", {}, view, update);

This sets the size of our screen (20 by 20 squares), and a title, and gets things ready for our game. Notice that we start our game by writing game.start.

Once you’ve typed in the code:

Mousepad with the code I just told you to type in

click “File” and then “Save”:

Choosing "File", then "Save"

Now we’re going to try it out! Go back to the minisnake folder:

Switching back to file manager, which is on the minisnake folder still

and double-click on the file called index.html:

Double-clicking index.html

If you typed everything correctly, a title screen should appear. To figure out any problems, we will want the Developer tools open. To do that, click the three dots in the top right, then “More tools”, and “Developer tools”:

Choosing Chromium's burger menu in the top right, then clicking More tools, then Developer Tools

The tool we want is the Console, so click “Console” to see it:

Choosing the Console tab in Developer tools

If everything went well, the Console should be empty:

Our game on the left, and the Console on the right, with no error messages showing

If you see red error messages in the Console, you probably have a typo: double-check everything you typed, and compare it with this version: game-01-nothing.js.

Now we have everything ready to get started, and you can see our game already has a title screen!

The game is going to be made from three things:

  • a model,
  • a view, and
  • an update function.

Let’s start with the model and the view.

The Model and the View

The first thing we’re going to do is add an apple to the game. To do that, we need to do two things:

  • say where the apple is, and
  • draw the apple on the screen.

To say where the apple is, we need a model. We actually already have a model, but it’s just empty.

Switch back to your code:

Switching back to Mousepad by clicking game.js in the top bar

Look at the last line of code we typed, and find where you typed {}. This is the “model”, or the state of out game when we start it off. Now, we’re going to replace that with newModel(), which means call a function called newModel. (A function is a piece of code that we can re-use, and “calling” a function means running that code.)

So we’re going to write a function called newModel, and then call it. Replace the very last line of your code with this:

function newModel() {
    return {
        apple: [5, 5]
    };
}

game.start("minisnake", newModel(), view, update);

The top part makes a function called newModel, and the bottom part is the same as you had before, except {} is replaced by newModel().

Have another look at the newModel function we wrote (the top part). Can you see where we made the apple? After the word return, we have some stuff inside curly brackets ({ and later, }). When we use curly brackets like that, we are making what JavaScript calls an object. Objects have properties. So far, our object only has one property, which is called “apple”. The right-hand part, [5, 5] is how we are say where our apple is – it’s at the position 5 squares in from the left, and 5 squares down from the top of our screen.

Now we have said where our apple is, we also need to draw it on the screen. That happens inside the view function. Scroll up and find the part that says function view() and add a new line between the opening { and the closing one:

function view(screen, model) {
    screen.set(model.apple[0], model.apple[1], [255, 0, 0]);
}

We are calling a function called screen.set, which draws a square on the screen. We are passing in three things to it: two points for the position to draw, and then the colour to use. The position is model.apple[0], which means take the first part of the [5, 5] we typed before, and model.apple[1], which means take the second part. The color is [255, 0, 0] which means lots of red, and no green or blue, because this is a red-green-blue (RGB) colour.

So let’s try this out and see whether the apple is drawn on the screen. First save your code in Mousepad, then switch back to our game (in Chromium), and click the refresh button to reload it.

Switching back to Chromium in the top bar, then clicking the refresh button

You should see your game saying “Enter to start”. Click on it, and the game should start, and look like this:

The game window is black, except there is a red square for the apple

Well done, you have drawn an apple!

If you see red error messages in the Console, you probably have a typo: double-check everything you typed, and compare it with this version: game-02-just-apple.js.

Drawing a snake

Now that we have an apple, let’s follow the exact same pattern to add a snake. We’re going to add information about it to the model, and then use that information to draw it on the screen.

First, change the newModel function to look like this:

function newModel() {
    return {
        apple: [5, 5],
        body: [[10, 10], [10, 11], [10, 12], [10, 13], [10, 14]]
    };
}

Don’t miss the extra comma at end of the “apple” line!

To give the co-ordinates of the apple, we just used one [x, y] pair. Because the snake’s body is made from 5 points, we need 5 similar pairs.

We’ve described the snake’s body position, so now let’s draw it on the screen. Update the view function so it looks like this:

function view(screen, model) {
    // Snake
    for (const [x, y] of model.body) {
        screen.set(x, y, [0, 255, 0]);
    }

    // Apple
    screen.set(model.apple[0], model.apple[1], [255, 0, 0]);
}

We have added a for loop – it runs through all the points in body, and for each one it draws a square on the screen. This time the colour is [0, 255, 0], which means the snake will be green.

(By the way, did you notice the lines we added that start with // – these are “comments” – we can write anything we like after the slashes and it doesn’t do anything. We can use comments to add notes that help us remember what different bits of our program do.

Save the file, go back to the game in Chromium and click the Refresh button again. If all goes well, you should see the snake appear:

A black game screen, with a red dot and green line for the snake

If you see red error messages in the Console, you probably have a typo: double-check everything you typed, and compare it with this version: game-03-snake.js.

So now we have a snake and an apple, but nothing is really happening … let’s fix that next.

Making the snake move

We’ve made a model in newModel and we’ve drawn it on the screen in view, but how do we make things move around? That is where the update function comes in: this is where we change the model based on what is happening.

Let’s start by making the snake move forward forever. Change the update function to look like this:

function update(runningGame, model) {
    // Move the snake
    let newHead = Smolpxl.coordMoved(model.body[0], model.dir);
    let newTail = model.body.slice(0, -1);

    model.body = [newHead, ...newTail];
    return model;
}

and change the newModel function to look like this:

function newModel() {
    return {
        apple: [5, 5],
        body: [[10, 10], [10, 11], [10, 12], [10, 13], [10, 14]],
        dir: Smolpxl.directions.UP
    };
}

Again, notice the comma at the end of the body line!

We added dir to the model, which is the direction the snake is facing.

The update function makes newHead by moving the snake’s head (the first entry in its body, which it gets with model.body[0]) in the direction it is facing (model.dir). Then we create newTail, which is everything in the old model.body except the last entry in the list (this is what .slice(0, -1) means).

Finally we update the body by setting it to [newHead, ...newTail], which just means make a new list by sticking newHead on to the beginning of newTail.

Save, switch to the game in Chromium, and refresh. Because we set dir to Smolpxl.directions.UP inside newModel, the snake moves updards!

The green snake moves up the screen (and off the top)

If you see red error messages in the Console, you probably have a typo: double-check everything you typed, and compare it with this version: game-04-movement.js.

If the snake disappears off the top, click the refresh button to see it again.

The game isn’t too much fun yet. Let’s add some keyboard controls, and allow you to die when you go off-screen.

Controlling the snake

The update function moves the snake, but now it’s time to make it a bit cleverer, by changing direction when you press a key, and stopping you when you go off-screen. Change it so it looks like this:

function update(runningGame, model) {
    if (!model.alive) {
        return;
    }

    if (runningGame.receivedInput("LEFT"))  {
        model.dir = Smolpxl.directions.LEFT;
    } else if (runningGame.receivedInput("RIGHT")) {
        model.dir = Smolpxl.directions.RIGHT;
    } else if (runningGame.receivedInput("UP")) {
        model.dir = Smolpxl.directions.UP;
    } else if (runningGame.receivedInput("DOWN")) {
        model.dir = Smolpxl.directions.DOWN;
    }

    // Move the snake
    let newHead = Smolpxl.coordMoved(model.body[0], model.dir);
    let newTail = model.body.slice(0, -1);

    // Die if we hit the edge
    if (
        newHead[0] === runningGame.screen.minX ||
        newHead[0] === runningGame.screen.maxX ||
        newHead[1] === runningGame.screen.minY ||
        newHead[1] === runningGame.screen.maxY
    ) {
        model.alive = false;
    }

    model.body = [newHead, ...newTail];
    return model;
}

and change the newModel function to look like this:

function newModel() {
    return {
        alive: true,
        apple: [5, 5],
        body: [[10, 10], [10, 11], [10, 12], [10, 13], [10, 14]],
        dir: Smolpxl.directions.UP
    };
}

We keep track of whether the snake is alive in the model, and we immediately return from update if we are dead, meaning the snake stops moving. (The ! in if (!model.alive) means “not”, so we are saying what to do when we are not alive – when we are dead. The return here means immediately stop running the code in this function.)

The next new part of update allows us to check whether an arrow key was pressed (using the runningGame.receivedInput function), and if so, change the direction of the snake (model.dir).

Finally, nearer the end of update, we check whether the position of the snake’s head (newHead) is at one of the screen edges, by comparing its co-ordinates with the maximum and minimum co-ordinates on the screen. If we are off the edge, we set model.alive to false, meaning the snake is now dead.

Save, switch to the game in Chromium, and refresh. With all that, we can control the snake with the arrow keys, and it can die:

A snake moves around, turning because arrow keys were used to control it

Try clicking on your game and then pressing the arrow keys to control the snake.

If you see red error messages in the Console, you probably have a typo: double-check everything you typed, and compare it with this version: game-05-control.js.

This is kind-of a game, but surely it’s time to eat some apples?

Eating apples

If we’re going to be eating apples, they should probably not always be in the same place, right?

Let’s start off by making a brand new function. Make some space immediately above the update function, and type in this code:

function randApple() {
    return [Smolpxl.randomInt(1, 18), Smolpxl.randomInt(1, 18)];
}

This function gives us some random co-ordinates where we can place an apple. Update newModel to place the first apple at a random place, by making it look like this:

function newModel() {
    return {
        alive: true,
        apple: randApple(),
        body: [[10, 10], [10, 11], [10, 12], [10, 13], [10, 14]],
        dir: Smolpxl.directions.UP
    };
}

So instead of writing the exact co-ordinates we want ([5, 5]), now we’re calling our new randApple function, which gives us back some random co-ordinates.

Now we can place apples randomly, let’s change the update function to allow us to eat apples. While we’re there, let’s check whether we crashed into our own body too:

function update(runningGame, model) {
    ... All the same stuff as before ...

    // Die if we hit the edge
    if (
        newHead[0] === runningGame.screen.minX ||
        newHead[0] === runningGame.screen.maxX ||
        newHead[1] === runningGame.screen.minY ||
        newHead[1] === runningGame.screen.maxY
    ) {
        model.alive = false;
    }

    // If we hit the apple we get longer
    if (Smolpxl.equalArrays(newHead, model.apple)) {
        for (let i = 0; i < 5; i++) {
            newTail.push([-1, -1]);
        }
        model.apple = randApple();
    }

    // If we hit our own body, we die
    if (Smolpxl.arrayIncludesArray(newTail, newHead)) {
        model.alive = false;
    }

    model.body = [newHead, ...newTail];
    return model;
}

We use the Smolpxl.equalArrays function* to ask whether the new position of the snake’s head (newHead) is the same as the position of the apple (model.apple).

* The function is called “equalArrays” because both newHead and model.apple are lists of two co-ordinates (x and y), so we are storing them inside JavaScript “arrays”. You can spot an array because it is surrounded by square brackets ([).

If the head is on top of the apple, we do two things: add new items to the end of newTail, by using a for loop that runs 5 times. Each time it runs, it uses push to add another body part on the to the tail. (It actually adds an off-screen point [-1, -1] every time – this means they won’t get drawn at first, but as the snake moves forward, they will gradually get replaced by on-screen points, and we’ll see the snake gradually get longer.)

The second thing we do if the head is on the apple is move the apple by setting model.apple to another random point given to us by randApple.

We also check whether we have hit our own body using the Smolpxl.arrayIncludesArray function (asking whether newHead is the same point as one of the points in newTail) and set model.alive to false, meaning we’re dead, if so.

Save, switch to the game in Chromium, and refresh. Now we can really play a game of snake!

A snake moves around the screen, eating apples and getting longer when it does

If you see red error messages in the Console, you probably have a typo: double-check everything you typed, and compare it with this version: game-06-eating-apples.js.

We are nearly done. The last jobs are to make the game look a little nicer, and handle starting a new game after we crash.

Finishing off

We need to draw the walls around the edge that you crash into, and it would also be nice to show your score at the top of the screen. We can do both of these by adding a bit more at the beginning of the view function:

function view(screen, model) {
    screen.messageTopLeft(`Score: ${model.body.length}`);

    // Walls
    for (const x of screen.xs()) {
        screen.set(x, screen.minY, [150, 150, 150]);
        screen.set(x, screen.maxY, [150, 150, 150]);
    }
    for (const y of screen.ys()) {
        screen.set(screen.minX, y, [150, 150, 150]);
        screen.set(screen.maxX, y, [150, 150, 150]);
    }

    ... All the same stuff as before ...
}

We use the screen.messageTopLeft function to display a message at the top. To format the message we write some text inside backticks (`), which allows us to substitute values in. Here we typed ${model.body.length}, which means our score is how many points there are inside our body (which is the length of the array model.body). The ${} part just means “substitute this value in here”.

We make use of the screen.xs() and screen.ys() functions that give us a list of all the x and y co-ordinates on the screen to draw the wall.

Save, switch back to the game, and refresh to see how this looks. If you want to check what you have typed so far, compare it against game-07-walls-and-score.js.

The very last thing we need to do is handle restarting the game after we crash.

First, let’s display a message on the screen by updating the view function one last time:

function view(screen, model) {
    ... All the same stuff as before ...

    // Snake
    for (const [x, y] of model.body) {
        screen.set(x, y, [0, 255, 0]);
    }

    // Death
    if (!model.alive) {
        screen.set(model.body[0][0], model.body[0][1], [0, 0, 255]);
        screen.dim();
        const score = model.body.length;
        screen.message(["Well done!", `Score: ${score}`, "Press <SELECT>"]);
    }

    // Apple
    screen.set(model.apple[0], model.apple[1], [255, 0, 0]);
}

When the snake is not alive we draw a blue dot ([0, 0, 255]) where its head is, then make the screen dark with the screen.dim function, and write a message on the screen with the screen.message function.

Last, if we press Enter while the death message is visible, we want to go back to the start. We can do that with a small change to the update function:

function update(runningGame, model) {
    if (!model.alive) {
        if (runningGame.receivedInput("SELECT")) {
            runningGame.endGame();
            return newModel();
        } else {
            return;
        }
    }

    if (runningGame.receivedInput("LEFT"))  {
    ... All the same stuff as before ...

Notice that we deleted the line that just said return; and replaced it with the new if code. We check whether the user pressed the SELECT button (which means the Enter key) and if so, we tell Smolpxl to go back to the title screen by calling runningGame.endGame, and then we reset everything in the model by returning newModel().

Save, switch to the game in Chromium, and refresh. Our game is finished!

A finished snake game being played

If you see red error messages in the Console, you probably have a typo: double-check everything you typed, and compare it with this version: game-08-finished.js.

Well done!

If you got that working, you should be extremely pleased with yourself. Typing all that code correctly is a major challenge, and your determination has paid off.

Take a breath, and have a think about whether you can show what you’ve done to someone else. I’m pretty sure they will be impressed that you coded an entire game!

Challenges

When you’ve properly taken the time to enjoy the great work you’ve done, try reading through the code, and reading back through this blog post to try and understand how the program works.

To learn more about JavaScript and making web sites, you can follow this much more comprehensive tutorial: Getting started with the Web.

You’ve done really well. If you want an extra challenge, try improving your game using these challenges. Bear in mind though, working these out on your own will be much harder than what we’ve done so far:

  • Challenge 1: Display a different message if you don’t eat any apples. If you crash before eating any apples, you might need some extra encouragement: change the part of the view function where we display a “Well done!” message, and if our score is 5 (if (model.body.length === 5) then display a different message – maybe “Bad luck!”. Look for ifelse statements we have already written, to see how they work.
  • Challenge 2: Prevent going back on yourself. Update your code so that when snake is going right, and the user presses left, we ignore it. You will need to change the update function, where we use receivedInput to check what key the user pressed. Instead of just setting the direction, you will need to add a new if statement that check what direction we are facing already.
  • Challenge 3: Remember the high score. Remember the best score anyone has got, and display it at the top-right of the screen. At the moment in our code, we return newModel(); in update, when we want to restart the game. That means we have a totally fresh model, forgetting everything else that happened before. If we want to remember a high score, we can’t do that! If you want some extra inspiration, have a look at the more complete version of snake that comes with the Smolpxl games. It’s actually included in the code you downloaded with the git clone command you typed at the very beginning.

What next?

To learn more, try:

Remember to have fun, and be kind to the people you meet on the way.

Play and create little retro games at Smolpxl

Andy Balaam from Andy Balaam&#039;s Blog

I love simple games: playing them and writing them.

But, it can be overwhelming getting started in the complex ecosystems of modern technology.

So, I am writing the Smolpxl library, which is some JavaScript code that makes it quite simple to write simple, pixellated games. It gives you a fixed-size screen to draw on, and it handles your game loop and frames-per-second, so you can focus on two things: updating your game “model” – the world containing all the things that exist in your game, and drawing onto the screen.

I am also writing some games with this library, and publishing them at smolpxl.artificialworlds.net:

I am hoping this site will gradually gain more and more Free and Open Source games, and start to rival some of the advertising-supported sites for the attention of casual gamers, especially kids.

Smolpxl is done for fun, and for its educational value, so it should be a safer place for kids than a world of advertising, loot boxes and site-wide currencies.

As I write games, I want to show how easy and fun it can be, so I will be streaming myself live doing it on twitch.tv/andybalaam, and putting the recordings up on peertube.mastodon.host/accounts/andybalaam and youtube.com/user/ajbalaam.

I am hoping these videos will serve as tutorials that help other people get into writing simple games.

Would you like to help? If so: