Posted on

Improving my vimrc live on stream

Andy Balaam from Andy Balaam's Blog

I was becoming increasingly uncomfortable with how crufty my neovim config was getting, and especially how I didn’t understand parts of it, so I decided to wipe it clean and rebuild it from scratch.

I did it live on stream, to make it feel like a worthwhile activity:

Headline features of the new vimrc:

  • A new theme, using the base16 theme framework
  • A file browser (NERDTree)
  • A minimalist status line with vim-airline
  • Search with ripgrep
  • Rust language support with Coc

Note: after the stream I managed to resolve the remaining issues with highlight colours not showing by triggering re-applying them after the theme has been applied:

augroup tabs_in_make
    autocmd!
    autocmd ColorScheme * highlight MatchParen cterm=none ctermbg=none ctermfg=green
augroup END

You can find my current neovim config at gitlab.com/andybalaam/configs/-/tree/main/.config/nvim.

Posted on

Comparison of Matrix events before and after “Extensible Events”

Andy Balaam from Andy Balaam's Blog

(Background: Matrix is the awesome open standard for messaging that I get to work on now that I work at Element.)

The Extensible Events (MSC1767) Matrix Spec Change proposal describes a new way of structuring events in matrix that makes it easy to send events that have multiple representations (e.g. something clever like an interactive map, and something simpler like an image of a map).

The main purpose of the change is to make it easy for clients that don’t support some amazing new feature to display something that is still useful.

Since there is an implementation of this change out in the wild (in Element), it seems reasonably likely that this change will be accepted into the Matrix spec.

I really like this change, but I find it hard to understand, so here is a simple example that I have found helpful to think it through.

An old event, and a new event

Here is an old-fashioned event, followed by a new, shiny, extensible version:

{
    "type": "m.room.message",
    "content": {
        "body": "This is the *old* way",
        "format": "org.matrix.custom.html",
        "formatted_body": "This is the <b>old</b> way",
        "msgtype": "m.text"
    },
    ... other properties not relevant to this, e.g. "sender" ...
}

{
    "type": "m.message",
    "content": {
        "m.message": [
            {"mimetype": "text/plain", "body": "This the *new* way"},
            {"mimetype": "text/html", "body": "This is the <b>new</b> way"}
        ],
    }
    ... other properties not relevant to this, e.g. "sender" ...
}

Notice that in the new extensible events, the property within content is the same as the message type (here: m.message).

The point is that as well as the primary event type (here, m.message) we can other representations of the same message, such as an image, location co-ordinates, or something completely different. The client will render the primary event type if it understands it (and is able to show it), but if not, it can look for other types that it does understand.

For example, in Polls when you send a new poll question, it could look like this:

{
    "type": "m.poll.start",
    "content": {
        "m.poll.start": {
            ... The actual poll question etc. ...
        },
        "m.message": [
            ... A text version of the question ...
        ]
    },
    ... other properties not relevant to this, e.g. "sender" ...
}

So clients that don’t know m.poll.start can still display the poll question (if they understand extensible events), instead of completely ignoring event types they don’t know about.

An abbreviated form of the new event

Of course, life is not quite as simple as that.

Because this is a lot of typing:

{
    "type": "m.message",
    "content": {
        "m.message": [
            {"mimetype": "text/plain", "body": "This the *new* way"},
            {"mimetype": "text/html", "body": "This is the <b>new</b> way"}
        ],
    }
    ... other properties not relevant to this, e.g. "sender" ...
}

We have an abbreviated form:

{
    "type": "m.message",
    "content": {
        "m.text": "This the *new* way",
        "m.html": "This is the <b>new</b> way"
    }
    ... other properties not relevant to this, e.g. "sender" ...
}

These two are exactly equivalent.

m.text is an abbreviation for an m.message containing an entry with "mimetype": "text/plain" and the relevant body. Similarly, m.html is an abbreviation for an m.message containing an entry with "mimetype": "text/html" and the relevant body. If you declare both, they effectively get squashed together into one m.message with both entries.

Those 2 are the only abbreviations listed, so they are special cases.

Backwards compatibility

Of course, life is way more complicated than that, so what we’re likely to see around if/when this gets widely adopted is some kind of mashed-together event like this:

{
    "type": "m.room.message",
    "content": {
        "msgtype": "m.text",
        "body": "Hello World",
        "format": "org.matrix.custom.html",
        "formatted_body": "<b>Hello</b> World",
        "m.text": "Hello World",
        "m.html": "<b>Hello</b> World"
    }
}

Note that the type here is m.room.message, where extensible events says it should be m.message. The idea is that an extensible-events-aware client will see "msgtype": "m.text" and know to look for m.message as the primary type. (This is further complicated here by the fact that there isn’t actually a m.message property – this is because m.text and m.html are abbreviated forms of it.)

Also, clients that want to display old events will need to preserve their code that parses the old event types in perpetuity.

Posted on

Air-Source Heat Pump – our experience so far, 2 months in

Andy Balaam from Andy Balaam&#039;s Blog

Summary: less energy, more money

2 months ago, we replaced our gas boiler with an air-source heat pump, which uses electricity to heat our home and boiler. This is a report of our experience so far.

We expected it to reduce our environmental impact, and cost us more money, and we were right.

It works: our house is comfortable. We use a lot less energy, and it costs us significantly more money (because electricity costs way more than gas).

The house

Our house is a beautiful, leaky old house, with a modern extension. Half of it is well-insulated. The other half was built around 1890, and while we do have double-glazing and decent loft insulation, the walls have no cavities and feel cold to the touch, and there are drafts everywhere.

The new half has underfloor heating. The old half and the upstairs are heated by radiators. We have a hot water cylinder.

The air-source heat pump

Our air-source heat pump uses electricity to extract heat from the outside air and heats water for radiators and hot water, directly replacing our gas boiler.

Our heat pump was installed by Your Energy Your Way and I must declare in interest: my wife is a director of the company.

The heat pump is an LG 16kW “THERMA V” model. It looks like a very large air conditioning unit, which sits outside our house in the yard to the side. It is about as tall as my shoulder height, with two big fans on it.

A large air-source heat pump

It stands on a soak-away area with some stones on it that the installers made by removing some patio tiles. This is needed because it drips a small amount of liquid as part of its normal operation. The outdoor unit makes noise, but our house is next to the main road, so we don’t hear it. It is not audible indoors.

Standing next to the outdoor unit you can feel a cold breeze, like opening the fridge door. This is unpleasant on cold days.

That outdoor unit connects through the wall to an indoor part that is a bit smaller than our old boiler.

The controller box has a terrible user interface and is very hard to decipher, but we did eventually manage to programme it to turn the target temperature up in the daytime and down at night. Your Energy Your Way advised us that it is more efficient to keep the house at a cool-ish 17 degrees at night, rather than letting it get cold and having to work hard heat it up again in the morning, so that is how we have set it up.

The controller box’s built-in thermostat does not work properly (it reports the wrong temperature), so we had to add an external thermostat, which works well.

We didn’t need to change anything about our hot water cylinder, or our underfloor heating.

When planning the installation, Your Energy Your Way estimated the heat loss of our rooms, and recommended upgrading our radiators. In an old house like ours this is sometimes needed, because it is way more efficient to heat a house with cooler water running through the radiators, but if the water is cooler, you need more radiator surface area to heat the house effectively. In a newer house with existing radiators, they are probably fine as-is.

We kept most of the existing radiators, and added some more in the coldest rooms.

How comfortable is the house?

The house is more comfortable than it was before, for two reasons: firstly the radiators we had were not really adequate, and secondly the cooler water in the radiators makes a less irritating heat, meaning the house is nicely comfortable most of the time, instead of bouncing between feeling cold and feeling oppressively over-heated.

On cold days, the old part of the house is a bit cold, but I think on average it’s a little better than it was before.

We do find mornings can be chilly, particularly because the system stops heating the radiators if the hot water cylinder needs heating up after people have had showers. We could improve this situation by getting a larger cylinder, which we are considering.

However, it’s worth pointing out that we needed engineers to visit four or five times to make adjustments before we felt the system was working well enough. There are a lot of things that can be tweaked, and it took some time for it to work well.

My advice: don’t pick the cheapest quote – pick the people you think you can trust to do the work well: especially the heat loss calculations before installation and the adjustments afterwards.

How much energy are we using? (The good news)

So far, it looks like we are using about two-thirds less energy in our household than we were before:

The above chart is stacked, so the top line represents the total energy usage. We switched to the air-source heat pump exactly when our gas usage was about to skyrocket (because it’s cold in winter), and it remained relatively low.

This is absolutely fantastic: our house is more comfortable than before, and we have reduced the amount of energy we are using by 66%. This is the total energy usage of our house, not just for heating, so the reduction of energy used for heating is even more dramatic than it looks.

Even better, the energy we use is at least partly produced from renewable sources, so our carbon footprint is much lower. Previously we were directly releasing carbon by burning imported gas – now we use mostly UK-produced electricity, and as the grid decarbonises, our carbon footprint reduces even if we make no further changes.

How much money are we spending? (The bad news)

Excluding standing charges*, we are spending about one third more on energy than we were before. This is because electricity is so much more expensive than gas: our electricity costs 19p per kWh and our gas costs 4p per kWh.

* Note: our energy provider wanted to charge us £350 to remove our gas meter, so we refused, and are still paying the gas standing charge. I’m not sure how we’re going to resolve this, especially since our energy provider is now in administration.

The above chart is stacked, so the top line represents the total cost (excluding standing charges). When we switched to the air-source heat pump, our energy costs increased faster than they did the same time last year, and were consistently higher. We think the peak in November might be misleading as it may have been when the system was not set up correctly, but we are not sure.

Because air-source heat pumps are more efficient when the weather is warmer, we do expect to fare better in the summer than we are right now.

I would not suggest getting a heat pump if you want to save money. Maybe this will change as gas prices are expected to rise significantly this year.

An installation like ours, including new radiators, costs £10-15K. A decent chunk of that will be paid back to us by the government, spread out over the next 7 years, under the soon-to-be-gone Renewable Heat Incentive (RHI). RHI will be replaced by the
Boiler Upgrade Scheme (BUS), which will be limited to a £5K grant for air-source heat pumps, although it is paid up-front. We would have received much less money under BUS than RHI. It is almost certainly too late for you to get a heat pump under RHI, by the way – all the installers are booked up until end of March 2022, when it ends.

Thoughts

If you think it’s surprising (and deeply concerning) that taking the step of significantly reducing our carbon footprint cost us a one-third increase in our energy bills, I would agree with you.

I am told that the tax taken on electricity is much higher than on gas, even though these taxes are apparently intended help decarbonise our energy.

Meanwhile, the government is replacing (with great fanfare) RHI with the much less generous (although more timely) BUS, making it even more economically punishing to reduce your carbon footprint.

I think this should be addressed urgently: money should be provided to help people install heat pumps, and the tax regime should be changed to make it cheap to use low-carbon fuels.

The technology is available, but the financial situation makes this a vanity project for people like me who can afford it, instead of what it could be: a feasible plan to get our national carbon usage down, fast.

On a positive note, our house is nice and warm, and I feel a bit less guilty about how much carbon we’re using to keep it cosy.

Posted on

Providing MapLibre-compatible style JSON from openstreetmap-tile-server

Andy Balaam from Andy Balaam&#039;s Blog

[Previous: Self-hosting maps on my laptop]

In the previous post I showed how to run OSM tile server stack locally.

Now I’ve managed to connect a MapLibre GL JS front end to my local tile server and it’s showing maps!

(It’s running inside Element Web, the awesome Matrix messenger I am working on. NOTE: this is a very, very early prototype!)

In the previous post I ran a docker run command to launch the tile server.

This time, I had to create a file style.json:

{
  "version": 8,
  "sources": {
    "localsource": {
      "type": "raster",
      "tiles": [
        "http://127.0.0.1:8080/tile/{z}/{x}/{y}.png"
      ]
    }
  },
  "layers": [
    {
      "id": "locallayer",
      "source": "localsource",
      "type": "raster"
    }
  ]
}

and then I launched the tile server with that file available in the document root:

docker run \
    -p 8080:80 \
    -v $PWD/style.json:/var/www/html/style.json \
    -v openstreetmap-data:/var/lib/postgresql/12/main \
    -v openstreetmap-rendered-tiles:/var/lib/mod_tile \
    -e THREADS=24 \
    -e ALLOW_CORS=enabled \
    -d overv/openstreetmap-tile-server:1.3.10 \
    run

Now I can point my MapLibre GL JS at that style file with code something like this:

this.map = new maplibregl.Map({
    container: my_container,
    style: "http://127.0.0.1:8080/style.json",
    center: [0, 0],
    zoom: 13,
});

Very excited to be drawing maps without any requests leaving my machine!

Posted on

Self-hosting maps on my laptop

Andy Balaam from Andy Balaam&#039;s Blog

As part of my research for working on location sharing for Element Web, the Matrix-based instant messenger, I have been learning about tile servers.

I managed to get OSM tile server stack working on my laptop:

Here are a couple useful pages I read during my research:

Today I managed to run a real tile server on my laptop, using data downloaded from OpenStreetMap in a way that I think complies with their terms of use.

To run these commands you will need Docker, and hopefully nothing much else.

Download the data

I downloaded the UK data like this:

wget 'https://download.geofabrik.de/europe/great-britain-latest.osm.pbf'

You can find downloads for other regions at download.geofabrik.de/

Import it

Then I ran an import, which converts the PBF data into tiles that can be shown in a UI:

docker volume create openstreetmap-data
docker volume create openstreetmap-rendered-tiles
docker run \
    -v $PWD/great-britain-latest.osm.pbf:/data.osm.pbf \
    -v openstreetmap-data:/var/lib/postgresql/12/main \
    -v openstreetmap-rendered-tiles:/var/lib/mod_tile \
    -e THREADS=24 \
    overv/openstreetmap-tile-server:1.3.10 \
    import

(Change “great-britain” to match what you downloaded.)

On my quite powerful laptop this took 39 minutes to run.

Run the tile server

Finally, I launched the server:

(Make sure you’ve done the “Import it” step first.)

docker run \
    -p 8080:80 \
    -v openstreetmap-data:/var/lib/postgresql/12/main \
    -v openstreetmap-rendered-tiles:/var/lib/mod_tile \
    -e THREADS=24 \
    -e ALLOW_CORS=enabled \
    -d overv/openstreetmap-tile-server:1.3.10 \
    run

This should launch the docker container in the background, which you can check with docker ps.

Test it

You can now grab a single file by going to http://127.0.0.1:8080/tile/0/0/0.png, or interact with the map properly at http://127.0.0.1:8080.

It was quite unresponsive at first, but once it had cached the tiles I was looking at, it was very smooth.

Posted on

Streaming to Twitch and PeerTube simultaneously using nginx on Oracle cloud

Andy Balaam from Andy Balaam&#039;s Blog

Simulcasting RTMP using NGINX

I want people to be able to watch my Matrix and Rust live coding streams using free software, so I’d like to simulcast to PeerTube as well as Twitch.

This is possible using NGINX and its RTMP module. It does involve building NGINX from source, but I actually found that reasonably easy to do.

Why Oracle cloud?

I would never recommend using Oracle for anything, but they do provide up to two virtual machines in their cloud for free, and the one I am using has been consistently available with very good connectivity, in a London data centre since I set it up several months ago.

So, we are making our lives more difficult by trying to do this on Oracle Linux, which is a derivative of RHEL.

Building NGINX and its RTMP module on Oracle Linux

I ran these commands on my Oracle cloud instance (running Oracle Linux):

sudo yum install git pcre-devel openssl-devel
mkdir nginx
cd nginx
wget http://nginx.org/download/nginx-1.21.4.tar.gz
git clone https://github.com/arut/nginx-rtmp-module.git
cd nginx-1.21.4
./configure --add-module=../nginx-rtmp-module/
make
sudo make install

After all this NGINX was installed to /usr/local/nginx/.

Creating the NGINX config file for RTMP simulcasting

Next I edited the NGINX config file by typing:

sudo nano /usr/local/nginx/conf/nginx.conf

And pasted in this config at the bottom of the file:

rtmp {
    server {
        listen 1935;
        chunk_size 4096;
        application live {
            live on;
            record off;
            push rtmp://live.twitch.tv/app/live_INSERT_TWITCH_STREAM_KEY;
            push rtmp://diode.zone:1935/live/INSERT_PEERTUBE_STREAM_KEY;
        }
    }
}

Notice that you will need to get your Twitch stream key from Twitch -> Creator Dashboard -> Settings -> Stream, then Copy next to the Primary Stream Key.

To get a PeerTube stream ID, you will need to go to your PeerTube page and click Publish, then Go Live, choose your channel and choose Go Live. Note that if you want the streams to record and be available later, you have to create a new stream key each time you start a stream, and change it in nginx.conf.

If you use a different PeerTube server (I use diode.zone) then you’ll need to change the server name in the config file above too.

Make sure your config file is saved with the right URLs in it.

Opening ports

To send RTMP traffic to my server, I needed to open the right port to the Oracle cloud instance. That involved creating an ingress rule, and adding a firewall rule.

Creating an ingress rule

In the web interface, I went to the menu in the top left, clicked Compute, then Instances.

I clicked on my instance’s name, then I clicked on the name of the subnet in the details (on the right).

I clicked on Default security list for…, then Add Ingress Rules.

I made an ingress rule with Source Type=CIDR, Source CIDR=0.0.0.0/0, IP Protocol=TCP, Source Port Range=(blank, meaning all), Destination Port Range=1935

Adding a firewall rule

Then I ssh’d into the machine and ran these commands to create a firewall rule allowing the traffic:

sudo firewall-cmd --zone=public --permanent --add-port=1935/tcp
sudo firewall-cmd --reload

Stop and Start NGINX

After creating the config file and opening the right port, I needed to start NGINX.

Every time I change the config file, I need to restart it.

If it’s already running, I stop it with:

sudo /usr/local/nginx/sbin/nginx -s stop

and then I start it up again with

sudo /usr/local/nginx/sbin/nginx

I can check whether it’s happy by looking at the log files, for example to see any errors:

less /usr/local/nginx/logs/error.log

Starting the stream

Now I go into OBS and go to File -> Settings -> Stream and choose the type as Custom, and the Server as rtmp://1.1.1.1/live. (But instead of 1.1.1.1 I put the public IP address of my instance, which I found by clicking the name of the instance in the Oracle cloud management console.)

Posted on

New game: Tron – frantic multiplayer retro action

Andy Balaam from Andy Balaam&#039;s Blog

My newest game is out now on Smolpxl Games – Tron:

Pixellated lines fight each other to stay alive

Play at smolpxl.gitlab.io/tron.

It’s a frantic multiplayer retro pixellated thingy playable in your browser. Try to stay alive longer than everyone else!

This version allows many players (up to 16 if you can manage it), and is quite pure in its implementation.

There are bots to play against, and you can gather your friends around a keyboard to play together.

Part of the motivation for writing this game was to test my new smolpxl-remote remote-play system, but this is not enabled yet, so watch this space…

I love playing games with other people – preferably at least 3 other people. In theory you could have 8 players around a keyboard playing this – send me a picture if you try!

One feature I worked on in the Smolpxl library for this game: saving configuration to local storage (and asking permission to do so). I ended up with a very ugly hack to do this, so a bit more work is needed before I merge it into the library.

Posted on

Preventing Virgin Media hijacking my DNS

Andy Balaam from Andy Balaam&#039;s Blog

Yesterday I learned that Virgin Media is inserting itself into some of my DNS requests. Much as I am not a fan of how powerful Cloudflare are, if they are telling the truth about their DNS, then it’s safe, so I followed their instructions on how to use their DNS and then removed the default DNS and hopefully my Internet will work now.

From the serverfault answer by lauc.exon.nod:

nmcli con mod "Wired connection 1" ipv4.dns "1.1.1.1 1.0.0.1"
nmcli con mod "Wired connection 1" ipv4.ignore-auto-dns yes
nmcli con down "Wired connection 1"
nmcli con up "Wired connection 1"
Posted on

Letter to my MP about climate emergency

Andy Balaam from Andy Balaam&#039;s Blog

[Introduction including details about my own air source heat pump install, and mention of the ending of the RHI funding in April 2022.]

After I have installed an air source heat pump, I will pay more money to heat my home, even though I am using less energy, because electricity is more expensive than gas. So this change will hurt me financially over both the short and longer terms.

Do you agree with me that climate emergency is the most important issue the government is now facing?

Do you also agree with me that we urgently need people to switch their heating and home insulation to reduce our dependence on burning gas?

Please do all you can to persuade the Prime Minister to introduce initiatives before COP26 that make it financially viable for families without spare cash to insulate their home and heat them with renewable energy.

Please pass my letter on to the Prime Minister and any government departments you consider relevant.

Thank you very much for your time.

Posted on

New Job at Element (Matrix)

Andy Balaam from Andy Balaam&#039;s Blog

I started a new job today at Element!

It has been a long-standing ambition of mine to work in Free and Open Source software, and I am very excited to work for a company that is the main developer of a really important project: the Matrix communication network.

I don’t know much about what I’ll be doing yet, but finding an open source company with a decent business model that is prepared to pay me is very exciting. The fact that they have offices that are close enough for me to go for is another huge bonus.

Wish me luck, and I’ll let you know what I’m working on when it becomes more clear.