Elm JSON decoder examples

Andy Balaam from Andy Balaam's Blog

I find JSON decoding in Elm confusing, so here are some thoughts and examples.

Setup

$ elm --version
0.19.0
$ mkdir myproj; cd myproj
$ elm init
...
$ elm install elm/json
...

To run the “Demo” parts of the examples below, type them into the interactive Elm interpreter. To try them out, start it like this:

$ elm repl

and import the library you need:

import Json.Decode as D

Scroll to “Concepts” at the bottom for lots of waffling about what is really going on, but if you’re looking to copy and paste concrete examples, here we are:

Examples

JSON object to Record

type alias MyRecord =
    { i : Int
    , s : String
    }

recordDecoder : D.Decoder MyRecord
recordDecoder =
    D.map2
        MyRecord
        (D.field "i" D.int)
        (D.field "s" D.string)

Demo:

> type alias MyRec = {i: Int, s: String}
> myRecDec = D.map2 MyRec (D.field "i" D.int) (D.field "s" D.string)
<internals> : D.Decoder MyRec
> D.decodeString myRecDec "{\"i\": 3, \"s\": \"bar\"}"
Ok { i = 3, s = "bar" }
    : Result D.Error MyRec

JSON array of ints to List

intArrayDecoder : D.Decoder (List Int)
intArrayDecoder =
    D.list D.int

Demo:

> myArrDec = D.list D.int
<internals> : D.Decoder (List Int)
> D.decodeString myArrDec "[3, 4]"
Ok [3,4] : Result D.Error (List Int)

JSON array of strings to List

stringArrayDecoder : D.Decoder (List String)
stringArrayDecoder =
    D.list D.string

Demo:

> myArrDec2 = D.list D.string
<internals> : D.Decoder (List String)
> D.decodeString myArrDec2 "[\"a\", \"b\"]"
Ok ["a","b"] : Result D.Error (List String)

JSON object to Dict

intDictDecoder : D.Decoder (Dict String Int)
intDictDecoder =
    D.dict D.int

Demo:

> myDictDecoder = D.dict D.int
<internals> : D.Decoder (Dict.Dict String Int)
> D.decodeString myDictDecoder "{\"a\": \"b\"}"
Err (Field "a" (Failure ("Expecting an INT") <internals>))
    : Result D.Error (Dict.Dict String Int)
> D.decodeString myDictDecoder "{\"a\": 3}"
Ok (Dict.fromList [("a",3)])
    : Result D.Error (Dict.Dict String Int)

To build a Dict of String to String, replace D.int above with
D.string.

JSON array of objects to List of Records

type alias MyRecord =
    { i : Int
    , s : String
    }

recordDecoder : D.Decoder MyRecord
recordDecoder =
    D.map2
        MyRecord
        (D.field "i" D.int)
        (D.field "s" D.string)


listOfRecordsDecoder : D.Decoder (List MyRecord)
listOfRecordsDecoder =
    D.list recordDecoder

Demo:

> import Json.Decode as D
> type alias MyRec = {i: Int, s: String}
> myRecDec = D.map2 MyRec (D.field "i" D.int) (D.field "s" D.string)
<internals> : D.Decoder MyRec
> listOfMyRecDec = D.list myRecDec
<internals> : D.Decoder (List MyRec)
> D.decodeString listOfMyRecDec "[{\"i\": 4, \"s\": \"one\"}, {\"i\": 5, \"s\":\"two\"}]"
Ok [{ i = 4, s = "one" },{ i = 5, s = "two" }]
    : Result D.Error (List MyRec)

Concepts

What is a Decoder?

A Decoder is something that describes how to take in JSON and spit out something. The “something” part is written after Decoder, so e.g. Decoder Int describes how to take in JSON and spit out an Int.

The Json.Decode module contains a function that is a Decoder Int. It’s called int:

> D.int
<internals> : D.Decoder Int

In some not-all-all-true way, a Decoder is sort of like a function:

-- This is a lie, but just pretend with me for a sec
Decoder a : SomeJSON -> a
-- That was a lie

To actually run your a Decoder, provide it to a function like decodeString:

> D.decodeString D.int "45"
Ok 45 : Result D.Error Int

So the actually-true way of getting an actual function is to combine decodeString and a decoder like int:

> D.decodeString D.int
<function> : String -> Result D.Error Int

When you apply decodeString to int you get a function that takes in a String and returns either an Int or an error. The error could be because the string you passed was not valid JSON:

> D.decodeString D.int "foo bar"
Err (Failure ("This is not valid JSON! Unexpected token o in JSON at position 1") )
    : Result D.Error Int

or because the parsed JSON does not match what the Decoder you supplied expects:

> D.decodeString D.int "\"45\""
Err (Failure ("Expecting an INT") )
    : Result D.Error Int

(We supplied a String containing a JSON string, but the int Decoder expects to find a JSON int.)

Side note: ints and floats are treated as different, even though the JSON Spec treats them all as just “Numbers”:

> D.decodeString D.int "45.2"
Err (Failure ("Expecting an INT") )
    : Result D.Error Int

What is a Value?

Elm has a type that represents JSON that has been parsed (actually, parsed and stored in a JavaScript object) but not interpreted into a useful Elm type. You can make one using the functions inside Json.Encode:

> import Json.Encode as E
> foo = E.string "foo"
 : E.Value

You can even turn one of these into a String containing JSON using encode:

> E.encode 0 foo
"\"foo\"" : String

or interpret the Value as useful Elm types using decodeValue:

> D.decodeValue D.string foo
Ok "foo" : Result D.Error String

(When JSON values come from JavaScript, e.g. via flags, they actually come as Values, but you don’t usually need to worry about that.)

However, what you can’t do is pull Values apart in any way, other than the standard ways Elm gives you. So any custom Decoder that you write has to be built out of existing Decoders.

How do I write my own Decoder?

If you want to make a Decoder that does custom things, build it from the existing magic Decoders, give it a type that describes the type it outputs, and insert your code using one of the mapN functions.

For example, to decode only ints that are below 100:

> under100 i = if i < 100 then D.succeed i else (D.fail "Not under 100")
<function> : number -> D.Decoder number
> intUnder100 = D.int > D.andThen under100
<internals> : D.Decoder Int
> D.decodeString intUnder100 "50"
Ok 50 : Result D.Error Int
> D.decodeString intUnder100 "500"
Err (Failure ("Not under 100") <internals>)
    : Result D.Error Int

Here, we use the andThen function to transform the Int value coming from calling the int function into a Decoder Int that expresses success or failure in terms of decoding. When we do actual decoding using the decodeString funcion, this is transformed into the more familiar Result values like Ok or Err.

If you want to understand the above, pay close attention to the types of under100 and intUnder100.

If you want to write a Decoder that returns some complex type, you should build it using the mapN functions.

For example, to decode strings into arrays of words split by spaces:

> splitIntoWords = String.split " "
<function> : String -> List String
> words = D.map splitIntoWords D.string
<internals> : D.Decoder (List String)
> D.decodeString words "\"foo bar baz\""
Ok ["foo","bar","baz"]
    : Result D.Error (List String)

Above we used map to transform a Decoder String (the provided string function) into a Decoder (List String) by mapping it over a function (splitIntoWords) that transforms a String into a List String.

Again, to understand this, look carefully at the types of splitIntoWords
and words.

How do I build up complex Decoders?

Complex decoders are built by combining simple ones. Many functions that make decoders take another decoder as an argument. A good example is “JSON array of objects to List of Records” above – there we make a Decoder MyRecord and use it to decode a whole list of records by passing it as an argument to list, so that it returns a Decoder (List MyRecord) which can take in a JSON array of JSON objects, and return a List of MyRecords.

Why is this so confusing?

Because Decoders are not functions, but they feel like functions. In fact they are opaque descriptions of how to interpret JSON that the Elm runtime uses to make Elm objects for you out of Values, which are opaque objects that underneath represent a piece of parsed JSON.

Graft Animation Language on Raspberry Pi

Andy Balaam from Andy Balaam&#039;s Blog

Because the Rapsberry Pi uses a slightly older Python version, there is a special version of Graft for it.

Here’s how to get it:

  • Open a terminal window by clicking the black icon with a “>” symbol on it at the top near the left.
  • First we need to install a couple of things Graft needs, so type this, then press Enter:
    sudo apt install python3-attr at-spi2-core
  • If you want to be able to make animated GIFs, install one more thing:
    sudo apt install imagemagick
  • To download Graft and switch to the Raspberry Pi version, type in these commands, pressing Enter after each line.
    git clone https://github.com/andybalaam/graft.git
    cd graft
    git checkout raspberry-pi
  • Now, you should be able to run Graft just like on another computer, for example, like this:
    ./graft 'd+=10 S()'
  • If you’re looking for a fun way to start, why not try the worksheet “Tell a story by making animations with code”?

    For more info, see Graft Raspberry Pi Setup.

My experience upgrading to Elm 0.19

Andy Balaam from Andy Balaam&#039;s Blog

Elm is unstable, so upgrading to the next version can be painful. Here’s what I needed to do to upgrade from 0.18 to 0.19.

  • Replace elm-package.json and tests/elm-package.json with elm.json – e06f5a1728
  • Switch to the new elm-testb964b7c7a
  • Re-arrange Main, and how we call it from JavaScript – 0c118c49f
  • Stop using eeue56/elm-all-dict (since it’s not ported to 0.19 and porting it looked hard due to a lack of Debug.crash) – fe100f256
  • Replace toString with String.fromX or Debug.toString – 9e78163d0a3
  • Stop “shadowing” names by making new variables with the same name as another in the scope – 9688a621de
  • Adapt to the changed Html.style function – b991ab4f
  • Stop using Debug.crash – f98a70ad1
  • Adapt to the changes in the Regex module – 856762a4
  • Stop using tuples with more than 3 parts – 472c0bb7

The lack of Debug.crash is really, really painful, especially for a library like eeue56/elm-all-dict that has lots of invariants that are hard or impossible to enforce via the type system. On the other hand, if Elm can give a hard guarantee that there will be no runtime errors, this seems pretty cool. The problem is that some code may well have to return the wrong answer silently, instead of crashing, which could be much worse than crashing in some use-cases.

I was annoyed by the lack of more-than-3-part tuples, but even as I did the work to change my code, I saw it get better, so it’s hard to argue with.

The hardest part to work out was how to run the tests. Fortunately the tests themselves needed almost no changes. I just needed to do this:

rm -r tests/elm-stuff
rm tests/elm-package.json
sudo npm install -g elm-test@0.19.0-beta8
elm-test install
elm-test

My next job is to check out the –optimize compiler flag, and the advice on making the code smaller and faster.

Bulk adding items to Wunderlist using wunderline on Ubuntu MATE

Andy Balaam from Andy Balaam&#039;s Blog

If you use Wunderlist and want to be able to bulk-add tasks from a text file, first install and set up wunderline.

Now, to be able to right-click a text file containing one task per line on Ubuntu MATE, create a file called “wunderlist-bulk-add” in ~/.config/caja/scripts/ and make it executable. Paste the code below into that file.

(Note: it’s possible that this would work in GNOME if you replaced “caja” with “nautius” in that path – let me know in the comments.)

#!/usr/bin/env bash

set -e
set -u
set -o pipefail

function nonblank()
{
    grep -v -e '^[[:space:]]*$' "$@"
}

for F in "$@"; do
{
    COUNT=$(nonblank "$F" | wc -l | awk '{print $1}')
    if [ "$COUNT" = "" ]; then
    {
        zenity --info --no-wrap --text="File $F does not exist"
    }
    else
    {
        set +e
        zenity --question --no-wrap \
            --text="Add $COUNT items from $F to Wunderlist?"
        ANSWER=$?
        set -e

        if [ "$ANSWER" = "0" ]; then
        {
            set +e
            nonblank "$F" | \
                wunderline add --stdin | \
                zenity --progress --text "Adding $COUNT items to Wunderlist"
            SUCCEEDED=$?
            set -e
            if [ "$SUCCEEDED" = "0" ]; then
            {
                zenity --info --no-wrap --text "Added $COUNT items."
            }
            else
            {
                zenity --error --no-wrap \
                    --text "Failed to add some or all of these items."
            }; fi
        }; fi
    }; fi
}; done

Writing a new Flarum extension on Ubuntu

Andy Balaam from Andy Balaam&#039;s Blog

In a previous post I described how to install Flarum locally on Ubuntu.

Here is how I set up my development environment on top of that setup so I was able to write a new Flarum extension and test it on my local machine.

Recap: I installed Apache and PHP and enabled mod_rewrite, installed MariaDB and made a database, installed Composer, and used Composer to install Flarum at /var/www/html/flarum.

More info: Flarum Contribution Guide, extension development guide, extension development quick start, workflow discussion.

I decided to call my extension “rabbitescape/flarum-ext-rabbitescape-leveleditor”, so that is the name you will see below.

Here’s what I did:

cd /var/www/html/flarum

Edit /var/www/html/flarum/composer.json (see the workbench explanation for examples of the full file).

At the end of the “require” section I added a line like this:

“rabbitescape/flarum-ext-rabbitescape-leveleditor”: “*@dev”

(Remembering to add a comma at the end of the previous line so it remained valid JSON.)

After the “require” section, I added this:

    "repositories": [
        {
            "type": "path",
            "url": "workbench/*/"
        }
    ],

Next, I made the place where my extension would live:

mkdir workbench
cd workbench
mkdir flarum-ext-rabbitescape-leveleditor
cd flarum-ext-rabbitescape-leveleditor

Now I created a file inside flarum-ext-rabbitescape-leveleditor called composer.json like this:

{
    "name": "rabbitescape/flarum-ext-rabbitescape-leveleditor",
    "description": "Allow viewing and editing Rabbit Escape levels in posts",
    "type": "flarum-extension",
    "keywords": ["rabbitescape"],
    "license": "MIT",
    "authors": [
        {
            "name": "Andy Balaam",
            "email": "andybalaam@artificialworlds.net"
        }
    ],
    "support": {
        "issues": "https://github.com/andybalaam/flarum-ext-rabbitescape-leveleditor/issues",
        "source": "https://github.com/andybalaam/flarum-ext-rabbitescape-leveleditor"
    },
    "require": {
        "flarum/core": "^0.1.0-beta.5"
    },
    "extra": {
        "flarum-extension": {
            "title": "Rabbit Escape Level Editor"
        }
    }
}

In the same directory I created a file bootstrap.php like this:

<?php

return function () {
    echo 'Hello, world!';
};

Then I told Composer to refresh the project like this:

cd ../..   # Back up to the main directory
composer update

Among the output I saw a line like this which convinced me it had worked:

  - Installing rabbitescape/flarum-ext-rabbitescape-leveleditor (dev-master): Symlinking from workbench/flarum-ext-rabbitescape-leveleditor

Now I went to http://localhost/flarum, signed in as admin, adminpassword (as I set up in the previous post), clicked my username in the top right and chose Administration, then clicked Extensions on the left.

I saw my extension in the list, and turned it on by clicking the check box below it. This made the whole web site disappear, and instead just the text “Hello world!” appeared. This showed my extension was being loaded.

Finally, I edited bootstrap.php and commented out the line starting with “echo”. I refreshed the page in my browser and saw the Flarum site reappear.

Now, my extension is installed and ready to be developed! See flarum.org/docs/extend/start for how to get started making it do cool stuff.

Ubuntu “compose” key for easy unicode character input

Andy Balaam from Andy Balaam&#039;s Blog

I found out on Mastodon recently that the Compose key exists, allowing you to enter special characters using easy-to-remember key sequences (e.g. “<compose>/=” gives you “≠”).

To do this on Ubuntu (and probably many other systems), hold SHIFT, then press AltGr, release them both, and type something like “/=”. You should see the ≠ symbol appear.

In Ubuntu MATE, you can customise what key is the compose key by opening “Keyboard”, clicking “Layouts”, then “Options…” and checking a box under “Position of compose key”. I checked “Right Alt”, meaning my “AltGr” key is now the compose key.

More info: Ubuntu ComposeKey docs.

Want to explore? See the list of the default sequences you can type.

Redirecting all requests to https and www using .htaccess in Apache

Andy Balaam from Andy Balaam&#039;s Blog

I want all requests to artificialworlds.net/rabbit-escape/levels/ to get redirected to use the https protocol, and to include “www.” at the beginning of the URL, and I found lots of Stack Overflow articles, but nothing that worked perfectly for me. Here is how I managed it.

I edited the .htaccess file in the directory where I want this to apply, and added this at the top:

<IfModule mod_rewrite.c>
  RewriteEngine On
  RewriteCond %{HTTP_HOST} ^artificialworlds\.net$ [NC]
  RewriteRule ^ https://www.artificialworlds.net%{REQUEST_URI} [R=301,L]

  RewriteCond %{HTTPS} off
  RewriteRule ^ https://www.artificialworlds.net%{REQUEST_URI} [R=301,L]
</IfModule>

Example of a systemd service file

Andy Balaam from Andy Balaam&#039;s Blog

Here is an almost-minimal example of a systemd service file, that I use to run the Mastodon bot of my generative art playground Graft.

I made a dedicated user just to run this service, and installed Graft into /home/graft/apps/graft under that username. Now, as root, I edited a file called /etc/systemd/service/graft.service and made it look like this:

[Service]
ExecStart=/home/graft/apps/graft/bot-mastodon
User=graft
Group=graft
[Install]
WantedBy=multi-user.target

Now I can start the graft service like any other service like this:

sudo systemctl start graft

and find out its status with:

sudo systemctl status graft

If I want it to run on startup I can do:

sudo systemctl enable graft

and it will. Easy!

If I want to look at its output, it’s:

sudo journalctl -u graft

As a reward for reading this far, here’s a little animation you can make with Graft:

Installing Flarum on Ubuntu 18.04

Andy Balaam from Andy Balaam&#039;s Blog

I am setting up a forum for sharing levels for my game Rabbit Escape, and I have decided to try and use Flarum, because it looks really usable and responsive, has features we need like liking posts and following authors, and I think it will be reasonably OK to write the custom features we want.

So, I want a dev environment on my local Ubuntu 18.04 machine, and the first step to that is a standard install.

Warning: at the time of writing the Flarum docs say it does not work with PHP 7.2, which is what is included with Ubuntu 18.04, so this may not work. (So far it looks OK for me.)

Here’s how I got it working (as far as the web installer stage, anyway):

sudo apt install \
    apache2 \
    libapache2-mod-php \
    mariadb-server \
    php-mysql \
    php-json \
    php-gd \
    php-tokenizer \
    php-mbstring \
    php-curl

php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"

# Get the neat line from https://getcomposer.org/download/
# Don't copy it exactly!
php -r "if (hash_file('SHA384', 'composer-setup.php') === '544e09ee996cdf60ece3804abc52599c22b1f40f4323403c44d44fdfdd586475ca9813a858088ffbc1f233e9b180f061') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;"

mkdir ~/bin
php composer-setup.php --install-dir=~/bin/ --filename=composer
rm composer-setup.php

cd /var/www/html
sudo mkdir flarum
sudo chown $(whoami) flarum

# Log out and in again here to get composer to be in your PATH
cd flarum
composer create-project flarum/flarum . --stability=beta

sudo chgrp -R www-data .
sudo chmod -R 775 .

sudo systemctl restart apache2

Go to http://localhost/flarum in your browser, and follow the instructions there to get set up.

If I get further, I will update this post, including on how to set up the MySQL database.

If you want to find and share levels for Rabbit Escape, check up on our progress setting up the forum at https://artificialworlds.net/rabbit-escape/levels.