Posted on

Migrating videos from YouTube to PeerTube inside a Docker container

Andy Balaam from Andy Balaam's Blog

I have quite a few videos hosted on YouTube that I would like to upload to my new PeerTube location, but I don’t want to install all the PeerTube dependencies on my machine, so I did it all inside a Docker image.

First I built and started a Docker container:

$ git clone https://github.com/chocobozzz/PeerTube /tmp/peertube
$ cd /tmp/peertube
$ docker build . -f ./support/docker/production/Dockerfile.stretch --tag peertube
$ docker run --tty --interactive peertube bash

Then I ran these commands inside it:

# yarn install --production=false
# node dist/server/tools/import-videos.js -u "https://peertube.mastodon.host" -U "andybalaam" -t "https://www.youtube.com/watch?v=TG0qRDrUPpA"

Of course, it would be better to write this up into its own Dockerfile to make this a one-liner.

References: PeerTube Docker setup, PeerTube video import.

Posted on

How to write a programming language articles

Andy Balaam from Andy Balaam's Blog

Recent Overload journal issues contain my new articles on How to Write a Programming Language.

Part 1: How to Write a Programming Language: Part 1, The Lexer

Part 2: How to Write a Programming Language: Part 2, The Parser

PDF of the latest issue: Overload 146 containing part 2.

This is all creative-commons licensed and developed in public at github.com/andybalaam/articles-how-to-write-a-programming-language

Posted on

Allow drag-to-side, but not drag-to-top in Ubuntu MATE (Marco)

Andy Balaam from Andy Balaam's Blog

I love the “tiling” feature in many window managers including Marco that means I can drag windows to the side of the screen and get them covering one half.

However, I never use the similar feature that allows dragging a window to the top, and it often triggers when I just want to move a window upwards.

Today I discovered that Marco (the non-fancy window manager in Ubuntu MATE and probably other places) does allow me to have one without the other, even though the UI configuration tools don’t expose the option.

Here’s what I did:

gsettings set org.mate.Marco.general allow-top-tiling false

Now my windows can be dragged to the side for half-screen maximisation, but not to the top for full-screen!

Posted on

Connecting to Slack from an IRC client using slirc

Andy Balaam from Andy Balaam's Blog

I tried to get back to an IRC interface to Slack using Matrix, and it had some problems. Thanks to Colin Watson’s comment on that post, I tried Daniel Beer’s slirc, and so far it seems to be working pretty well.

Here’s what I did:

Get a Slack legacy token which slirc will use to connect to Slack as you. Follow the instructions given at that link, and you should end up with a token that looks something like “abcd-123456768-ETC-ETC”. Keep a note of it.

Install the prerequisites for slirc, and download it:

sudo cpan AnyEvent AnyEvent::HTTP AnyEvent::Socket AnyEvent::WebSocket::Client URI::Encode Data::Dumper JSON
mkdir slirc
cd slirc
wget -q 'https://www.dlbeer.co.nz/articles/slirc/slirc-20180515.pl'
chmod +x slirc-20180515.pl

Create a file in the slirc directory you created above, called rc.conf, and make it look like this:

slack_token=abcd-123456768-ETC-ETC
password=somepassword
port=6667

Replace “abcd-123456768-ETC-ETC” with the Slack legacy token you noted down earlier.

Replace “somepassword” with something you’ve made up (not your Slack password) – this is what you will type as the password in your IRC client.

Run slirc and leave it running:

./slirc-20180515.pl rc.conf

(Make sure you are inside the slirc dir when you run that.)

Start your IRC client (e.g. HexChat) and add a server with address “localhost” and port 6667, with your slack username and the password you added in the rc.conf (which you wrote instead of “somepassword”).

This mostly works for me, except it has a tendency to open a load of ad-hoc chats as channels, so I have to close them all to get a usable list.

Posted on

Using Matrix to connect to Slack from an IRC client on Ubuntu

Andy Balaam from Andy Balaam's Blog

I like using HexChat to talk to my colleagues, like one other guy. It is fast, and it pops up a new window when someone sends me a direct/private message.

Recently, Slack shut down their IRC gateway, forcing me to use their slow UI, and making me unable to be responsive to direct messages without spending my entire life checking said UI, or being disturbed by notifications.

So, I decided to set up a Matrix bridge, because how hard can it be?

My system is Ubuntu MATE 18.04.

What I did

Install Synapse

Install Synapse, a Matrix homeserver.

wget -qO - https://matrix.org/packages/debian/repo-key.asc | sudo apt-key add -
sudo apt-add-repository http://matrix.org/packages/debian/
sudo apt update
sudo apt install matrix-synapse

Edit /etc/matrix-synapse/homeserver.yaml to change the line containing “registration_shared_secret” to look something like:

registration_shared_secret: "some secret password you made up"

(Replacing “some secret password you made up” with something secret.)

Make a new user:

register_new_matrix_user -c /etc/matrix-synapse/homeserver.yaml http://localhost:8008

Enter a username and password, and take a note of them.

Install Riot

(Technically, this step is optional, but I definitely recommend it to check everything is working.)

Install Riot, a client that can talk to a homeserver.

curl https://riot.im/packages/debian/repo-key.asc | sudo apt-key add -
sudo apt-add-repository https://riot.im/packages/debian/
sudo apt update
sudo apt install riot-web

Start Riot from the menu or by typing riot-web in a console.

Choose to log in to a Custom Server (not create a new user), and enter the Home Server URL as http://localhost:8008, leave the Identity Server URL as it is, and enter the username and password you entered in the previous step.

Riot should be able to connect to the Synapse server, even though it won’t have any rooms in it yet.

If this doesn’t work, try running riot-web in a console, and check the Synapse log file at /var/log/matrix-synapse/homeserver.log

Install matrix-puppet-slack

(Note: after this step, the homeserver can only be for you now, because it is logged into Slack as you. If you want to make a new Matrix server for lots of people, I found that matrix-appservice-slack works, but this only connects channels, not direct messages.)

Get a Slack legacy token which matrix-puppet-slack will use to connect to Slack as you. Follow the instructions given at that link, and you should end up with a token that looks something like “abcd-123456768-ETC-ETC”. Keep a note of it.

Install matrix-puppet-slack which allows you to connect a homeserver to a Slack instance, pretending it is you:

cd
git clone https://github.com/matrix-hacks/matrix-puppet-slack.git
cd matrix-puppet-slack/
npm install
cp config.sample.json config.json

Edit config.json so it looks something like this:

{
  "slack": [{
    "team_name": "Name of my team",
    "user_access_token": "abcd-123456768-ETC-ETC"
  }],
  "registrationPath": "slack-registration.yaml",
  "port": 8090,
  "bridge": {
    "homeserverUrl":"http://localhost:8008",
    "domain": "localhost",
    "registration": "slack-registration.yaml"
  }
}

Now generate a config file that Synapse will use, called slack-registration.yaml, by typing this command (still in the matrix-puppet-slack directory):

node index.js -r -u "http://localhost:8090"

When asked, type in the username and password you entered in the first step (when you ran the register_new_matrix_user command).

This creates a config file telling Synapse that matrix-puppet-slack is running on the local machine on port 8090.

Now, run it:

node index.js

and edit Synapse’s config file /etc/matrix-synapse/homeserver.yaml to point at the new config file you just made, by editing the line mentioning app_service_config_files so that it looks like this:

app_service_config_files: [
    "/path/to/matrix-puppet-slack/slack-registration.yaml"
]

Make sure you replace “/path/to/matrix-puppet-slack” in the above with the directory where you put matrix-puppet-slack, which is /home/yourusername/matrix-puppet-slack if you followed the instructions exactly.

Restart Synapse, and when you go back into Riot, you should see your Slack channels appear gradually (as people talk in them):

sudo service matrix-synapse restart

If not, check the output from the node index.js command, and the Synapse log file at /var/log/matrix-synapse/homeserver.log.

Install matrix-ircd

To allow connecting to Matrix from an IRC client, install matrix-ircd:

cd
git clone https://github.com/matrix-org/matrix-ircd.git
cd matrix-ircd
cargo build
cargo run -- --url "http://localhost:8008"

(Note this works on Ubuntu 18.04, but operating systems with older versions of Rust may need to get the latest first – see the matrix-ircd page for details.)

Connect from an IRC client

Now use an IRC client to connect to matrix-ird. I recommend HexChat if you’re looking for one.

matrix-ircd runs on port 5999 by default, so you should be able to connect to it from your IRC client by setting the server to localhost/5999 and using the username and password from the first step (when you ran the register_new_matrix_user command). Use plain username/password authentication.

If it works, you should see your Slack channels appear in your IRC client. If not check the logs mentioned before, and the error messages and logs from your IRC client.

What works and doesn’t work

  • Messages typed by me and others into IRC, Riot and Slack appear in all the other places, including direct messages.
  • Synapse and Riot are installed fully, so Synapse will be running after a reboot, and Riot is available in the menus, but matrix-puppet-slack and matrix-ircd are just running in a console, and have to be manually started. It should be reasonably simple to make custom systemd files to start them automatically.
  • matrix-puppet-slack crashed once and I had to restart it, so this may not be very reliable.
  • The Slack channel names look terrible in my IRC client. I think matrix-ircd needs to find the pretty names (which are displayed correctly in Riot) and use them instead of the coded names used under the cover in Matrix.
  • My first ever direct message to someone does not work, even though I can choose them and attempt to send a message. As soon as they say something to me, they appear as a channel in Riot and IRC, and I can talk back and forth no problem from then on.
  • Direct messages look like normal channels in Matrix, which means I can’t use HexChat to pop up notifications for them, so this was all pointless.

Please leave comments below if you know how I can fix any of these problems.

Conclusions

  • Matrix seems really cool
  • It is basically possible to bridge IRC-Matrix-Slack as I wanted
  • But, some remaining bugs mean I have to keep using Slack’s UI for now :-(
  • Please comment telling me how to do this better
Posted on

Examples of SQL join types (LEFT JOIN, INNER JOIN etc.)

Andy Balaam from Andy Balaam's Blog

I have 2 tables like this:

> SELECT * FROM table_a;
+------+------+
| id   | name |
+------+------+
|    1 | row1 |
|    2 | row2 |
+------+------+

> SELECT * FROM table_b;
+------+------+------+
| id   | name | aid  |
+------+------+------+
|    3 | row3 |    1 |
|    4 | row4 |    1 |
|    5 | row5 | NULL |
+------+------+------+

INNER JOIN cares about both tables

INNER JOIN cares about both tables, so you only get a row if both tables have one. If there is more than one matching pair, you get multiple rows.

> SELECT * FROM table_a a INNER JOIN table_b b ON a.id=b.aid;
+------+------+------+------+------+
| id   | name | id   | name | aid  |
+------+------+------+------+------+
|    1 | row1 |    3 | row3 | 1    |
|    1 | row1 |    4 | row4 | 1    |
+------+------+------+------+------+

It makes no difference to INNER JOIN if you reverse the order, because it cares about both tables:

> SELECT * FROM table_b b INNER JOIN table_a a ON a.id=b.aid;
+------+------+------+------+------+
| id   | name | aid  | id   | name |
+------+------+------+------+------+
|    3 | row3 | 1    |    1 | row1 |
|    4 | row4 | 1    |    1 | row1 |
+------+------+------+------+------+

You get the same rows, but the columns are in a different order because we mentioned the tables in a different order.

LEFT JOIN only cares about the first table

LEFT JOIN cares about the first table you give it, and doesn’t care much about the second, so you always get the rows from the first table, even if there is no corresponding row in the second:

> SELECT * FROM table_a a LEFT JOIN table_b b ON a.id=b.aid;
+------+------+------+------+------+
| id   | name | id   | name | aid  |
+------+------+------+------+------+
|    1 | row1 |    3 | row3 | 1    |
|    1 | row1 |    4 | row4 | 1    |
|    2 | row2 | NULL | NULL | NULL |
+------+------+------+------+------+

Above you can see all rows of table_a even though some of them do not match with anything in table b, but not all rows of table_b – only ones that match something in table_a.

If we reverse the order of the tables, LEFT JOIN behaves differently:

> SELECT * FROM table_b b LEFT JOIN table_a a ON a.id=b.aid;
+------+------+------+------+------+
| id   | name | aid  | id   | name |
+------+------+------+------+------+
|    3 | row3 | 1    |    1 | row1 |
|    4 | row4 | 1    |    1 | row1 |
|    5 | row5 | NULL | NULL | NULL |
+------+------+------+------+------+

Now we get all rows of table_b, but only matching rows of table_a.

RIGHT JOIN only cares about the second table

a RIGHT JOIN b gets you exactly the same rows as b LEFT JOIN a. The only difference is the default order of the columns.

> SELECT * FROM table_a a RIGHT JOIN table_b b ON a.id=b.aid;
+------+------+------+------+------+
| id   | name | id   | name | aid  |
+------+------+------+------+------+
|    1 | row1 |    3 | row3 | 1    |
|    1 | row1 |    4 | row4 | 1    |
| NULL | NULL |    5 | row5 | NULL |
+------+------+------+------+------+

This is the same rows as table_b LEFT JOIN table_a, which we saw in the LEFT JOIN section.

Similarly:

> SELECT * FROM table_b b RIGHT JOIN table_a a ON a.id=b.aid;
+------+------+------+------+------+
| id   | name | aid  | id   | name |
+------+------+------+------+------+
|    3 | row3 | 1    |    1 | row1 |
|    4 | row4 | 1    |    1 | row1 |
| NULL | NULL | NULL |    2 | row2 |
+------+------+------+------+------+

Is the same rows as table_a LEFT JOIN table_b.

No join at all gives you copies of everything

If you write your tables with no JOIN clause at all, just separated by commas, you get every row of the first table written next to every row of the second table, in every possible combination:

> SELECT * FROM table_b b, table_a;
+------+------+------+------+------+
| id   | name | aid  | id   | name |
+------+------+------+------+------+
|    3 | row3 | 1    |    1 | row1 |
|    3 | row3 | 1    |    2 | row2 |
|    4 | row4 | 1    |    1 | row1 |
|    4 | row4 | 1    |    2 | row2 |
|    5 | row5 | NULL |    1 | row1 |
|    5 | row5 | NULL |    2 | row2 |
+------+------+------+------+------+
Posted on

Fixing Slack emojis in HexChat

Andy Balaam from Andy Balaam's Blog

If, like me, you are this person:

(Source: xkcd.com/1782)

You may want to fix the stupid :slightly_smiling_face: messages you receive from Slack via the IRC gateway. Obviously, I’d prefer they went away entirely, but it’s still better to see a character than being spammed with colon abominations all over the place.

You’ll need the Python emoji package, and a HexChat plugin like this:

# Replace all the horrible :slightly_smiling_face: rubbish that Slack inserts
# into horrible Unicode emoji symbols.
# Author: Andy Balaam
# License: CC0 https://creativecommons.org/publicdomain/zero/1.0/

# Requires https://pypi.python.org/pypi/emoji - I used 0.4.5
# I manually copied the emoji dir into:
# /home/andrebal/.local/lib/python2.7/site-packages
import emoji
import hexchat

__module_name__ = "slack-emojis"
__module_version__ = "1.0"
__module_description__ = "Translate emojis from Slack with colons into emojis"

print "Loading slack-emojis"
chmsg = "Channel Message"
prmsg = "Private Message to Dialog"

def preprint(words, word_eol, userdata):
    txt = word_eol[1]
    replaced = emoji.emojize(txt, use_aliases=True)
    if replaced != txt:
        hexchat.emit_print(
            userdata["msgtype"],
            words[0],
            replaced.encode('utf-8'),
        )
        return hexchat.EAT_HEXCHAT
    else:
        return hexchat.EAT_NONE

hexchat.hook_print(chmsg, preprint, {"msgtype": chmsg})
hexchat.hook_print(prmsg, preprint, {"msgtype": prmsg})

According to the page linked above, Slack are retiring the IRC gateway, which will make me very unhappy.

Update: added support for private messages too.

Posted on

Ideas on how lexing will work in Pepper3

Andy Balaam from Andy Balaam's Blog

I am trying to practice documentation-driven development in Pepper3, so every time I start on an area, I will write documentation explaining how it works, and include examples that are automatically verified during the build.

I’ve started work on lexing, since you can’t do much before you do that, but in fact, of course, I need to have a command line interface before I can verify any of the examples, so I’m working on that too.

Lexing is the process that takes a stream of characters (e.g. from a file) and turns it into a stream of “tokens” that are chunks of code like a variable name, a number or a string. (There is more on lexing in my mini programming language, Cell.)

My thoughts so far about lexing are in lexing.md, and current ideas about command line interface are at command_line.md. All very much subject to change.

Headlines:

  • Ordinary programmers can write their own lexing rules.
  • Operators (functions like “+” that find their arguments on their left and right, instead of between brackets like normal functions) are defined at the lexing phase, so any symbol (e.g. “in”) can be an operator if you want.
  • Anything you might want to do with a pepper program, including running it, compiling it, packaging it for an distribution system, should be available as a sub-command of the main pepper3 command line.
  • The command is “pepper3”, never “pepper”. If a new, incompatible version comes out, it will be called “pepper4”, and they will be parallel-installable, with no confusion.
Posted on

Deleting commits from the git history

Andy Balaam from Andy Balaam's Blog

Today I wanted to fix a Git repo that contained some bad commits (i.e. git fsck complained about them). [I wanted to do this because GitLab was not allowing me to push the bad commits.]

I wanted the code to look exactly as it did before, but the history to look different, so the bad commits disappeared, and (presumably) the work done in the bad commits to look like it was done in the commits following them.

Here’s what I ran:

git filter-branch -f --commit-filter '

    if [ "${GIT_COMMIT}" = "abdcef012345abcdef012345etcetcetc" ];
    then
        echo "Skipping GIT_COMMIT=${GIT_COMMIT}" >&2;
        skip_commit "$@";
    else
        git commit-tree "$@";
    fi
' --tag-name-filter cat -- --all

(Where abdcef012345abcdef012345etcetcetc was the ID of the commit I wanted to delete.)

Of course, you can make this cleverer to exclude multiple commits at a time, or run this several times, putting in the right commit ID each time.