PG Webhooks Tool (beta) – I want your feedback!

Paul Grenyer from Paul Grenyer


PG Webhooks Tool

https://pg-webhooks.herokuapp.com/

PG Webhooks is a tool designed to help with the testing of webhooks. It allows you to receive and view messages from a webhook until you're ready to setup the server which will receive the real webhooks for your application.

Register the URL uniquely generated for you by PG Webhooks with your webhook. Fire the webhook and see the messages appear. Select each message to see the details, including the body and header.

PG Webhooks sits on a free Heroku instance so may take a few moments to start the first time you use it. It is backed by a non-persistent Redis database, so your messages are only stored temporarily.

PG Webhooks beta is very much a prototype and has plenty of rough edges. I would appreciate any feedback in the comments below. 



New Profile: Written for a talk on ‘Entrepreneurship’

Paul Grenyer from Paul Grenyer

Husband, father, software engineer, metaller, Paul has been writing software for over 35 years and professionally for more than 20. In that time he has worked for and in all sorts of companies from two man startups to world famous investment banks and insurance companies. He has built and run three limited companies, none of which made him a millionaire and two of which threatened his sanity on more than one occasion.

Paul was a founding member of both SyncNorwich and Norfolk Developers, two of the most successful tech and startup based community groups in the East. He created and chaired the hugely successful Norfolk Developers Conference (nor(DEV):con) for seven years bringing in speakers and delegates in the sphere of software engineering from around the globe.

Paul is currently a Senior Platform Engineer at the Fintech startup Snoop and the founder of the tea finding app, Find My Tea. He loathes the word Entrepreneur, not least because he struggles to spell it and it reminds him of Del Boy from the 80s sitcom Only Fools and Horses. He sees Entrepreneurship as a side effect of the creative process of problem solving, rather than a career path in its own right.

Despite having dealt with the world of business from directors of the board down, Paul has kept both feet firmly on the ground even when his head has been in the clouds with healthy doses of Heavy Metal, Science Fiction and Formula One and long hair until it started falling out in 2013.

Oh, and he loves good tea too!

A Review of Absolution Gap: Ignore the naysayers. Read this book. Love this book.

Paul Grenyer from Paul Grenyer

Alastair Reynolds ISBN-13 : 978-0575083165

I re-read Absolution Gap a decade or more after the first time in anticipation of the next part coming out in July of this year (2021). It was always the weakest of the trilogy, but not nearly as bad as I remember. In fact this time I devoured it in a relatively, for me, short period of time.

It’s true, as some other reviewers have said, that the story here could have been told in far fewer words, but then much of the texture of the story telling would have been lost and I think this is what makes this such a great book!


The characters and themes are believable in this universe. I think the story could have been very different if certain characters had not been killed off so early or at all. It’s always a shame when a lot of the main thrust of a previous book (Redemption Ark) is undone, but this is often how things play out in the real world.


I achieved what I wanted by re-reading Absolution Gap, I’m up to speed ready for the next instalment. The problem is that there is so much in the Revelation Space universe I now feel the need to go back and read it all again.


Ignore the naysayers. Read this book. Love this book.


Red Rebel Day: Did you know that 1 in 10 women can’t afford sanitary protection?

Paul Grenyer from Paul Grenyer

When Neelam Sultan asked me to support RedRebelDay I was keen to do what I could. It’s what you do when someone asks. It’s what I like to do when someone asks.

Then I attended the ‘ambassadors’ briefing, where a heavy emphasis was put on telling people what Period Poverty means to you. As a man this was quite daunting, especially as until recently the subject was still quite taboo for me. Especially as at least one of my former employees took great delight, and still does, in telling me when she had her period just to see the uncomfortable look on my face. What got me over it? Attending a Zoom call with 100+ women and a handful of men who were all talking very openly about periods and the problems 1 in 10 women face in affording sanitary protection.

Beyond the facts, that still didn’t help me understand what Period Poverty really means, let alone what it means to me. Then someone on the call described not being able to go to work or go to school or about your everyday life for a few days every month because you couldn’t afford something as simple as a sanitary product. This struck home for me, because I take being able to do these things for granted and I know how it feels when something beyond my control stops me from doing something I want to do. This is what Period Poverty means to me.

Will you help women live their lives in the way they deserve by donating to Red Rebel Day today? I have: 


Tea

A review: Permafrost

Paul Grenyer from Paul Grenyer

 

Permafrost

by Alastair Reynolds

ASIN : B07HF26D1H

Alistair Reynolds continues his run of form with this fantastic novella, Permafrost. One of the things I love most about Reynold’s Revelation Space series is how the stories flip between different times. For me this is one of the biggest and best parts of weaving a space opera, and it’s in abundance in Permafrost. I’m still in two minds about whether I enjoy first person writing though, I’m definitely more of a fan of third person. However, first person worked well here. Novellas, by their very nature are short, which means the character building is fast and the stories are fast paced too. I really enjoyed that aspect of Permafrost.

Given that the story is based on the effects of climate change, as well as time travel, I was concerned it would be over the top, but it actually describes brilliantly and concisely the effects of climate change and how they form the bedrock for the story. Then Reynolds moves on and concentrates on the story.

The climax is a page turner (or button clicker if you read on a kindle like I do) and moves very quickly - which I loved. Without wishing to give too much away, I did feel that defeating the evil (or maybe not so evil) machines was a bit too easy, but this didn’t detract from the climax.

If you’ve never read Alistair Reynolds before, then this would be an excellent place to start!

There’s a new Revelation Space novel out later this year and in preparation I’m rereading Absolution Gap. I’ve already read the first two paragraphs, one of my most memorable scenes from Revelation Space, and I am very, very, excited!

Just when you thought it wasn’t safe to search for tea…

Paul Grenyer from Paul Grenyer

There hasn’t been anything from Find My Tea HQ for a while. We’ve been working hard behind the scenes improving the existing infrastructure and fixing bugs in the app, not least of which, the lack of functioning maps on iPhone. The good news is the maps are back and there’s a new version of the app to download for iPhone users!

Early next year we hope to release a feature which will let users add new locations for other users to search for. Stick with us, we really appreciate your support so far.

Please connect with us on social media in the meantime:

Website: http://findmytea.co.uk/ (including links to the apps)

Facebook: https://www.facebook.com/Findmytea-113424607013669

Linkedin: https://www.linkedin.com/company/findmytea

Twitter: https://twitter.com/FindMyTea


Review: Bone Silence Delivers

Paul Grenyer from Paul Grenyer

Bone Silence (Revenger, #3)Bone Silence by Alastair Reynolds
My rating: 5 of 5 stars

Alistair Reynolds is still by far my favorite author and he has continued his form, which rebooted with Revenger, in Bone Silence. A friend once said to me that Alistair Reynolds struggles to know how to write an ending and, disappointingly, I think this is still true with Bone Silence. This trilogy has the constant unanswered questions which drew me into and kept me hooked on the Revelation Space stories. There is a kind of answer at the end and a bit of a twist which, if I’m honest, left me underwhelmed. The answer lacks detail and explanation of the reasons and then the Ness sisters ride off into the sunset and Reynolds announces that he’s done with the pair.

That aside I loved Bone Silence and the entire trilogy. Each book is different and describes different aspects of the universe in which it is set. The characters are diverse and interesting and the story wide, far reaching and mostly unpredictable. The sisters' respect for each other is clear throughout, but like real siblings there are tensions and fights. There is a lot more to this universe and so much scope for expansion. I’d love Alistair Reynolds to return to it some day and expand this story, fill in some glossed over gaps and detail and tell some of the other stories, even if the cliched seafaring language did irritate me throughout.


View all my reviews

Piping Software for Less: Why, What & How (Part 1)

Paul Grenyer from Paul Grenyer

Developing software is hard and all good developers are lazy. This is one of the reasons we have tools which automate practices like continuous integration, static analysis and measuring test coverage. The practices help us to measure quality and find problems with code early. When you measure something you can make it better. Automation makes it easy to perform the practices and means that lazy developers are likely to perform them more often, especially if they’re automatically performed every time the developer checks code in.

This is old news. These practices have been around for more than twenty years. They have become industry standards and not using them is, quite rightly, frowned upon. What is relatively new is the introduction of cloud based services such as BitBucket Pipelines, CircleCI and SonarCloud which allow you to set up these practices in minutes, however with this flexibility and efficiency comes a cost.

Why

While BitBucket Pipelines, CircleCI and SonarCloud have free tiers there are limits.

With BitBucket Pipelines you only get 50 build minutes a month on the free tier. The next step up is $15/month and then you get 2500 build minutes.

On the free CircleCI tier you get 2500 free credits per week, but you can only use public repositories, which means anyone and everyone can see your code. The use of private repositories starts at $15 per month.

With SonarCloud you can analyse as many lines of code as you like, but again you have to have your code in a public repository or pay $10 per month for the first 100,000 lines of code.

If you want continuous integration and a static analysis repository which includes test coverage and you need to keep your source code private, you’re looking at a minimum of $15 per month for these cloud based solutions and that’s if you can manage with only 50 build minutes per month. If you can’t it’s more likely to be $30 per month, that’s $360 per year.

That’s not a lot of money for a large software company or even a well funded startup or SME, though as the number of users goes up so does that price. For a personal project it’s a lot of money. 

Cost isn’t the only drawback, with these approaches you can lose some flexibility as well. 

The alternative is to build your own development pipelines. 

I bet you’re thinking that setting up these tools from scratch is a royal pain in the arse and will take hours; when the cloud solutions can be set up in minutes. Not to mention running and managing your own pipeline on your personal machine and don’t they suck resources when they’re running in the background all the time? And shouldn’t they be set up on isolated machines? What if I told you, you could set all of this up in about an hour and turn it all on and off as necessary with a single command? And if you wanted to, you could run it all on a DigitalOcean Droplet for around $20 per month. 

Interested? Read on.

What

When you know how, setting up a continuous integration server such as Jenkins and a static analysis repository such as SonarQube in a Docker container is relatively straightforward. As is starting and stopping them altogether using Docker Compose. As I said, the key is knowing how; and what I explain in the rest of this article is the product of around twenty development hours, a lot of which was banging my head against a number of individual issues which turned out to have really simple solutions.

Docker

Docker is a way of encapsulating software in a container. Anything from an entire operating system such as Ubuntu to a simple tool such as the scanner for SonarQube. The configuration of the container is detailed in a Dockerfile and Docker uses Dockerfiles to build, start and stop containers. Jenkins and SonarQube all have publically available Docker images, which we’ll use with a few relatively minor modifications, to build a development pipeline.

Docker Compose

Docker Compose is a tool which orchestrates Docker containers. Via a simple YML file it is possible to start and stop multiple Docker containers with a single command. This means that once configured we can start and stop the entire development pipeline so that it is only running when we need it or, via a tool such as Terraform, construct and provision a DigitalOcean droplet (or AWS service, etc.) with a few simple commands and tear it down again just as easily so that it only incurs cost when we’re actually developing. Terraform and DigitalOcean are beyond the scope of this article, but I plan to cover them in the near future. 

See the Docker and Docker Compose websites for instructions on how to install them for your operating system.

How

In order to focus on the development pipeline configuration, Over this and a few other posts I’ll describe how to create an extremely simple Dotnet Core class library with a very basic test and describe in more detail how to configure and run Jenkins and SonarQube Docker containers and setup simple projects in both to demonstrate the pipeline. I’ll also describe how to orchestrate the containers with Docker Compose. 

I’m using Dotnet Core because that’s what I’m working with on a daily basis. The development pipeline can also be used with Java, Node, TypeScript or any other of the supported languages. Dotnet Core is also free to install and use on Windows, Linux and Mac which means that anyone can follow along.

A Simple Dotnet Core Class Library Project

I’ve chosen to use a class library project as an example for two reasons. It means that I can easily use a separate project for the tests, which allows me to describe the development pipeline more iteratively. It also means that I can use it as the groundwork for a future article which introduces the NuGet server Baget to the development pipeline.

Open a command prompt and start off by creating an empty directory and moving into it.

mkdir messagelib
cd messagelib

Then open the directory in your favorite IDE, I like VSCode for this sort of project. Add a Dotnet Core appropriate .gitignore file and then create a solution and a class library project and add it to the solution:

dotnet new sln
dotnet new classLib --name Messagelib
dotnet sln add Messagelib/Messagelib.csproj

Delete MessageLib/class1.cs and create a new class file and class called Message:

using System;

namespace Messagelib
{
    public class Message
    {
        public string Deliver()
        {
            return "Hello, World!";
        }
    }
}

Make sure it builds with:

dotnet build

Commit the solution to a public git repository or you can use the existing one in my bitbucket account here: https://bitbucket.org/findmytea/messagelib

A public repository keeps this example simple and although I won’t cover it here, it’s quite straightforward to add a key to a BitBucket or GitHub private repository and to Jenkins so that it can access them.

Remember that one of the main driving forces for setting up the development pipeline is to allow the use of private repositories without having to incur unnecessary cost.


Read the next parts here:




Sidebar 1


Continuous Integration 

Continuous Integration (CI) is a development practice where developers integrate code into a shared repository frequently, preferably several times a day. Each integration can then be verified by an automated build and automated tests. While automated testing is not strictly part of CI it is typically implied.


Static Analysis

Static (code) analysis is a method of debugging by examining source code before a program is run. It’s done by analyzing a set of code against a set (or multiple sets) of coding rules.


Measuring Code Coverage

Code coverage is a metric that can help you understand how much of your source is tested. It's a very useful metric that can help you assess the quality of your test suite.



Sidebar 2: CircleCI Credits

Credits are used to pay for your team’s usage based on machine type and size, and premium features like Docker layer caching.



Sidebar 3: What is Terraform?

Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions.

Configuration files describe to Terraform the components needed to run a single application or your entire datacenter. Terraform generates an execution plan describing what it will do to reach the desired state, and then executes it to build the described infrastructure. As the configuration changes, Terraform is able to determine what changed and create incremental execution plans which can be applied.

The infrastructure Terraform can manage includes low-level components such as compute instances, storage, and networking, as well as high-level components such as DNS entries, SaaS features, etc.


Piping Software for Less: Jenkins (Part 2)

Paul Grenyer from Paul Grenyer



Run Jenkins in Docker with Docker Compose

Why use Jenkins I hear you ask? Well, for me the answers are simple: familiarity and the availability of an existing tested and officially supported Docker image. I have been using Jenkins for as long as I can remember. 

The official image is here: https://hub.docker.com/r/jenkins/jenkins

After getting Jenkins up and running in the container we’ll look at creating a ‘Pipeline’ with the Docker Pipeline plugin. Jenkins supports lots of different ‘Items’, which used to be called ‘Jobs’, but Docker can be used to encapsulate build and test environments as well. In fact this is what BitBucket Pipelines and CircleCI also do.

To run Jenkins Pipeline we need a Jenkins installation with Docker installed. The easiest way to do this is to use the existing Jenkins Docker image from Docker Hub. Open a new command prompt and create a new directory for the development pipeline configuration and a sub directory called Jenkins with the following Dockerfile in it: 

FROM jenkins/jenkins:lts

USER root
RUN apt-get update
RUN apt-get -y install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common

RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -

RUN add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/debian \
   $(lsb_release -cs) \
   stable"

RUN apt-get update
RUN apt-get install -y docker-ce docker-ce-cli containerd.io
RUN service docker start

# drop back to the regular jenkins user - good practice
USER jenkins

You can see that our Dockerfile imports the existing Jenkins Docker image and then installs Docker for Linux. The Jenkins image, like most Docker images, is based on a Linux base image.

To get Docker Compose to build and run the image, we need a simple docker-compose.yml file in the root of the development pipeline directory with the details of the Jenkins service:

version: '3'
services:
  jenkins:
    container_name: jenkins
    build: ./jenkins/
    ports:
      - "8080:8080"
      - "5000:5000"
    volumes:
        - ~/.jenkins:/var/jenkins_home
        - /var/run/docker.sock:/var/run/docker.sock

Note the build parameter which references a sub directory where the Jenkins Dockerfile should be located. Also note the volumes. We want the builds to persist even if the container does not, so create a .jenkins directory in your home directory:

mkdir ~/.jenkins

Specifying it as a volume in docker-compse.yml tells the Docker image to write anything which Jenkins writes to /var/jenkins_home in the container to ~/.jenkins on the host - your local machine. If the development pipeline is running on a DigitalOcean droplet, DigitalOcean Volumes can be used to persist the volumes even after the droplet is torn down.

As well as running Jenkins in a Docker container we’ll also be doing our build and running our tests in a Docker container. Docker doesn’t generally like being run in a Docker container itself, so by specifying /var/run/docker.sock as a volume, the Jenkins container and the test container can be run on the same Docker instance.

To run Jenkins, simply bring it up with Docker compose:

docker-compose up

(To stop it again just use ctrl+c)

Make sure the first time you note down the default password. It will appear in the log like this:

Jenkins initial setup is required. An admin user has been created and a password generated.

Please use the following password to proceed to installation:

<password>

This may also be found at: /var/jenkins_home/secrets/initialAdminPasswor

To configure Jenkins for the first time open a browser and navigate to:

http://localhost:8080

Then:

  1. Paste in the default password and click continue.
  2. Install the recommended plugins. This will take a few minutes. There is another plugin we need too which can be installed afterwards.
  3. Create the first admin user and click Save & Continue.
  4. Confirm the Jenkins url and click Save & Finish.
  5. Click Start Jenkins to start Jenkins.

You now have Jenkins up and running locally in a Docker container! 

  1. To use Docker pipelines in Jenkins we need to install the plugin. To do this:
  2. Select Manage Jenkins from the left hand menu, followed by Manage Plugins.
  3. Select the ‘Available’ tab, search for ‘Docker Pipeline’ and select it,
  4. Click ‘Download now and install after restart’. 
  5. On the next page put a tick in the ‘restart after download’ check box and wait for the installation and for Jenkins to restart. Then log in again.

Next we need to create the Docker Pipeline for the Messagelib solution. 

  1. Select ‘New Item’ from the left hand menu, enter ‘Messagelib’ as the name, select ‘Pipeline’ and click ok.
  2. Scroll to the ‘Pipeline’ section and select ‘Pipeline script from SCM’ from the ‘Definition’ dropdown. This is because we’re going to define our pipeline in a file in the Messagelib solution. 
  3. From the ‘SCM’ dropdown, select ‘Git’ and enter the repository URL of the Messagelib solution. 
  4. Then click Save.


Jenkins is now configured to run the Messagelib pipeline, but we need to tell it what to do by adding a text file called Jenkinsfile to the root of the Messagelib solution.

/* groovylint-disable CompileStatic, GStringExpressionWithinString, LineLength */

pipeline
{
    agent
    {
        docker { image 'pjgrenyer/dotnet-build-sonarscanner:latest' }
    }
    stages
    {
        stage('Build & Test')
        {
            steps
            {
                sh 'dotnet clean'
                sh 'dotnet restore'
                sh 'dotnet build'
            }
        }
    }
}

This very simple Groovy script tells the Jenkins pipeline to get the latest ‘dotnet-build-sonarscanner’ Docker image and then use it to clean, restore and build the dotnet project. ‘dotnet-build-sonarscanner’ is a Docker image I built and pushed to Docker Hub using the following Dockerfile:

FROM mcr.microsoft.com/dotnet/core/sdk:latest AS build-env
WORKDIR /
RUN apt update
RUN apt install -y default-jre
ARG dotnet_cli_home_arg=/tmp
ENV DOTNET_CLI_HOME=$dotnet_cli_home_arg
ENV DOTNET_CLI_TELEMETRY_OPTOUT=1
ENV PATH="${DOTNET_CLI_HOME}/.dotnet/tools:${PATH}"
ENV HOME=${DOTNET_CLI_HOME}
RUN dotnet tool install --global dotnet-sonarscanner
RUN chmod 777 -R ${dotnet_cli_home_arg}

This creates and configures a development environment for Dotnet Core and Sonar Scanner, which requires Java. 

There is a way to use the Dockerfile directly, rather than getting it from Docker Hub, described here: https://www.jenkins.io/doc/book/pipeline/docker/

Once the Jenkins file is added to the project and committed, set the build off by clicking ‘Build now’ from the left hand menu of the MessageLib item. The first run will take a little while as the Docker image is pulled (or built). Future runs won’t have to do that and will be quicker. You should find that once the image is downloaded, the project is built quickly and Jenkins shows success.


Read the next parts here:


Part 2: Piping Software for Less: Jenkins (Part 2)
Part 3: Piping Software for Less: SonarQube (Part 2)