Norfolk Developers Business Dinners Return

Rain Crowson from Paul Grenyer



We are pleased to announce that Norfolk Developers Business (NorDevBiz!) is making a return. NorDevBiz is aiming is to increase awareness of Norfolk Developers member's businesses and stimulate internal and external referrals.

Our second evening meeting will be on Monday 3rd of July 2017 at the Library Restaurant in Norwich at 7.30pm.

Arrive is for 7pm for informal networking and a 7.30pm sharp start. The evening will (provisionally) be as follows:

  • Informal networking (from 7pm)
  • Introductions  - an introduction to the group from Dom Davis followed by a one minute introduction from each attendee (7.30pm)
  • Main Course
  • Spotlight -  Sean Billings will give a 10 to 15 minute introduction to his business and take questions
  • Pudding
  • Close - Dom Davis

This is a business focussed event where we will learn more about each other’s businesses with the aim of generating both internal and external referrals. Most content will not be of a technical nature.

Dress code is comfortable.

Please get in touch if you are interested in attending our business dinner. Please also note that places are limited.

paul@norfolkdevelopers.com

Women Who Code workshop on “Write your own programming language”

Andy Balaam from Andy Balaam's Blog

On Wednesday 28th June 2017 a group of people from OpenMarket went to the Fora office space in Clerkenwell, London to run a workshop with the Women Who Code group, who work to help women achieve their career goals.

OpenMarket provided the workshop “Write your own programming language” and funded the food, and the venue was provided gratis by Fora.

We started the evening with some networking and food:

networking

food

but most of the time was spent coding:

coding

with lots of help from our OpenMarket helpers:

helpers

The feedback we got was very positive:

Everyone seemed to be having fun, so we hope we might get invited back to do more in future.

Why do this?

At OpenMarket we want to improve our diversity, and we have started by looking at gender diversity specifically. By being involved with events like this we hope to learn how we can make our company better at welcoming and supporting employees, encourage people from under-represented groups to apply to work here, and improve the general climate in our industry.

Thank you

A huge thank you to the OpenMarket people (from London and Guadalajara!) who helped out – I think people felt welcome and there was plenty of help available for the attendees – you did a great job.

Thank you also for the great response from everyone in our London office – several people in the office wanted to come but couldn’t make it on the night – I am hoping we will get more opportunities in future.

We’re also really grateful to OpenMarket for funding the food, to Fora for providing the space, and to Women Who Code for doing such great work to improve our industry.

Links

[Photos by David Lawson.]

Tell, don’t ask

Tim Pizey from Tim Pizey

More than twelve years ago Tim Joyce passed on some programming wisdom:

With programs tell don't ask, vice versa for people.

This was a bit abstract for me at the time but last night it came back to me as what is wrong with the code I am currently working on. We store our application configuration in a table in the system's target database and whenever some configuration is needed it is looked up in the database. There was no problem with this approach when the code was written because JUnit had not been invented and testing was not the main part of our discipline. However to write a test we would need a database present, which is an obstacle to fast, distinct, unit tests and has been a blocker to writing tests.

Noncompliant Code Example

public class Example { 
private String path;
public void logPath() {
try {
path = CachedSystemParameter.getInstance().
getParameterValue("PATH");
} catch (SystemParameterException e) {
logger.error("[BUSINESS] Error while retrieving system parameter PATH", e);
}
logger.info("Path: " + path);
}
}

Compliant Code Example

By adding sftpPath to the class constructor we can test the business logic without the need for a database fixture.
public class Example { 

private String path;

public Example() {
this(CachedSystemParameter.getInstance().
getParameterValue("PATH"));
}

public Example(String path) {
this.path = path;
}

public void logPath() {
logger.info("Path: " + path);
}
}

Node.js the Right Way: Practical Server-Side JavaScript That Scales

Paul Grenyer from Paul Grenyer

By Jim R. Wilson
ISBN-13: 978-1937785734

Node.js the Right Way is a fantastic little book. It’s a small book (but then it’s Pragmatic exPress) and it doesn’t go into anything in much detail, but then that’s what makes it fantastic. It gives a useful and practical overview of writing Node.js server side applications and explains many of the tools and JavaScript patterns which will be useful to Node.js programmers.

It starts off with examples of manipulating the local file system using Node.js. This struck me a little odd as the only thing I tend to use the local file system for is reading configuration files. If I need to write a file I tend to put it in Amazon S3. However, this is genius and looking at how to manipulate the filesystem gives some useful insights into Node.js programming.

The book then goes on to look at networking with sockets, something which is often neglected in a world where we expect everything to be RESTful. There’s then a tour through scalable messaging, including clustering, how to access databases and how to write web services, including JavaScript promises and generators!

The final chapter covers writing a web application with a single page front end and authentication. This is the only place the book falls down. Too much is covered in two short a chapter. It’s still quite useful though.

This is not a book for a novice JavaScript or even a novice Node.js developer, but for once a little knowledge is not a dangerous thing and Node.js the Right Way will help increase that knowledge. It even led me to believe JavaScript might actually be the future.

Running a virtualenv with a custom-built Python

Andy Balaam from Andy Balaam's Blog

For my attempt to improve the asyncio.as_completed Python standard library function I needed to build a local copy of cpython (the Python interpreter).

To test it, I needed the aiohttp module, which is not part of the standard library, so the easiest way to get it was using virtualenv.

Here is the recipe I used to get a virtualenv and install packages using pip with a custom-built Python:

$ ~/code/public/cpython/python -m venv env
$ . env/bin/activate
(env) $ pip install aiohttp
(env) $ python mycode.py

Adding a concurrency limit to Python’s asyncio.as_completed

Andy Balaam from Andy Balaam's Blog

Series: asyncio basics, large numbers in parallel, parallel HTTP requests, adding to stdlib

In the previous post I demonstrated how the limited_as_completed method allows us to run a very large number of tasks using concurrency, but limiting the number of concurrent tasks to a sensible limit to ensure we don’t exhaust resources like memory or operating system file handles.

I think this could be a useful addition to the Python standard library, so I have been working on a modification to the current asyncio.as_completed method. My work so far is here: limited-as_completed.

I ran similar tests to the ones I ran for the last blog post with this code to validate that the modified standard library version achieves the same goals as before.

I used an identical copy of timed from the previous post and updated versions of the other files because I was using a much newer version of aiohttp along with the custom-built python I was running.

server looked like:

#!/usr/bin/env python3

from aiohttp import web
import asyncio
import random

async def handle(request):
    await asyncio.sleep(random.randint(0, 3))
    return web.Response(text="Hello, World!")

app = web.Application()
app.router.add_get('/{name}', handle)

web.run_app(app)

client-async-sem needed me to add a custom TCPConnector to avoid a new limit on the number of concurrent connections that was added to aiohttp in version 2.0. I also need to move the ClientSession usage inside a coroutine to avoid a warning:

#!/usr/bin/env python3

from aiohttp import ClientSession, TCPConnector
import asyncio
import sys

limit = 1000

async def fetch(url, session):
    async with session.get(url) as response:
        return await response.read()

async def bound_fetch(sem, url, session):
    # Getter function with semaphore.
    async with sem:
        await fetch(url, session)

async def run(r):
    with ClientSession(connector=TCPConnector(limit=limit)) as session:
        url = "http://localhost:8080/{}"
        tasks = []
        # create instance of Semaphore
        sem = asyncio.Semaphore(limit)
        for i in range(r):
            # pass Semaphore and session to every GET request
            task = asyncio.ensure_future(
                bound_fetch(sem, url.format(i), session))
            tasks.append(task)
        responses = asyncio.gather(*tasks)
        await responses

loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.ensure_future(run(int(sys.argv[1]))))

My new code that uses my proposed extension to as_completed looked like:

#!/usr/bin/env python3

from aiohttp import ClientSession, TCPConnector
import asyncio
import sys

async def fetch(url, session):
    async with session.get(url) as response:
        return await response.read()

limit = 1000

async def print_when_done():
    with ClientSession(connector=TCPConnector(limit=limit)) as session:
        tasks = (fetch(url.format(i), session) for i in range(r))
        for res in asyncio.as_completed(tasks, limit=limit):
            await res

r = int(sys.argv[1])
url = "http://localhost:8080/{}"
loop = asyncio.get_event_loop()
loop.run_until_complete(print_when_done())
loop.close()

and with these, we get similar behaviour to the previous post:

$ ./timed ./client-async-sem 10000
Memory usage: 73640KB	Time: 19.18 seconds
$ ./timed ./client-async-stdlib 10000
Memory usage: 49332KB	Time: 18.97 seconds

So the implementation I plan to submit to the Python standard library appears to work well. In fact, I think it is better than the one I presented in the previous post, because it uses on_complete callbacks to notice when futures have completed, which reduces the busy-looping we were doing to check for and yield finished tasks.

The Python issue is bpo-30782 and the pull request is #2424.

Note: at first glance, it looks like the aiohttp.ClientSession‘s limit on the number of connections (introduced in version 1.0 and then updated in version 2.0) gives us what we want without any of this extra code, but in fact it only limits the number of connections, not the number of futures we are creating, so it has the same problem of unbounded memory use as the semaphore-based implementation.

Call a dll from Python

Frances Buontempo from BuontempoConsulting

Letting VS2015 make a dll called SomeDLL for me with these implementations

// SomeDll.cpp : 
// Defines the exported functions for the DLL application.
//

#include "stdafx.h"
#include "SomeDll.h"


// This is an example of an exported variable
SOMEDLL_API int nSomeDll=0;

// This is an example of an exported function.
SOMEDLL_API int fnSomeDll(void)
{
    return 42;
}

// This is the constructor of a class that has been exported.
// see SomeDll.h for the class definition
CSomeDll::CSomeDll()
{
    return;
}
 

I then make a python script, using ctypes and loading the dll, using os to find it:

import os
import ctypes

os.chdir("C:\\Users\\sbkg525\\src\\SomeDll\\Debug")
SomeDll = ctypes.WinDLL("SomeDll.dll")


I can either use attributes of the library or use protoypes. I tried protoypes first.The function returns an int and takes no parameters:

proto = ctypes.WINFUNCTYPE(ctypes.c_int)
params = ()

answer = proto(("fnSomeDll", SomeDll), params)


Unfortunately this says

AttributeError: function 'fnSomeDll' not found

because C++ is name mangled. extern "C" FTW another time; for now

link.exe /dump /exports Debug\SomeDll.dll

          1    0 0001114F ??0CSomeDll@@QAE@XZ =

                  @ILT+330(??0CSomeDll@@QAE@XZ)
          2    1 00011244 ??4CSomeDll@@QAEAAV0@$$QAV0@@Z =

                  @ILT+575(??4CSomeDll@@QAEAAV0@$$QAV0@@Z)
          3    2 0001100A ??4CSomeDll@@QAEAAV0@ABV0@@Z =

                  @ILT+5(??4CSomeDll@@QAEAAV0@ABV0@@Z)
          4    3 0001111D ?fnSomeDll@@YAHXZ =

                  @ILT+280(?fnSomeDll@@YAHXZ)
          5    4 00018138 ?nSomeDll@@3HA =

                  ?nSomeDll@@3HA (int nSomeDll)

Looks like we want 4;

answer = proto(("?fnSomeDll@@YAHXZ", SomeDll), params)

print answer

>>> <WinFunctionType object at 0x0248A7B0>


Of course. It's a function. Let's call the function

print answer()
>>> 42


Done. I'll try attributes next. And functions which take parameters. Later, I'll look at other calling conventions.

First, the docs says, "foreign functions can be accessed as attributes of loaded shared libraries.
Given

extern "C"
{
    SOMEDLL_API void hello_world()
    {
        //now way of telling this has worked
    }

}

we call it like this

lib = ctypes.cdll.LoadLibrary('SomeDll.dll')
lib.hello_world()


It seems to assume a return type of int. For example,

extern "C"
{
    SOMEDLL_API double some_number()
    {
        return 1.01;
    }

}

called as follows

lib = ctypes.cdll.LoadLibrary('SomeDll.dll')
lib.hello_world()
val = lib.some_number()
print val, type(val)

Gives

-858993460, <type 'int'>


We need to specify the return type, since it's not an int:



lib.some_number.restype = ctypes.c_double
val = lib.some_number()
print val, type(val)


then we get what we want

1.01 <type 'float'>

We need to do likewise for parameters

extern "C"
{
    SOMEDLL_API double add_numbers(double x, double y)
    {
        return x + y;
    }
}


If we just try to call it with some floats we get an error

    total = lib.add_numbers(10.5, 25.7)
ctypes.ArgumentError: argument 1: <type 'exceptions.TypeError'>: Don't know how to convert parameter 1


Stating the parameter type fixes this

lib.add_numbers.restype = ctypes.c_double
lib.add_numbers.argtypes = [ctypes.c_double, ctypes.c_double]
total = lib.add_numbers(10.5, 25.7)
print total, type(total)


36.2 <type 'float'>

Just one starter thought on strings. Returning a const char * seems to give a str in python

SOMEDLL_API const char * speak()
{
    return "Hello";


called like this



lib.speak.restype = ctypes.c_char_p
print lib.speak()


says "Hello"; using parameters as strings and particularly as refrences that can be changed needs some investigation.

Does a similar approach using attributes of the loaded library work for non-extern "C" functions? Can I use the proto approach on the extern "C" functions?

Let's see...







Codes of Conduct

Frances Buontempo from BuontempoConsulting

There's been yet another thread about code of conducts at conferences on twitter, and I wanted to compare them to conditions of sale for gig or festival tickets. Here's an example:
Let's consider a sample:
1. I can't turn up before it starts? Why even say that? Oh, and 2. I have to leave when it's over. THIS IS PC GONE MAD. OK, not PC. But why do you need to spell this out? It's like putting a precondition of a string has to terminate with a null for you to call strlen.
3. Yeah, rulez. Whatever. Surprised you didn't say it's illegal to break the law. What is this?
Blahblahblah.
9. Yeah, fair enough. But what about flame throwers?
No, no, no...
16. Does that mean I can bring *Real* weapons?
Blahblahblah.

25. There might be swear words?! Why tell me that?

Or maybe some people will bring kids with them and that might need a sane conversation about a time and a place for certain behaviours. I know the bands, I know what kind of music to expect. I know what I'm walking in to.

I also know there's a clearly marked welfare tent on site just in case.

And wardens on the campsite wearing obvious vests.

And hundreds of programmers listening to their favourite rock stars.
(OK, not all the punters are programmers, but many are).

Why did I ever go to a field full of metallers? (Knowing I'd probably be one of the few women there). I went with friends, which made me feel safer. I wanted to listen to bands I knew, and discover new music. If I'd gone by myself, knowing in advance about wardens and the welfare tent would have made me feel OK. Hey, the conditions of sale show the organisers have thought about things that might go wrong or concern people, and that makes me feel safer.

Do they make people think, "This is PC gone mad? If you need to tell me not to set fire to myself and drink myself to death then you are insulting me and I want nothing to do with this?"

Not by the looks of the number of people who turn up. And there have been more women and kids recently which is great.

What are codes of conduct for? Quite frankly me. But not just me. They are not there to tell you how to behave because the organisers think white guys don't know how to conduct themselves, and that all such guys are potential rapists or murders. In some ways, the logic of stating a CoC is pointless because we know how to be nice taken to extreme could mean laws are pointless. Why say "You aren't allowed to commit murder?" Surely that doesn't need pointing out? And yet most countries have such a law.

Conditions of sale and code of conducts aren't laws, but they give a shared statement of expectation and make me feel OK about going to conferences/talks/festivals/gigs alone. And then I meet loads of new people and discover new music. And I can't wait til Bloodstock. Or the next tech talk/conference I go to.





Refactoring – Before or After?

Chris Oldwood from The OldWood Thing

I recently worked on a codebase where I had a new feature to implement but found myself struggling to understand the existing structure. Despite paring a considerable amount I realised that without other people to easily guide me I still got lost trying to find where I needed to make the change. I felt like I was walking through a familiar wood but the exact route eluded me without my usual guides.

I reverted the changes I had made and proposed that now might be a good point to do a little reorganisation. The response was met with a brief and light-hearted game of “Ken Beck Quote Tennis” - some suggested we do the refactoring before the feature whilst others preferred after. I felt there was a somewhat superficial conflict here that I hadn’t really noticed before and wondered what the drivers might be to taking one approach over the other.

Refactor After

If you’re into Test Driven Development (TDD) then you’ll have the mantra “Red, Green, Refactor” firmly lodged in your psyche. When practicing TDD you first write the test, then make it pass, and finally finish up by refactoring the code to remove duplication or otherwise simplify it. Ken Beck’s Test Driven Development: By Example is probably the de facto read for adopting this practice.

The approach here can be seen as one where the refactoring comes after you have the functionality working. From a value perspective most of it comes from having the functionality itself – the refactoring step is an investment in the codebase to allow future value to be added more easily later.

Just after adding a feature is the point where you’ve probably learned the most about the problem at hand and so ensuring the design best represents your current understanding is a worthwhile aid to future comprehension.

Refactor Before

Another saying from Kent Beck that I’m particularly fond of is “make the change easy, then make the easy change” [1]. Here he is alluding to a dose of refactoring up-front to mould the codebase into a shape that is more amenable to allowing you to add the feature you really want.

At this point we are not adding anything new but are leaning on all the existing tests, and maybe improving them too, to ensure that we make no functional changes. The value here is about reducing the risk of the new feature by showing that the codebase can safely evolve towards supporting it. More importantly It also gives the earliest visibility to others about the new direction the code will take [2].

We know the least amount about what it will take to implement the new feature at this point but we also have a working product that we can leverage to see how it’s likely to be impacted.

Refactor Before, During & After

Taken at face value it might appear to be contradictory about when the best time to refactor is. Of course this is really a straw man argument as the best time is in fact “all the time” – we should continually keep the code in good shape [3].

That said the act of refactoring should not occur within a vacuum, it should be driven by a need to make a more valuable change. If the code never needed to change we wouldn’t be doing it in the first place and this should be borne in mind when working on a large codebase where there might be a temptation to refactor purely for the sake of it. Seeing stories or tasks go on the backlog which solely amount to a refactoring are a smell and should be heavily scrutinised.

Emergent Design

That said, there are no absolutes and whilst I would view any isolated refactoring task with suspicion, that is effectively what I was proposing back at the beginning of this post. One of the side-effects of emergent design is that you can get yourself into quite a state before a cohesive design finally emerges.

Whilst on paper we had a number of potential designs all vying for a place in the architecture we had gone with the simplest possible thing for as long as possible in the hope that more complex features would arrive on the backlog and we would then have the forces we needed to evaluate one design over another.

Hence the refactoring decision became one between digging ourselves into an even deeper hole first, and then refactoring heavily once we had made the functional change, or doing some up-front preparation to solidify some of the emerging concepts first. There is the potential for waste if you go too far down the up-front route but if you’ve been watching how the design and feature list have been emerging over time it’s likely you already know where you are heading when the time comes to put the design into action.

 

[1] I tend to elide the warning from the original quote about the first part potentially being hard when saying it out loud because the audience is usually well aware of that :o).

[2] See “The Cost of Long-Lived Feature Branches” for a cautionary tale about storing up changes.

[3] See “Relentless Refactoring” for the changes in attitude towards this practice.