Planet Linux Australia
Celebrating Australians & Kiwis in the Linux and Free/Open-Source community...

April 17, 2019

Programming an AnyTone AT-D878UV on Linux using Windows 10 and VirtualBox

I recently acquired an AnyTone AT-D878UV DMR radio which is unfortunately not supported by chirp, my usual go-to free software package for programming amateur radios.

Instead, I had to setup a Windows 10 virtual machine so that I could setup the radio using the manufacturer's computer programming software (CPS).

Install VirtualBox

Install VirtualBox:

apt install virtualbox virtualbox-guest-additions-iso

and add your user account to the vboxusers group:

adduser francois vboxusers

to make filesharing before the host and the guest work.

Finally, reboot to ensure that group membership and kernel modules are all set.

Create a Windows 10 virtual machine

Create a new Windows 10 virtual machine within VirtualBox. Then, download Windows 10 from Microsoft then start the virtual machine mounting the .iso file as an optical drive.

Follow the instructions to install Windows 10, paying attention to the various privacy options you will be offered.

Once Windows is installed, mount the host's /usr/share/virtualbox/VBoxGuestAdditions.iso as a virtual optical drive and install the VirtualBox guest additions.

Installing the CPS

With Windows fully setup, it's time to download the latest version of the computer programming software.

Unpack the downloaded file and then install it as Admin (right-click on the .exe).

Do NOT install the GD driver update or the USB driver, they do not appear to be necessary.

Program the radio

First, you'll want to download from the radio to get a starting configuration that you can change.

To do this:

  1. Turn the radio on and wait until it has finished booting.
  2. Plug the USB programming cable onto the computer and the radio.
  3. From the CPS menu choose "Set COM port".
  4. From the CPS menu choose "Read from radio".

Save this original codeplug to a file as a backup in case you need to easily reset back to the factory settings.

To program the radio, follow this handy third-party guide since it's much better than the official manual.

You should be able to use the "Write to radio" menu option without any problems once you're done creating your codeplug.

April 13, 2019

Secure ssh-agent usage

ssh-agent was in the news recently due to the compromise. The main takeaway from that incident was that one should avoid the ForwardAgent (or -A) functionality when ProxyCommand can do and consider multi-factor authentication on the server-side, for example using libpam-google-authenticator or libpam-yubico.

That said, there are also two options to ssh-add that can help reduce the risk of someone else with elevated privileges hijacking your agent to make use of your ssh credentials.

Prompt before each use of a key

The first option is -c which will require you to confirm each use of your ssh key by pressing Enter when a graphical prompt shows up.

Simply install an ssh-askpass frontend like ssh-askpass-gnome:

apt install ssh-askpass-gnome

and then use this to when adding your key to the agent:

ssh-add -c ~/.ssh/key

Automatically removing keys after a timeout

ssh-add -D will remove all identities (i.e. keys) from your ssh agent, but requires that you remember to run it manually once you're done.

That's where the second option comes in. Specifying -t when adding a key will automatically remove that key from the agent after a while.

For example, I have found that this setting works well at work:

ssh-add -t 10h ~/.ssh/key

where I don't want to have to type my ssh password everytime I push a git branch.

At home on the other hand, my use of ssh is more sporadic and so I don't mind a shorter timeout:

ssh-add -t 4h ~/.ssh/key

Making these options the default

I couldn't find a configuration file to make these settings the default and so I ended up putting the following line in my ~/.bash_aliases:

alias ssh-add='ssh-add -c -t 4h'

so that I can continue to use ssh-add as normal and have not remember to include these extra options.

April 11, 2019

Using a MCP4921 or MCP4922 as a SPI DAC for Audio on Raspberry Pi


I’ve been playing recently with using a MCP4921 as an audio DAC on a Raspberry Pi Zero W, although a MCP4922 would be equivalent (the ’22 is a two channel DAC, the ’21 is a single channel DAC). This post is my notes on where I got to before I decided that thing wasn’t going to work out for me.

My basic requirement was to be able to play sounds on a raspberry pi which already has two SPI buses in use. Thus, adding a SPI DAC seemed like a logical choice. The basic circuit looked like this:

MCP4921 SPI DAC circuit

Driving this circuit looked like this (noting that this code was a prototype and isn’t the best ever). The bit that took a while there was realising that the CS line needs to be toggled between 16 bit writes. Once that had been done (which meant moving to a different spidev call), things were on the up and up.

This was the point I realised that I was at a dead end. I can’t find a way to send the data to the DAC in a way which respects the timing of the audio file. Before I had to do small writes to get the CS line to toggle I could do things fast enough, but not afterwards. Perhaps there’s a DMA option instead, but I haven’t found one yet.

Instead, I think I’m going to go and try PWM based audio. If that doesn’t work, it will be a MAX219 i2c DAC for me!


April 10, 2019

Audiobooks – March 2019

An Economist Gets Lunch: New Rules for Everyday Foodies by Tyler Cowen

A huge amount of practical advice and how and where to find the best food both locally and abroad. Plus good explanations as to why. 8/10

The Not-Quite States of America: Dispatches from the Territories and Other Far-Flung Outposts of the USA by Doug Mack

Writer tours the not-states of the USA. A bit too fluffy most of the time & too much hanging with US expats. Some interesting bits. 6/10

Shattered: Inside Hillary Clinton’s Doomed Campaign by Jonathan Allen & Amie Parnes

Chronology of the campaign based on background interviews with staffers. A ready needs a good knowledge of the race since this is assumed. Interesting enough. 7/10

Rush Hour by Iain Gatel

A history of commuting (from the early railway era), how it has driven changes in housing, work and society. Plus lots of other random stuff. Very pleasant. 8/10


April 08, 2019

1-Wire home automation tutorial from 2019, part 2


For the actual on-the-day work, delegates were handed a link to these instructions in github. If you’re playing along at home, you should probably read 1-Wire home automation tutorial from 2019, part 1 before attempting the work described here. Its especially important that you know the IP address of your board for example.

Relay tweaks

The instructions are pretty self explanatory, although I did get confused about where to connect the relay as I couldn’t find PC8 in my 40 pin header diagrams. That’s because the shields for the tutorial have a separate header which is a bit more convenient:

GPIO header

I was also a bit confused when the relay didn’t work initially, but that turns out because I’d misunderstood the wiring. The relay needs to be powered from the 3.3v pin on the 40 pin header, as there is a PCB error which puts 5v on the pins labelled as 3.3v on the GPIO header. I ended up with jumper wires which looked like this:

Cabling the relay

1-Wire issues

Following on the tutorial instructions worked well from then on until I tried to get 1-Wire setup. The owfs2mqtt bridge plugin was logging this:

2019-04-08 19:23:55.075: /opt/OWFS-MQTT-Bridge/lib/Daemon/ Connection to owserver failed: Can't connect owserver: Address not available

Debugging that involved connecting to the owfs2mqtt docker container (hint: ssh to the Orange Pi, do a docker ps, and then run bash inside the docker container for the addon). Running owserver with debug produces this:

owserver fails

Sorry to post that as an image, cut and paste for the hassos ssh server doesn’t like me for some reason. I suspect I have a defective DS2482, but I’ll have to wait and see what Allistair says.


1-Wire home automation tutorial from 2019, part 1


I didn’t get much of a chance to work through the home automation tutorial at 2019 because I ended up helping others in the room get their Orange Pi is booting. Now that things have settled down after the conference, I’ve had a chance to actually do some of the tutorial myself. These are my notes so I can remember what I did later…

Pre-tutorial setup

You need to do the pre-tutorial setup first. I use Ubuntu, which means its important that I use 18.10 or greater so that st-link is packaged. Apart from that the instructions as written just worked.

You also need to download the image for the SD card, which was provided on the day at the conference. The URL for that is from github. Download that image, decompress it, and then flash it to an SD card using something like Balena Etcher. The tutorial used 32gb SD cards, but the image will fit on something smaller than that.

hassos also doesn’t put anything on the Orange Pi HDMI port when it boots, so your machine is going to look like it didn’t boot. That’s expected. For the tutorial we provided a mapping from board number (mac address effectively) to IP address allocated in the tutorial. At home if you’re using an Orange Pi that isn’t from the conference you’re going to have to find another way to determine the IP address of your Orange Pi.

The way we determined MAC addresses and so forth for the boards used at the conference was to boot an Armbian image and then run a simple python script which performed some simple checks of each board by logging into the board over serial. The MAC addresses for the boards handed out on the day are on github.

An Aside: Serial on the Orange Pi Prime

As an aside, the serial on the Orange Pi Prime is really handy, especially with the hassos image. Serial is exposed by a three pin header on the board, which is sort of labelled:

Orange Pi Prime Serial PortThe Orange Pi Prime Serial Port

Noting that you’ll need to bend the pins of the serial header a little if you’re using the shield from the conference:

Serial port connected to a USB to serial converter

The advantage being that suddenly you get useful debugging information! The serial connection is 115200 baud 8N1 (8 data bits, no parity, 1 stop bit) by the way.

Serial debug information from an hassos boot

The hassos image used for the conference allows login as root with no password over serial, which dumps you into a hass interface. Type “login” to get a bash prompt, even though its not in the list of commands available. At this point you can use the “ip address” command to work out what address DHCP handed the board.

The actual on-the-day work

So at this point we’re about as ready as people were meant to be when they walked into the room for the tutorial. I’ll write more notes when I complete the actual tutorial.


April 05, 2019

Introducing GangScan


As some of you might know, I am a Scout Leader. One of the things I do for Scouts is I assist in a minor role with the running of Canberra Gang Show, a theatre production for young people.

One of the things Gang Show cares about is that they need to be able to do rapid roll calls and reporting on who is present at any given time — this is used for working out who is absent before a performance (and therefore needs an understudy), as well as ensuring we know where everyone is in an environment that sometimes has its fire suppression systems isolated.

Before I came along, Canberra Gang Show was doing this with a Windows based attendance tracking application, and 125kHz RFID tags. This system worked just fine, except that the software was clunky and there was only one badge reader — we struggled explaining to youth that they need to press the “out” button when logging out, and we wanted to be able to have attendance trackers at other locations in the theatre instead of forcing everyone to flow through a single door.

So, I got thinking. How hard can it be to build something a bit better?

Let’s start with some requirements: simple to deploy and manage; free software (both cost and freedom); more badge readers than what we have now; and low cost.

My basic proposal for such a thing is a Raspberry Pi Zero W, with a small LCD screen and a RFID reader. The device scans badges, and displays a confirmation of scan to the user. If the device can talk to a central server it streams events to it; otherwise it queues them until the server is available and then streams them.

Sourcing a simple SPI LCD screen and SPI RFID reader from ebay wasn’t too hard, and we were off! The only real wart was that I wanted to use 13.56mHz RFID cards, because then I could store some interesting (up to 1kb) data on the card itself. The first version was simply a ribbon cable:

v0.0, a ribbon cable

Which then led to me having my first PCB ever made. Let’s ignore that its the wrong size shall we?

v0.1, an incorrectly sized PCB

I’m now at the point where the software for the scanner is reasonable, and there is a bare bones server that does enough roll call that it should be functional. I am sure there’s more to be done, but it works enough to demo. One thing I learned while showing off the device at coffee the other day is that it really needs to make a noise when you scan a badge. I’ve ordered a SPI DAC to play with, which might be the solution there. Other next steps include a newer version of the PCB, and some sort of case solution. I’ll do another post when things progress further.

Oh yes, and I’ll eventually release the software too once its in a more workable state.


April 04, 2019

React Isn’t The Problem

As React (via Gutenberg) becomes more present in the WordPress world, I’m seeing some common themes pop up in conversations about it. I spoke a bit about this kind of thing at WordCamp US last year, but if you don’t feel like sitting through a half hour video, let me summarise my thoughts. 🙂

I agree that React is hard. I strongly disagree with the commonly contrasted view that HTML, CSS, PHP, or vanilla JavaScript are easy. They’re all just as hard to work with as React, sometimes more-so, particularly when having to deal with the exciting world of cross-browser compatibility.

The advantage that PHP has over modern JavaScript development isn’t that it’s easy, or that the tooling is better, or more reliable, or anything like that. The advantage is that it’s familiar. If you’re new to web development, React is just as easy anything else to start with.

I’m honestly shocked when someone manages to wade through the mess of tooling (even pre-Gutenberg) to contribute to WordPress. It’s such an incomprehensible, thankless, unreliable process, the tenacity of anyone who makes it out the other side should be applauded. That said, this high barrier is unacceptable.

I’ve been working in this industry for long enough to have forgotten the number of iterations of my personal development environment I’ve gone through, to get to where I can set up something for myself which isn’t awful. React wasn’t around for all of that time, so that can’t be the reason web development has been hard for as long as I remember. What is, then?

Doing Better

Over the past year or so, I’ve been tinkering with a tool to help deal with the difficulties of contributing to WordPress. That tool is called TestPress, it’s getting pretty close to being usable, at least on MacOS. Windows support is a little less reliable, but getting better. 🙂 If you enjoy tinkering with tools, too, you’re welcome to try out the development version, but it does still has some bugs in it. Feedback and PRs are always welcome! There are some screenshots in this issue that give an idea of what the experience is like, if you’d like to check it out that way.

TestPress is not a panacea: at best, it’s an attempt at levelling the playing field a little bit. You shouldn’t need years of experience to build a reliable development environment, that should be the bare minimum we provide.

React is part of the solution

There’s still a lot of work to do to make web development something that anyone can easily get into. I think React is part of the solution to this, however.

React isn’t without its problems, of course. Modern JavaScript can encourage iteration for the sake of iteration. Certainly, there’s a drive to React-ify All The Things (a trap I’m guilty of falling into, as well). React’s development model is fundamentally different to that of vanilla JavaScript or jQuery, which is why it can seem incomprehensible if you’re already well versed in the old way of doing things: it requires a shift in your mental model of how JavaScript works. This is a hard problem to solve, but it’s not insurmountable.

Perhaps a little controversially, I don’t think that React is guilty of causing the web to become less accessible. At worst, it’s continuing the long standing practice of web standards making accessibility an optional extra. Building anything beyond a basic, non-interactive web page with just HTML and CSS will inevitably cause accessibility issues, unless you happen to be familiar with the mystical combinations of accessible tags, or applying aria attributes, or styling your content in just the right way (and none of the wrong ways).

React (or any component-based development system, really) can improve accessibility for everyone, and we’re seeing this with Gutenberg already. By providing a set of base components for plugin and theme authors to use, we can ensure the correct HTML is produced for screen readers to work with. Much like desktop and mobile app developers don’t need to do anything to make their apps accessible (because it’s baked into the APIs they use to build their apps), web developers should have the same experience, regardless of the complexity of the app they’re building.

Arguing that accessibility needs to be part of the design process is the wrong argument. Accessibility shouldn’t be a consideration, it should be unavoidable.

Do Better

Now, can we do better? Absolutely. There’s always room for improvement. People shouldn’t need to learn React if they don’t want to. They shouldn’t have to deal with the complexities of the WCAG. They should have the freedom to tinker, and the reassurance that they can tinker without breaking everything.

The pre-React web didn’t arrive in its final form, all clean, shiny, and perfect. It took decades of evolution to get there. The post-React web needs some time to evolve, too, but it has the benefit of hindsight: we can compress the decades of evolving into a much shorter time period, provide a fresh start for those who want it, while also providing backwards compatibility with the existing ways of doing things.

April 03, 2019

Easybuild: Building Software with Ease

Building software from source is necessary for performance and development reasons. However, this can come with complex dependency and compiler requirements, which have to be explicitly stated in research computing to ensure replication of results. EasyBuild, originally developed by the Julich Supercomputing Centre, the University of Gent, and the Texas Advanced Computing Center, is a tool that allows the building of software with ease, managing the complex dependencies and toolchains, and integrating by default with the Lmod environment modules system.

This presentation will outline the need for tools like Easybuild, describe the framework of Easyblocks, Toolchains, and Easyconfig recipes, and extensions. It will also describe how to install and configure EasyBuild, write and contribute configuration files, and use the configurations to install software with a variety of optional parameters, such as rebuilds and partial builds. Finally, it will conclude with a discussion of some of the more advanced options and opportunities for involvement in the Easybuild community.

Easybuild: Building Software with Ease
Presentation to Linux Users of Victoria, 2nd April, 2010

April 01, 2019

Article Review: Curing the Vulnerable Parser

Every once in a while I read papers or articles. Previously, I've just read them myself, but I was wondering if there were more useful things I could do beyond that. So I've written up a summary and my thoughts on an article I read - let me know if it's useful!

I recently read Curing the Vulnerable Parser: Design Patterns for Secure Input Handling (Bratus, et al; USENIX ;login: Spring 2017). It's not a formal academic paper but an article in the Usenix magazine, so it doesn't have a formal abstract I can quote, but in short it takes the long history of parser and parsing vulnerabilities and uses that as a springboard to talk about how you could design better ones. It introduces a toolkit based on that design for more safely parsing some binary formats.


It's worth noting early on that this comes out of the LangSec crowd. They have a pretty strong underpinning philosophy:

The Language-theoretic approach (LANGSEC) regards the Internet insecurity epidemic as a consequence of ad hoc programming of input handling at all layers of network stacks, and in other kinds of software stacks. LANGSEC posits that the only path to trustworthy software that takes untrusted inputs is treating all valid or expected inputs as a formal language, and the respective input-handling routines as a recognizer for that language. The recognition must be feasible, and the recognizer must match the language in required computation power.

A big theme in this article is predictability:

Trustworthy input is input with predictable effects. The goal of input-checking is being able to predict the input’s effects on the rest of your program.

This seems sensible enough at first, but leads to some questionable assertions, such as:

Safety is predictability. When it's impossible to predict what the effects of the input will be (however valid), there is no safety.

They follow this with an example of Ethereum contracts stealing money from the DAO. The example is compelling enough, but again comes with a very strong assertion about the impossibility of securing a language virtual machine:

From the viewpoint of language-theoretic security, a catastrophic exploit in Ethereum was only a matter of time: one can only find out what such programs do by running them. By then it is too late.

I'm not sure that (a) I buy the assertions, or that (b) they provide a useful way to deal with the world as we find it.

Is this even correct?

You can tease out 2 contentions in the first part of the article:

  • there should be a formal language that describes the data, and
  • this language should be as simple as possible, ideally being regular and context-free.

Neither of these are bad ideas - in fact they're both good ideas - but I don't know that I draw the same links between them and security.

Consider PostScript as a possible counter-example. It's a Turing-complete language, so it absolutely cannot have predictable results. It has a well documented specification and executes in a restricted virtual machine. So let's say that it satisfies only the first plank of their argument.

I'd say that PostScript has a good security record, despite being Turing complete. PostScript has been around since 1985 and apart from the recent bugs in GhostScript, it doesn't have a long history of bugs and exploits. Maybe this just because no-one has really looked, or maybe it is possible to have reasonably safe complex languages by restricting the execution environment, as PostScript consciously and deliberately does.

Indeed, if you consider the recent spate of GhostScript bugs, perhaps some may be avoided by stricter compliance with a formal language specification. However, most seem to me to arise from the desirability of implementing some of the PostScript functionality in PostScript itself, and some of the GhostScript-specific, stupendously powerful operators exposed to the language to enable this. The bugs involve tricks to allow a user to get access to these operators. A non-Turing-complete language may be sufficient to prevent these attacks, but it is not necessary: just not doing this sort of meta-programming with such dangerous operators would also have worked. Storing the true values of the security state outside of a language-accessible object would also be good.

Is this a useful way to deal with the world as we find it?

My main problem with the general LangSec approach that this article takes is this: to get to their desired world, we need to rewrite a bunch of things with entirely different language foundations. The article talks about HTML and PDFs as examples of unsafe formats, but I cannot imagine the sudden wholesale replacement of either of these - although I would love to be proven wrong.

Can we get even part of the way with existing standards? Kinda-sorta, but mostly no, and to the authors' credit, they are open about this. They argue that formal definition parsing the language should be the "most restrictive input definition" - they specifically require you to "give up attempting to accept arbitrarily complex data", and call for "subsetting of many protocols, formats, encodings and command languages, including eliminating unneeded variability and introducing determinism and static values".

No doubt we would be in a better place if people took up these ideas for future programs. However, for our current set of programs and use cases, this is probably not tractable in any meaningful way.

The rest of the paper

The rest of the paper is reasonably interesting. Their general theory is that you should build your parsers based on a formal definition of a language, and that the parser should convert the input data to a set of objects, and then your business logic should deal with those objects. This is the 'recognizer pattern', and is illustrated below:

The recognizer pattern: separate code parses input according to a formal grammar, creating valid objects that are passed to the business logic

In short, the article is full of great ideas if you happen to be parsing a simple language, or are designing not just a parser but a full language ecosystem. They do also provide a binary parser toolkit that might be helpful if you are parsing a binary format that can be expressed with a parser combinator.

Overall, however, I think the burden of maintaining old systems is such that a security paradigm that relies on new code is pretty unlikely, and one that relies on new languages is fatally doomed from the outset. New systems should take up these ideas, yes. But I'd really like to see people grappling with how to deal with the complex and irregular languages that we're stuck with (HTML, PDF, etc) in secure ways.

Praise-Singing Poppler Utilities

Last year I gave a presentation at Linux Users of Victoria entitled Being An Acrobat: Linux and PDFs (there was an additional discussion not in the presentation about embedding Javascript in a PDF and some related security issues, but that's for another post). Part of this presentation was singing the praises of Poppler Utilities (named after the Futurama episode, "The Problem with Popplers"). This is probably the most single most common suite of tools I use when dealing with PDFs with the single exception of reading and creating. There is an enormous number of times I have encountered an issue with PDFs and resolved it with something from the Popper utils suite.

The entire relevant slide from the presentation is reproduced was as follows:

Derived from xpdf, Poppler is a software library for rendering PDF documents and is used in a variety of PDF readers (including Evince, KPDF, LibreOffice, Inkscape, Okular, Zathura etc). A collection of tools, poppler-utils, is built on Poppler’s API provides a variety of useful functions e.g.,

pdffonts - lists the fonts used in a PDF (e.g., pdfonts filename.pdf)
pdfimages - extract images from a PDF (e.g., pdfimages -png filename.pdf images/)
pdfseparate - extract single pages from a PDF (e.g., pdfseparate sample.pdf sample-%d.pdf)
pdftohtml - convert PDF to HTML format retaining formatting
pdftops - convert PDF to printable PS format
pdftotext - extract text from a PDF
pdfunite - merges PDFs (pdfunite page{01..13}.pdf combined.pdf)

Recently I had an experience with this that illustrates a practical example of one of the tools. I am currently doing a MSc in Information Systems at the University of Salford. The course content itself is conducted through the Robert Kennedy College in Swizterland. For each asssignment the course accepts uploads for one file and one file only, and only in particular formats as well (e.g., docx, pdf, rtf etc). An additional upload on the RKC system will overwrite one's previously submitted assignment file.

If you have multi-part components to an assignment, you will have to export them to a common format, combine them, and upload them as a single document. In a project management course, I ended with several files, as the assignment demanded a title page, a slideshow with nodes, a main body of the assignment (business case and project product plan), a Gannt chart (created through ProjLibre), and an reference and appendix file.

At the end of the assignment, I had a Title Page file, a Part A file, Part B Main Body file, Part B LibreProj file, and a Refs and Appendix File. First I converted them all to PDFs. This is one of the file formats accepted, and is a common export format for the text, slideshow, and project. Then I used the application pdfunite (Linux application) to combine them into a single file. e.g.,

pdfunite title.pdf parta.pdf partb.pdf gannt.pdf refs.pfs assignment.pdf

Quite clearly RKC has a limited and arguably poorly designed upload system. But the options are either complain about it, give up, or work around it. Information systems science demands the latter because we will encounter this all the time. We all have to learn how to workaround system limitations. When it comes to PDF limitations, I have found that Popper Utilities are one of the most useful tools available and I have found that I use the various utilities with almost alarming regularity. So here's a few more that I didn't mention in my initial presentation due to time constraints:

pdfdetach - extract embedded documents from a PDF (e.g., pdfdetach --saveall filename.pdf)
pdfinfo - print file information from a PDF (title, subject, keywords, author, creator etc) (e.g., pdfinfo filename.pdf)
pdftocairo - convert pdf to a png/jpeg/tiff/ps/eps/svg using cairo (e.g., pdftocairo -svg filename.pdf filename.svg)
pdftoppm - convert pdf to Portable Pixmap bitmaps (e.g., pdftoppm filename.pdf filename)

March 30, 2019

Digital government: it all starts with open

This is a short video I did on the importance of openness for digital government, for the EngageTech Forum 2018. I’ve had a few people reuse it for other events so I thought I should blog it properly :) Please see the transcript below. 

<Conference introductory remarks>

I wanted to talk about why openness and engagement is so critical for our work in a modern public service.

For me, looking at digital government, it’s not just about digital services, it’s about how we transform governments for the 21st century: how we do service delivery, engagement, collaboration, and how we do policy, legislation and regulation. How we make public services fit for purpose so they can serve you, the people, communities and economy of the 21st century.

For me, a lot of people think about digital and think about technology, but open government is a founding premise, a founding principle for digital government. Open that’s not digital doesn’t scale, and digital that’s not open doesn’t last. That doesn’t just mean looking at things like open source, open content and open APIs, but it means being open. Open to change. Being open to people and doing things with people, not just to people.

There’s a fundamental cultural, technical and process shift that we need to make, and it all starts with open.

<closing conference remarks>

March 29, 2019

Installing an xmonad Environment on NixOS

NixOS Coin by Craige McWhirter

Xmonad is a very fast, reliable and flexible window manager for Linux and other related operating systems. As I recently shifted from Debian + Propellor to NixOS + NixOps, I now needed to redefine my Xmonad requirements for the new platform.


  • Grab my xmonad.nix file and import it in your /etc/nixos/configuration.nix
  • You can also grab my related xmonad.hs, xmobarrc and session files to use the complete setup.

An Example

At the time of writing, I used the below xmonad.nix to install my requirements. My current version can be found here.

# Configuration for my xmonad desktop requirements

{ config, pkgs, ... }:


  services.xserver.enable = true;                        # Enable the X11 windowing system.
  services.xserver.layout = "us";                        # Set your preferred keyboard layout.
  services.xserver.desktopManager.default = "none";      # Unset the default desktop manager.
  services.xserver.windowManager = {                     # Open configuration for the window manager.
    xmonad.enable = true;                                # Enable xmonad.
    xmonad.enableContribAndExtras = true;                # Enable xmonad contrib and extras.
    xmonad.extraPackages = hpkgs: [                      # Open configuration for additional Haskell packages.
      hpkgs.xmonad-contrib                               # Install xmonad-contrib.
      hpkgs.xmonad-extras                                # Install xmonad-extras.
      hpkgs.xmonad                                       # Install xmonad itself.
    default = "xmonad";                                  # Set xmonad as the default window manager.

  services.xserver.desktopManager.xterm.enable = false;  # Disable NixOS default desktop manager.

  services.xserver.libinput.enable = true;               # Enable touchpad support.

  services.udisks2.enable = true;                        # Enable udisks2.
  services.devmon.enable = true;                         # Enable external device automounting.

  services.xserver.displayManager.sddm.enable = true;    # Enable the default NixOS display manager.
  services.xserver.desktopManager.plasma5.enable = true; # Enable KDE, the default NixOS desktop environment.

  # Install any additional fonts that I require to be used with xmonad
  fonts.fonts = with pkgs; [
    opensans-ttf             # Used in in my xmobar configuration

  # Install other packages that I require to be used with xmonad.
  environment.systemPackages = with pkgs; [
    dmenu                    # A menu for use with xmonad
    feh                      # A light-weight image viewer to set backgrounds
    haskellPackages.libmpd   # Shows MPD status in xmobar
    haskellPackages.xmobar   # A Minimalistic Text Based Status Bar
    libnotify                # Notification client for my Xmonad setup
    lxqt.lxqt-notificationd  # The notify daemon itself
    mpc_cli                  # CLI for MPD, called from xmonad
    scrot                    # CLI screen capture utility
    trayer                   # A system tray for use with xmonad
    xbrightness              # X11 brigthness and gamma software control
    xcompmgr                 # X composting manager
    xorg.xrandr              # CLI to X11 RandR extension
    xscreensaver             # My preferred screensaver
    xsettingsd               # A lightweight desktop settings server


This provides my xmonad environment with everything I need for xmonad to run as configured.


It occurs to me that I never wrote up the end result of the support ticket I opened with iiNet after discovering significant evening packet loss on our fixed wireless NBN connection in August 2017.

The whole saga took about a month. I was asked to run a battery of tests (ping, traceroute, file download and speedtest, from a laptop plugged directly into the NTD) three times a day for three days, then send all the results in so that a fault could be lodged. I did this, but somehow there was a delay in the results being communicated, so that by the time someone actually looked at them, they were considered stale, and I had to run the whole set of tests all over again. It’s a good thing I work from home, because otherwise there’s no way it would be possible to spend half an hour three times a day running tests like this. Having finally demonstrated significant evening slowdowns, a fault was lodged, and eventually NBN Co admitted that there was congestion in the evenings.

We have investigated and the cell which this user is connected to experiences high utilisation during busy periods. This means that the speed of this service is likely to be reduced, particularly in the evening when more people are using the internet.

nbn constantly monitors the fixed wireless network for sites which require capacity expansion and we aim to upgrade site capacity before congestion occurs, however sometimes demand exceeds expectations, resulting in a site becoming congested.

This site is scheduled for capacity expansion in Quarter 4, 2017 which should result in improved performance for users on the site. While we endeavour to upgrade sites on their scheduled date, it is possible for the date to change.

I wasn’t especially happy with that reply after a support experience that lasted for a month, but some time in October that year, the evening packet loss became less, and the window of time where we experienced congestion shrank. So I guess they did do some sort of capacity expansion.

It’s been mostly the same since then, i.e. slower in the evenings than during the day, but, well, it could be worse than it is. There was one glitch in November or December 2018 (poor speed / connection issues again, but this time during the day) which resulted in iiNet sending out a new router, but I don’t have a record of this, because it was a couple of hours of phone support that for some reason never appeared in the list of tickets in the iiNet toolbox, and even if it had, once a ticket is closed, it’s impossible to click it to view the details of what actually happened. It’s just a subject line, status and last modified date.

Fast forward to Monday March 25 2019 – a day with a severe weather warning for damaging winds – and I woke up to 34% packet loss, ping times all over the place (32-494ms), continual disconnections from IRC and a complete inability to use a VPN connection I need for work. I did the power-cycle-everything dance to no avail. I contemplated a phone call to support, then tethered my laptop to my phone instead in order to get a decent connection, and decided to wait it out, confident that the issue had already been reported by someone else after chatting to my neighbour.


Tuesday morning it was still horribly broken, so I unplugged the router from the NTD, plugged a laptop straight in, and started running ping, traceroute and speed tests. Having done that I called support and went through the whole story (massive packet loss, unusable connection). They asked me to run speed tests again, almost all of which failed immediately with a latency error. The one that did complete showed about 8Mbps down, compared to the usual ~20Mbps during the day. So iiNet lodged a fault, and said there was an appointment available on Thursday for someone to come out. I said fine, thank you, and plugged the router back in to the NTD.

Curiously, very shortly after this, everything suddenly went back to normal. If I was a deeply suspicious person, I’d imagine that because I’d just given the MAC address of my router to support, this enabled someone to reset something that was broken at the other end, and fix my connection. But nobody ever told me that anything like this happened; instead I received a phone call the next day to say that the “speed issue” I had reported was just regular congestion and that the tower was scheduled for an upgrade later in the year. I thanked them for the call, then pointed out that the symptoms of this particular issue were completely different to regular congestion and that I was sure that something had actually been broken, but I was left with the impression that this particular feedback would be summarily ignored.

I’m still convinced something was broken, and got fixed. I’d be utterly unsurprised if there had been some problem with the tower on the Sunday night, given the strong winds, and it took ’til mid-Tuesday to get it sorted. But we’ll never know, because NBN Co don’t publish information about congestion, scheduled upgrades, faults and outages anywhere the general public can see it. I’m not even sure they make this information consistently available to retail ISPs. My neighbour, who’s with a different ISP, sent me a notice that says there’ll be maintenance/upgrades occurring on April 18, then again from April 23-25. There’s nothing about this on iiNet’s status page when I enter my address.

There was one time in the past few years though, when there was an outage that impacted me, and it was listed on iiNet’s status page. It said “customers in the area of Herringback may be affected”. I initially didn’t realise that meant me, as I’d never heard for a suburb, region, or area called Herringback. Turns out it’s the name of the mountain our NBN tower is on.

LUV April 2019 Main Meeting: EasyBuild

Apr 2 2019 19:00
Apr 2 2019 21:00
Apr 2 2019 19:00
Apr 2 2019 21:00
Kathleen Syme Library, 251 Faraday Street Carlton VIC 3053


7:00 PM to 9:00 PM Tuesday, April 2, 2019
Kathleen Syme Library, 251 Faraday Street Carlton VIC 3053


  • Lev Lafayette: EasyBuild


Many of us like to go for dinner nearby after the meeting, typically at Brunetti's or Trotters Bistro in Lygon St.  Please let us know if you'd like to join us!

Linux Users of Victoria is a subcommittee of Linux Australia.

April 2, 2019 - 19:00

read more

LUV April 2019 Workshop: TBA

Apr 27 2019 12:30
Apr 27 2019 16:30
Apr 27 2019 12:30
Apr 27 2019 16:30
Infoxchange, 33 Elizabeth St. Richmond

Topic TBA.  Please feel free to email with suggestions!

There will also be the usual casual hands-on workshop, Linux installation, configuration and assistance and advice. Bring your laptop if you need help with a particular issue. This will now occur BEFORE the talks from 12:30 to 14:00. The talks will commence at 14:00 (2pm) so there is time for people to have lunch nearby.

The meeting will be held at Infoxchange, 33 Elizabeth St. Richmond 3121.  Late arrivals please call (0421) 775 358 for access to the venue.

LUV would like to acknowledge Infoxchange for the venue.

Linux Users of Victoria is a subcommittee of Linux Australia.

April 27, 2019 - 12:30

March 26, 2019

Brilliant Smart Wifi plug with Tasmota


A couple of weeks ago I was playing with Tuya derived smart light globes (Mirabella Genio from K-Mart Australia in my case, but there are a variety of other options as well). Now Bunnings has the Brillant Smart Wifi Plug on special for $20 and I decided to give that a go as well, given it is also Tuya derived.

The basic procedure for OTA flashing was the same as flashing the globes, except that you hold down the button on the device for five seconds to put it into flash mode. That all worked brilliantly, until I appear to have fat fingered my wifi details in Tasmota — when I rebooted the device it never appeared on my network.

That would be much more annoying on the globes, but it turns out these smart plugs are really easy to open and that Tuya has documented the pin out of the controlling microprocessor. So, I ended up temporarily soldering some cables to the microprocessor to debug what had gone wrong. It should be noted that as a soldering person I make a great software engineer:

Jumper wires soldered to the serial port.

Once you’ve connected with a serial console, its pretty obvious who can’t be trusted to type their wifi password correctly:

I can’t have nice things.

At this point I was in the process of working out how to use esptool to re-flash the plug when I got super lucky. However, first… Where the hell is GPIO0 (the way you turn on flashing mode)? Its not broken out on the pins for the MCU, but as a kind redditor pointed out, it is exposed on a pad on the back of the board:

The cunningly hidden GPIO0.

…and then I got lucky. You see, to put the MCU into flashing mode you short GPIO0 to ground. I was having trouble getting that to work with esptool, so I had a serial console attached to see what was happening. There I was, shorting GPIO0 out over and over trying to get the magic to work. However, Tasmota also setups up a button on GPIO0 by default, because the sonoffs have a hardware button on that pin. If you hit that button with four short presses, you put the device back into captive portal configuration mode

Once I was back in that mode I could just use a laptop over wifi to re-enter the wifi password and I’m good to go. In hindsight I didn’t need the serial port if I could have powered the device and shorted that pin four times, but it sure was nice to be told what was happening on the serial console while poking around.


March 22, 2019

Seeding sccache for Faster Brave Browser Builds

Compiling the Brave Browser (based on Chromium) on Linux can take a really long time and so most developers use sccache to cache objects files and speed up future re-compilations.

Here's the cronjob I wrote to seed my local cache every work day to pre-compile the latest builds:

30 23 * * 0-4   francois  /usr/bin/chronic /home/francois/bin/seed-brave-browser-cache

and here are the contents of that script:

set -e

# Set the path and sccache environment variables correctly
source ${HOME}/.bashrc-brave
export LANG=en_CA.UTF-8

cd ${HOME}/devel/brave-browser-cache

echo "Environment:"
echo "- HOME = ${HOME}"
echo "- PATH = ${PATH}"
echo "- PWD = ${PWD}"
echo "- SHELL = ${SHELL}"
echo "- BASH_ENV = ${BASH_ENV}"

echo $(date)
echo "=> Update repo"
git pull
npm install
npm run init

echo $(date)
echo "=> Delete any old build output"
rm -rf src/out

echo $(date)
echo "=> Debug build"
ionice nice timeout 4h npm run build

echo $(date)
echo "=>Release build"
ionice nice timeout 5h npm run build Release

echo $(date)
echo "=> Delete build output"
rm -rf src/out

March 16, 2019

I, Robot


Not the book of the movie, but the collection of short stories by Isaac Asimov. I’ve read this book several times before and enjoyed it, although this time I found it to be more dated than I remembered, both in its characterisations of technology as well as it’s handling of gender. Still enjoyable, but not the best book I’ve read recently.

I, Robot Book Cover I, Robot
Isaac Asimov

The development of robot technology to a state of perfection by future civilizations is explored in nine science fiction stories.


March 10, 2019

Running Home Assistant on Fedora with Docker

Home Assistant is a really great, open source home automation platform written in Python which supports hundreds of components. They have a containerised version called which can run on a bunch of hardware and has a built-in marketplace to make the running of addons (like Let’s Encrypt) easy.

I’ve been running Home Assistant on a Raspberry Pi for a couple of years, but I want something that’s more poweful and where I have more control. Here’s how you can use the official Home Assistant containers on Fedora (note that this does not include their marketplace).

First, install Fedora Server edition, which comes with the handy web UI for managing the system called Cockpit.

Once you’re up and running, install Docker and the Cockpit plugin.

sudo dnf install -y docker cockpit-docker

Now we can start and enable the Docker daemon and restart cockpit to load the Docker plugin.

sudo systemctl start docker && sudo systemctl enable docker
sudo systemctl restart cockpit

Create a location for the Home Assistant configuration and set the appropriate SELinux context. This lets you modify the configuration directly from the host and restart the container to pick up the change.

sudo mkdir -p /hass/config
sudo chcon -Rt svirt_sandbox_file_t /hass

Start up a container called hass using the Home Assistant Docker image which will start automatically thanks to the restart option. We pass through the /hass/config directory on the host as /config inside the container.

docker run --init -d \
--restart unless-stopped \
--name="hass" \
-v /hass/config/:/config \
-v /etc/localtime:/etc/localtime:ro \
--net=host \

You should be able to see the container starting up.

sudo docker ps

If you need to, you can get the logs for the container.

sudo docker logs hass

Once it’s started you should see port 8123 listening on the host

sudo ss -ltnp |grep 8123

Finally, enable port 8123 on the firewall to access the service on your network.

sudo firewall-cmd --zone=FedoraServer --add-port=8123/tcp
sudo firewall-cmd --runtime-to-permanent

Now browse to the IP address of your server on port 8123 and you should see Home Assistant. Create an account to get started!

March 09, 2019

Powerful Non Defensive Communication (PNDC)

How do we take the war out of our words? Sharon Strand Ellison's Powerful Non-Defensive Communication approach highlights our natural defensive reaction to feeling unsure, anxious or attacked when communicating with others.

She outlines these six different defensive reactions:

  1. Surrender-Betray
  2. Surrender-Sabotage
  3. Withdraw-Escape
  4. Withdraw-Entrap
  5. Counterattack-Justify
  6. Counterattack-Blame 

The first step toward neutralising our natural "fight, flight or freeze" response is to look within and identify our own individual "go to" response.

I know I tend to respond with counterattack-justify when I feel I'm being criticised, or challenged.  Learning to pause, and listen, is an ongoing project.
Ellison goes on to provide strategies we can all learn to use instead of our natural reflexes which can escalate to conflict.
She advocates three approaches:
  1. Questions
  2. Statements
  3. Predictions
Ellis book, "Taking the War out of our Words" is filled with examples, and exercises that will help you learn the skills to employ this approach.

March 08, 2019

Secure Data in Public Configuration Management With Propellor

Steampunk propeller ring by Daniel Proulx


List fields and contexts:

$ propellor --list-fields

Set a field for a particular context:

$ propellor --set 'SshAuthorizedKeys "myuser"' yourServers < authKeys

Dump a field from a specific context:

$ propellor --dump 'SshAuthorizedKeys "myuser"' yourServers

An Example

When using Propellor for configuration management, you can utilise GPG encryption to encrypt data sets. This enables you to leverage public git repositories for your centralised configuration management needs.

To list existing fields, you can run:

$ propellor --list-fields

which will not only list existing fields but will helpfully also list fields that would be used if set:

Missing data that would be used if set:
Field                             Context          Used by
-----                             -------          -------
'Password "myuser"'               'yourDesktops'
'CryptPassword "myuser"'          'yourServers'
'PrivFile "/etc/mail/dkim.key"'   'mailServers'    your.mail.server

You can set these fields with input from either STDIN or files prepared earlier.

For example, if you have public SSH keys you wish to distribute, you can place then into a file then use that file to populate the fields of an appropriate context. The contents of an example authorized_keys, we'll call authKeys, may look like this:

ssh-ed25519 eetohm9doJ4ta2Joo~P2geetoh6aBah9efu4ta5ievoongah5feih2eY4fie9xa1ughi you@host1
ssh-ed25519 choi7moogh<i2Jie6uejoo6ANoMei;th2ahm^aiR(e5Gohgh5Du-oqu1roh6Mie4shie you@host2
ssh-ed25519 baewah%vooPho2Huofaicahnob=i^ph;o1Meod:eugohtiuGeecho2eiwi.a7cuJain6 you@host3

To add these keys to the appropriate users for the hosts of a particular context you could run:

$ propellor --set 'SshAuthorizedKeys "myuser"' yourServers < authKeys

To verify that the fields for this context have the correct data, you can dump it:

$ propellor --dump 'SshAuthorizedKeys "myuser"' yourServers
gpg: encrypted with 256-bit ECDH key, ID 5F4CEXB7GU3AHT1E, created 2019-03-08
      "My User <myuser@my.domain.tld>"
      ssh-ed25519 eetohm9doJ4ta2Joo~P2geetoh6aBah9efu4ta5ievoongah5feih2eY4fie9xa1ughi you@host1
      ssh-ed25519 choi7moogh<i2Jie6uejoo6ANoMei;th2ahm^aiR(e5Gohgh5Du-oqu1roh6Mie4shie you@host2
      ssh-ed25519 baewah%vooPho2Huofaicahnob=i^ph;o1Meod:eugohtiuGeecho2eiwi.a7cuJain6 you@host3

When you next spin Propellor for the desired hosts, those SSH public keys with be installed into the authorized_keys_ filefor the user myuser for hosts that belong to the allServers context.

Setting and Storing Passwords

One of the most obvious and practical uses of this feature is to set secure data that needs to be distributed, such as passwords or certificates. We'll use passwords for this example.

Create a hash of the password you wish to distribute:

$ mkpasswd -m sha-512 > /tmp/deleteme
$ cat /tmp/deleteme

Now that we have that file, we can use it as input for Propellor:

$ propellor --set 'CryptPassword "myuser"' yourServers < /tmp/deleteme
Enter private data on stdin; ctrl-D when done:
gpg: encrypted with 256-bit ECDH key, ID 5F4CEXB7GU3AHT1E, created 2019-03-08
      "My User <myuser@my.domain.tld>"
gpg: WARNING: standard input reopened
Private data set.

Tidy up:

$ rm /tmp/deletem

You're now ready to deploy that password for that user to those servers.

Mirabella Genio smart lights with Tasmota and Home Assistant


One of the things I like about Home Assistant is that it allows you to take hardware from a bunch of various vendors and stitch it together into a single consistent interface. So for example I now have five home automation vendor apps on my phone, but don’t use any of them because Home Assistant manages everything.

A concrete example — we have Philips Hue lights, but they’re not perfect. They’re expensive, require a hub, and need to talk to a Philips data centre to function (i.e. the internet needs to work at my house, which isn’t always true thanks to the failings of the Liberal Party).

I’d been meaning to look at the cheapo smart lights from Kmart for a while, and finally got around to it this week. For $15 you can pickup a dimmable white globe, and for $29 you can have a RGB one. That’s heaps cheaper than the Hue options. Even better, the globes are flashable to run the open source Tasmota stack, which means no web services required!

So here are some instructions on flashing these globes to be useful:

Buy the globes. I bought this warm while dimmable and this RBG option.

Flash to tasmota. This was a little bit fiddly, but was mostly about getting the sequence to put the globes into config mode right (turn off for 10 seconds, turn on, turn off, turn on, turn off, turn on). Wait a few seconds and then expect the lamp to blink rapidly indicating its in config mode. For Canberra people I now have a raspberry pi setup to do this easily, so we can run a flashing session sometime if people want.

Configure tasmota. This is really up to you, but the globes need to know local wifi details, where your MQTT server is, and stuff like that.

And then configure Home Assistant. The example of how to do that from my house is on github.


March 06, 2019

Authentication in WordPress

WebAuthn is now a W3C recommendation, bringing us one step closer to not having to use passwords anymore. If you’re not familiar with WebAuthn, here’s a little demo (if you don’t own a security key, it’ll probably work best on an Android phone with a fingerprint reader).

That I needed to add a disclaimer for the demo indicates the state of WebAuthn authenticator support. It’s nice when it works, but it’s clearly still in progress, and that progress varies. WebAuthn also doesn’t cover how the authenticator device works, that falls under the proposed CTAP standard. They work together to form the FIDO2 Project. Currently, the most reliable option is to purchase a security key, but quality varies wildly, and needing to carry around an extra dongle just for logging in to sites is no fun.

What WordPress Needs

Anything that replaces passwords needs to provide some extra benefit, without losing the strengths of the password model:

  • Passwords are universally understood as an authentication model.
  • They’re portable: you don’t need a special app or token to use them anywhere.
  • They’re extendable: strong passwords can be enforced as needed. Additional authentication (2FA codes, for example) can be added, too.

Magic login links are an interesting step in this direction. The WordPress mobile apps added magic login support for accounts a while ago, I’d love to see this working on all WordPress sites.

A WebAuthn-based model would be a wonderful future step, once the entire user experience is more polished.

The password-less future hasn’t quite arrived yet, but we’re getting closer.

March 04, 2019

What If?


More correctly titled “you die horribly and it probably involves plasma”, this light hearted and fun read explores serious answers to silly scientific questions. The footnotes are definitely the best bit. A really enjoyable read.

What If? Book Cover What If?
Randall Munroe
Houghton Mifflin Harcourt
September 2, 2014

The creator of the incredibly popular webcomic xkcd presents his heavily researched answers to his fans' oddest questions, including “What if I took a swim in a spent-nuclear-fuel pool?” and “Could you build a jetpack using downward-firing machine guns?”


Audiobooks – February 2019

Tamed: Ten Species that Changed our World by Alice Roberts

Plenty of content (14 hours) and not too dumbed down. About 8 of the 10 species are the ones you’d expect. 8/10

It Won’t Be Easy: An Exceedingly Honest (and Slightly Unprofessional) Love Letter to Teaching by Tom Rademacher

A breezy little book about the realities of teaching (at least in the US). Interesting to outsiders & hopefully useful to those in the profession. 7/10

The Hobbit by J. R. R Tolkien, Read by Rob Inglis

A good audio-edition of the book. Unabridged & really the default one for most people. I alternated chapters of this with the excellent Prancing Pony Podcast commentaries on those chapters. 9/10

The Life of Greece: The Story of Civilization, Volume 2 (The Story of Civilization series) by Will Durant

32 hours on the history of Ancient Greece. Seemed to cover just above everything. Written in the 1930s so probably a little out-of-date in places. 7/10


March 03, 2019

Problems with Dreamhost


This site is hosted at Dreamhost, and for reasons I can’t explain right now isn’t accessible from large chunks of Australia. It seems to work fine from elsewhere though. Dreamhost certainly has an explaination — they allege in their emails that take 24 hours that you can’t reply to that its because wordpress is using too much RAM.

However, they don’t explain why that’s suddenly happened when its been previously fine for years, and they certainly don’t explain why it works from some places but not others and why other Dreamhost sites are also offline from the sites having issues.

Its time for a new hosting solution I think, although not bothering to have hosting might also be that solution.


LPCNet Quantiser – wideband speech at 1700 bits/s

I’ve been working with Neural Net (NN) speech synthesis using LPCNet.

My interest is digital voice over HF radio. To get a NN codec “on the air” I need a fully quantised version at 2000 bit/s or below. The possibility of 8kHz audio over HF radio is intriguing, so I decided to experiment with quantising the LPCNet features. These consist of 18 spectral energy samples, pitch, and the pitch gain which is effectively a measure of voicing.

So I have built a Vector Quantiser (VQ) for the DCT-ed 18 log-magnitude samples. LPCNet updates these every 10ms, which is a bit too fast for my target bit rate. So I decimate to say 30ms, then use linear interpolation to reconstruct the 10ms frames at the decoder. The spectrum changes slowly (most of the time), so I quantise the difference between frames to save a few bits.

Detailed Results

I’ve developed a script that generates a bunch of samples, plots various statistics, and builds a HTML page to summarise the results. Here is the current page, including samples for the fully quantised prototype codec at three bit rates between around 2000 and 1400 bits/s. If anyone would like more explanation of that page, just ask.

Discussion of Results

I can hear “birch” losing some quality at the 20ms decimation step. When training my own NN, I have had quite a bit of trouble with very rough speech when synthesising “canadian”. I’m learning that roughness in NN synthesis means more training required, the network just hasn’t experienced this sort of speaker before. The “canadian” sample is quite low pitch so I may need some more training material with low pitch speakers.

My quantisation scheme works really well on some of the carefully spoken Harvard sentences (oak, glue), in ideal recording conditions. However with more realistic, quickly spoken speech with real world background noise (separately, wanted) it starts to sound vocoder-ish (albeit a pretty good vocoder).

One factor is the frame rate decimation from 10 to 20-30ms, which I used to get the bit rate beneath 2000 bit/s. A better quantisation scheme, or LPCNet running on 20ms frames could improve this. Or we could just run it at greater that 2000 bit/s (say for VHF/UHF two way radio).

Comparison to Wavenet

Source Listen
Wavenet, Codec 2 encoder, 2400 bits/s Listen
LPCnet unquantised, 10ms frame rate Listen
Quantised to 1733 bits/s (44bit/30ms) Listen

The “separately” sample from the Wavenet team sounds better to me. Ironically, the these samples use my Codec 2 encoder, running at just 8kHz! It’s difficult to draw broad conclusions from this, as we don’t have access to a Wavenet system to try many different samples. All codecs tend to break down under certain conditions and samples.

However it does suggest (i) we can eventually get higher quality from NN synthesis and (ii) it is possible to encode high quality wideband speech with features covering a narrow spectral range (e.g. 200-3800Hz for the Codec 2 encoder). The 18 element vectors (covering DC to 8000Hz) I’m currently using ultimately set the bit rate of my current system. After a few VQ stages the elements are independent Gaussians and reduction in quantiser noise is very slow as bits are added.

The LPCNet engine has several awesome features: it’s open source, runs in real time on regular CPUs, and is available for us to test on wide variety of samples. The speech quality I am achieving with even my first attempts is rather good compared to any other speech codecs I have played with at these bit rates – in either the open or closed source worlds.

Tips and Observations

I’ve started training my own models, and discovered that if you get rough speech – you probably need more data. For example when I tried training on 1E6 vectors, I had a few samples sounding rough when I tested the network. However with 5E6 vectors, it works just fine.

The LPCNet dump_data –train mode program helps you by being very clever. It “fuzzes” the speech frequency, gain, and adds a little noise. If the NN hasn’t experienced a particular combination of features before, it tends to get lost – and you get rough sounding speech.

I found that 10 Epochs of 5E6 vectors gives me good speech quality on my test samples. That takes about a day with my somewhat underpowered GPU. In fact, most of the training seems to happen on the first few Epochs:

Here is a plot of the training and validation loss for my training database:

This plot shows how much the loss changes on each Epoch, not very much, but not zero. I’m unsure if these small gains lead to meaningful improvements over many Epochs:

I looked into the LPCNet pitch and voicing estimation. Like all estimators (including those in Codec 2), they tend to make occasional mistakes. That’s what happen when you try to fit neat signal processing models to real-world biological signals. Anyway, the amazing thing is that LPCNet doesn’t care very much. I have some samples where pitch is all over the place but the speech still sounds OK.

This is really surprising to me. I’ve put a lot of time into the Codec 2 pitch estimators. Pitch errors are very obvious in traditional, model based low bit rate speech codecs. This suggest that with NNs we can get away with less pitch information – which means less bits and better compression. Same with voicing. This leads to intriguing possibilities for very low bit (few 100 bit/s) speech coding.

Conclusions, Further Work and FreeDV 2020

Overall I’m pleased with my first attempt at quantisation. I’ve learnt a lot about VQ and NN synthesis and carefully documented (and even scripted) my work. The learning and experimental experience has been very satisfying.

Next I’d like to get one of these candidates on the air, see how it sounds over real world digital radio channels, and find out what happens when we get bit errors. I’m a bit nervous about predictive quantisation on radio channels, as it causes errors to propagate in time. However I have a good HF modem and FEC, and some spare bits to add some non-predictive quantisation if needed.

My design for a new, experimental “FreeDV 2020” mode employing LPCNet uses just 1600 Hz of RF bandwidth for 8kHz bandwidth speech, and should run at 10dB SNR on a moderate fading channel.

Here is a longer example of LPCNet at 1733 bit/s compared to HF SSB at a SNR of 10dB (we can send error free LPCNet through a similar HF channel). The speech sample is from the MP3 source of the Australian weekly WIA broadcast:

Source Listen
SSB simulation at 10dB SNR Listen
LPCNet Quantised to 1733 bits/s (44bit/30ms) Listen
Mixed LPCNet Quantised and SSB (thanks Peter VK2TPM!) Listen

This is really new technology, and there is a lot to explore. The work presented here represents my initial attempt at quantisation with the LPCNet synthesis engine, and is hopefully useful for other people who would like to experiment in the area.


Thanks Jean-Marc for developing the LPCnet technology, making the code open source, and answering my many questions.


LPCnet introductory page.

The source code for my quantisation work (and notes on how to use it) is available as a branch on the GitHub LPCNet repo.

WaveNet and Codec 2

March 01, 2019

Connecting a VoIP phone directly to an Asterisk server

On my Asterisk server, I happen to have two on-board ethernet boards. Since I only used one of these, I decided to move my VoIP phone from the local network switch to being connected directly to the Asterisk server.

The main advantage is that this phone, running proprietary software of unknown quality, is no longer available on my general home network. Most importantly though, it no longer has access to the Internet, without my having to firewall it manually.

Here's how I configured everything.

Private network configuration

On the server, I started by giving the second network interface a static IP address in /etc/network/interfaces:

auto eth1
iface eth1 inet static

On the VoIP phone itself, I set the static IP address to and the DNS server to I then updated the SIP registrar IP address to

The DNS server actually refers to an unbound daemon running on the Asterisk server. The only configuration change I had to make was to listen on the second interface and allow the VoIP phone in:

    access-control: refuse
    access-control: allow
    access-control: allow

Finally, I opened the right ports on the server's firewall in /etc/network/iptables.up.rules:

-A INPUT -s -p udp --dport 5060 -j ACCEPT
-A INPUT -s -p udp --dport 10000:20000 -j ACCEPT

Accessing the admin page

Now that the VoIP phone is no longer available on the local network, it's not possible to access its admin page. That's a good thing from a security point of view, but it's somewhat inconvenient.

Therefore I put the following in my ~/.ssh/config to make the admin page available on http://localhost:8081 after I connect to the Asterisk server via ssh:

Host asterisk
    LocalForward 8081

February 28, 2019

LUV March 2019 Workshop: 30th Anniversary of the Web / Federated Social Media

Mar 16 2019 12:30
Mar 16 2019 16:30
Mar 16 2019 12:30
Mar 16 2019 16:30
Infoxchange, 33 Elizabeth St. Richmond

This month we will celebreate the 30th anniversary of the World Wide Web with a discussion of its past, present and future.  Andrew Pam will also demonstrate and discuss the installation, operation and use of federated social media platforms including Diaspora and Hubzilla.

The meeting will be held at Infoxchange, 33 Elizabeth St. Richmond 3121.  Late arrivals please call (0421) 775 358 for access to the venue.

LUV would like to acknowledge Infoxchange for the venue.

Linux Users of Victoria is a subcommittee of Linux Australia.

March 16, 2019 - 12:30

read more

LUV March 2019 Main Meeting: ZeroTier / Ethics in the computer realm

Mar 5 2019 19:00
Mar 5 2019 21:00
Mar 5 2019 19:00
Mar 5 2019 21:00
Kathleen Syme Library, 251 Faraday Street Carlton VIC 3053


7:00 PM to 9:00 PM Tuesday, March 5, 2019
Kathleen Syme Library, 251 Faraday Street Carlton VIC 3053


  • Adrian Close: ZeroTier
  • Enno Davids: Ethics - Weasel words, weasel words, weasel words, bad thing


Many of us like to go for dinner nearby after the meeting, typically at Brunetti's or Trotters Bistro in Lygon St.  Please let us know if you'd like to join us!

Linux Users of Victoria is a subcommittee of Linux Australia.

March 5, 2019 - 19:00

read more

Propagating Native Plants

by Alec M. Blomberry & Betty Maloney

Propagating Australian Plants

I have over 1,000 Diospyrus Geminata (scaly ebony) seedlings growing in my shade house (in used toilet rolls). I'd collected the seeds from the (delicious) fruit in late 2017 (they appear to fruit based on rainfall - no fruit in 2018) and they're all still rather small.

All the literature stated that they were slow growing. I may have been more dismissive of this than I needed to be.

I'm growing these for landscape scale planting, it's going to be a while between gathering the seeds (mid 2017) and planting the trees (maybe mid 2019).

So I needed to look into other forms of propagation and either cutting or aerial layering appear to be the way to go, as I already have large numbers of of mature Diospyros Geminata on our property or nearby.

The catch being that I know nothing of either cutting or aerial layering and in particular I want to do this at a reasonable scale (ie: possibly thousands).

So this is where Propagating Australian Plants comes in.

Aerial Layering

It's a fairly dry and academic read, that feels like it may be more of an introductory guide for botanic students than a lay person such as myself.

Despite being last published in 1994 by a publisher that no longer exists and having a distinct antique feel to it, the information within is crisp and concise with clear and helpful illustrations.

Highly recommended if you're starting to propagate natives as I am.

Although I wish you luck picking up a copy - I got mine from an Op Shop. At least the National Library of Australia appears to have a copy.

February 27, 2019

FreeDV QSO Party 2019

My local radio club, the Amateur Radio Experimenters Group (AREG), have organised a special FreeDV QSO Party Weekend from April 27th 0300z to April 28th 0300z 2019. This is a great chance to try out FreeDV, work Australia using open source HF digital voice, and even talk to me!

All the details including frequencies, times and the point scoring system over on the AREG site.

February 26, 2019

5 axis cnc fun!

The 5th axis build came together surprisingly well. I had expected much more resistance getting the unit to be known to both fusion360 and LinuxCNC. There is still some tinkering to be done for sure but I can get some reasonable results already. The video below gives an overview of the design:

Shown below is a silent movie of a few test jobs I created to see how well tool contact would be maintained during motion in A and B axis while the x,y,z are moved to keep the tool in the right position. This is the flow toolpath in Fusion360 in action. Non familiarity with these CAM paths makes for a learning curve which is interesting when paired with a custom made 5th that you are trying to debug at the same time.

I haven't tested how well the setup works when cutting harder materials like alloy yet. It is much quieter and forgiving to test cutting on timber and be reasonably sure about the toolpaths and that you are not going to accidentally crash to deep into the material after a 90 degree rotation.

New Dark Age: Technology and the End of the Future

by James Bridle

New Dark Age: Technology and the End of the Future

It may be my first book for 2019 but I'm going to put it out there, this is my must read book for 2109 already. Considering it was published in 2018 to broad acclaim, it may be a safe call.

tl;dr; Read this book. It's well resourced, thoroughly referenced, well thought out with the compiled information and lines drawn potentially redrawing the way you see our industry and the world. If you're already across the issues, the hard facts will still cause you to draw breath.

I read this book in bursts over 4 weeks. Each chapter packing it's own informative punch. The narrative first grabbed my attention on page 4 where the weakness of learning to code alone was fleshed out.

"Computational thinking is predominant in the world today, driving the worst trends in our societies and interactions, and must be opposed by real systemic literacy." page 4

Where it is argued that systemic literacy is much more important than learning to code - with a humorous but fitting plumbing analogy in the mix.

One of the recurring threads in the book is the titular "Dark New Age", with points being drawn back to the various actions present in modern society that are actively reducing our knowledge.

"And so we find ourselves today connected to vast repositories of knowledge, and yet we have not learned to think. In fact, the opposite is true: that which was intended to enlighten the world in practice darkens it. The abundance of information and the plurality of world-views now accessible to us through the Internet are not producing a coherent consensus reality, but one riven by fundamentalist insistence on simplistic narratives, conspiracy theories, and post-factual politics." page 10

Also covered are more well known instances of corporate and government censorship, the traps of the modern convenience technologies.

"When an ebook is purchased from an online service, it remains the property of the seller, it's loan subject to revocation at any time - as happened when Amazon remotely deleted thousands of copies of 1984 and Animal Farm from customers' Kindles in 2009. Streaming music and video services filter the media available by legal jurisdiction and algorithmically determine 'personal' preferences. Academic journals determine access to knowledge by institutional affiliation and financial contribution as physical, open-access libraries close down." page 39

It was the "Climate" chapter that packed the biggest punch for me, as an issue I considered myself rather well across over the last 30 years, it turns out there was a significant factor I'd missed. A hint that surprise was coming came in an interesting diversion into clear air turbulence.

"An advisory circular on preventing turbulence-related injuries, published by the US Federal Aviation Administration in 2006, states that the frequency of turbulence accidents has increased steadily for more than a decade, from 0.3 accidents per million flights in 1989 to 1.7 in 2003." page 68

The reason for this increase was laid at the feet of increased CO2 levels in the atmosphere by Paul Williams of the National Centre for Atmospheric Science and the implications were expounded upon in his paper Nature Climate Change (2013) thusly:

" winter, most clear air turbulence measures show a 10-40 per cent increase in the median strength...40-70 per cent increase in the frequency of occurrence of moderate or greater turbulence." page 69

The real punch in the guts came on page 73, where I first came across the concept of "Peak Knowledge" and how the climate change was playing it's defining role in that decline, where President of the American Meteorological Society William B Gail wonders if:

"we have already passed through 'peak knowledge", just as we may have already passed 'peak oil'." page 73

Wondering what that claim was based on, the next few paragraphs of information can be summarised in the following points:

  • From 1000 - 1750 CE CO2 was at 275-285 parts / million.
  • 295ppm by the start of the 20th century
  • 310ppm by 1950
  • 325ppm in 1970
  • 350ppm in 1988
  • 375ppm by 2004
  • 400ppm by 2015 - the first time in 800,000 years
  • 1,000ppm is projected to be passed by the end of this century.

"At 1,000ppm, human cognitive ability drops by 21%" page 74

Then a couple of bombshells:

"CO2 already reaches 500ppm in industrial cities"

"indoors in poorly ventilated schools, homes and workplaces it can regularly exceed 1,000ppm - substantial numbers of schools in California and Texas measured in 2012 breached 2,000ppm."

The implications of this are fairly obvious.

All this is by the end of chapter 3. It's a gritty, honest look at where we're at and where going. It's not pretty but as the old saying goes, to be forewarned is to be forearmed.

Do yourself a favour, read it.

A (simplified) view of OpenPOWER Firmware Development

I’ve been working on trying to better document the whole flow of code that goes into a build of firmware for an OpenPOWER machine. This is partially to help those not familiar with it get a better grasp of the sheer scale of what goes into that 32/64MB of flash.

I also wanted to convey the components that we heavily re-used from other Open Source projects, what parts are still “IBM internal” (as they relate to the open source workflow) and which bits are primarily contributed to by IBMers (at least at this point in time).

As such, let’s start with the legend of the diagram:

Now, the diagram:

Simplified development flow for OpenPOWER firmware

The end thing that a user with a machine will download and apply (or that comes shipped with a box) is the purple “Installable Firmware Release” nodes (bottom center). In this diagram, there are 4 of them. One for POWER9 systems such as the just-announced AC922 system (this is the “OP910 Release” node, which is the witherspoon_defconfig in the op-build tree); one for the p9dsu platform (p9dsu_defconfig in op-build) and one is for IBM FSP based systems such as the S812L and S822L systems (or S812/S822 in OPAL mode).

There are more platforms out there, but this diagram is meant to be simplified. The key difference with the p9dsu platform is that this is produced by somebody other than IBM.

All of these releases are based off the upstream op-build project, op-build is the light blue box in the center of the diagram. We do regular X.Y releases and sometimes do X.Y.Z releases. It’s primarily a pull request based workflow currently, so everything goes via a pull request. The op-build project brings together all the POWER specific firmware components (pretty much everything in every other light blue/blue box) along with a Linux kernel and buildroot.

The kernel and buildroot are the two big yellow boxes on the top right. Buildroot brings together a lot of open source components that are in our firmware image (including some power specific ones that we get through upstream buildroot).

For Linux, this is a pretty simplified view of the process, but we primarily ship the stable tree (with maybe up to half a dozen patches).

The skiboot and petitboot components both use a mailing list based workflow (similar to kernel) as well as X.Y and X.Y.Z releases (again, similar to the linux kernel).

On the far left of the diagram, we have Hostboot, SBE and OCC. These are three firmware components that come from the traditional IBM POWER Firmware group, and are shared with the IBM non-OpenPOWER POWER systems (“traditional” POWER). These components have part of their code from from an (internal) repository called “ekb” which also goes into a (very) low level debug tool and the FSP based systems. There’s also an (internal) gerrit instance that’s the primary place where code review/development discussions are for these components.

In future posts, I’ll probably delve into more specifics of the current development process, and how we may try and change things for the better.

How I do email (at work)

Recently, I blogged on my home email setup and in that post, I hinted that my work setup was rather different. I have entirely separate computing devices for work and personal, a setup I strongly recommend. This also lets me “go home” from work even when working from home, I use a different physical machine!

Since I work for IBM I have (at least) two email accounts for work: a Lotus Notes one and a internet standards compliant one. It’s “easy” enough to get the Notes one to forward to the standards compliant one, from which I can just use fetchmail or similar to pull down mail.

I run mail through a rather simple procmail script: it de-mangles some URL mangling that can happen in the current IBM email infrastructure, runs things through SpamAssassin and deliver to a date based Maildir (or one giant pile for spam).

My ~/.procmailrc looks something like this:


DATE=`date +"%Y%m"`

| magic_script_to_unmangle_things

| spamc

* ^X-Spam-Status: Yes

I use tail -f mail_log as a really dumb kind of biff replacement.

Now, what do I read and write mail with? Notmuch! It is the only thing that even comes close to being able to deal with a decent flow of mail. I have a couple of saved searches just to track how much mail I pull in a day/week. Today (on Monday), it says 442 today and 10,403 over the past week.

For the most part, my workflow is kind of INBOX-ZERO like, except that I currently view victory as INBOX 2000. Most mail does go into my INBOX, the notable exceptions are two main mailing lists I’m subscribed to mostly as FYI and to search/find things when needed. Those are the Linux Kernel Mailing List (LKML) and the buildroot mailing list. Why notmuch rather than just searching the web for mailing list archives? Notmuch can return the result of a query in less time it takes light to get to and from the United States in ideal conditions.

For work, I don’t sync my mail anywhere. It’s just on my laptop. Not having it on my phone is a feature. I have a notmuch post-new hook that does some initial tagging of mail, and as such I have this in my ~/.notmuch-config:


My post-new hook looks like this:


# immediately archive all messages from "me"
notmuch tag -new -- tag:new and

# tag all message from lists
notmuch tag +devicetree +list -- tag:new and
notmuch tag +inbox +unread -new -- tag:new and tag:devicetree and
notmuch tag +listinbox +unread +list -new -- tag:new and tag:devicetree and not

notmuch tag +linuxppc +list -- tag:new and
notmuch tag +linuxppc +list -- tag:new and
notmuch tag +inbox +unread -new -- tag:new and tag:linuxppc
notmuch tag +openbmc +list -- tag:new and
notmuch tag +inbox +unread -new -- tag:new and tag:openbmc

notmuch tag +lkml +list -- tag:new and
notmuch tag +inbox +unread -new -- tag:new and tag:lkml and
notmuch tag +listinbox +unread -new -- tag:new and tag:lkml and not tag:linuxppc and not

notmuch tag +qemuppc +list -- tag:new and
notmuch tag +inbox +unread -new -- tag:new and tag:qemuppc and
notmuch tag +listinbox +unread -new -- tag:new and tag:qemuppc and not

notmuch tag +qemu +list -- tag:new and
notmuch tag +inbox +unread -new -- tag:new and tag:qemu and
notmuch tag +listinbox +unread -new -- tag:new and tag:qemu and not

notmuch tag +buildroot +list -- tag:new and
notmuch tag +buildroot +list -- tag:new and
notmuch tag +buildroot +list -- tag:newa nd
notmuch tag +inbox +unread -new -- tag:new and tag:buildroot and
notmuch tag +listinbox +unread -new -- tag:new and tag:buildroot and not

notmuch tag +ibmbugzilla -- tag:new and

# finally, retag all "new" messages "inbox" and "unread"
notmuch tag +inbox +unread -new -- tag:new

This leaves me with both an inbox and a listinbox. I do not look at the overwhelming majority of mail that hits the listinbox – It’s mostly for following up on individual things. If I started to need to care more about specific topics, I’d probably add something in there for them so I could easily find them.

My notmuch emacs setup has a bunch of saved searches, so my notmuch-hello screen looks something like this:

This gets me a bit of a state-of-the-world-of-email-to-look-at view for the day. I’ll often have meetings first thing in the morning that may reference email I haven’t looked at yet, and this generally lets me quickly find mail related to the problems of the day and start being productive.

How I do email (at home)

I thought I might write something up on how I’ve been doing email both at home and at work. I very much on purpose keep the two completely separate, and have slightly different use cases for both of them.

For work, I do not want mail on my phone. For personal mail, it turns out I do want this on my phone, which is currently an Android phone. Since my work and personal email is very separate, the volume of mail is really, really different. Personal mail is maybe a couple of dozen a day at most. Work is… orders of magnitude more.

Considering I generally prefer free software to non-free software, K9 Mail is the way I go on my phone. I have it set up to point at the IMAP and SMTP servers of my mail provider (FastMail). I also have a google account, and the gmail app works fine for the few bits of mail that go there instead of my regular account.

For my mail accounts, I do an INBOX ZERO like approach (in reality, I’m pretty much nowhere near zero, but today I learned I’m a lot closer than many colleagues). This means I read / respond / do / ignore mail and then move it to an ARCHIVE folder. K9 and Gmail both have the ability to do this easily, so it works well.

Additionally though, I don’t want to care about limits on storage (i.e. expire mail from the server after X days), nor do I want to rely on “the cloud” to be the only copy of things. I also don’t want to have to upload any of past mail I may be keeping around. I also generally prefer to use notmuch as a mail client on a computer.

For those not familiar with notmuch, it does tags on mail in Maildir, is extremely fast and can actually cope with a quantity of mail. It also has this “archive”/INBOX ZERO workflow which I like.

In order to get mail from FastMail and Gmail onto a machine, I use offlineimap. An important thing to do is to set “status_backend = sqlite” for each Account. It turns out I first hacked on sqlite for offlineimap status a bit over ten years ago – time flies. For each Account I also set presynchook = ~/Maildir/maildir-notmuch-presync (below) and a postsynchook = notmuch new. The presynchook is run before we sync, and its job is to move files around based on the tags in notmuch and the postsynchook lets notmuch catch any new mail that’s been fetched.

My maildir-notmuch-presync hook script is:

notmuch search --output=files not tag:inbox and folder:fastmail/INBOX|xargs -I'{}' mv '{}' "$HOME/Maildir/INBOX/fastmail/Archive/cur/"

notmuch search --output=files folder:fastmail/INBOX and tag:spam |xargs -I'{}' mv '{}' "$HOME/Maildir/INBOX/fastmail/Spam/cur/"
ARCHIVE_DIR=$HOME/Maildir/INBOX/`date +"%Y%m"`/cur/
mkdir -p $ARCHIVE_DIR
notmuch search --output=files folder:fastmail/Archive and date:..90d and not tag:flagged | xargs -I'{}' mv '{}' "$ARCHIVE_DIR"

# Gmail
notmuch search --output=files not tag:inbox and folder:gmail/INBOX|grep 'INBOX/gmail/INBOX/' | xargs -I'{}' rm '{}'
notmuch search --output=files folder:gmail/INBOX and tag:spam |xargs -I'{}' mv '{}' "$HOME/Maildir/INBOX/gmail/[Gmail].Spam/cur/"

So This keeps 90 days of mail on the fastmail server, and archives older mail off into month based archive dirs. This is simply to keep directory sizes not too large, you could put everything in one directory… but at some point that gets a bit silly.

I don’t think this is all the most optimal setup I could have, but it does let me read and answer mail on my phone and desktop (as well as use a web client if I want to). There is a bit of needless copying of messages by offlineimap under certain circumstances, but I don’t get enough personal mail for it to be a problem.

pwnm-sync: Synchronizing Patchwork and Notmuch

One of the core bits of infrastructure I use as a maintainer is Patchwork (I wrote about making it faster recently). Patchwork tracks patches sent to a mailing list, allowing me as a maintainer to track the state of them (New|Under Review|Changes Requested|Accepted etc), combine them into patch bundles, look at specific series, test results etc.

One of the core bits of software I use is my email client, notmuch. Most other mail clients are laughably slow and clunky, or just plain annoying for absorbing a torrent of mail and being able to deal with it or just plain ignore it but have it searchable locally.

You may think your mail client is better than notmuch, but you’re wrong.

A key feature of notmuch is tagging email. It doesn’t do the traditional “folders” but instead does tags (if you’ve used gmail, you’d be somewhat familiar).

One of my key work flows as a maintainer is looking at what patches are outstanding for a project, and then reviewing them. This should also be a core part of any contributor to a project too. You may think that a tag:unread and to:project-list@foo query would be enough, but that doesn’t correspond with what’s in patchwork.

So, I decided to make a tool that would add tags to messages in notmuch corresponding with the state of the patch in patchwork. This way, I could easily search for “patches marked as New in patchwork” (or Under Review or whatever) and see what I should be reviewing and looking at merging or commenting on.

Logically, this wouldn’t be that hard, just use the (new) Patchwork REST API to get the state of everything and issue the appropriate notmuch commands.

But just going one way isn’t that interesting, I wanted to be able to change the tags in notmuch and have them sync back up to Patchwork. So, I made that part of the tool too.

Introducing pwnm-sync: a tool to sync patchwork and notmuch.

notmuch-hello tag counts for pwnm-sync tagged
patches in patchwork
notmuch-hello tag counts for pwnm-sync tagged
patches in patchwork

With this tool I can easily see the patchwork state of any patch that I have in my notmuch database. For projects that I’m a maintainer on (i.e. can change the state of patches), If I update the patches of that email and run pwnm-sync again, it’ll update the state in patchwork.

I’ve been using this for a few weeks myself and it’s made my maintainer workflow significantly nicer.

It may also be useful to people who want to find what patches need some review.

The sync time is mostly dependent on how fast your patchwork instance is for API requests. Unfortunately, we need to make some improvements on the Patchwork side of things here, but a full sync of the above takes about 4 minutes for me. You can also add a –epoch option (with a date/time) to say “only fetch things from patchwork since that date” which makes things a lot quicker for incremental syncs. For me, I typically run it with an epoch of a couple of months ago, and that takes ~20-30 seconds to run. In this case, if you’ve locally updated a old patch, it will still sync that change up to patchwork.

Implementation Details

It’s a python3 script using the notmuch python bindings, the requests-futures module for asynchronous HTTP requests (so we can have the patchwork server assemble the next page of results while we process the previous one), and a local sqlite3 database to store state in so we can work out what changed locally / server side.

Getting it

Head to or just:

git clone

Optimizing database access in Django: A patchwork story

tl;dr: I made Patchwork a lot faster by looking at what database queries were being generated and optimizing them either by making Django produce better queries or by adding better indexes.

Introduction to Patchwork

One of the key bits of infrastructure a bunch of maintainers of Open Source Software use is a tool called Patchwork. We use it for a bunch of OpenPOWER firmware development, several Linux subsystems use it as well as

The purpose of Patchwork is to supplement the patches-to-a-mailing-list development work flow. It allows a maintainer to see all the patches that have been posted on the list, How many Acked-by/Reviewed-by/Tested-by replies they have, delegate responsibility for the patch to a co-maintainer, track (and change) the state of the patch (e.g. to “Under Review”, “Changes Requested”, or “Accepted”), and create bundles of patches to help in review and testing.

Since patchwork is an open source project itself, there’s several instances of it out there in common use. One of the main instances is which is (funnily enough) used by a bunch of people connected to OzLabs for projects that are somewhat connected to OzLabs. e.g. the linuxppc-dev project and the skiboot and petitboot projects. There’s also a instance, which is used by some kernel subsystems.

Recent versions of Patchwork have added some pretty cool features such as the ability to integrate with CI systems such as Snowpatch which helps maintainers see if patches submitted are likely to break things.

Unfortunately, there’s also been some complaints that recent version of patchwork have gotten slower than previous ones. This may well be the case, or it could just be that the volume of patches is much higher and there’s load on the database. Anyway, after asking a few questions about what the size and scope was of the patchwork database on, I went “hrm… this sounds like it shouldn’t really be a problem… perhaps I should look into this”.

Attacking the problem…

Every so often it is revealed that I know a little bit about databases.

Getting a development environment up for Patchwork is amazingly easy thanks to Docker and the great work of the Patchwork maintainers. The only thing you need to load in is an example dataset. I started by importing mail from a few mailing lists I’m subscribed to, which was Good Enough(TM) for an initial look.

Due to how Django forces us to design a database schema though, the suggested method of getting a sample data set will not mirror what occurs in a production system with multiple lists. It’s for this reason that I ended up using a copy of a live dataset for much of my work rather than constructing an artificial one.

Patchwork supports both a MySQL and PostgreSQL database backend. Since the one on is backed by PostgreSQL, I ended up loading a large dataset into PostgreSQL for most of my work, although I also did some testing with MySQL.

The current instance has a database of around 13GB in side, with about a million patches. You may think this is big, my database brain goes “no, this is actually quite small and everything should be a lot faster than it is even on quite limited hardware”

The problem with ORMs

It turns out that Patchwork is written in Django, an ORM (Object-Relational Mapping) framework in Python – and thus something that pretty effectively obfuscates application code from the SQL being run.

There is one thing that Django misses that could be a pretty big general performance boost to many applications: it doesn’t support composite primary keys. For some databases (e.g. MySQL’s InnoDB engine) the PRIMARY KEY is a clustered index – that is, the physical layout of the rows on disk reflect primary key order. You can use this feature to your advantage and have much higher cache hits of your database pages.

Unfortunately though, we cannot do that with Django, so we lose a bunch of possible performance because of it (especially for queries that are going to have to bring in data from disk). In fact, we’re forced to use an ID field that’ll scatter our rows all over the place rather than do something efficient. You can somewhat get back some of the performance by creating covering indexes, but this costs in terms of index maintenance and disk space.

It should be noted that PostgreSQL doesn’t have a similar concept, although there is a (locking) CLUSTER statement that can (as an offline operation for the table) re-arrange existing rows to be in index order. In my testing, this can give a bit of a boost to performance of some of the Patchwork queries.

With MySQL, you’d look at a bunch of statistics on what pages are being brought in and paged out of the InnoDB buffer pool. With PostgreSQL it’s a bit more complex as it relies heavily on the OS page cache.

My main experience is with MySQL like environment, so I’ve had to re-learn a bunch of PostgreSQL things in this work which was kind of fun. It may be “because of my upbringing” but it seems as if there’s a lot more resources and documentation out in the wild about optimizing MySQL environments than PostgreSQL ones, especially when it comes to documentation around a bunch of things inside the database server. A lot of credit should go to the MySQL Documentation team – I wish the PostgreSQL documentation was up to the same standard.

Another issue is that fetching BLOBs is generally an expensive operation that you want to avoid unless you’re going to use them. Thus, fetching the whole “object” at once isn’t always optimal. The Django query generation appears to be somewhat buggy when it comes to “hey, don’t fetch these columns, I don’t need them”, so you do have to watch what query is produced not just what query you expect to be produced. For example, [01/11] Improve patch listing performance (~3x).

Another issue with Django is how you go from your Python code to an actual SQL query, especially when the produced SQL query is needlessly complex or inefficient. I’ve seen Django always produce an ORDER BY for one table, even when not needed, I’ve also seen it always join tables even when you’re getting no columns from one of them and there’s no way you’re asking for it. In fact, I had to revert to raw SQL for one of my performance improvements as I just couldn’t beat it into submission: [10/11] Be sensible computing project patch counts.

An ORM can be great for getting apps out quickly, or programming in a familiar way. But like many things, an understanding of what is going on underneath is key for extracting maximum performance.

Also, if you ever hear something like “ORM $x doesn’t scale” then maybe that person just hasn’t looked at how to use the ORM better. The same goes for if they say “database $y doesn’t scale”- especially if it’s a long existing relational database such as MySQL or PostgreSQL.

Speeding up listing current patches for a project

17 SQL queries in 4477msMore than 4 seconds in the database
does not make page load time great.

Fortunately though, the Django development environment lets you really easily dive into what queries are being generated and (at least roughly) where they’re being generated from. There’s a sidebar in your browser that shows how many SQL queries were needed to generate the page and how long they took. The key to making your application go faster is to run fewer queries in less time.

I was incredibly impressed with how easy it was to see what queries were run, where they were run from, and the EXPLAIN output for them.

By clicking on that SQL button on the right side of your browser, you get this wonderful chart of what queries were executed, when, and how long they took. From this, it is incredibly obvious which query is the most problematic: the one that took more than four seconds!

In the dim dark days of web development, you’d have to turn on a Slow Query Log on the database server and then grep through your source code or some other miserable activity. I am so glad I didn’t have to do that.

More than four seconds for a single database query does not make for a nice UX.

This particular query was a real hairy one, the EXPLAIN output from PostgreSQL (and MySQL) was certainly long and involved and would most certainly not feature in the first half of an “Introduction to query optimization” day long workshop. If you haven’t brushed up on various bits of documentation on understanding EXPLAIN, you should! The MySQL EXPLAIN FORMAT=JSON is especially fantastic for getting deep details as to what’s going on with query execution.

The big performance gain here was to have the database be able to execute the query in a much more efficient way by creating two covering indexes for part of the query. To work out what indexes to create, one has to look at the EXPLAIN output and work out why the database is choosing to do either a sequential scan of a large table, or use an index that doesn’t exclude that many rows. In this case, I tweaked the code to slightly change the query that was generated as well as adding a covering index. What we ended up with is something that is dramatically faster.

The main query is ~350x faster than before

You’ll notice that it appears that the first query there takes a lot more time but it doesn’t, it just takes a lot more time relative to the main query.

In fact, this particular page is one that people have mentioned at being really, really slow to load. With the main query now about 350 times faster than it was originally, it shouldn’t be a problem anymore.

A future improvement would be to cache the COUNT() for the common case, as it’s pretty easily computed when new patches come in or states change.

The patches that help this particular page have been submitted upstream here:

Making viewing a patch with comments faster

Now that we can list patches faster, can we make other pages that Patchwork has quicker?

One such page is viewing a patch (or cover letter) that has a lot of comments on it. One feature of Patchwork is that it will display all the email replies to a patch or cover letter in the Web UI. But… this seemed slow

On one of the most commented patches I could find, we ended up executing one hundred and seventy seven SQL queries to view it! If we dove into it, a bunch of the queries looked really really similar…

I’ve got 99 queries where I only need 1.

The problem here is that the Patchwork UI is wanting to find out the name of each person who submitted a comment, and is doing that by querying the ID from a table. What it should be doing instead is a SQL JOIN on the original query and just fetching all that information in one go: make the database server do the work, it’s really good at it.

My patch [02/11] 4x performance improvement for viewing patch with many comments   does just that by using the select_related() method correctly, as well as being explicit about what information we want to retrieve.

We’re now only a few milliseconds to grab all the comments

With that patch, we’re down to a constant number of queries and around a 3x-7x faster time executing them depending if we have a warm cache or not.

The one time I had to use raw SQL

When viewing a project page (such as ) it displays the number of patches (archived and not archived) for the project. By looking at what SQL queries are executed to collect these numbers, you’ll notice two things. First, here are the queries:

COUNT() queries can be expensive

First thing you’ll notice is that they took a loooooong time to execute (more than a second each). The second thing, if you look closer, is that they contain a join which is completely unneeded.

I spent a good long while trying to make Django behave, and I just could not. I believe it’s due to the model having some inheritance in it. Writing the query by hand ended up being the best solution, and it gave a significant performance improvement:

Unfortunately, only 4x faster.

Arguably, a better way would be to precompute the count for the archived/non-archived patches and just display them. I (or someone else who knows more about Django) may want to look at that for a future improvement.

Conclusion and final thoughts

There’s a few more places where there could be some optimizations, but currently I cannot get any single page to take more than between 40-400ms in the database when running on my laptop – and that’s Good Enough(TM) for now.

The next steps are getting these patches through a round or two of review, and then getting them into a Patchwork release and deployed out on and see if people can find any new ways to make things slow.

If you’re interested, the full patchset with cover letter is here: [00/11] Performance for ALL THE THINGS!

The diffstat is interesting, as most of the added code is auto-generated by Django for database migrations (adding of indexes).

 .../migrations/ | 23 +++++++++++++++++
 .../           | 19 ++++++++++++++
 .../     | 19 ++++++++++++++
 patchwork/                           | 21 ++++++++++++++--
 patchwork/templates/patchwork/submission.html | 16 ++++++------
 patchwork/views/                   |  8 +++++-
 patchwork/views/                      |  5 ++++
 patchwork/views/                      |  7 ++++++
 patchwork/views/                    | 25 ++++++++++++++++---
 9 files changed, 128 insertions(+), 15 deletions(-)
 create mode 100644 patchwork/migrations/
 create mode 100644 patchwork/migrations/
 create mode 100644 patchwork/migrations/

I think the lesson is that making dramatic improvements to performance of your Django based app does not mean you have to write a lot of code or revert to raw SQL or abandon your ORM. In fact, use it properly and you can get a looong way. It’s just that to use it properly, you’re going to have to understand the layer below the ORM, and not just treat the database as a magic black box.

ccache and op-build

You may have heard of ccache (Compiler Cache) which saves you heaps of real world time when rebuilding a source tree that is rather similar to one you’ve recently built before. It’s really useful in buildroot based projects where you’re building similar trees, or have done a minor bump of some components.

In trying to find a commit which introduced a bug in op-build (OpenPOWER firmware), I noticed that hostboot wasn’t being built using ccache and we were always doing a full build. So, I started digging into it.

It turns out that a bunch of the perl scripts for parsing the Machine Readable Workbook XML in hostboot did a bunch of things like foreach $key (%hash) – which means that the code iterates over the items in hash order rather than an order that would produce predictable output such as “attribute name” or something. So… much messing with that later, I had hostboot generating the same output for the same input on every build.

Next step was to work out why I was still getting a lot of CCACHE misses. It turns out the default ccache size is 5GB. A full hostboot build uses around 7.1GB of that.

So, if building op-build with CCACHE, be sure to set both BR2_CCACHE=y in your config as well as something like BR2_CCACHE_INITIAL_SETUP="--max-size 20G"

Hopefully my patches hit hostboot and op-build soon.

Switching to iPhone: Part 1

I have used Android phones since the first one: the G1. I’m one of the (relatively) few people who has used Android 1.0. I’ve had numerous Android phones since then, mostly the Google flagship.

I have fond memories of the Nexus One and Galaxy Nexus, as well as a bunch of time running Cyanogen (often daily builds, because YOLO) to get more privacy preserving features (or a more recent Android). I had a Sony Z1 Compact for a while which was great bang for buck except for the fact the screen broke whenever you looked at it sideways. Great kudos to the Sony team for being so friendly to custom firmware loads.

I buy my hardware from physical stores. Why? Well, it means that the NSA and others get to spend extra effort to insert hardware modifications (backdoors), as well as the benefit of having a place to go to/set the ACCC on to get my rights under Australian Consumer Law.

My phone before last was a Nexus 5X. There were a lot of good things about this phone; the promise of fast charging via USB-C was one, as was the ever improving performance of the hardware and Android itself. Well… it just got progressively slower, and slower, and slower – as if it was designed to get near unusable by the time of the next Google phone announcement.

Inevitably, my 5X succumbed to the manufacturing defect that resulted in a boot loop. It would start booting, and then spontaneously reboot, in a loop, forever. The remedy? Replace it under warranty! That would take weeks, which isn’t a suitable timeframe in this day and age to be without a phone, so I mulled over buying a Google Pixel or my first ever iPhone (my iPhone owning friends assured me that if such a thing happens with an iPhone that Apple would have swapped it on the spot). Not wanting to give up a lot of the personal freedom that comes with the Android world, I spent the $100 more to get the Pixel, acutely aware that having a phone was now a near $1000/year habit.

The Google Pixel was a fantastic phone (except the price, they should have matched the iPhone price). The camera was the first phone camera I actually went “wow, I’m impressed” over. The eye-watering $279 to replace a cracked screen, the still eye-watering cost of USB-C cables, and the seat to process the HDR photos were all forgiven. It was a good phone. Until, that is, less than a year in, the battery was completely shot. It would power off when less than 40% and couldn’t last the trip from Melbourne airport to Melbourne city.

So, with a flagship phone well within the “reasonable quality” time that consumer law would dictate, I contacted Google after going through all the standard troubleshooting. Google agreed this was not normal and that the phone was defective. I was told that they would mail me a replacement, I could transfer my stuff over and then mail in the broken one. FANTASTIC!! This was soooo much better than the experience with the 5X.

Except that it wasn’t. A week later, I rang back to ask what was going on as I hadn’t received the replacement; it turns out Google had lied to me, I’d have to mail the phone to them and then another ten business days later I’d have a replacement. Errr…. no, I’ve been here before.

I rang the retailer, JB Hi-Fi; they said it would take them at least three weeks, which I told them was not acceptable nor a “reasonable timeframe” as dictated by consumer law.

So, with a bunch of travel imminent, I bought a big external USB-C battery and kept it constantly connected as without it the battery percentage went down faster than the minutes ticked over. I could sort it out once I was back from travel.

So, I’m back. In fact, I drove back from a weekend away and finally bit the bullet – I went to pick up a phone who’s manufacturer has a reputation of supporting their hardware.

I picked up an iPhone.

I figured I should write up how, why, my reasons, and experiences in switching phone platforms. I think my next post will be “Why iPhone and not a different Android”.

CVE-2019-6260: Gaining control of BMC from the host processor

This is details for CVE-2019-6260 – which has been nicknamed “pantsdown” due to the nature of feeling that we feel that we’ve “caught chunks of the industry with their…” and combined with the fact that naming things is hard, so if you pick a bad name somebody would have to come up with a better one before we publish.

I expect OpenBMC to have a statement shortly.

The ASPEED ast2400 and ast2500 Baseboard Management Controller (BMC) hardware and firmware implement Advanced High-performance Bus (AHB) bridges, which allow arbitrary read and write access to the BMC’s physical address space from the host, or from the network if the BMC console uart is attached to a serial concentrator (this is atypical for most systems).

Common configuration of the ASPEED BMC SoC’s hardware features leaves it open to “remote” unauthenticated compromise from the host and from the BMC console. This stems from AHB bridges on the LPC and PCIe buses, another on the BMC console UART (hardware password protected), and the ability of the X-DMA engine to address all of the BMC’s M-Bus (memory bus).

This affects multiple BMC firmware stacks, including OpenBMC, AMI’s BMC, and SuperMicro. It is independent of host processor architecture, and has been observed on systems with x86_64 processors IBM POWER processors (there is no reason to suggest that other architectures wouldn’t be affected, these are just the ones we’ve been able to get access to)

The LPC, PCIe and UART AHB bridges are all explicitly features of Aspeed’s designs: They exist to recover the BMC during firmware development or to allow the host to drive the BMC hardware if the BMC has no firmware of its own. See section 1.9 of the AST2500 Software Programming Guide.

The typical consequence of external, unauthenticated, arbitrary AHB access is that the BMC fails to ensure all three of confidentiality, integrity and availability for its data and services. For instance it is possible to:

  1. Reflash or dump the firmware of a running BMC from the host
  2. Perform arbitrary reads and writes to BMC RAM
  3. Configure an in-band BMC console from the host
  4. “Brick” the BMC by disabling the CPU clock until the next AC power cycle

Using 1 we can obviously implant any malicious code we like, with the impact of BMC downtime while the flashing and reboot take place. This may take the form of minor, malicious modifications to the officially provisioned BMC image, as we can extract, modify, then repackage the image to be re-flashed on the BMC. As the BMC potentially has no secure boot facility it is likely difficult to detect such actions.

Abusing 3 may require valid login credentials, but combining 1 and 2 we can simply change the locks on the BMC by replacing all instances of the root shadow password hash in RAM with a chosen password hash – one instance of the hash is in the page cache, and from that point forward any login process will authenticate with the chosen password.

We obtain the current root password hash by using 1 to dump the current flash content, then using to extract the rootfs, then simply loop-mount the rootfs to access /etc/shadow. At least one BMC stack doesn’t require this, and instead offers “Press enter for console”.

IBM has internally developed a proof-of-concept application that we intend to open-source, likely as part of the OpenBMC project, that demonstrates how to use the interfaces and probes for their availability. The intent is that it be added to platform firmware test
suites as a platform security test case. The application requires root user privilege on the host system for the LPC and PCIe bridges, or normal user privilege on a remote system to exploit the debug UART interface. Access from userspace demonstrates the vulnerability of systems in bare-metal cloud hosting lease arrangements where the BMC
is likely in a separate security domain to the host.

OpenBMC Versions affected: Up to at least 2.6, all supported Aspeed-based platforms

It only affects systems using the ASPEED ast2400, ast2500 SoCs. There has not been any investigation into other hardware.

The specific issues are listed below, along with some judgement calls on their risk.

iLPC2AHB bridge Pt I

State: Enabled at cold start
Description: A SuperIO device is exposed that provides access to the BMC’s address-space
Impact: Arbitrary reads and writes to the BMC address-space
Risk: High – known vulnerability and explicitly used as a feature in some platform designs
Mitigation: Can be disabled by configuring a bit in the BMC’s LPC controller, however see Pt II.

iLPC2AHB bridge Pt II

State: Enabled at cold start
Description: The bit disabling the iLPC2AHB bridge only removes write access – reads are still possible.
Impact: Arbitrary reads of the BMC address-space
Risk: High – we expect the capability and mitigation are not well known, and the mitigation has side-effects
Mitigation: Disable SuperIO decoding on the LPC bus (0x2E/0x4E decode). Decoding is controlled via hardware strapping and can be turned off at runtime, however disabling SuperIO decoding also removes the host’s ability to configure SUARTs, System wakeups, GPIOs and the BMC/Host mailbox

PCIe VGA P2A bridge

State: Enabled at cold start
Description: The VGA graphics device provides a host-controllable window mapping onto the BMC address-space
Impact: Arbitrary reads and writes to the BMC address-space
Risk: Medium – the capability is known to some platform integrators and may be disabled in some firmware stacks
Mitigation: Can be disabled or filter writes to coarse-grained regions of the AHB by configuring bits in the System Control Unit

DMA from/to arbitrary BMC memory via X-DMA

State: Enabled at cold start
Description: X-DMA available from VGA and BMC PCI devices
Impact: Misconfiguration can expose the entirety of the BMC’s RAM to the host
AST2400 Risk: High – SDK u-boot does not constrain X-DMA to VGA reserved memory
AST2500 Risk: Low – SDK u-boot restricts X-DMA to VGA reserved memory
Mitigation: X-DMA accesses are configured to remap into VGA reserved memory in u-boot

UART-based SoC Debug interface

State: Enabled at cold start
Description: Pasting a magic password over the configured UART exposes a hardware-provided debug shell. The capability is only exposed on one of UART1 or UART5, and interactions are only possible via the physical IO port (cannot be accessed from the host)
Impact: Misconfiguration can expose the BMC’s address-space to the network if the BMC console is made available via a serial concentrator.
Risk: Low
Mitigation: Can be disabled by configuring a bit in the System Control Unit

LPC2AHB bridge

State: Disabled at cold start
Description: Maps LPC Firmware cycles onto the BMC’s address-space
Impact: Misconfiguration can expose vulnerable parts of the BMC’s address-space to the host
Risk: Low – requires reasonable effort to configure and enable.
Mitigation: Don’t enable the feature if not required.
Note: As a counter-point, this feature is used legitimately on OpenPOWER systems to expose the boot flash device content to the host

PCIe BMC P2A bridge

State: Disabled at cold start
Description: PCI-to-BMC address-space bridge allowing memory and IO accesses
Impact: Enabling the device provides limited access to BMC address-space
Risk: Low – requires some effort to enable, constrained to specific parts of the BMC address space
Mitigation: Don’t enable the feature if not required.

Watchdog setup

State: Required system function, always available
Description: Misconfiguring the watchdog to use “System Reset” mode for BMC reboot will re-open all the “enabled at cold start” backdoors until the firmware reconfigures the hardware otherwise. Rebooting the BMC is generally possible from the host via IPMI “mc reset” command, and this may provide a window of opportunity for BMC compromise.
Impact: May allow arbitrary access to BMC address space via any of the above mechanisms
Risk: Low – “System Reset” mode is unlikely to be used for reboot due to obvious side-effects
Mitigation: Ensure BMC reboots always use “SOC Reset” mode

The CVSS score for these vulnerabilities is:

There is some debate on if this is a local or remote vulnerability, and it depends on if you consider the connection between the BMC and the host processor as a network or not.

The fix is platform dependent as it can involve patching both the BMC firmware and the host firmware.

For example, we have mitigated these vulnerabilities for OpenPOWER systems, both on the host and BMC side. OpenBMC has a u-boot patch that disables the features:

Which platforms can opt into in the following way:

The process is opt-in for OpenBMC platforms because platform maintainers have the knowledge of if their platform uses affected hardware features. This is important when disabling the iLPC2AHB bridge as it can be a bit of a finicky process.

See also for a WIP OpenBMC Security Architecture document which should eventually contain all these details.

For OpenPOWER systems, the host firmware patches are contained in op-build v2.0.11 and enabled for certain platforms. Again, this is not by default for all platforms as there is BMC work required as well as per-platform changes.

Credit for finding these problems: Andrew Jeffery, Benjamin
Herrenschmidt, Jeremy Kerr, Russell Currey, Stewart Smith. There have been many more people who have helped with this issue, and they too deserve thanks.

Tracing flash reads (and writes) during boot

On OpenPOWER POWER9 systems, we typically talk to the flash chips that hold firmware for the host (i.e. the POWER9) processor through a daemon running on the BMC (aka service processor) rather than directly.

We have host firmware map “windows” on the LPC bus to parts of the flash chip. This flash chip can in fact be a virtual one, constructed dynamically from files on the BMC.

Since we’re mapping windows into this flash address space, we have some knowledge as to what IO the host is doing to/from the pnor. We can use this to output data in the blktrace format and feed into existing tools used to analyze IO patterns.

So, with a bit of learning of the data format and learning how to drive the various tools, I was ready to patch the BMC daemon (mboxbridge) to get some data out.

An initial bit of data is a graph of the windows into PNOR opened up during an normal boot (see below).

PNOR windows created over the course of a normal boot.

This shows us that over the course of the boot, we open a bunch of windows, and switch them around a fair bit early on. This makes sense as early in boot we do not yet have DRAM working and page in firmware on-demand into L3 cache.

Later in boot, you can see the loading of larger chunks of firmware into memory. It’s also possible to see that this seems to take longer than it should – and indeed, we have a bug there.

Next, by modifying the code again, I introduced recording of when we used a window that the BMC had already cached. While the host will only see one window at a time, the BMC can keep around the ones it prepared earlier in order to avoid IO to the actual flash chips (which are SPI flash, so aren’t incredibly fast).

Here we can see that we’re likely not doing the most efficient things during boot, and there’s probably room for some optimization.

Normal boot but including re-used Windows rather than just created ones

Finally, in order to get finer grained information, I reduced the window size from one megabyte down to 4096 bytes. This will impose a heavy speed penalty as it’ll mean we will have to create a lot more windows to do the same amount of IO, but it means that since we’re using the page size of hostboot, we’ll see each individual page in/out operation that it does during boot.

So, from the next graph, we can see that there’s several “hot” areas of the image, and on the whole it’s not too many pages. This gives us a hint that a bit of effort to reduce binary image size a little bit could greatly reduce the amount of IO we have to do.

4096 byte (i.e. page) size window, capturing the bits of flash we need to read in several times due to being low on memory when we’re L3 cache constrained.

The iowatcher tool also can construct a video of the boot and what “blocks” are being read.

Video of what blocks are read from flash during booting

So, what do we get from this adventure? Well, we get a good list of things to look into in order to improve boot performance, and we get to back these up with data rather than guesswork. Since this also works on unmodified host firmware, we’re measuring what we really boot rather than changing it in order to measure it.

What you need to reproduce this:

Switching to iPhone Part 2: Seriously?

In which I ask of Apple, “Seriously?”.

That was pretty much my reaction with Apple sticking to Lightning connectors rather than going with the USB-C standard. Having USB-C around the place for my last two (Android) phones was fantastic. I could charge a phone, external battery, a (future) laptop, all off the same wall wart and with the same cable. It is with some hilarity that I read that the new iPad Pro has USB-C rather than Lightning.

But Apple’s dongle fetish reigns supreme, and so I get a multitude of damn dongles all for a wonderfully inflated price with an Australia Tax whacked on top.

The most egregious one is the Lightning-to-3.5mm dongle. In the office, I have a good set of headphones. The idea is to block out the sound of an open plan office so I can actually get some concentrating done. With tiny dedicated MP3 players and my previous phones, these sounded great. The Apple dongle? It sounds terrible. Absolutely terrible. The Lighting-to-3.5mm adapter might be okay for small earbuds but it is nearly completely intolerable for any decent set of headphones. I’m now in the market for a Bluetooth headphone amplifier. Another bunch of money to throw at another damn dongle.

Luckily, there seems to be a really good Bluetooth headphone amplifier on Amazon. The same Amazon that no longer ships to Australia. Well, there’s an Australian seller, for six times the price.


February 25, 2019

What Do You Mean "No"?

Quite often when building small Linux images having separate user accounts isn't always at the top of the list of things to include. Petitboot is no different; the most common operations like mounting disks, configuring interfaces, and calling kexec all require root and Petitboot generally only exists long enough to boot into the next thing, so why not run it all as root?

The picture is less clear when we start to think about what is possible to do in Petitboot by default. If someone comes across an open Petitboot console they're only a few keystrokes away from wiping disks, changing files, or even flashing firmware. Depending on how your system is used that may or may not be something you care about, but over time there have been a few requests to "add a password screen to Petitboot" to at least make it so that the system isn't open season for whoever sees it.

Enter Password:

The most direct way to avoid this would be to slap a password prompt onto Petitboot before any changes can be made. There are two immediate drawbacks to this:

  • The Petitboot UI still runs as root, and
  • Exiting to the shell gives the user root permissions as well.

There is already a mechanism to prevent the user exiting to the shell, but this puts all of our eggs in the basket of petitboot-nc being a secure program. If a user can accidentally or otherwise find a way to exit or crash the UI then they're immediately in a root shell, and while petitboot-nc is a good UI it was never designed to be a hardened program protecting the system.

You Have No Power Here

The idea instead as of Petitboot v1.10.0 is not to care if the user drops to the shell; because now it's completely unprivileged.

Normal shell

The only process now that runs as root is pb-discover itself; the console, UI, and helper scripts run as a new 'petituser'. For the server and clients to still communicate the "petitiboot.ui" socket permissions are modified to allow processes that are part of the 'petitgroup' to connect. However now if pb-discover notices that a client in the petitgroup is connecting (or more accurately the client isn't running as root) by default it ignores any commands from it that would configure or boot the system.

A new command, PB_PROTOCOL_ACTION_AUTHENTICATE, lets a client send a password to the server to then be allowed to send all the usual commands like updating the config or booting a specific option. This keeps all the authentication on the server side, avoiding writing any "secure" ncurses code. In the UI the biggest difference is that when trying to change something the user will hit a password field:


Then the password is sent to the server, checked, and if correct the action goes ahead.

Whose Passwords?

But where does this password come from? Technically it's just the root password. The server computes a hash of the supplied password and compares it against the system's root password. Similarly in the shell the user can run sudo with the root password to enter a full shell if needed:


Petitboot of course runs in memory, and writing a root password into the image itself would mean recompiling to change the password, so instead Petitboot pulls the root password from NVRAM. On startup Petitboot reads the petitboot,password parameter which is the hash of the root password and updates /etc/shadow with it. This happens before any clients are up or can connect to the server.

Don't Panic

By default no password is set. After all we don't want people upgrading and then being somehow locked out of their system. For ease of use, and for testing-purposes, if no password is configured and the user drops to the shell it is automatically upgraded to a root shell:


To set a password there is a new subscreen in System Configuration:

New Password

This sends an authentication command to the server, and assuming the client is authenticated with the current password as well pb-discover updates the shadow file, and writes the hash back to NVRAM.

User support exists in Petitboot v1.10.0 onwards. You will also need some support in your build system to set up the users, see how op-build did it for an example.

There are a few items on the TODO list still which would be good to have. For example storing the password hash in an attached TPM if available, as well as splitting out more of what runs as root; for example the bootloader parsers in pb-discover preferably wouldn't run with root privileges, but they are all part of the one binary.

As always, comments, suggestions, and patches welcome on the list!

February 24, 2019

Audiobooks – January 2019

The Grandmaster: Magnus Carlsen and the Match That Made Chess Great Again by Brin-Jonathan Butler

Not a lot about the match. The author rolls out a bunch of random chess stories and profiles instead. 4/10

The Next American City: The Big Promise of Our Midsize Metros by Mick Cornett

The four-term Mayor of Oklahoma City goes over projects OC and other mid-sized cities implemented to improve their cities & fight back against the large metros. 8/10

21 Lessons for the 21st Century by Yuval Noah Harari

Apparently a lot of expanded essays but still a lot of interesting stuff in there. The good ones are great and the bad ones are okay. 7/10

Chasing the Demon: A Secret History of the Quest for the Sound Barrier, and the Band of American Aces Who Conquered It by Dan Hampton

Covers some early aircraft and aerodynamics history, then the lives of pilots who would break the sound barrier & then the actual event (or events!). 7/10


February 18, 2019

Abaddon’s Gate


This is the third book in the Leviathan Wakes series by James SA Corey. Just as good as the first two, this is a story about how much a daughter loves her father, perhaps beyond reason, moral choices, and politics — just as much as it is the continuation of the story arc around the alien visitor.

Another excellent book, with a bit more emphasis on space battles than previously and an overall enjoyable plot line. Worth a read, to be honest I think the series is getting better.

Abaddon's Gate Book Cover Abaddon's Gate
James S. A. Corey
Hachette UK
June 4, 2013

The third book in the New York Times bestselling Expanse series. NOW A MAJOR TV SERIES FROM NETFLIX For generations, the solar system - Mars, the Moon, the Asteroid Belt - was humanity's great frontier. Until now. The alien artefact working through its program under the clouds of Venus has emerged to build a massive structure outside the orbit of Uranus: a gate that leads into a starless dark. Jim Holden and the crew of the Rocinante are part of a vast flotilla of scientific and military ships going out to examine the artefact. But behind the scenes, a complex plot is unfolding, with the destruction of Holden at its core. As the emissaries of the human race try to find whether the gate is an opportunity or a threat, the greatest danger is the one they brought with them.


5th axis getting up to speed

After a little bit of calibration and tinkering in the fusion360 cps output filter files the cnc is starting to take advantage of the new 5th axis. Below is a "Flow" multiaxis toolpath finishing a sphere with a flat endmill.

There are still some issues that I have to address. It has been a surprisingly good experience so far. Starting out cutting 5 sides of a block and moving on to cutting the edges at an angle. This is the third test where I rough out the sphere using a 3d adaptive path and then use a flow multiaxis path to clean things up. Things look better in video and there is some of that to come.

February 17, 2019

Using D-STAR with Codec 2

Antony Chazapis, SV9OAN, has been extending the D-STAR protocol to use Codec 2. Below is a brief explanation of the project, please also see his Github page. Over to Antony:

Instead of designing a new protocol, I propose a D-STAR protocol extension that allows to use Codec 2 in frames instead of AMBE. There are many Codec 2 variants, so I started by using the highest bitrate one (3200 mode) without FEC which is fine for over-the-Internet communication. I then modified xlxd to add a new listener for the D-STAR protocol variant (named “DExtra Open”) and ambed to transcode from 1 codec to 2 (eg. AMBE in, Codec 2/AMBE+ out).

In theory you could do transmissions with the new protocol, but the problem is that there are no real clients that implement it – yet. Actually, the only clients available now are my Python-based dv-player and dv-recorder which connect to xlxd, play and record dvtool files respectively. I am thinking about software-based implementations at the moment: Android, iOS, Mac OS, Linux, etc.

Another direction would be to figure out a FEC implementation and do real over-the-air experiments with MMDVM based devices, assuming a microphone input and a speaker output.

73 de SV9OAN

About Antony

Antony Chazapis, SV9OAN, is licensed since 2009. He is a member of RAAG, RASC, ARRL, and the volunteer group HARES (Hellenic Amateur Radio Emergency Service). Since 2017, he has been serving as the Association Manager for the SOTA award program in Greece. Antony enjoys DXing and exploring what makes digital modes tick. He is the author of PocketPacket, an APRS client application for macOS and iOS. Recently, he has been working on enabling D-STAR communications with Codec 2. He holds a PhD in Computer Science and is currently employed by a Toronto-based tech startup implementing bleeding-edge storage solutions.

Update – Feb 2019

About two months later, and Antony has introduced a protocol variant based on Codec 2’s 2400 mode with added FEC. He has also implemented Estrella, a DExtra Open client, that allows direct communication through xlxd reflectors (macOS and iOS versions are available now, with more platforms to follow).

Estrella for macOS
Estrella for iOS

February 15, 2019

Thematic Review 3 : Practical Implementations of Information Systems

Following examples of a successful IS implementation, the Bankers Trust of Australia, and an on-going failure with the example of the UK's Universal Credit project, it is opportune to consider a subset of IS that is common to both projects and determine whether there is a general rule that can be applied. In particular, attention is drawn to the fact that the IT contract for the IS projects in the two cases was either largely developed in-house for the successful project and fully outsourced in the failed project. Common-sense should dictate that this cannot be universalised; surely not every introduction of an IS system must have the IT component developed in-house? One wonders whether there are any thorough studies with practical implementations to explore when it in-house or outsourced development is appropriate?

Traylor (2006) looks at this very issue, arguing "Decades of trial, error, and egghead analysis have yielded a consensus conclusion: Buy when you need to automate commodity business processes; build when you're dealing with the core processes that differentiate your company", however immediately pointing out that such a theoretical model (doubtless built on numerous empirical examples), may encounter problems with reality, such as in-house software for standard processes may come with very high transition costs, whereas external commercial packages may actually be a very good fit for strategic plans. Traylor continues: "'Buy to standardize, build to compete' may be terrific as an ideology, but the choices that real companies face are a lot messier ... and more interesting."

Following the arguments of several senior executives from companies with significant IT investments, some variation is noted. The former CIO of Pricewaterhouse Coopers argues that "[E]verybody knows" that standardised and off-the-shelf purchases are more cost-effective for implementation and maintenance, whereas the IT chief architect at MCI argues for in-house development where one can get incremental revenue or a competitive advantage, whilst at the same time encouraging code re-use. Whilst decisions metrics (cost, skill-sets, strategic value etc) remain the same in either approach, and most firms have a combination of both, Traylor also points out - almost fifteen years ago now - that open source software offers the best of both, providing standards-compliance and bespoke elements. Emphasis is also made towards the software lifecycle with the stated claim that 70 percent of software costs occur after implementation.

Even large and security-conscious companies, such as Visa, who have an enormous amount of in-house development will purchase commercial solutions when there is no competitive advantage to be achieved and costs would be greater to follow the in-house method. Nevertheless, when it comes to commercial software a clear warning is expressed about developing components outside that commercial package which are nevertheless dependent on it. The result of such an approach will be to generate in-house software with critical dependencies which is outside of the control of the organisation, which is both wasteful and dangerous. In many cases, commercial applications offer their own extensions which can fit existing varied business processes.

This is not to say that in-house development should not occur alongside or even with commercial offerings. It should just not be for the same business process (e.g., an accounting package with an in-house accounting extension). An example is given of the District of Columbia city government which has migrated from a myriad of in-house applications to commercial applications, yet still has specific business processes (such as Business Intelligence) which are developed in-house, because commercial applications did not meet the needs of the organisation. Sometimes opportunities even exist for commercialisation with existing vendors to develop new applications, such as the work between the University of Pittsburgh Medical Center and Stentor for a picture-archiving communications system.

Traylor argues the IT Planning Software is taking into account these decisions, and helping managers determine whether to build or buy, and gives an example of the questions confronting MCI on a system to track third-party services inventory. It comes across as being a lot less detailed or sophisticated as one would hope for in a for a purchase of such importance. One is interested about the series of posed questions is that they are almost entirely in reference to an IT business approach, rather than an IS approach - there is little evidence that the question of existing business processes, staff involvement etc are included in such software packages. Whilst does suggest that whilst they provide some contribution to the decision, it is also evidence that an IS decision must be of broader scope than an IT decision.


Traylor, Polly (2006). To build or to buy IT applications?, InfoWorld, 13 Feb, 2006

February 14, 2019

Thematic Review 2 : Practical Implementations of Information Systems

When reviewing practical implementations of information systems (IS), incredible failures provide very valuable lessons even if they are ongoing. At an estimated £12.8bn, far in excess of the originally estimated £2.2bn (Ballard, 2013), the UK's Universal Credit project will be the single-most expensive failed or overbudget custom software projects, although when adjusted for inflation the UK's NHS Connecting for Health project (mostly abandoned in 2011), also cost around £12bn. Apparently if one wishes to study exceptional failures in IS, government in general, the UK in particular, and the subcategory of health and welfare is a good place to start. Whilst this may seem slightly snide, it is backed by empirical evidence. Only the US's FAA Advanced Automation System (c$4.5b USD, 1994) is really within a similar sort of scale of failure.

Universal Credit, like many such projects, on a prima facie level, seems to designed on reasonable principles. Announced in 2010, with the objective to simplify working-age welfare benefits and encourage taking up paid work, it would replace and combine six different means-tested benefits, and roll them into a single monthly payment and which, as paid work was taken up, would be gradually be reduced, rather than having an all-or-nothing approach, following the "negative income tax" proposals, as proposed by Juliet Rhys William and Milton Friedman (Forget, 2012). The project was meant to start in 2013 and completed by 2017. Under current plans (there have been at least seven timetable completion changes), this has been pushed out to 2023 (Butler, Walker, 2016).

It is important to note that the majority of problems that confront the Universal Credit project were not just limited by IT, although there are plenty of those. Attention to detail is evidently lacking, perhaps overlooked by decision-makers who have a minimal visceral understanding of those who require welfare. For example, part-time work can lead to a situation where people can work more and be paid less (Toynbee, 2013) and the self-employed could lose thousands of pounds per annum. The transition period from old welfare payments to Universal Credit meant that people on already marginal incomes experiencing delays in payments, with the National Audit reporting to eight months (Richardson, 2018). As a result, there are numerous instances of people falling behind in rent payments, food banks requests rocketing, (Savage, Jayanetti, 2017) and some even turning to prostitution (Quayle, Box, 2018). These are, in an economic model, a producer-consumer issue where the producer is a monopoly provider of a good (welfare) which has near-vertical demand due to need. From a business perspective in IS, it reflects not just administrative errors, but also a lack of project planning with sufficient input from users.

However, in addition to these errors, the project has fraught with IT issues. With 15 different suppliers providing components of the outsourced IT systems in the initial stages (DWP, 2012), the suppliers themselves voiced concerns at the capacity to build a nation-wide reporting system within the short time-frame initially provided (Boffey, 2011). The system requires real-time data from a new Pay as You Earn (PAYE) system being developed with HM Revenue & Customs. The resulting IT system has been described as "completely unworkable, badly designed" and already "out of date" (Jee, 2014) and a subsequent survey of staff found that 90% found the system "inadequate" (Jee, 2015). An audit of the system found that the IT systems "depend heavily on manual intervention and will only handle a small number of claims".

Information Systems needs to founded on quantitative and realistic expectations of the capacity of a technology to deliver a good or service, and with project plans that incorporate end-users, as they will be people who "feel the pain" of a poorly designed system. In the case of welfare, this is no mere consumable, but rather pain in a very visceral and, without having to engage in hyperbole, even mortal requirements. The evidence provided so far is that the Universal Credit system has engaged neither in a realistic assessment of the IT requirements and capabilities, was designed without sufficient input of the final recipients of the service, and as a result, has delivered a sub-par project. Overall, it constitutes the worst IS project in history.


Ballard, Mark (2013), Universal Credit will cost taxpayers £12.8bn, Computer Weekly, 03 June, 2013.

Boffey, Daniel (2011), Universal credit's 2013 delivery could be derailed by complex IT system, The Observer, 19 June 2011

Butler, Patrick., Walker, Peter (2016) Universal credit falls five years behind schedule, The Guardian, 20 July, 2016.

DWP Central Freedom of Information Team (2012), FoI request from Mr. Robinson, 18 October, 2012

Forget, Evelyn L. (2012), Advocating negative income taxes: Juliet Rhys-Williams and Milton Friedman, Proceedings of the History of Political Economy Conference Spring 2012.

Jee, Charlotte (2014), Leaked memo says Universal Credit IT 'completely unworkable', Computer World UK, October 27, 2014

Jee, Charlotte (2015), 90 percent of Universal Credit staff say IT systems 'inadequate', Computer World UK, March 9, 2015

Richardson, Hannah (2018), Universal Credit 'could cost more than current benefits system', BBC News Education, 15 June 2018

Savage, Michael., Jayanetti, Chaminda (2017), Revealed: universal credit sends rent arrears soaring, The Observer, 16 September, 2017

Toynbee, Polly (2013), Universal credit is simple: work more and get paid less, The Guardian, 12 July, 2013.

Quayle, Jess., Box, Dan (2018), 'I was forced into prostitution by Universal Credit', BBC News, 19 November, 2018.

February 13, 2019

Thematic Review 1 : Practical Implementations of Information Systems

Following definitional foundations and theoretical models in Information Systems (IS) there is a great desire to find some detailed practical applications. The first, the Arcadia project at Bankers Trust Australia limited (BTAL) for the derivatives group is almost ancient history as far as computer technology is concerned - it was initiated in 1994. However, as a very successful project it provides excellent information on the processes of implementing new systems into an organisation (Baster et. al., (2001)).

The distinction at this time was described as a "B-T gap" ("Business-Technology gap"), where business users assumed that IT solutions were unable to capture the "dynamic, holistic, and imprecise" nature of business information and decisions, whereas IT users tended to place the blame on 'error between keyboard and chair'. Naturally enough, recognising the gap and the need to overcome it was an excellent starting point of the project along with stating an objective of an information system that was "extensible, adaptable, and responsive to rapid changes".

In more contemporary times what was being built would be considered an Enterprise Resource Planning (ERP) system, which makes some of the management choices particularly interesting in that context. Because of stated needs, BTAL rejected the possibility of a vendor-based solution, which was estimated to be more expensive and less flexible, giving "BTAL less control for satisfying internal decision-making strategies and responding quickly to business changes." To further assist the flexibility a component-based approach was applied, allowing for well-structured interfaces between specific business and technology procedures. The component-based model combined both business and technological logic into a layered model, where the core technological infrastructure was subject to minimal changes, with high-level GUI-presentation layer being areas of most iterative functional elaborations, and with databases and scripting incorporating the business logic desired.

One means of bridging the B-T gap was to have project implementation carried out by a combination of IT and business users, with developers, business drivers, and business users who provided feedback in testing and validating the components, whilst the drivers had detailed expertise in the business functionality that was desired. This was seen a different to much theoretical literature which suggested high-level abstraction, whilst the project used an understanding of complex details and development through rapid iteration.

The success of the Arcadia project was based on continuing feedback, follow-up steps to reduce the B-T gap, and the interest expressed by global operations of the BTAL. The major lessons learned from the project include understanding the existence of a B-T gap and the need to bridge it, understanding and leveraging user behaviour and skills whilst minimising interface changes, keeping the project low-profile whilst recruiting detail-orientated staff as testers. Overall, this detail-orientated layered approach which combined business needs with technological capability should be perceived as a model case study for Information Systems.

Figure 1. Business-Technology layers in the Arcadia project


Baster, Greg., Konana, Prabhudev., Scott, Judy E., Business components: A case study of bankers trust Australia limited, Communications of the ACM 44(5):92-98 · May 2001

Video Review of a Social Media Webinar

A video review of the webinar/video presentation of Meet Edgar's "Ten Social Media Tips for 2019 Success", for the MSc course in Information Systems (University of Salford).

Video produced through slides made with LibreOffice Impress and audio with GNOME SoundRecorder, and composed with OpenShot.

Whilst a transcription is not available, the following is the content from the slideshow presentation.

Ten Social Media Tips for 2019

* This is a review of Meet Edgar’s “Ten Social Media Tips for 2019”
* The video (and transcript) is available from the following URL
* Presenters are Megan and Christina Meet Edgar Community Social Media
* First argument is that social media is a free platform where you can connect with your followers and create a community.
* The presentation does not describe or differentiate between specific social media platforms, but rather gives general advice common to all.
* This immediately limits the value the presentation; the sort of engagement that one has on Facebook is different to Twitter is different to Youtube is different to Livejournal (does anyone outside of Russia still actively use LJ?), or even Usenet, due to the posting restrictions (word count, imagery), style, and especially community norms.

Tip 1: More social less media

* Purpose of this suggestion is to encourage authenticity in marketing, and that “personal branding is now part of company branding”, so engage directly with followers, add more personality into their social media posts. “People buy from people, not businesses”.
* It is claimed that engagement provides for opportunities to connect with followers, find out what drives them, establish brand-loyalty on the basis of brand personality.
* This is all largely true, but over-stated. For example the webinar has completely overlooked the importance of business-to-business transactions (even if these are not usually via social media). The assertion of “brand personality” is backwards; actors (even corporate actors) have personality. A brand is an identifier of a personality.

Tip 2: More Listening, Less Broadcasting

* Discover pain points of consumer, find out what their ideal image of themselves in the future and try to create a situation where your products and services match that image.
* There’s more content on social media than is possible for a consumer to process. It should be stated that in a factor leading to a greater balance of market power between the supplier of goods and services and the consumer of goods and services (other factors, such as a concentration of capital, are a countervailing trend).
* Argues that producers need to “listen and reply and engage with your followers not just broadcast out your promotions”.

Tip 3 More Video, Less Text

* Argues that "everything we read, every trend we're seeing, every analytics that we're looking at goes to this idea that video is where it's at on social media".
* Produce exclusive video content for followers, and encourage comments.
* Claims that live streams generates the most views.
* Drive engagement through emotional reactions.
* Ironically, the video provided by Meet Edgar is essentially a presentation with images and a voice-over (like this one too, but at least I’m knowingly self-referentially ironic).
* Video is a great media for immersion in exciting live content. It is not live streams that generates the views but the content of the live streams (otherwise Andy Warhol’s Empire would be a great film).
* In reality text is ten times faster than video, can be easily referenced,and is technologically a more independent media.

Tip 4 More new experiences

* Argues that marketing agents must keep an eye on what’s coming up next in social media, find out which platforms are winning, and direct experiments on those.
* If the platform or the social media posts fail to connect then use it as a learning experience and incorporate those lessons in the base networks.
* This advice is horribly vague and could be a massive sinkhole of time and effort if followed as presented. Instead they should be providing a metric for experimentation, trigger levels for greater levels of investment etc

Tip 5 More Content Creation, Less Analytics

* Claims that checking in analytics can be addictive; instead of doing it daily, do it every week or so. Instead, concentrate on content creation and follow meaningful trends over weeks, months, etc.
* This is fairly sound advice, although further practical elaboration on how to correlate analytics information with action would be appreciated. Advice is simply to “really connect the dots on where things are going in your marketing strategy”. That’s not exactly helpful from an information systems point of view.

Tip 6 More education and emotion and less selling

* The proposal is to walk through the “buyer’s journey” (see also Tip 9) and teach them how to use the product to its fullest, and give them an emotional boost when they succeed with it.
* This makes sense with complex products with high price tags that can cover the cost of servicing the customer (for example a supercomputer with training included). Not exactly sure how this is especially relevant for consumer products sold through social media, unless it is mass transmission (e.g., training videos) rather than individual coaching.
* Something that probably should have been included is the potential of conflict between emotion and education. Many emotional decisions are not educated decisions. The ideal customer is one who is deeply committed to a product with grounded reasons, rather than just rash feelings.

Tip 7 More Values, Less Guessing

* A corporate body should have “brand values” which become known to the social media followers.
* Consistency and committed to the values in every social media post.
* Values can change through conversations with consumers.
* Again, brands don’t have values but represent values. Values represent the moral integrity of an organisation. Good corporate practise to liaise with independent organisations and potential critics (e.g., unions, environmental groups, workplace advocates etc).

Tip 8 More customer hero, less product hero

* The presentation argues that the structure of the Hero’s journey can be applied to the customer and social media marketing; “people are going throughout something, they come to a struggle, they overcome it and they are better off for it at the end.”
* Part of the process is to get user-generated content on success stories and share them as social proof in product reassurance.
* The great insight of Joseph Campbell, “The Hero with A Thousand Faces” in generating the monomyth (the hero ventures from the ordinary world to the supernatural, confronts great and magical adversaries, returns and bestows new powers to the community), has been converted to a shopping expedition.
* Raising any customer or product to a heroic level is worthy of ridicule. Mortal bravery is required, not a high-limit credit card.

Tip 9 More answers, less questions

* Suggestions to seek follower questions to develop a list of frequent questions; make it a consistent fun activity to invite followers to participate in regular question and answer sessions. Note that private message requests are increasing faster than requests on public forums.
* Curious that the authors are suggesting developing a Frequently Asked Questions (FAQ) which have been on the Internet (originally by NASA) since the early 1980s and arguably in publications since the 17th century.
* The matter of private messaging is probably a result of increased awareness of privacy issues.

Tip 10 More Traffic, Less Work

* Propose "working smarter" and "systemizing your social media strategy". The advice doesn't really come down to more than having a time-based plan for social media content.
* Even if one does adopt a project-like approach to social media one has to consider time, product, and cost (the classic project management triad) along with contingencies, or the classic marketing approach of product, price, and place.
* Just because we’re in the supposedly new world of social media and individualised market segments these hard issues can’t be hand-waved away.
* The best way to develop “more traffic, less work” is to have a good product at a good price when users need it, and then they’ll do a lot of the marketing for you!

Concluding Remarks

* There was some good material in the presentation, with regards to individual approaches and customer engagement, along with correctly identifying these as part of other trends in online communication.
* However much of the material was seriously short on elaboration, over-stated the case significantly, and lack a quantitative evaluation. The signal to noise ratio was very low.
* It is extremely difficult to see how the authors could seriously argue that this material reflected “Ten Social Media Tips for 2019”. Much of it was not related to social media, let alone material that is particularly important for 2019.
* Really, the presentation could have done with an information systems perspective. That is, looking at social media marketing as being a system, with a technological and business workflow, with market segment inputs decision points, triggers and contingencies, and so forth. The positive aspects of the presentation were random, the negative comes down to a lack of a systemic perspective.

February 12, 2019

Two White Paper Reviews

Information Systems and Enterprise Resource Planning

If a broad definition of information systems is taken as "usage and adaptation of the IT and the formal and informal processes by all of its users" (Paul, 2007), then Enterprise Resource Planning (ERP) must be recognised as a major IT application which seeks to combine a very wide range of business processes in an organisation in a technologically-mediated manner. Integration is of primary importance, for example, so that the disparate and siloed software applications that manage customers, sales, procurement, production, distribution, accounting, human resources, governance etc are provided common associations through a database system and from which decision-makers can engage in effective and informed business intelligence and enterprise management.

As can be imagined with such scope, effective ERP systems are highly sought after, with a range of well-known major providers (e.g., Oracle, SAP, Infor, Microsoft, Syspro, Pegasus etc), and a number of free and open-source solutions as well (e.g., LedgerSMB, metasfresh, Dolibarr etc). The main advantages of ERP systems should be self-evident; forecasting, tracking, a systems consolidation, a comprehensive workflow of activities, and business quality, efficiency, and collaboration. What is perhaps less well-known is the disadvantages; the twins of expensive customisation or business process restructuring for the software, the possibility of vendor lock-in and transition costs, and, of course, cost.

Panorama Consulting Services is a consulting company, that specialises in ERP that has been in operation since 2005, providing advice to business and government. They describe themselves as "[o]ne-hundred percent technology agnostic and independent of vendor affiliation" (Panorama Consulting, 2019), who provide a significant number of corporate "White Papers" on ERP-related subjects which, on a prima facie level, makes them a good candidate to begin some ERP comparisons. In particular, two White Papers are reviewed, "Ten Tips for a Successful ERP Implementation" (Panorama Consulting, 2015) and "Clash of the Titans 2019: An Independent Comparison of SAP, Oracle, Microsoft Dynamics and Infor" (2018), the selection in part to have a sense of a foundational paper followed by a practical implementation, and to determine whether their own principles of assessment in the former are actually applied in practical cases.

As an aside, mentioned must be made of the difference between White Papers in government and business. In the Commonwealth tradition, a White Paper is an official policy statement that is developed from an initial brief, and an initial statement designed for public consultation (a "Green Paper"). This differs significantly in the business world where a White Paper has come to mean a brief statement of businesses policy on a complex topic, and is much closer to a marketing tool (Graham, 2015). Government White Papers are significantly longer, more complex, and, arguably, are often the result of more extensive research.

Review: Ten Tips for a Successful ERP Implementation

The first recommendation from Panorama Consulting is for organisations to establish their own business requirements, which must arise from having defined business processes. It is not possible to define ERP requirements without having the business processes (and presumably the technology) already in place. This relates heavily to the major issues noted by Panorama that 75% of ERP projects take longer than expected, 55% are over-budget, and 41% realise less than half of their expected business benefits (based on Panorama's 2015 Annual Report). The reason for these problems are based on "unrealistic expectations", "mismanagement of scope", "unrealistic plans", "failure to manage the software vendor and project scope".

These issues form the basis of the stated tips, i.e., "focus on business processes and requirements first", "quantify expected benefits" to achieve a healthy ERP return-in-investment, "[e]nsure strong project management and resource commitment", "solicit executive buy-in", "take time to plan up front", "[e]nsure adequate training and change management", "[u]nderstand why you are implementing ERP", "[f]ocus on data migration early in the process", "[l]everage the value of conference room pilots", and finally "clear communication ... to all stakeholders" via a project charter.

Much of the White Paper is dedicated to issues that are less related to ERP as such and more on the project management aspects. Certainly there is some justification for this, after all, the implementation of ERP is a significant project. The highest level recommendation, however, ties in well with an information systems approach, that is, ensuring that business processes exist in the first place. If a business has not mapped its own processes the introduction of an ERP will not solve their problems as well as it could because neither the business procedures or culture are in place to effectively use the ERP.

Review: Clash of the Titans 2019

The second, and far more extensive, Panorama ERP White Paper was written some four years after the "Ten Tips". It analyses the major enterprise ERP software providers, (SAP, Oracle, Microsoft Dynamics, and Infor), based on their own ERP Benchmark Survey which was conducted between September 2018 and October 2018. The survey had some 263 respondents, with some 30% using SAP, 29% using Microsoft Dynamics, 25% using Oracle, and 16% using Infor. The top-level comparison from Panorama to organisations is to "consider software functionality, deployment options and vendor and product viability" and, returning to the previous White Paper, "[a]n informed decision also requires business process mapping and requirements definition".

The evaluations taken in the survey include; (i) implementation duration., (ii) operational disruption., (iii) single-system vs best-of-breed., (iv) internal vs external resource requirements., (v) significance of ERP in the organisation's digital strategy., and (vi) business initiatives included in the digital strategy. The results included some interpretation by Panorama, although it is worth noting the questions were more based around business results rather than the underlying technological tools.

From the ERP systems, SAP had the longest duration for implementation (average of 14.7 months), whereas Infor had the shortest (11.2), and by the same token SAP had the longest average operational disruption (128.5 days) and Infor had the shortest (120.6 days). However SAP clients tend towards large, complex, and global organisations, whereas Infor is designed for less cutomisation, and also as a result, tended to be a single ERP system (90.5% of solution set) whereas SAP had the lowest (86.2%), although it should be pointed out that the variation between the providers was not great.

Of some concern is human resource investment required; "vendors typically recommend a team of at least 8-12 full-time internal resources, this isn't feasible for most organizations". Oracle customers have a high of 50.9% of resources sourced internal, whilst Microsoft Dynamics is at at the other end of the scale at 45.4%. Panorama argues that simple customisation can be done in-house, whereas complex configurations require external support. Also of concern is whether or not ERP played a significant role in the organisations digital strategy, which ranged from 71.4% (Infor) to a low of 45% (Oracle), although the latter is explained as Oracle offers diverse components for specific functions. When functions themselves were analysed, a high of 84.6% included ERP in their business strategy, with a low of 8.3% using eCommerce from an Infor installation.

Critique of the Two White Papers

The initial proposal of conducting a review of the second paper based on the suggestions of the first is difficult as the survey as those questions are not directly addressed. Whilst the second White Paper does reiterate the concerns of planning and project management in both the short and long-term for an organisation considering implementing an ERP system, it does not associate the difficulties with the evident disruptions of implementation that ERP systems cause. In a sense, this can serve as a critique of the second White Paper in its own right, that they survey whilst broad in the range of the questions asked related to usage is also shallow, insofar that it relied on some fairly superficial interpretations of Panorama Consulting on why particular results were generated.

A major issue that is not directly addressed by either paper is whether ERP systems are justified, or have a limited justification from an enterprise software perspective. Whilst it would be challenging for an organisation such as Panorama Consulting to give advice that essentially says "No, an ERP system is not for you", it is surprising that in neither White Paper consideration is given to building bespoke systems. These do not need be as "complex but easy" (easy to use, complex to change) expensive enterprise tools would suggest. After all, an ERP is essentially a graphic interface connecting queries to database with stored procedures that incorporates business logic in a stateful manner. An ERP that is developed in-house but with proprietary-free storage and logic engines is relatively easy to change as the business itself develops. Rather than a "solution" it may be worthwhile looking at a "toolkit". It would certainly help with one of the often-overlooked matter that major ERP solutions embody significant cultural biases (Newman, Zhao, 2008).

Further, neither paper engaged in a comparison of technological resources required and consumed, such as requisite operating system and existing software requirements, physical system requirements, or latency and bandwidth. Given the effect that this has in the end-user experience of the system, the performance of the system (especially consider distance between data and processing) and therefore the speed of business intelligence, and the capacity of the system to cover the business needs at scale, one would have thought that such quantitative and objective measures would have had a primary consideration.

The implementation of an ERP system, which suggests an organisation-wide repository of related business data that is updated from various input functions, certainly seems enticing. However, as the first White Paper has correctly argued, an ERP system can only perform as well as the existing business logic and data repositories. The degree that these are incomplete suggests a need that will arise during implementation. Whilst the two White Papers from Panorama Consulting provide some information necessary for the implementation (organisational buy-in, project management) and some information concerning utilising of major packages, the papers lack the necessary systematic perspective for either implementation or comparison, part of which directly comes from the lack of critical quantitative and logical evaluations.


Graham, Gordon (2015). The White Paper FAQ,, Retrieved 16 March 2015.

Newman, M. and Zhao, Y. (2008), The Process of ERP Implementation and BPR: A Tale from two Chinese SMEs. Information Systems Journal, 18 (4): 405-426.

Panorama Consulting (2015), Ten Tips for a Successful ERP Implementation.

Panorama Consulting (2018), Clash of the Titans 2019: An Independent Comparison of SAP, Oracle, Microsoft Dynamics and Infor

Panorama Consulting (accessed 2019), We Are Panorama From:

Paul, Ray J., (2007), Challenges to information systems: time to change, European Journal of Information Systems, 16, p193-195Two Panorama ERP White Paper Reviews

February 09, 2019

The Disciplinary Vagaries of Information Systems


More than thirty years ago, Professor Peter Checkland of the University of Lancaster, raised the question whether information systems (IS) and systems thinking could be united (Checkland, 1988). Almost twenty years later, Ray J. Paul, senior lecturer at the London School of Economics and Political Science also raised the disciplinary status of the subject, as editor the European Journal of Information Systems (Paul, 2007). These two papers are both illustrative of several others (e.g., Banville and Laundry, (1988)., George et. al., (2005)., Firth et. al., (2011)., Annabi and McGann, (2015)) from information systems as it attempts to find its own disciplinary boundaries among the crowd of academia, research, and vocational activities (c.f., Abraham et. al., (2006)., Benamati et. al., (2010).

The two papers are selecting not only to provide an at-a-glance illustration of the time-period of foundational issues within Information Systems as a discipline, but also the temporal context of each paper, and the differences in their views which, at least in part, is reflective of those different times. Drawing from these illustrative comments and from other source material mentioned, some critical issues facing the field of information systems is identified. Rather than attempting to enforce a niche for information systems, a philosophical reconstruction is carried out using formal pragmatics, as developed by the philosopher Karl-Otto Apel (1980) and the social theorist Jurgen Habermas (1984).

Checkland and Paul: A Review

Checkland argues that human beings are "unique" in converting data into information and attributing meaning. Whilst recent studies in animal studies suggest that a more relative term would be more appropriate (c.f., Pika, et. al, (2018)), the point is made. Information Systems are defined as "organized attempts" by institutions to provide information, dominated by the use of computers as the dominant means to carry this into effect. For Checkland a potential conflict between technical experts who avoid the social aspect, and agents of social change who side-step technical expertise, with the discipline lacking in a "Newton" to bring the field together.

In elucidating how this situation came about Checkland notes that the recent history of Information Systems begins with the approach of engineers and statisticians of the 1940s (e.g., Fisher, Wiener, Weaver, Shannon) were primarily concerned with signal-transmission of messages and their distortion, with no reference to meaning. This would be developed in a "semantic information theory" by Stamper et. al., at the London School of Economics, to include semantic analysis. Nevertheless Checkland argues that information systems have "tacitly followed the systems thinking of the 1950s and 1960s". In taking up this argument Checkland applies a phenomenological epistemology where the system is derived from concepts which themselves are a "mutually-creating relationship between perceived reality .. and intellectual concepts (which include 'system')".

Generalized approaches as systems to entities as an "autonomous whole" applies in a variety of fields as diverse as biology, ecology, engineering, economics, sociology etc, from the 1940s to the 1960s and beyond. Checkland argues that this approach correlated with the information technology systems at the time, where such "hard" systems theory where "organizations are conceptualized as goal-seeking machines and information systems are there to enable the information needs associated with organizational goals to be met". In the 1980s new technologies were making it increasingly possible to view organizations as"discourses, cultures, tribes, political battlegrounds, quasi-families, or communication and task networks", where functionalism was being replaced with "phenomenology and hermeneutics", which correlates with the development of Soft Systems Methodology (SSM), meaning-attribution, and emergent properties, and where SSM is a learning system as a whole, and which makes system models.

With this transformation in technology and systems methodology, Checkland argues in the future computer projects will rarely use a project life cycle, except for when "relatively mechanical administrative procedures are computerized". Instead, projects will be increasingly orientated around tasks of perception and meaning where processes and information flows "increasingly need a social and political perspective", which places computing in the hands of the end-user, and where systems are developed so that they can learn and build their own information system. Checkland concludes that this "process orientation offers help with some of the crux problems of information provision in organizations in the 1990s".

Paul's editorial is quite different from Checkland's paper, with the most notable differences being between the difference of twenty years in time and the size of the publication; the editorial is a mere three pages compared to Checkland reaching just over ten. Despite this Paul managed to raise a great number of critical issues with both brevity and power, and with a great deal of hindsight wisdom when compared to the zeitgeist optimism of Checkland. Thematically Paul is concerned with the two topics of challenge and change, noting a variety of past editorials that are concerned with these matters, and notes a connection between them: It is clear to me that there are challenges to IS, and that it is time to address them – it is time to change.

Five challenges are identified which strike at the very core of Information Systems as a discipline and research project highlighting its continuing problematic status. Specifically, Paul states the following: "Nobody seems to know who were are outside the IS community", "Demand from students to study IS is generally dropping", "Research publications in IS do not appear to be publishing the right sort or content of research". In addition. Paul identifies "journal league tables" as being particularly troubling for the discipline, and finally, noting that what are information systems is still subject to significant debate.

Initially referring to the first four challenges, which are less elaborated, Paul suggests that the lack of IS's public recognition is because of its lack of distinct identity, often being located either within business studies or computer science, and suggests that a stronger IS community could provide the strength to overcome these barriers. Whilst the second challenge is an empirical measure it is suggested that community strength would overcome also help with declining enrolments. The third challenge is one of real-world applications versus pure research, with Paul strongly leaning towards the former. As for "journal league tables" Paul makes a case for journals to have a diversity of roles and fitness of purpose.

The final matter is the definition of IS. It is obviously difficult for a discipline to find its place in academia when it is unsure of its own core subject matter, Paul addresses this last point as a matter of major concentration. Taking an approach of via negativa, Paul tries to define information systems by what it is not. Thus, information systems are not information technology, which are the devices that provide the delivery mechanism for information systems. Nor is information systems the processes being used, and nor is the people using the technology and the processes. Instead, to Paul, information systems are the combination of these contributing factors. "The IS is what emerges from the usage and adaptation of the IT and the formal and informal processes by all of its users."

Critique and Reconstruction

Despite the differences in time, size, and even approach, both the papers from Checkland and Paul refer to the issue of identity for IS as a discipline and research project. Checkland provides a historical and philosophical approach to describing this matter and concludes with an optimistic zeitgeist of transformation, partially developed from a degree of technological determinism. Paul's editorial, in contrast, is more concerned with immediate issues and context, although this too is ultimately grounded in ontological and epistemological issues that confront Information Systems.

Checkland's historical argument of the transformation from "hard systems" approaches to SSM are largely accepted within the discipline, although the argument that this correlates with organisations moving from goal-seeking institutions to some sort of extended family is pretty speculative to say the least. Likewise is the claim of processes moving away from functionalism to more phenomenological and hermeneutic approaches, as if these are at variance to each other. After all, functionalism has both phenomenology and hermeneutics as a philosophical foundation. Whilst Checkland is close to transcending these subject-centred approaches with references to speech-act theory and the generation of meaning, they have not engaged in the core principle of linguistic philosophy which notes that shared symbolic values cannot be systematically generated on account of their intersubjectivity (Apel, 1980., Habermas, 1984).

Likewise, Paul's five challenges are not without problems. Two at least are not exactly IS unique problems but rather universal to all academic disciplines, that is the matter of conflict between pure research and practical application (see also Benamati et. al., (2010)) and the issue of journal league tables and fitness to purpose. The matter of IS lacking its own distinct disciplinary identity is certainly a challenge however as Paul's definitional conclusion indirectly notes, this is because it is a multi-disciplinary subject that draws together the organisational requirements in business studies with the technology of computer science. Finally, Paul takes a fairly idealistic approach in arguing that the drop in IS enrolments is due to a lack of an IS community. As Abraham et al (2006) point out, there were particularly historical macroeconomic issues as a major factor, specifically the "" crash of 2000-2002.

A reconstruction of the disciplinary identity of IS is founded on the philosophy of formal (or universal) pragmatics. Initiated by the "linguistic turn" of Wittgenstein (1953), this approach identifies language and meaning as something that is generated between people, and rather than taking a subject-centered view of the world elaborates this to an intersubjective approach. Uniting epistemological considerations with ontological ones formal pragmatics takes a rationalisation of world orientations with specific verification claims.

Unverifiable Metaphysics Physicalist, Symbolist, Idealist Theology
Verifiable Reality Logical and Empirical Philosophy
Orientations/Worlds (verification) 1. Objective or External World 2.Intersubective or Social World 3. Subjective or Internal World
1. Statements of Truth - Sciences (correspondence) Scientific facts Social facts Unverifiable
2. Statements of Justice - Laws (consensus) Unverifiable Legal Norms Moral Norms
3. Statements of Beauty – Arts (sincerity) Aesthetic Expressions Unverifiable Sensual Expressions

Table 1: Elaborated from Habermas (1984), illustrating the pragmatic complexes

In doing so, IS is placed within multiple complexes; not only does it have to deal with the world-orientation of objective facts (information technology), it is equally involved in the world-orientation of social facts (business processes); whereas earlier IS focussed more on the former, SSM was focussed more on the later. Both of these are part of DIKW hierarchy, which refers to the relationship between data, information, knowledge, and wisdom (Rowley (2007). IS is thus a subset of the wider body of knowledge management, which contains all the specific academic disciplines and world-orientations. Whilst the importance of IS will continue to grow due to the increased development of business processes, information technology, and their integration, it will always be a multi-disciplinary subject as it crosses pragmatic boundaries.

One sociological effect of formal pragmatics refers to an understanding of society as a combination of institutional systems and a lifeworld of communities. Procedural action occurs through institutional systems, whereas meaning generation occurs through the lifeworld. In this sense, a system cannot generate meaning and, despite the best efforts of public relations experts, cannot determine the response of communities to their slogans, neologisms, etc. It is very much asystematic, and the best that organisations can hope for is that they have sufficient inputs from the wild world of external communities to be attentive to communities think of them. In this sense, the hopes of Checkland that somehow institutional systems will transform themselves from procedural to meaning-generating bodies is implausible.


Taking the divergent papers of Checkland and Paul a common theme was noted as both attempted to find a foundation on which to base Information Systems as a discipline and research project. Checkland's approach was to apply philosophical considerations to the history of IS and in particular note the transformation of the engineering and biological perspectives of the 1950s and 1960s to the Soft Systems Methodology of the 1970s and 1980s with the possibility of institutional transformation aided by changes in technological devices. In contrast, Paul noted difficulties of identity within IS as it crossed disciplinary boundaries and lacked a referential ability to define itself in distinction to other fields of inquiry.

Both these papers contribute to an ongoing debate on the nature and function of IS. A contribution is made here using contemporary pragmatic and linguistic philosophy with the identification that the challenges confronting IS are actually required for the existence of IS. As long as there is data that is stored in a structured manner as information and that that information is utilised by institutions, then it is inevitable that IS will be crossing the boundaries of instrumental and social technologies. Likewise, as long meaning is generated intersubjectively then there can be no way that an external body can enforce and define the interpretation of shared symbolic values.

By way of conclusion, it must be said that formal pragmatics is obviously not just applicable to IS. As a theory that looks at whether statements of affairs are even verifiable in a pragmatic sense it must incorporate the entirety of the knowledge of our species; not in the content of course, but rather in the categorisation of what sort of statements are possible in the first place. Certainly, there is many things in the universe which we do not know yet, but at least formal pragmatics provides the tools of what sort of questions can be asked, and when propositions are made, whether or not that content seeks verification of truth, justice, or beauty. There are, perhaps further questions outside these areas of verification, but for them, perhaps it is best to refer to two aphorisms of the early Wittgenstein (1922). "The limits of my language mean the limits of my world"; "Whereof one cannot speak, thereof one must be silent."


Abraham, T., Beath, C., Bullen, C., Gallagher, K., Goles, T., Kaiser, K., and Simon, J. (2006) IT Workforce Trends: Implications For IS Programs, Communications of the Association for Information Systems, (17)50, p. 1147-1170

Annabi, H., McGann, Sean T., (2015), MIS-Understood: A Longitudinal Analysis of MIS Major Misconceptions, Proceedings of SIGED: IAIM Conference 2015

Apel, Karl-Otto, (1980), (trans. Glyn Adey, David Frisby), Towards a Transformation of Philosophy, Routledge and Kegal Paul, FP 1972

Benamati, J. H., Ozdemir, Z. D., and Smith, H. J. (2010), Aligning Undergraduate IS Curricula with Industry Needs, Communications of the Association for Information Systems, (53)3, p. 152-156

Banville, C., Landry, M., (1989) Can the Field of MIS be Disciplined?, Communications of the ACM, 32 (1), p48-60

Checkland, P.B., (1988) Information Systems and Systems Thinking: Time to Unite?, International Journal of Information Management, 8, p239-248

George, J. F., Valacich, J. S., and Valor, J. (2005), Does Information Systems Still Matter? Lessons for a Maturing Discipline, Communications of the Association for Information Systems (16)8, pp. 219-232

Habermas, J., The Theory of Communicative Action, Volume 1: Reason and the Rationalisation of Society, Beacon Press, 1984 [FP 1981]

Firth, D., King, J., Koch, H., Looney, C. A., Pavlou, P., and Trauth, E. M. (2011), Addressing the Credibility Crisis in IS, Communications of the Association for Information Systems, (28)13, p. 199-212

Paul, Ray J., (2007), Challenges to information systems: time to change, European Journal of Information Systems, 16, p193-195

Pika, Simone., Wilkinson, Ray, Kendrik, Kobin H., Vernes, Sonja C., (2018), Taking turns: bridging the gap between human and animal communication, Proceeedings of the Royal Society B, 285(1880)

Rowley, Jennifer (2007). The wisdom hierarchy: representations of the DIKW hierarchy. Journal of Information and Communication Science. 33 (2): p163–180.

Wittgenstein, Ludwig (1922), (trans. Frank P. Ramsey and Charles Kay Ogden), Tractatus Logico-Philosophicus, Kegan Paul, FP 1921

Wittgenstein, Ludwig (1953), (trans. G. E. M. Anscombe), Philosophical Investigations, Macmillan Publishing Company.

February 06, 2019

Encrypted connection between SIP phones using Asterisk

Here is the setup I put together to have two SIP phones connect together over an encrypted channel. Since the two phones do not support encryption, I used Asterisk to provide the encrypted channel over the Internet.

Installing Asterisk

First of all, each VoIP phone is in a different physical location and so I installed an Asterisk server in each house.

One of the server is a Debian stretch machine and the other runs Ubuntu bionic 18.04. Regardless, I used a fairly standard configuration and simply installed the asterisk package on both machines:

apt install asterisk

SIP phones

The two phones, both Snom 300, connect to their local asterisk server on its local IP address and use the same details as I have put in /etc/asterisk/sip.conf:


Dialplan and voicemail

The extension number above (1000) maps to the following configuration blurb in /etc/asterisk/extensions.conf:

exten => 1000,1,Dial(SIP/1000,20)
exten => 1000,n,Goto(in1000-${DIALSTATUS},1)
exten => 1000,n,Hangup
exten => in1000-BUSY,1,Hangup(17)
exten => in1000-CONGESTION,1,Hangup(3)
exten => in1000-CHANUNAVAIL,1,VoiceMail(1000@mailboxes,su)
exten => in1000-CHANUNAVAIL,n,Hangup(3)
exten => in1000-NOANSWER,1,VoiceMail(1000@mailboxes,su)
exten => in1000-NOANSWER,n,Hangup(16)
exten => _in1000-.,1,Hangup(16)

the internal context maps to the following blurb in /etc/asterisk/extensions.conf:

include => home
include => iax2users
exten => 707,1,VoiceMailMain(1000@mailboxes)

and 1000@mailboxes maps to the following entry in /etc/asterisk/voicemail.conf:

1000 => 1234,home,

(with 1234 being the voicemail PIN).

Encrypted IAX links

In order to create a virtual link between the two servers using the IAX protocol, I created user credentials on each server in /etc/asterisk/iax.conf:


then I created an entry for the other server in the same file:


The second machine contains the same configuration with the exception of the server name (server1 instead of server2) and hostname ( instead of

Speed dial for the other phone

Finally, to allow each phone to ring one another by dialing 2000, I put the following in /etc/asterisk/extensions.conf:

include => home
exten => 2000,1,Set(CALLERID(all)=Francois Marier <2000>)
exten => 2000,2,Dial(IAX2/server1/1000)

and of course a similar blurb on the other machine:

include => home
exten => 2000,1,Set(CALLERID(all)=Other Person <2000>)
exten => 2000,2,Dial(IAX2/server2/1000)

Firewall rules

Since we are using the IAX protocol instead of SIP, there is only one port to open in /etc/network/iptables.up.rules for the remote server:

# IAX2 protocol
-A INPUT -s x.x.x.x/y -p udp --dport 4569 -j ACCEPT

where x.x.x.x/y is the IP range allocated to the ISP that the other machine is behind.

If you want to restrict traffic on the local network as well, then these ports need to be open for the SIP phone to be able to connect to its local server:

# VoIP phones (internal)
-A INPUT -s -p udp --dport 5060 -j ACCEPT
-A INPUT -s -p udp --dport 10000:20000 -j ACCEPT

where is the static IP address allocated to the SIP phone.

January 31, 2019

20 Years of [Memoirs]

On the first night I arrived in Christchurch, New Zealand for 2019, a group of around a dozen attendees went to dinner. Amongst them were Steve Hanley and Hugh Blemmings, whom I have known since the early 2000’s at various LCAs around the region. They asked for some memoirs of LCA – something small; what follows was my throughts, far longer than expected

Dateline: Just after the Year 2000. The Y2K bug. The first billion seconds of the Unix™ epoch (Sept 9 2001)…

In the summer of 2001, some friends from Perth and I made a trip to a new conference we had heard about called I was a new Debian Linux developer, my friends were similarly developers, sysadmins, etc. What met us was one of the best interactions of like minded individuals we had seen; deeply technical discussions and presentations by key individuals who not only knew their subject matter, but wrote the code, created the community, or otherwise steered a section of the Open Source software movement from around the world. 2001 – day 1

Living on the opposite side of Australia in Perth meant we were intellectually starved of being able to talk face-to-face to key people from this new world of Open Source and Free Software. The distance across the county is almost the same as East to West coast United States, and not many visitors to Melbourne or Sydney make the long trek over the Great Australian Bight to reach Western Australia’s capital.

We found ourselves asking the LCA 2011 organisers if it would be possible in future to run Linux.Conf.Au in Perth one day.

Having had the initial conference (then called the Conference of Australian Linux Users, or CALU) in 1999 in Melbourne, and then 2001 in Sydney, it seemed a natural progression to having LCA roam around different cities year; it felt almost unfair to those who could not afford to travel to Melbourne or Sydney.

The result from 2001 was that in 2002 it would run in Brisbane, but that we should make a proposal and get organised

MiniConfs at LCA

In 1999 I went to AusWeb in Ballina, NSW, and ApacheCon 2K in London.

Closing of ApacheCon 2000 in London – developers on stage Q&A

I also went to DebConf 1 in Bordeaux, France. DebConf was run as an adjunct to the larger French Libre Software Meeting (LSM), as Debian felt that its gathering of developers was too small to warrant the organisational overhead for its own conference at that time.

Debian Developers at DebConf1, Bordeaux 2001

I liked the idea of a pre-conference gathering for Debian for 2002 in Brisbane – a Mini Conference.

So in parallel to talking about running LCA in Perth for 2003, I asked Raymond Smith, LCA 2002 lead organiser, and the rest of the Brisbane organising team if I could turn up a few days early to Brisbane for the 2002 conference, use one of the rooms for a small pre-event.

The principle was simple: minimal organisation overhead; don’t get in the way of those setting up for LCA.

LCA2003 Bid Preparation

The Puffin/Penguin suit, and a small TuxPuffin/Penguin suit

In December 2001 we found what was probably closer to a full-size puffin costume at a fancy dress shop – close enough that it could pass for a penguin.

We started to plan a video as a welcome video – to show some of Perth, and what could be expected in coming to the West.

With a logo I designed from a classic Australian yellow road sign, we had a theme of the Road Trip to Perth.

So with the Puffin/Penguin suit in hand, and a few phone calls, we found ourselves with camera kit on New Years’ Day 2002 at the arrivals hall of Perth airport to film segments for a video to play at the close of LCA2002: the story of tux arriving and making his/her/their way to the conference venue at UWA. Much of the costume performance was Nick Bannon, but also Mark Tearle, and others. I filmed and rough scripted, Tony Breeds edited video, and sought licensing for the music, generously donated by the band credited.

LCA 2002

The MiniConf at LCA ran smoothly. People arrived from around the world, including then DPL Bdale Garbee.

Debian mini-conf at LCA 2002 in BrisbaneJames Bromberger (c) and Bdale Garbee (r) at the Dbeian Mini-Conf, 2002

The main conference was awesome, as always.

Rusty Russell speaking at 2002
Jeremy Allison (l) and Andrew Tridgell (r) tlaking Samba at 2002
Raymond Smith, lead organiser LCA 2002
Ted T’so talking filesystems at LCA 2002. “In Ted we trust”.
LCA2003 closing
James Bromberger (l) and Tony Breeds (r) – co-chairs of LCA 2003
LCA2003 invite video being played at LCA2002 Closing

Post 2002 prep for 2003

We ran monthly, then weekly face to face meetings, we split into teams – web site, papers committee, travel & accommodation, swag, catering, venue, AV and more. Bernard Blackham made significant changes to get us able to process the crypto to talk to the CommSecure payment gateway so we could process registrations (and send signed receipts).

We thought that not many people would come to Perth, a worry that drove us to innovate. Sun Microsystems agreed to sponsor a program we devised called the Sun Regional Delegate Programme, funding a sizable amount of money to fly people from across the state down to Perth to attend

I left my full time job in November 2002 to work full time on the conference, having planned to start travelling in Europe sometime after LCA in 2003. Hugh Blemmings (then at IBM) sponsored Linus Torvalds to attend, which we kept under wraps.

Tux at 2003

A small group worked on making a much better, full size Penguin costume, which days before the opening we proposed to put Linus in as part of the opening welcome.

Holy Penguin Pee, LCA 2003

I sourced some white label wine, designed and had printed some custom labels, naming this the Holy Penguin Pee, which was to be our conference dinner wine (amongst other beverages). While at the time this was a nice little drop; the bottles I now hold some 16 years later are a little less palatable.

Perth 2003

Miniconfs had blown out to several sessions. Attendance was projected to exceed 500 attendees (excluding speakers and organisers).

As the audience gathered for the welcome in the Octagon Theatre at UWA, we had amongst us Tove Torvalds, and their three small girls: Patricia (6), Daniella (4), and Celeste (2).

The opening of 2003James Bromberger (l), Rusty Russel (c), Linus Torvalds (r

As the conference opened, I took to the stage, and the Penguin waddled on. I commented that we have a mascot, and he’s here; Rusty then joined me, and removed the head of the Penguin to reveal Linus within.

Linus in Penguin costume talking to his daughters

Along with Linus, we also had Tove Torvalds, and their three small girls: Patricia (6), Daniella (4), and Celeste (2). During the earlier rehearsal, the girls were so amused to see their Daddy in a penguin suit; there were some lovely photos of them inspecting the suit, and looking at their Dad change the world while having fun.

On that opening welcome – the morning in the Penguin suit – the temperature was over 40°C.

For a non-profit event, we had too much money left over that it was decided to reduce the profit by ordering pizza for lunch on the last day. Days ahead, we drove to a local pizza hut branch, and asked what would be required to order some 300 pizzas, and could they deliver them effectively. We cut a cheque (remember those) two days in advance, and on the day, two minivans stuffed to the roof turned up.

LCA spelt with Pizza Boxes, a slang name for a common form factor of servers around this time, and one of the LCA yellow banners.

Prior to recycling, I suggested we spell out the name of this even in Pizza boxes as a fun tribute to the amount of pizza we all consumed as we cut code, and changed the world. This photo embodies LCA (and appears on the Wikipedia page). I think I took the image from the library balcony, but I may be wrong.

LCA2003 was the first time we had full audio recordings of all main conference sessions. Ogg Speex was the best codec at the time, and video was just beyond us. A CD was produced containing all recordings, plus a source copy of the Speex codec.

LCA2003 closed on 25 January 2003.

Then on the 26th (Australia Day) my then girlfriend and I grabbed our bags, and moved to the UK for 1 year (PS: it was 8).

Roaming around the northern hemisphere

My time in Europe got me to FOSDEM and DebConf many times. I was at UKUUG, tripping through Cambridge occasionally, seeing people whom I had previously met from the Debian community at the LSM in Bordeaux. I met new people as well, some of whom have since made the trip to Australia in order to present at LCA.

James Bromberger (far l), Barry White (l), Pierre Denis (c), Simon Wardley (far r), at the Fotango Christmas party (skiiing) in Chamonix in 2004.

I spent time at Fotango (part of Canon Europe) working with some awesome Perl developers, and running the data centres and infrastructure.

Returning Home

Upon my return to Perth in 2010, I went back to PLUG, to find a new generation of people who were going to LCA 2011 in Brisbane.

I started a cunning plan with the PLUG crew; we put forward a proposal to the Lottery commission for $10,000 to get equipment for us to set up a single stream for video recording using DVSwitch in order to record the regular PLUG meetings.

Euan (l) and Jason (r) playing with DVSwitch at Perth Artifactory in 2011

It worked; a crew came together, and PLUG had some practice at what running a Video Team required (at the time).  I managed to convince Luke John to put forward a proposal to run LCA – it had been nearly 10 years since it had been in Perth – and thus it came to pass.

I, however, was not going to be front and centre in 2014 (though I did give a presentation on Debian and AWS at the 2014 conference).

But I found a new role I could play. With the additional video kit, and a bit of organising, we grabbed a couch and for one year, created LCA TV – an opportunity to grab on video some of the amazing people who come to While we now have great video of presentations, it’s nice to have a few minutes for chat with those amongst us, captured for posterity.

LCA TV 2014

I want to thank LA Council who have had the courage to have LCA wander the region year to year. I want to thank the LCA crews I have worked with in 2003 and 2014, but I want to thank the crew from every year, the speakers who have stood up and spoken, the video teams, and the volunteers.

Looking forward; I want to thank people who haven’t done what they are going to do yet: those who will run LCA in future, and those who will give their time to share their knowledge with others, across countries, languages, companies and more.

Linux.Conf.Au has been central to the success of technical talent in the Linux and Open Source space in this region.

Arriving at CHC Airport, LCA 2019 was present for conference registrations in the terminal.

I have one more person to thank. My then-girlfriend in 2003, now my wife of many years who has put up with me spending so much time attending, planning and running a technical conference over the years. Thanks Andrea.

The Gentle Art of Swedish Death Cleaning


We’ve owned this book for a while, but ironically Catherine lost it for a bit. It seems very topical at the moment because of the Marie Kondo craze, but its been floating around our house for probably a year.

The book is written by an 80+ year old and explains the Swedish tradition of sorting your stuff out before you keel over, which seems like a totally reasonable thing to do when the other option is leaving your grieving kids to work out what on earth to do. The book isn’t as applicable to people not at the end of the lives — it for example recommends starting with large things like furniture and younger people are unlikely to have heaps of unneeded furniture.

That said, there is definitely advice in here that is applicable to other life stages.

The book is composed of a series of generally short chapters. They read a bit like small letters, notes, or blog posts. This makes the book feel very approachable and its a quite fast read.

I enjoyed the book and I think I got some interesting things out of it.

The Gentle Art of Swedish Death Cleaning Book Cover The Gentle Art of Swedish Death Cleaning
Margareta Magnusson, Jane Magnusson,
October 19, 2017

D�st�dning, or the art of death cleaning, is a Swedish phenomenon by which the elderly and their families set their affairs in order. Whether it's sorting the family heirlooms from the junk, downsizing to a smaller place, or setting up a system to help you stop misplacing your keys, death cleaning gives us the chance to make the later years of our lives as comfortable and stress-free as possible. Whatever your age, Swedish death cleaning can be used to help you de-clutter your life, and take stock of what's important. Margareta Magnusson has death cleaned for herself and for many others. Radical and joyous, her guide is an invigorating, touching and surprising process that can help you or someone you love immeasurably, and offers the chance to celebrate and reflect on all the tiny joys that make up a long life along the way.


Introduction to Information Systems & Systems Thinking

What Are Information Systems? (video)

Reflective One Minute Paper

The primary objective here is to define information systems. To do so, one must differentiate between raw, unorganised data, to processed, organised, and structured information that is meaningful. Information is necessary to for behaviour, decisions, and outcomes and can be valued by various metrics (timeliness, appropriateness, accuracy, etc). Information has a life-cycle: Creation (internal or external capture), Existence (Store/Retrieve, Use), Termination (Archive or Destroy).

The scholarly history of Information Systems has developed initially from positivist approaches (business and economics), to include interpretative (sociological) and critical (feminist, environmentalist). The use of Information Systems as a discipline has been especially effective in software development, and in the restructuring of business processes. Pragmatic processes are concerned with capturing and understanding business processes for analysis of that process, with notational systems (e.g., Business Process Model and Notation) being employed.

Three significant questions arise from this review.

Firstly, is there a possibility of transcendence or overcoming (aufhebung) to break the information life-cycle, where iteration leads to qualitative improvement (destruction of information, conversely, would be a qualitative destruction). If this so, then information (and data) should always be at the very least archived and never destroyed.

Secondly, where is Information Systems placed in academia given the influences? This is a major issue for researchers and theorists for decades (e.g., Checkland (1988), Banville and Landry (1989), Paul (2007)), which in part reflects its history of attempting to be positivist, interpretative and critical in approaches. Perhaps overlooked in this analysis is the importance of the earlier Habermas-Luhmann debates between systems and critical approaches in social theory (Habermas, Luhmann, 1971)

Finally, the possibility is raised for the insights of critical approaches in information systems and the iterative techniques of agile project management (Highsmith, 2010) to open up formal mappings such as the Business Process Model and Notation. Applying new decision points that allow for democratic inputs and dynamically changing the model maps would allow for dynamic projects as well as established operations, whilst at the same time providing rigour to agile project management.


Checkland, P.B. (1988), Information Systems and Systems Thinking: Time to Unite?, International Journal of Information Management, 8 p239-248

Banville, Claude., Landry, Maurice (1989), Can the Field of MIS be Disciplined?, Communications of the ACM, Vol 32 No 1, p48-60

Habermas, Jurgen., Luhmann, Niklas (1971), Theorie der Gesellschaft oder Sozialtechnologie?, Suhrkamp

Highsmith, Jim (2010), Agile Project Management, 2nd edition, Addison-Wesley

Paul, Ray J. (2007), Challenges to information systems: time to change, European Journal of Information Systems 16, p193–195. doi:10.1057/palgrave.ejis.3000681

Systems Thinking (video)

Reflective One Minute Paper

Systems Thinking

Reflective One Minute Paper

The following provides a review of the history and approaches to the concept of "systems". Etymologically, the word derives from the Greek sustēma, from sun- 'with' and histanai 'set up' meaning uniting, putting together. The first scientific use of the word comes from Carnot, referring to steam thermodynamics in the early-mid 19th century, with systems concepts being applied in evolutionary and biological sciences (e.g., Darwin, 1850s., Tansley 1910s) with the biologist Bertalanffy (from Bogdanov) developing a "general systems theory" in the 1930s. Wiener provided the notion of cybernetics, the general study of control and communication, in the 1940s, alongside computing sciences with von Neumann, Turing, and Shannon.

'Cybernetics', is derived from the Greek word for steersman or helmsman, who provides the control system for a boat or ship. Note also applies for government, "kyber-ment'. Cybernetics can be applied to itself, and as such second-order cybernetics was developed by Beer et al in the 1960s, which included the role of the observer. Soft systems methodology developed by Checkland, Wilson et al in the 1970s allowed for normative evaluations to be applied within an interrogative systems approach. Finally, Kauffman, Botkin et al in the 1990s developed a systems approach to complexity which identifies disequilibria dynamics in self-organisation.

Overlooked in the lecture is the extremely important contribution of systems theory within sociology and social theory, deriving from the functional sociology of Weber (1900s), the structuralism of Parsons (1950s), the systems theory of Luhmann (1960s), and the neofunctionalism of Alexander (2000s). Second-order cybernetics also has relevancy with Giddens' (1987) double hermeneutic, a distinguishing difference between social and natural sciences, where the observer is observed as well as the subject.

The structure of a system is a static property and refers to the constituent elements of the system and their relationship to each other. The behaviour is a dynamic property and refers to the effect produced by a system in operation. Feedback is information about the results of a process which is used to change the process. The homeostat, the human being, and the thermostat all are said to maintain homoeostasis or equilibrium, through feedback loops, which was promoted as a theory of everything (e.g., Odum, 1959). Whilst feedback plus systems behaviour was meant to provide a self-regulating system, the observed result is disequilibrium and dynamism at least in nature (equilibrium is present in machines).

Of note was the misuse of language from the computer engineering processes to systems theory (e.g., the use of "memory" for "information storage" (primary, secondary, tertiary, dynamic, fixed). Likewise the use of "information theory" instead of "signal transmission theory". Apropos, Shannon (1948) even described "A Mathematical Theory of Communication", an exceptional paper on signal transmission and noise, but which did not touch upon the pragmatics or semantics of language.


Giddens, A., (1987), Social Theory and Modern Sociology, Polity Press, p20-21

Odum, Howard (1959) "The relationships between producer plants and consumer animals, between predator and prey, not to mention the numbers and kinds of organisms in a given environment, are all limited and controlled by the same basic laws which govern non-living systems, such as electric motors and automobiles.", Fundamentals of Ecology, 2nd ed p44

Shannon, C.E., (1948), A Mathematical Theory of Communication, The Bell System Technical Journal, Vol. 27, pp. 379-423 and 623-656, 1948.

LUV February 2019 Main Meeting: All users will eventually die / ZeroTier

Feb 5 2019 19:00
Feb 5 2019 21:00
Feb 5 2019 19:00
Feb 5 2019 21:00
Kathleen Syme Library, 251 Faraday Street Carlton VIC 3053


7:00 PM to 9:00 PM Tuesday, February 5, 2019
Training Room, Kathleen Syme Library, 251 Faraday Street Carlton VIC 3053


Many of us like to go for dinner nearby after the meeting, typically at Brunetti's or Trotters Bistro in Lygon St.  Please let us know if you'd like to join us!

Linux Users of Victoria is a subcommittee of Linux Australia.

February 5, 2019 - 19:00

read more

LUV February 2019 Workshop: TBA

Feb 23 2019 12:30
Feb 23 2019 16:30
Feb 23 2019 12:30
Feb 23 2019 16:30
Infoxchange, 33 Elizabeth St. Richmond


Topic to be announced

There will also be the usual casual hands-on workshop, Linux installation, configuration and assistance and advice. Bring your laptop if you need help with a particular issue. This will now occur BEFORE the talks from 12:30 to 14:00. The talks will commence at 14:00 (2pm) so there is time for people to have lunch nearby.

The meeting will be held at Infoxchange, 33 Elizabeth St. Richmond 3121.  Late arrivals please call (0421) 775 358 for access to the venue.

LUV would like to acknowledge Infoxchange for the venue.

Linux Users of Victoria is a subcommittee of Linux Australia.

February 23, 2019 - 12:30

January 29, 2019

Save the Dates! Linux Security Summit Events for 2019.

There will be two Linux Security Summit (LSS) events again this year:

Stay tuned for CFP announcements!

Distributed Storage is Easier Now: Usability from Ceph Luminous to Nautilus

On January 21, 2019 I presented Distributed Storage is Easier Now: Usability from Ceph Luminous to Nautilus at the 2019 Systems Administration Miniconf. Thanks to the incredible Next Day Video crew, the video was online the next day, and you can watch it here:

If you’d rather read than watch, the meat of the talk follows, but before we get to that I have two important announcements:

  1. Cephalocon 2019 is coming up on May 19-20, in Barcelona, Spain. The CFP is open until Friday February 1, so time is rapidly running out for submissions. Get onto it.
  2. If you’re able to make it to FOSDEM on February 2-3, there’s a whole Software Defined Storage Developer Room thing going on, with loads of excellent content including What’s new in Ceph Nautilus – project status update and preview of the coming release and Managing and Monitoring Ceph with the Ceph Manager Dashboard, which will cover rather more than I was able to here.

Back to the talk. At 2018, Sage Weil presented “Making distributed storage easy: usability in Ceph Luminous and beyond”. What follows is somewhat of a sequel to that talk, covering the changes we’ve made in the meantime, and what’s still coming down the track. If you’re not familiar with Ceph, you should probably check out A Gentle Introduction to Ceph before proceeding. In brief though, Ceph provides object, block and file storage in a single, horizontally scalable cluster, with no single points of failure. It’s Free and Open Source software, it runs on commodity hardware, and it tries to be self-managing wherever possible, so it notices when disks fail, and replicates data elsewhere. It does background scrubbing, and it tries to balance data evenly across the cluster. But you do still need to actually administer it.

This leads to one of the first points Sage made this time last year: Ceph is Hard. Status display and logs were traditionally difficult to parse visually, there were (and still are) lots of configuration options, tricky authentication setup, and it was difficult to figure out the number of placement groups to use (which is really an internal detail of how Ceph shards data across the cluster, and ideally nobody should need to worry about it). Also, you had to do everything with a CLI, unless you had a third-party GUI.

I’d like to be able to flip this point to the past tense, because a bunch of those things were already fixed in the Luminous release in August 2017; status display and logs were cleaned up, a balancer module was added to help ensure data is spread more evenly, crush device classes were added to differentiate between HDDs and SSDs, a new in-tree web dashboard was added (although it was read-only, so just cluster status display, no admin tasks), plus a bunch of other stuff.

But we can’t go all the way to saying “Ceph was hard”, because that might imply that everything is now easy. So until we reach that frabjous day, I’m just going to say that Ceph is easier now, and it will continue to get easier in future.

At in January 2018, we were half way through the Mimic development cycle, and at the time the major usability enhancements planned included:

  • Centralised configuration management
  • Slick deployment in Kubernetes with Rook
  • A vastly improved dashboard based on ceph-mgr and openATTIC
  • Placement Group merging

We got some of that stuff done for Mimic, which was released in June 2018, and more of it is coming in the Nautilus release, which is due out very soon.

In terms of usability improvements, Mimic gave us a new dashboard, inspired by and derived from openATTIC. This dashboard includes all the features of the Luminous dashboard, plus username/password authentication, SSL/TLS support, RBD and RGW management, and a configuration settings browser. Mimic also brought the ability to store and manage configuration options centrally on the MONs, which means we no longer need to set options in /etc/ceph/ceph.conf, replicate that across the cluster, and restart whatever daemons were affected. Instead, you can run `ceph config set ...` to make configuration changes. For initial cluster bootstrap, you can even use DNS SRV records rather than specifying MON hosts in the ceph.conf file.

As I mentioned, the Nautilus release is due out really soon, and will include a bunch more good stuff:

  • PG autoscaling:
  • More dashboard enhancements, including:
    • Multiple users/roles, also single sign on via SAML
    • Internationalisation and localisation
    • iSCSI and NFS Ganesha management
    • Embedded Grafana dashboards
    • The ability to mark OSDs up/down/in/out, and trigger scrubs/deep scrubs
    • Storage pool management
    • A configuration settings editor which actually tells you what the configuration settings mean, and do
    • Embedded Grafana dashboards
    • To see what this all looks like, check out Ceph Manager Dashboard Screenshots as of 2019-01-17
  • Blinky lights, that being the ability to turn on or off the ident and fault LEDs for the disk(s) backing a given OSD, so you can find the damn things in your DC.
  • Orchestrator module(s)

Blinky lights, and some of the dashboard functionality (notably configuring iSCSI gateways and NFS Ganesha) means that Ceph needs to be able to talk to whatever tool it was that deployed the cluster, which leads to the final big thing I want to talk about for the Nautilus release, which is the Orchestrator modules.

There’s a bunch of ways to deploy Ceph, and your deployment tool will always know more about your environment, and have more power to do things than Ceph itself will, but if you’re managing Ceph, through the inbuilt dashboard and CLI tools, there’s things you want to be able to do as a Ceph admin, that Ceph itself can’t do. Ceph can’t deploy a new MDS, or RGW, or NFS Ganesha host. Ceph can’t deploy new OSDs by itself. Ceph can’t blink the lights on a disk on some host if Ceph itself has somehow failed, but the host is still up. For these things, you rely on your deployment tool, whatever it is. So Nautilus will include Orchestrator modules for Ansible, DeepSea/Salt, and Rook/Kubernetes, which allow the Ceph management tools to call out to your deployment tool as necessary to have it perform those tasks. This is the bit I’m working on at the moment.

Beyond Nautilus, Octopus is the next release, due in a bit more than nine months, and on the usability front I know we can expect more dashboard and more orchestrator functionality, but before that, we have the Software Defined Storage Developer Room at FOSDEM on February 2-3 and Cephalocon 2019 on May 19-20. Hopefully some of you reading this will be able to attend :-)

Update 2019-02-04: Check out Sage’s What’s new in Ceph Nautilus FOSDEM talk for much more detail on what’s coming up in Nautilus and beyond.

January 27, 2019

Best Foot Forward


Catherine and I have been huge fans of Adam Hills for ages, so it wasn’t a surprise to me that I’d like a book by him. As an aside, we’ve never seen him live — we had tickets for his show in Canberra in 2013, but some of us ended up in labor in hospital instead, so we had to give those tickets away. One day we’ll manage to see him live though, he just needs to get back to touring Australia more!

Anyways, I enjoyed this book which as mentioned above wasn’t a surprise. What was a surprise is that he said something interesting which I have been pondering for the last few days…

Basically, its nice to get on stage and say things, either entertaining the audience or in my case perhaps educating them a little (I give technical conference talks). However, that’s not the most important thing. You need to work out why you’re on that stage before you go out there. What is the overall thing you’re trying to convey? Once you know that, everything else falls into place. I think this is especially true for keynote speeches, which need to appeal to a more general audience than a conference talk where people can pick from a menu.

What Adam seems to be saying in his comedy (at least to me) is to embrace life and be good to each other. Adam is a super positive guy, which is delightful. There is something very special about someone who lifts up those around them. I hope to be that person one day.

Best Foot Forward Book Cover Best Foot Forward
Adam Hills
Hachette Australia


January 26, 2019

Fedora 29 LXC setup on Ubuntu Bionic 18.04

Similarly to what I wrote for Debian stretch and jessie, here is how I was able to create a Fedora 29 LXC container on an Ubuntu 18.04 (bionic) laptop.

Setting up LXC on Ubuntu

First of all, install lxc:

apt install lxc
echo "veth" >> /etc/modules
modprobe veth

turn on bridged networking by putting the following in /etc/sysctl.d/local.conf:


and applying it using:

sysctl -p /etc/sysctl.d/local.conf

Then allow the right traffic in your firewall (/etc/network/iptables.up.rules in my case):

# LXC containers
-A FORWARD -d -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -d -s -j ACCEPT
-A INPUT -d -s -j ACCEPT
-A INPUT -d -s -j ACCEPT
-A INPUT -d -s -j ACCEPT

and apply these changes:


before restarting the lxc networking:

systemctl restart lxc-net.service

Create the container

Once that's in place, you can finally create the Fedora 29 container:

lxc-create -n fedora29 -t download -- -d fedora -r 29 -a amd64

To see a list of all distros available with the download template:

lxc-create --template=download -- --list

Logging in as root

Start up the container and get a login console:

lxc-start -n fedora29 -F

In another terminal, set a password for the root user:

lxc-attach -n fedora29 passwd

You can now use this password to log into the console you started earlier.

Logging in as an unprivileged user via ssh

As root, install a few packages:

dnf install openssh-server vim sudo man

and then create an unprivileged user with sudo access:

adduser francois -G wheel
passwd francois

Now login as that user from the console and add an ssh public key:

mkdir .ssh
chmod 700 .ssh
echo "<your public key>" > .ssh/authorized_keys
chmod 644 .ssh/authorized_keys

You can now login via ssh. The IP address to use can be seen in the output of:

lxc-ls --fancy

Enabling all necessary locales

To ensure that you have all available locales and don't see ugly perl warnings such as:

perl: warning: Setting locale failed.
perl: warning: Falling back to the standard locale ("C").

install the appropriate language packs:

dnf install langpacks-en.noarch
dnf reinstall dnf

January 25, 2019 should rethink how they present hardware content at


I was originally going to pursue this as a bit of a project in 2019, but that’s no longer possible as I resigned from the papers committee late on Thursday to spend more time with my family. So instead I’ll just throw this out there and maybe something will come of it, but also maybe not.

Should hardware events such as the Open Hardware miniconf or this year’s Home Automation workshop be part of Certainly they’re very popular, often selling out and having to maintain waiting lists in case someone cancels.

Let me take you on an journey and explain why is hard to take hands on hardware content for the conference… I feel qualified to have an opinion on all this having served on the LCA papers committee since 2004, and having been on a conference core team twice.


Hardware assembly sessions are much more complicated to run well than a conference talk. While we expect conference speakers to be well researched and have depth in their field, we don’t expect them to perform flawlessly in an environment with a lot of variables. A static slide deck or maybe a demo or two is all we ask of them.

On the other hand, we do expect hardware assembly sessions to take a group of people with varying skill levels, and get them to assemble non-trivial hardware and have it work at the end. These devices have now reached a level of complexity where they have surface mount components, microprocessors equivalent to to the one in your phone, and software stacks often as large as that on your laptop.

Under recognition of the number of people involved

I haven’t asked the Open Hardware miniconf how many people hours were spent preparing for this year, but I am sure it was a lot. I know that Alastair D’Silva spent a truly huge amount of time on his preparation — think all his spare time for several months while having a small child.

How do we reward that preparation effort? With a single conference ticket. This discourages there from being a team contributing to the project, because at the end of the day only one person can be rewarded with a discount to LCA. This happens because the conference has budget concerns, and can’t give away 20 free tickets to a miniconf without running the risk of failing to break even. It is also because the assumption in the past has been that miniconfs are “conference light” and require less preparation than a main conference talk, and therefore the reduced subsidy for the event was justified.

Lead time

The conference papers review process is fairly complicated these days, with a lot of effort put into promoting the event so that we get the widest range of speakers possible, as well as a diverse review team attempting to ensure that we have good representation of the topic areas you expect to have covered at LCA, as well as diversity in our speaker population. This means that while we normally advertise our call for papers in July, we don’t normally inform speakers that they have been accepted until September.

So, we have complicated hardware projects with a lot of variables, and we give the presenters three months runway to prepare.

One of the obvious ways to lengthen the runway is to just assume your proposal will be accepted, and start preparing earlier. However, that brings me to my next concern…

Out of pocket expenses

Let’s assume you start preparing for your hardware event earlier than September. Let’s also assume that because your hardware is a little complicated and involves surface mount components, that you’ll go through a few prototype rounds and get the final board partially fabricated professionally. This is how things work at the moment.

Guess who gets to pay for that? The organisers of the event. One of those organisers this year was personally out of pocket over $10,000. If the event doesn’t go ahead, or if no one registers for it, then that money is gone. That doesn’t strike me as fair when Linux Australia has $800,000 in the bank at the moment.

The registration process is a nightmare

I don’t know if you noticed, but the registration process for hardware events is also a mess. Again, its up to the individuals running the event to collect money from participants and ensure that there is a fair registration process. The first indications of the level of interest from conference attendees are in the few weeks before the conference itself, when its way too late to course correct if that’s needed.

This is harder than it looks, and comes at a time when the team is flat out ensuring the event runs at all. Worse, then people get annoyed and complain.

Finally on registration, the events are too expensive. In an attempt to simplify registration, there’s only one option — buy the hardware. However, if you already own a Raspberry Pi, why would you buy another one? That’s an extra $60 or so that you’re charged for hardware that you probably don’t really want. If you do want another Raspberry Pi great, but it should be an add on to the event registration.

In summary

Conference organisers generally phrase things in terms of risk versus reward when trying to run a conference. So, the hardware events we have at LCA are high reward for those who attend them, but hugely risky for the conference. Failure of such an event would reflect poorly on LCA, which has worked for 20 years to have a reputation as one of the best technical conferences in the world.

So this is why we shouldn’t run these hardware events — they’re risky, as well as being a super hard slog for the people running them.


But… They’re also hugely popular and an important part of what LCA has become. In a way, LCA is what it is because it embraces a diversity of interests. Without the hardware events, LCA would be diminished from what we have today.

So instead…

If we can’t accept not having these hardware events, we’re only really left with one other option. Let’s blow up the status quo and find a better way of running them.

That’s what originally wrote this post to do — propose something concrete that minimises the impact on the conference organising team while reducing the risk of these events failing and hopefully making life easier for the event organisers.

So here’s Michael’s simple plan for hardware events at LCA:

There should be a separate call for hardware content from now on, for a specific small number of hardware slots at the conference. This should be done really really early. Like stupidly early. As in, opening next month for 2020 and closing in in March. One of the things which holds the call for papers for the main conference up is having the conference web site ready. So let’s not do that — we can use a Google form or something instead. The number of proposals is going to be small, and we can use a “higher touch” process for the review and approval process.

Linux Australia should provide fiscal sponsorship for seed funding to accepted events, based on a proposed bill of materials that is submitted as part of the proposal, on similar terms as is provided for LCA itself and other conference. In other words, Linux Australia should bear the financial risk of failure. Linux Australia should also directly fund conference tickets for hardware event organisers — obviously, we need to somehow validate how many people actually did some work and there need to be limits, but that can be worked through without too much pain.

The LCA website itself should offer registration for hardware events. I see this as relatively trivial in that we can just treat these events as new types of t-shirts and the code to handle that has been around for a really long time.

And finally, we as a community should recognise better the huge amount of effort that has gone into these events in the past (and hopefully in the future) and make a point of thanking the organisers as often as possible. Sometimes there might be warts, but these are human beings like most of us working hard to do something complicated and exciting while still holding down day jobs.

As I said at the start, I had intended to have a go at fixing these issues this year, but events have overtaken me and that will no longer be possible. Hopefully someone else will have a go.

Share 2019 – Friday – Lightning talks and Conference Close

Closing Stuff

  • Special Badge given out
  • Projects from Junior Group from Club Penguin
  • Rusty Wrench award to Joshua Hesketh

Lightning Talks

  • 3 minutes each
  • Martin Kraft
    • Digital trust and trust in the real world
    • In real world it is wired into our brains
    • Aunt’s Laptop. Has Trusted CAs, Debian.
    • Doesn’t know what lock on browser means
    • Imaging that trust is a competition that happens in real time, that takes interactions, mood, random stuff.
    • Maybe when you visit a good vs bad website the browser looks visably different
    • Machine Learning
  • Brimly Collings-Stone
    • Maori language not available on AAC outputs
    • Need a device that speaks Maori and represents Maori grammar accurately
  • Mathew Sherborn
    • RSI
    • Got it in the past, tried various keyboards
    • Type-Matrix but it broke
    • ErgoDox – open source keyboard
    • – Keyboard in batch orders
    • Like the ErgoDox-E – $500 but good – web app to program
    • Change the Dvorak keyboard with special keyboard
  • Emma Springkmeier
    • What do I do when it all goes wrong
    • Potentially stressful situations – phone calls, meetings.
    • eg last year’s lightning talk
    • What I do to cope
    • Talk to friends, explain how I feel to others, listening to calming music, breathing techniques ( 4s in, 4s hold, 4 out, 4 holding, repeat )
  • Karl Kitcher
    • Secretary of the NZ Open Source Society
    • Charity since 2008
    • Reducing in interest in the recent years
    • Open source is not so prevalent, people not really caring, trying to maintain the momentium
    • Open vs Fauxpem
    • – signup to the mailing list
    • Various services to projects
  • Leon Wright
    • About Leon’s badge
    • Twitter bot hooked to hug detector in his badge
    • 2017 badge detects hugs
    • 2018 version 2 . So good twitter shadow banned his account
    • 2019 – Docker containers and other crazy stuff
  • Talia White
    • At LCA since 2018 – Was only 8. Now 12
    • Ordered a robot kit for ardiano
    • Made various projects
    • Don’t give up, struggled to start with coding, got better
  • Brenda Wallace
    • Works for the NZ Govt
    • Sometimes abigious
    • Going to publish for some legislation as python rules
    • eg Social welfare rules,
    • Unit tests
  • Paul Gunn Stephen
    • GDP per km of coastline
    • %coastline length for area
    • Means hard to get Tsunami warning systems
    • Cheaper
    • ETC Lali system approach
    • Every Village has a local warning system
    • Redundant system
  • E Dunham
    • You should speak at conferences
    • 54th talk in 5 years
    • Promotes your company
    • Intersection: What you know, what conference needs and what the attendees needs
    • Find conference want to attend
    • Write abstract
    • Submit a lot, get rejected a lot
    • Each reject is how you dodged a bullet
  • Charell
    • CVE-2019-3462
    • Bug in apt that allows injection of bad content
    • Why https
    • Attestation
    • apt-transport-https – enable
  • Jen Zajac
    • Project scaffolding eg Cookiecutter, yoeman
    • Lots of generating options
    • Creates templates for a project
  • Hugh Blemmings
    • Ardionu and Beagleboard
    • Cool but not high performance
    • A True open and HP computer
    • Open Hardware, Open software stack, no bin blobs, No unexpected software, No cost/perf penality
  • Benno Rice
    • Cobol
    • Over 50 years old
    • Not used much
    • What Language is the new Cobol?
    • PHP is the new COBOL
    • Perl is the new COBOL
    • Python2 ?
    • Javascript ?
    • C ?
    • Y2K – Maybe the real Cobol is the maintenance we incurred along the way
    • Maybe you should support software before it bites you back

Closing Stuff

  • 652 people attended
  • 2.4TB transferred over the SSID
  • 3113 Coffee vouchers

Lots of sponsors and suppliers and staff thanked 2020 is in …. Gold Coast

  • 21st birthday!
  • Gold Coast convention and Exhibition centre
  • 13 – 17th January 2020

Share 2019 – Friday – Session 2

OpenLI: Lawful Intercept Without the Massive Price Tag
– Shane Alcock

Shane Alcock
  • Police get Warrent to ISP
  • ISP Obligations
    • Can’t tip off person being intercepted
    • Both current and past intercepts must be private
    • Can’t accept other people’s communications
    • Must accept all communications
  • NZ Lawful Intercept
    • All Providers with more than 4000 customers must be LI capable
    • Must be streamed live
    • TCP/IP over tunnel
    • Higher level agencies have extra requirements
    • 2 seperate handovers – IRI metadata for calls, IP sessions. CC= data packets
  • Open LI
    • $10,000s – $100,000s costs to impliment and license from vendors
    • WAND had some experise in packet collection
    • Known my NZ Network Operator community
    • Voluntary contributions from NZ Network Operators
    • $10k+ each
    • Buys 50% of my time to work on it for a year.
    • Avoiding Free Rider problem
      • Early access for supporters
      • Dev assistence with deployment
      • Priority support for bugs and features
  • Building Blocks
    • Developed and tested on Debian
    • Should work on other Linux flavours
    • Written in C – fast and likes writing C
    • Use libtrace from WAND
    • Data Plane Develop Kit
  • Provisioner
    • Interface for operators
    • Not very busy
  • Collector
    • Comms from Provisioner
    • Intercept instructions
    • Recommended run on bare-metal
    • 1RU Server with 10G interface with DPDK support
    • Supports multiple collectors
  • Mediator
    • Gets data from Collector
    • Forwards to Agency based on instructions from Provisioner
  • Target Identification
    • Nothing on the packets linked to target user
    • People get dynamic IPs, can change
    • For VOIP calls need to know RDP port
    • SIP for VIP , Radius to IP to ID the user’s IPs/Ports
    • Deriving caller identities from SIP packets can be tricky. Other headers can be used, depends on various factors
  • Performance Matters
    • 1Gb/s plans are available to residential customers
    • ISP may have multiple customers being intercepted. Collector must not drop packets
    • Aim to support multiple Gb/s of data
    • libtrace lets use spread load across multiple interfaces, cpus etc
    • But packets now be in multiple threads
    • Lots of threads to keep things all in sync
  • Status
  • Future
    • Build user-driver community around the software
  • Questions
    • Can it handle a hotel? – maybe
    • ISPs or police contributing? – Not yet
    • What have people been doing so far? – They have been gettign away with saying they will use this
    • What about bad guys using? – This probably doesn’t give them any more functionality
    • Larger Operators? – Gone with Vendor Solutions
    • Overseas Interest? – One from Khazakstan , but targetted at small operators
    • Why not Rust, worry about parsing data – Didn’t have time to learn Rust

But Mummy I don’t want to use CUDA – Open source GPU compute
Dave Airlie

Dave Airlie
  • Use Cases
    • AI/ML – Tensorflow
    • HPC – On big supercomputers
    • Scientific – Big datasets, maybe not on big clusters
  • What APIs Exist
    • CUDA
      • NVIDIA defined
      • Closed Source
      • C++ Based single source
      • Lots of support libraries ( BLAS, CiDNN ) from NVIDIA
    • API – HIP
      • AMD Defined
      • Sourcecode released on github
      • C++ based single source
    • OPenCL
      • Khronos Standard
      • Open and Closed implimentations
      • 1.2 v 2.0
      • OpenCL C/C++ Not single source (GPU and CPU code separate)
      • Online vs offline compilation (Online means final compilation at run time)
      • SPIR-V kernel
    • SYCL
      • Khronos Standard
      • C++ Single source
      • CPU Launch via OpenMP
      • GPU launch via OpenCL
      • Closed (codeplay) vs Open(triSYS)
      • Opening of implementation in Progress (from Intel – Jan 2019)
    • Others
      • C++AMP – MS
      • OPenMP – Gettign better for GPUs
      • OpenACC
      • Vulkan Compute
        • Low level submission API
        • Maybe
    • Future
      • C++ standard
      • C++ ISO standards body, ongoing input from everybody
      • Implementations must be tested
      • Still needs execution environment
  • Components of GPU stack
    • Source -> Compiler
    • Output of GPU and CPU code
  • IR
    • Intermediate representation
    • Between source and final binary
    • NVIDIA PTX – liek assemble
  • OpenCL Stacks
    • Vendor Specific
    • LLVM Forks
  • Open Source
    • Development vs Release Model
    • Vendors don’t want to support ports to competitors hardware
    • Distro challenges
      • No idea on future directions
      • Large bodies of code
      • Very little common code
      • Forked llvm/clang everywhere in code
  • Proposed Stack
    • Needs reference implementation
    • vendor neutral, runs on multiple vendors
    • Shared Code based (eg one copy of clang, llvm)
    • Standards based
    • Common API for runtime
    • Common IR as much as possible
    • Common Tooling – eg single debugger
    • SPIR-V in executable -> NIR -> HW Finaliser
    • Maybe Intel’s implementation will do this
  • Questions
    • Vulkan on top of Metal/Molten ? – Don’t know
    • Lots of other questions I didn’t understand enough to write

Share 2019 – Friday – Session 1

Preventing the IoT Dystopia with Copyleft- Bradley M. Kuhn

Bradley M. Kuhn
  • The S in IoT stands for Security
  • Many stories of people hacking into baby monitors and home cameras
  • IoT Devices often phone home to manufactorers website in order that you can access then remotely. “I suppose there are Chinese hackers watching my Dogs all day, I hope they will call me if they need water etc”
  • Open source people have historically worked to get around problems like this.
  • 1992 – If you wanted Linux, you downloaded the software onto floppies and installed it yourself. And Often had to work hard to make it work.
  • Today only a small percentage of laptops sold have Linux on it.
  • But Linux is commonly installed on IoT devices – 90% odd
  • But
    • No [easy] way to reinstall it yourself
    • Much worse than laptops
    • GPL includes “The scripts used to control the compilation and install of the executable”
    • “Freedom to Study” is not enough
  • Linksys Wifi router
    • OpenWRT Project
    • Release forced from Linksys and Cisco
    • “Source as received from Linksys from GPL enforcement”
    • Is OpenWRT a Unicorn
      • Few projects with serious alternative firmware project
    • Still sold new after 20 years
  • BusyBox Lawsuits
    • Before IoT was even a term
    • At least one model of Samsung TV ->
    • “Baffles me as to why do the manufactorers want us to buy more hardware”
  • Linux focuses to much on big corp users and ignores hobbyist users
    • Kernel peopel only care about the .c files. Don’t care about the install scripts etc.
    • People at top of Linux now got their start hacking on the devices in front of them.
    • The next generation of developers will be those hackers not from IBM and other big companies
    • You didn’t need anything but a computer and an internet connection to become and upstream developer in those days. This is becoming less true.
    • If the only thing you can install Linux on is a rackmount server, a cloud server or maybe a laptop and none of the IoT devices around you then things don’t look good….
  • Linux was successful because users could install it on their own devices
  • Linux won’t remain the most important GPL program if users can’t install their modifications. Tinkering is what makes Free software great.
  • Upstream matters of course, but downstream matters more.
    • There may be 1000s of Linux developers
    • Put 2 billion people have Linux on their phone – Which is locked down and they can’t reinstall
  • We don’t need a revolution to liberate IoT devices
    • because the words are already there in the GPL
    • We just have to take up our rights
  • What you can do.
    • Request Linux sources on every device you own – Companies have figured out people almost never ask
    • Try to build and install them. If you can’t ask a friend or ask Conservancy for help
    • If it doesn’t build/install it is a GPL violation, report it Conservancy
    • Step up as a leader of a project devices that matter to you.
  • Why this will work
    • The problem seems insurmountable now, only because we have been led astray
    • First and absolutely necessary step towards privacy and scurity on those devices
    • When the user controls the OS again, the balance of power can be restored
  • Questions
    • Best way to ask for source code? Try email, the manual should say.
    • How to get the new code on the device? Needs some push onto industry
    • What if writing requires expensive equipment? Fairly rare, many devices allow over-the-air upgrades, we should be able to go the same way.
    • Is there a list of compliant devices? – Proposed in past. Want to go softly at first in many cases
    • Am I exposed to liability if I modify and distribute code I receive? – Almost certainly note, contact Conservatory if you are threatened.

Web Security 2019 – James Bromberger

James Bromberger
  • History of browser
    • No images
    • Images
    • Netscape with crappy ‘International Security”
    • https takeup is growing
    • Chrome is hitting 60-70%
    • 82% of browser are “modern”, crossover of chrome users to new version is about 3 months.
  • PCI
    • Remove early TLS in mid 2018
    • TLS 1.1 and higher allowed
  • The legacy browser has gone in the real world
    • Some envs still behind, but moving ahead
  • What can we do with as little changes as possible?
  • 0. Don’t use http, use https
    • Use letsencrypt
    • Stds reducing max length of certs from 5 years
  • 1. TLS protocols
    • 7 versions out there (old ones SSL).
    • Most over 10+ years old
    • Only 6 in the wild
    • 3 not-known to be comprimised ( 1.1 1.2 1.3 )
    • Very few clients only support 1.1 and not 1.2 (small gap in 2006-2008 ). IE supports 1.2. So maybe disable 1.1
    • Log the protocol being used so you have data on your users
    • OTOH not much supports 1.3 yet
    • Use 1.2 and 1.3
    • Turn off on the Browsers to
    • Looks at which libraries you are using in code that makes https connections
  • 2. Cypher Suite Optimisation
    • New EC certs for key exchange
    • New certs getting changed to ECDSA
    • AES is standard for bulk encryption. GCM mode is best although windows 9 can’t do (Upgrade to 10!)
    • MAC/Cehecksum – remove MD5, SHA1, remove SHA2-256+ , New ones coming
  • Security Header
    • Content-Security-Policy
    • Referer-Policy – Usually locked down
    • Feature-Policy – lots of stuff
    • ” X-Content-Type-Options: no-sniff ” – don’t guess content type
  • 4. CAA
    • Around 200 Cert Authorities
    • Authorized record type (CAA record) lists what CAs are allowed to issue certs for you.
    • DNS Sec is useful – But during US Govt shutdown DNS keys are expring
  • 5. Sub Resource Integrity
    • Scripts included by html
    • Can include checksums in html calling to varify
  • 6. Cookies
    • Secure – httpsonly
    • “SameSite=Strict” – Reduces cross site request forgery
  • 7. Http2
    • Binary wire protocol
    • Apache 2.4 on debian
    • Forces better protocols
  • 8. Lots more
    • New compression algorithms
    • Network error logs


January 24, 2019 2019 – Friday – Keynote: A Story – Rusty Russell

Rusty Russell
  • Bitcoin Billionare
  • 1992
    • The days of SunOS
    • Read the GNU Manafesto
  • 1995
    • Using GPP compiler at work
    • First patch accepted on November 1995
  • 1997
    • USENIX Conference in california
    • UseLinux – Had a track for Linux
    • Hung around a bunch of top guys in Linux talked about added SMP to Linux
    • Talk on porting Linux to Sparc by David Miller & Miguel de Icaza. Going into improvements and showing how Linux port to sparc bet Solaris in the Lmbench benchmarks on same hardware.
    • Relaized lived in a world where students could create and port OS that bet the original OS from the vendor
  • 1997 – 1998
    • Wrote (with another guy) and got ipchains added to Linux
    • “I woke up one morning and I was kernel firewall maintainer”
    • Got job people paid to work on Linux firewall code
  • 1998
    • Decided needed an Australian Linux conference
    • Oct-Nov visited a bunch of LUGS to invite people and find person to collect money.
    • People not sure what they wanted to go to a Linux conference ( $380 bucks)
    • Invited John Maddog Hall
    • Created and ran a slashdot ad
    • Created card got into $14k negative
    • Last session of the 3rd day, reran the 3 best talks
  • Three stories from 1998
    • Tutorial Books for each of the tutorials- Couldn’t get photocopies from commercial facility, so had to make 400 copies of books via 4 coin operated photocopiers
    • Tridge bought up a triple-CD burner. People ran it in relays
    • Somebody said. “I can’t believe you don’t have conference tshirts”. He bought white tshirts, got them screen printed and sold them.
  • End of conference Tridge organised a gift from the Speakers to Rusty. Pewter Beer mug
  • after 1999
    • 2001 scheduled 3 talks from Rusty. At the same time
    • Met Tridge at LCA – Moved to Canberra they did AusLabs
  • How Great Projects
    • Smart and Capable enough to complete them
    • They are Dumb enough to try
    • When somebody tells you about a project?
      • That sounds Great, Tell me more
      • What can I do to help
    • Enable people’s enthusiasms
    • Collaboration is a super Power
    • Get along with people is a skill
    • “Constructive absenteeism”
  • Headwinds to collaboration
    • Signs are welcoming to some people
    • Other people get signs that they are not so welcoming
    • Good are seeing them when they are aimed at them, not so good are even seeing they exist when they are not aimed at them.

Share 2019 – Thursday – Session 3

Open Source Tools for Publishing and Processing Earth Observation Imagery – Paul Haesler

Paul Haesler
  • Golden age of satelite imagery
  • Geostationary – One area – Good for weather
  • Circum-polar orbits – all over earth every 10-16 days
  • Data processing Chain
    • Level 0 – Raw
    • Level 1 – Geo rectify – Measure surface radience
    • Level 2 – Curroect for sun, sat angle, atmostphere – ARD – Records surface reflectance
  • Landsat-8 (25-30m , 8-16 day cycle, data since 1982 ) Sentinel-2 ( better, 5 day cycle, 10m resolution)
  • Digital Earth Australia
  • The Problem
    • Open Data Cube
    • Python, based on xarry
    • Postgres for metadata
    • Actual satellite sata from local or network repo (transparently)
    • GUI maintained by CEOS
    • Aims to publish all Aus OpenData that can be mapped
    • Based on TerriaJS
    • Some DEA data was already being publish but need for additional stuff
  • COGs – Cloud Optimised GeoTIFFs
  • DataCube_ows
    • Lightweight web application server
    • Developed by Datda61 for GA
    • WMS
      • OGC Web Map service
      • Good for general-usage web apps
      • Returns standard images (eg png)
      • Support 1.3 well, works with most clients
      • Styles for band-mapping
      • on-the-fly solar angle correction
    • WCS
      • Version 1 supported
      • Works well with TerriaJS , works okay with QGIS or ArcGIS
  • Next Steps
    • WPS for on-the-fly processing is regularly discussed
    • Better ingegration with datacube-core
    • More recent WCS versions inc WCS-2EO
    • Sparse Data problems

The Tragedy of systemd – Benno Rice

  • References to Contempt Culture
  • Ancestry of Systemd
    • Unix: Happy accident, place, time, reaction to the previous thing
    • housekeeping functions – “mounting filesystems and starting daemons”
    • inetd – Super Daemon for all sockets – “worked well until The Internet Happened”
  • Then the Internet happened
    • forking a process per connection doesn’t scale
    • Lots of persistent state for things like databases
    • Service
      • Might be a bunch of processors
      • Init starts but doesn’t manage
      • initab can restart things in SystemV
  • System Config vs Service bootstrap
    • Mixed in togeather
    • Service management needs more
  • Windows NT
    • Service model there from beginning
  • MacOs
    • Application Model means lot richer interaction with the host
    • Application delegate
    • launchd
  • The Idea of Systemd
    • launchd
      • Service handling in MacOS
      • Took over init, inetd, cron
      • Can listen on ports. Start stuff. doesn’t need to start on boot, boot gets faster, power reduced, security improved
      • Move system services to daemons, then start daemons as needed
    • From Launchd to systemd
      • upstart
        • event driven
        • shell based
      • Rethinking PID 1 – Lennart
      • “Start less” , “Start more in parallel” , “listen to hardware and software changes”
      • cites launchd
  • System management
    • Everything is a lot more dynamic
    • Hotplug , DHCP , etc
    • Don’t install 15 different packages that all behave differently
    • But systemd will have to do things in a different way to those 15 other things
  • The reality of systemd
    • Widely adopted ( 2011 – 2015 )
    • Arguments
      • Violates the unix philosphy – actually systemd actually is many binaries
      • It is bloated and monolithic – Well it does do a lot of thing
      • It is buggy – So is all software, actually a good failure mode
      • I can’t stand Lennart Poettering – He’s delivered. “I won’t defend his community interaction”
      • It is not portable – UNIX is dead – Posix isn’t really a thing anymore, there are not a bunch of crazy Linux variations. “These days you have Linux and some rounding errors”
    • cgroups
    • User-Level units
  • Change – System is a lot of disruptive change
  • The Tragedy of Change
    • Nerds love change as long as we are the ones doing it
    • System boot ups using shell script interaction is like the old blanky we should of got rid of 20 years ago
    • The Knee-jerk – Abuse is not Cool
  • The Next Generation
    • They See a lot more APIs
    • Thinking in Containers is different from thinking in not-containers
  • What does Systemd have that FreeBSD (or even future Linux) could use, or could do better
    • Message Transport
    • RPC Framework
    • Kernel and Use-space services should look similarly to the services above them
    • Service Lifescycle
    • Automation via API – Easier for vendors to write appliances
    • Containers
    • The System Layer
      • Doesn’t have to be the only implementation of theis
    • Consistent Device Naming
    • Better Log/Event/Audit Handling
    • A new model of an application ( a bunch of things managed as a Unit, See the MacOS model)
  • Questions
    • Launchd option – Too MacOS specific
    • Dynamic Libraries = DLL Hell – Containers avoid, different problems
    • Is reaction to systemd scaring other big changes off – Possible, hard to write, very hard to handle the social issues to push though
    • Where is FreeBSD at? – A long way away, no consensus this sort of change needed
    • Should everything have been swallowed up – Thought experiment, If systemd had instead defined an API for separate projects instead of writing them itself, would that have worked? And now we do know what is needed could we switch to a separate model with APIs?
    • Enbeded Devices need systemd – Anything Dynamic needs it
    • What Push back from FreeBSD – Something like that but not systemd. Some like launchd
    • What needs to change in community and systemd team to make things better – See Adam Harvey’s talk on language changes. Hard since everythign is asking for different stuff, systemd people.
    • What should systemd go further into – Messageing and RPC stuff more pervasive and more thought about. Something into the kernel.

Share 2019 – Thursday – Session 2

Go All-In! – Bdale Garbee

Bdale Garbee
  • How to get companies involved in FOSS
  • First contribution of source code was almost 40 years ago
  • Used Vax BSD at CMU – HAd the deal with an obscure priesthood
  • KA9Q TCP/IP stack for amateur radio in the 80s
  • Appearance of RMS in my world
    • GNU Manifesto
    • 4 Freedoms
    • GPL
  • Debian
  • Worked as LInux CTO from one of the largest IT companies in the world
  • Collaborative Development Model
    • Spread out besides just making FOSS
    • No one company in charge
    • Diverse range of contributors, massively different motivations
    • We get a software commons we all get to benifit from
  • Free Software means Freedom of Choice
    • Reduced barriers between users and producers of software
    • Any user can be a dev, or pay someone to dev
    • If upsteams goes bad, things can be forked
  • What it means to be successful when you are operating in an open and collaborative model?
    • The goal of a trad company is for investment to yeld technological control points
    • First mover advantage
    • Differentiated features, preferably patentable
    • Collaborative dev model allows us to recognise the benifit of collaroation on all the non-differentialting elements. Leavign more value to the users / customers
    • Thinks less about control points, more about points of affinity. What is it that would make a customer want to user you products or services?
  • Innovation these days largely takes place in the open space
  • Wrights/Goddard – They didn’t get told to to the next new thing, they just started it as a hobby
  • Free Software enables people who we don’t know exist to create innovation and invent things we can’t imagine
  • Long Tail of Contribution
    • Example: People who did one Linux Kernel contribution, often to fix on specific thing that was causing them problems.
    • No company on earth that can hire that resource
    • Needs to be easy for people to access the code and make contributions
  • Attributed of Successful Communities
    • Active contribution and collaboration
    • Diverse participation
    • Solid core of code
    • Recognizable mainline trunk
    • Unified, cohesive structure
    • Low barriers to entry
  • Choosing the right license
    • Businesses can only be successful with permissive licenses
    • The most successful projects seems to be communities built around open contribution
    • Share-alike licenses stop possible problem of Closed Corporate fork while the original project withers

Beach Wreck Ignition: Challenges in open source voice –
Kathy Reid

Kathy Reid
  • MycroftAI – One of the few open source voice stacks
  • Introduction to a Voice Stack
    • Wake Word – eg “Hey Alexie”
    • Utterance – Phrase of command
    • Speech2text processor
    • Looks for keywords etc
    • Runs a command
    • Dialog – acknowledger + response
  • Wake Word
    • PocketPhinx, Snowbox, Mycroft Ai Precise
    • Some use Phonemes (smallest units of sounce in a language)
    • Hard to tell differences between all words
    • Always listening, connected to internet
    • Some use Use Neural networks
    • Low accuracy can cause frustration
      • Bias towards male speaker (10:1 male:female in dataset). Also more with American than other accents
      • To unbias the sample had to tag the samples with ethnicity, gender etc. Which was a problem with ethics of taggign samples/speakers
  • Speech to Test
    • Kaldi – no network needed, compute heavy
    • Deep Speech – From mozilla
    • Challenges
      • Lots of accents out there. Hard
      • Only trained for most common accents
      • Also problem with regional slang
      • Need to train on individual speaker
      • But need lots of data to understand a speaker
  • Endangered Languages
    • No commercial imperative to cover them
    • Mycroft Translate using Pootle to translate command words to 40 languages
    • Issues for gendered languages, formality
  • Intent Parsers
    • Rasa, Mycroft Adapt, Mycroft Padatious
    • Intent Collisions – Use confidence scoring depending on how explicit the request is.
  • Text to Speech
    • Mary TTS, Espeak, Mycroft Mimic, Mycroft Mimic 2
    • Mimic recording studio, Need 40-60 hours audio
    • Challenges
      • Natural sounding voice – making the voice sounds not robotics
      • Pronunciation – often requires after creation

Share 2019 – Thursday – Session 1

A Long Day’s Journey Into Backups – Rachel Kelly

Rachael Kelly
  • A journey in four stages
  • Version 1 – State of Backups Aug 2017
    • Needed to look though old logs to see how far back problem wentDaily diffing from duply/duplicity – But where was the original?
    • Tried to restore data from old backups
    • Couldn’t restore from backup since original was too old
    • Couldn’t get it to work, needed something new fast
  • Version 2
    • Created tarball, uploaded to AWS S3, via daily cron
    • Done quickly
    • Not reliable sending to s3
    • Needed ongoing work
    • Big Win: But at least complient, and we could get data back out
  • Try 3 : Shiny EFS
    • EFS is AWS’s NFS solution
    • tarball created on local EFS disk, easypeasy
    • Big Win: Reliable backups, incorporated into infrastructure, retrievable data. 8 weeks off backups
    • Miscalculated cost: About 10x original estimate
  • Try 4: Tarball to s3 redux
    • Tarball to s3. PLUS infrastructure
    • Would work going forward
    • S3 approx 1/10 the cost of EFS
    • Big Win: Reliable, inexpensive, functional, dependable
    • Discovered that EFS not encrypted
    • Able to manage well
  • Current Solution
    • cron job calls a bash script
    • Safety in bash: set -euo pipefail
    • tar up the filesystem (with exclusions)
    • Set it to s3 with aws cli (with hosts’s hostname)
    • After 56 days is sent to glacier
    • restore script to get a file (uses AWS credientials of current instance)
  • What’s Next?
    • I could work on backups forever and they will never be perfect
    • Ability to restore to a new instance
    • Want to be able to get files from anywhere
    • Microservice backups
    • Deglacierisation
    • What we need
    • Better CLI tool – safer
  • What I really really want
    • A decent enterprise solution
    • NIH is a dangerous habit
    • Speaker knows no one using a hosted enterprise solution
    • Vendor solutions seem to be crazy overkill
  • Feedbacks
  • Amanda Recomended
  • Every morning restore dev enviroment from anon prodouting one. Comet Backup
  • Wasabi compatable with s3
  • Recoment “Retrospec”
  • tarsnap
  • bacula
  • rsnapshot
  • Looking at Borg Back
  • rsync to zfs, zfs send
  • Personal backups, using duplicity
  • Industry 4.0
    • After mechanization, mass production and automation
    • The machines run everything and we reap all the benefits – maybe
  • Robot Hype
    • Post AI winter
  • Implementoer’s bias (top down design of neutral network setup, choose number of layers etc)


January 23, 2019 2019 – Thursday – Keynote: Shannon Morse

Personal Branding for the Security Conscious – Shannon Morse

Shannon Morse
  • Who am I
    • Youtube videos on Infosec, Travel
  • Imposter Syndrome
    • Work hard to beat it.
    • Say Yes to offers
    • Work hard to make somethign I am proud of
    • Surround yourself with positive people
  • Accomplishment
    • Keep a list of them, be proud of them
  • Backstory
    • No background in Linux, hacking, infosec
    • Mom and Dad supported me
  • RTFM Sucks
    • Lots of egos and elitism in forums and community online
    • Decided to become the resource for learners
  • Starting your career
    • What do companies need, what hiring for
    • How has industry changed?
    • Diversity numbers?
    • Can you change industry in a positive way
    • Review CERT holders numbers vs Openings looking for those certs
    • Look at job titles being advertised
    • Industry growing -> lots of beginners
  • How can you get good at it
    • Understand what is your best way to learn
    • Read books, classes, videos, whatever
    • Compile your list of passions
    • Get list of influencers / thought leaders / speakers in the area
    • Follow them on social media
    • Learn from your role models
    • you might end up being a thought leader in their eyes
    • Follow people in other areas too
  • Keep learning
    • Do it every single day
    • Make it become a habit
    • Make it a routine
  • Resume
    • Create a one-pager
    • Business cards
    • Dropped out of college put “Huitus”
  • Build you platform
    • Youtube, write articles, videos, whatever
    • If you can afford it, offer free classes for under-represented groups
  • Personal brand
    • Develop the blocks
      • skillsets, values, what does it mean for you to succeed
      • What obstacles have you overcome
      • what are you passions
      • what makes you unique
      • write and live by your vision statement
    • If you don’t control and manage your brand others will do it for you
      • Where do you draw that privacy line?
      • Quiz yourself
      • Eg how public are you on you income?
    • Resources
      • password managers
      • 2FA
      • Guest vs home Networks
    • Clean up your social media accounts, delete old junk
    • Smart sharing
      • Share stuff but not barcodes, addresses
      • Have a plan
    • Be ready to deal with targeted harassment
      • Keep notes, screenshots, know who to contact
      • Trolls? Block; banhammer
      • Troll back (YMMV)
  • Why I don’t quit
    • Do it because you love it

Share 2019 – Wednesday – Session 3

Why continues to amaze 20 years on!
Steven Hanley, Angus Lees, Hugh Blemings

Steven, Angus, Hugh
  • Three people who have attended every
  • What has happened to keep it relivant and how it has evolved
  • Focus on Open Source Software , These days hardware too
  • Pre-history
    • Australia big in early Linux
    • late 90s Linux and LUGS growing in Aus
  • CALU
    • In 1998 Rusty toured LUGS and organised CALU in July 99 in Monash
    • 9 July – 11 July
    • 6 tutorials and 16 talks
  • Post-CALU
    • Lots of excitement
    • Height of dotcom boom, Big Linuxcare Expo in Darling Harbour in 2000. Very commercial
    • Push for another one
  • 2001
    • UNSW interested in Lions memorial conference
    • CALU type conference would be good
    • Sydney big due to Olympics
    • Domain picked
  • Challenge of early events
    • Everythign was new
    • CFP wasn’t formalised
    • Linux Aus still in infancy
    • Open Source was new and Few conferences
    • Few people worked fulltime in FOSS
    • sceduled over a weekend cause people didn’t get holidays
  • Early objectives we hold on to
    • Community rather than commercial, modest size
    • Easier for students and hobbists, low price
    • tech/non-tech balance to encource interesting delegates
    • Miniconfs
    • Speakers treated well
    • Timeing to make conference attractive (Summer!)
    • University venue, dorms, communal accom, Holidays
  • Miniconfs
    • First in Brs in 2002. more in 03, formalised in 2004
    • Open to all delegates
    • Incubate possible future conferences
    • Fill the week without adding more effort
    • Try out nice topics, extended BOFS
    • Practice ground for new speakers
  • Growth and Roaming
    • Change of location helps
    • New team help avoids burnout and bring fresh ideas
    • Allows more people to contribute
    • Repeat city visits with new people involved
    • Allows people who are less-technical to help out
    • Bid process introduced, overseen by LA
    • Specifics of venue and location help teams structure their event
    • New locations add tourism aspect, encourages aspect
    • Positive experience shared by work-of-mounth
  • Expansion
    • Will anyone go to Perth? NZ?
    • Infinite growth is not a goal (complexity, conference atmosphere)
    • New activities and events are continuously evaluated
  • Call for papers
    • Originally ad-hoc
    • Seperate Panel since 2005
    • See Mary’s blog post in 2006
    • 4-5 times proposals to slots
    • Process
      • Actively solicit hot speakers / topics
      • Review submissions individually
      • face-to-face to reach final as group
    • Conference organisers set overall theme, choose keynote speakers
  • Conference MNGT software
    • Much NIH
    • New software in 2004, 2005, 2007, 2017
    • Mainly for CFP has hooks to other conference components (scheduling, badges, website)
  • Ghosts of conferences Past
    • Mailing list to ask
    • Visit conference organisers
    • Some people have remained over multiple years
  • Giving Back
    • Donation to a charity, action off a shirt singned by speakers
  • Event Style and Flow
    • The current event is typical
    • Moved from Weekend to week as people get paid by employer
    • Less Beer, More Food
  • Giving back to the technical commons
    • Regional delegate program ( 2004 for a few years )
    • Stories of laptops being fixed by the guy who dev’d it
    • git is an outcome of “no more bitkeeper” in 2005
    • Encouraging work done once talk has been accepted
    • The Debian couch had no back so he had to sit around and support each other
  • Sponsorship
    • Some Very long term
    • Some years had to educate sponsors on how to participate
    • Local vendors and sponsors
    • Nurturing sponsosrs takes consious effort
  • Learning lessons
    • Floods (Brisbane 2011)
    • Budget issues (where LA comes into play)
    • CFP feedback loop has sought to tweak technical/non-tech focus
    • Code of conduct has made conf better
    • People step up to make things happen
  • Resources
    • Mirror
    • Mary Gardiner’s post on getting a talk accepted
    • Simon Lyall’s guide

Right to Not Broadcast – Karen Sandler

Karen Sandler
  • Thought the problem with the propitiatory device in her heart was about transparency, now feels it is more about control
  • Got a new device where the programmer (controller) only worked when it was touching, rather than over a distance.
  • Research team bought a device on ebay that had patients data still in it.
  • On the other hand the Keynote Speaker this morning had to hack into her own device to get the info.
  • Sleep Apnea machines transmitting data to both doctors and insurance companies
  • Smart TVs
    • Listening for wake words all the time
    • Sending viewing data to 3rd parties
  • Various Legislation
    • HIPAA , NZ Health Information Privacy Code, Aus Privacy Act, GDPR
    • GDPR – Europe’s gift to the rest of the world
  • ” Incorporating connectivity means we can never be totally in control of our critical information “
  • The environment/reason we provide the data in now may change
  • Often the non-connected option is marginal or doesn’t even exist.
  • [ Laptop ran out of battery here ]
  • Things need to be worked on
    • Raising the awareness of the non-networking, privacy-first issue
    • Even among the LCA-type crowd

Share 2019 – Wednesday – Session 2

Around the world in 80 Microamps: ESP32 and LoRa for low-power IoT – Christopher Biggs

Christopher Biggs
  • Promise of IOT
    • Control everything
    • Sensors everywhere
    • Reduce cognitive load
  • Problem
    • Computers everywhere = wires everywhere
    • Can’t be done every time
    • But em if you got em
    • eg Power over ethernet, Ethernet over power, Ethernet over coax
    • Wifi is great for connections. But what about power?
  • Batteries are bad
    • Lead acid – obsolete just about everywhere
    • Single-use dry cells – leak
    • Nickel Metal-hydride – Some stuff
    • Lithium – Everything else
    • Sample battery
      • Chap-stick battery = 2 Amp/hours
      • 3.7 volts
      • Labels on batteries often lie – you need to always verify
      • Energy capacity is quoted for 20h discharge, not linear relationship
    • But they are geting better due to phones, scooters, drones pushing
  • Off the shelf solutions for packs
    • Smart ones may turn themselves off if draw very low
    • Cell plus simple system works
    • Cell with “Battery Managmnet system” is a bit more complex
    • Solar panels are useless, needs to be a4, a4, a5 size at least
    • Linux systems too much draw for non-wired sensors, need to be used as hubs
  • Computers can spend most of time asleep
    • Config one or more wake-interupts
    • Arduino deep sleep – Sleep consumption as low as 6 micro amps. ( 38 years with Chapstick battery)
    • Watch out for stock voltage regulators (eats 10 mA)
  • ESP 8266 Sleep modes
    • Several levels off sleep modes
    • Wake up every 5 minutes = 1 week battery life
  • ESP 32
    • Sleepier modes
    • Complex sleep patterns.
    • Ultra-low coprocessor
      • 4 register, 10 instructions, 16-bit, special slow memory
      • Can be configed to wake up at intervals
      • Can go back to sleep or wake the main cores
    • ULP in practice
      • Write code, load into the ULP processor
      • Enough code to decide to go back to sleep or wake up main processes
  • Aim for efficiency
    • Sleep as much as possible
    • Use interrupts not polling where possible
  • Nasty Surprises
    • Simple resistors ladders leak power
    • Linear regulators leak power
    • Poor antennas cost watts
    • Beware: USB programming bridges that are always on
    • Almost all the off-the-shelf IoT boards are no good for permanent installation
  • Solutions
    • Turn of everything you are not using
    • radio turn off when not in use, receiver turn it on now and then. Do store-and-forward
    • slow down the cpu, turn off bluetooth
    • Reduce brightness of lights
    • BE careful about cutting out safety features
  • Case Study – Smart water meter in multi-tenant building
    • Existing meter has a physical rotating dial, can count rotations
    • In cellar with no power
    • Create own
      • ESP-32
      • Wifi for setup or maint
      • LoRA for comms every 15 minutes
      • ULP monitors 4 sensors
      • ULP wakes CPU after number of elasped minutes and/or pulse
      • Transmits to Linux-based hub covers building
      • 150mA WiFi
      • 100mA over LoRA
      • 50mA when idle with radio on
      • 40mA when idle with radio off
      • 80uA in deep sleep
      • Average under 1mA , lifetime = 1-5 years
  • Recap
    • Wires are hard
    • Measure and understand usage
    • The basics off deep-sleep
    • ESP32 Ultra-low-Power co-processor
    • Design your own battery-friendly systems (see Arts Miniconf presentation)
    • Project and monitor your battery lifetime
    • Website

Deep Learning, Not Deep Creepy – Jack Moffitt

Jack Moffitt
  • What is machine learning
    • Make decisions based on statistics
  • How is deep learning different?
    • Many layers of neurons each learning more sophisticated representations of features in the data
    • Transfer learning – reuse N-network for similar task where less training data
    • Generative Adversarial networks
  • The dark side of deep learning
    • Works better with more data. Incentive for companies to get a huge amount of data
    • Computationally very expensive – Creates incentive to move things to large clouds
    • Inaccessible to smaller players
    • Hard to debug, black boxes.
    • Amplify biases in training data, somemays to fix but not generally fixable
    • Data may be low quality
  • Machine learning @mozilla
    • Deepspeech and common voice
    • Deepspeech – state-of-art speech detection
      • Existing solutions owned by big companies. Costs $ and in cloud
      • Opening up models and train data will allow innovation
      • Based on baidu’s deep speech paper
      • pre-trained models for english
      • runs real-time on mobile
      • word error rate of 6.48% on librivox
      • streaming support
    • Common voice
      • Crowd source voice data for new applications
      • 20 languages launched
      • 1800 hours collected so far
    • Deepproof – spelling and grammar checker
      • Existing one is basically a keylogger ( Grammarly )
      • Needs to be small enough to run on device
      • Learn by example, rather than few rules
        • Less language-specific tuning
        • More scaleable
      • Local interface to avoid sending private text to several
      • 12 million 300-character chunks from wikipedia
      • Inject plausible mistakes
      • Real-life data
      • Maybe improve with federated learning without disclosing text
    • Lpcnet
      • lots of test-to-speech are end-to-end
      • a separate network converts spectrograms to audio
      • GRiffin-Lim sounds bad
      • WaveNet / WaveRNN needs 10s GFLOPS
      • needs something for efficient for on-device
      • Currently 1.5-6 GFLOPS
      • real-time on mobile
      • Works okay
      • Other applications
        • Speech compression
        • noise compression
  • Questions:
    • Does the audio slow-down work on non-speech? Not really
    • How do you deal with region variations of speech and grammar – Common voice is collecting

Share 2019 – Wednesday – Session 1

Filesender: Sending large files across facilities – Ben Martin

Dr Ben Martin
  • 10 Year old project
  • Web based File Sharing
    • Quick 1-to-few file sharing between people
    • Files go away after a month by default
    • Simpler to run than anon-ftp etc
    • Stats of downloads available to sharer
    • User only needs web browser
    • Upload resume, important with TB sized files
    • Notifications
    • share with explicit groups
    • Browser-to-browser encryption of data AES-256
    • SAML for auth scale
    • GDPR by default, about privacy page
  • Overview
    • Server side is PHP
    • Client side JS with light widgets
    • MariaDB
  • Server Storage
    • Chunked 5MB files
    • Cept used at aarnet
  • Downloading
    • On the fly zip64 archive creation
    • One of more files listed per transfer
    • Links for console download if needed
  • Dragons
    • Auto Downloads and fast uploads cross browser is HARD
    • Mixed browsers
    • Long uploads can exceed auth sessions times
    • Web crypto support w3c
    • People use ancient versions of databases
  • Lots of details on the Database and Encryption. Sounds like both have improved to a good state
  • Future
    • UI refresh
    • Mobile App
    • E2E Encryption
    • Docker image for easy setup
    • More SAML info, apache config?
    • Integration of Endpoints ( auto youtube etc )
    • Session Clone to investigate problems (but privacy?)
    • Run the whole thing in the cloud
  • Questions
    • Command line? – REST API (php and client)
    • RSYNC for slightly changing files – Being investigated

Hot Potato – James Forman and Callum Dickinson

James and Callum
  • What is Hot potato
    • Not a monitoring System
    • monitoring System -> Hot potato -> On-call person
    • Web app in python and flask
    • Tells you things and stays out of the way
  • Why ?
    • Spark shutdown paging Network
    • Needed quick version
  • Goals
    • Don’t get in the way
    • Alert reduction
    • Highly available
    • Support any System – Nagios family now, Prometheus later
    • Support methods – Pushover, SMS, Paging
  • What else can it do
    • Failure notifications when contacts are not working
    • Heartbeats so know when monitoring system is down
  • Planning stuff to add to it
    • Teams – put everyone on call
    • Team escalations
    • Planned work ( go to person working before oncall, extend windows )
    • Support Hotline integration
    • Mobile App
    • Adding German and Italian
  • How it works
    • Flask App
    • RabbitMQ
    • Database (cockroachDB)
    • Apps talk via the Databases
    • Alert -> Object in DB -> Put on Queue -> Worker
    • Worker -> Get details to send to -> Try to send -> Store result in DB
    • If Failure fails then work it’s way though the list.
  • Questions:
    • ACK can use pushover so don’t have to login to app
    • Looking at teams functions
    • CockroachDB picked since it seems very reliable
    • Not sure about restoring/calendaring features going in since need to make it generic?
    • Endpoints fairly modular so should be extendable to new ones.


January 22, 2019 2019 – Wednesday – Keynote

#WeAreNotWaiting: how open source is changing healthcare
– Dana Lewis

Dana Lewis
  • Getting diagnosed with a chronic disease is like being struck by lighting.
  • Insulin takes a while to kick in
  • Manual diabetes
    • Have to check level over and over again
    • Have to judge trend and decide more insulin, food, exercise, etc
    • Constantly
  • Device
    • Windows-only interface software to access
    • Alarm not great, various other limitations
  • Idea
    • Pull data off the device, create smarter app
    • Hard to do
  • First version
    • Device -> WinPC -> dropbox -> app -> pushover -> phone
  • 2nd Version
    • Press button to indicate what she is doing (eating, sleeping) when get alert
  • 3rd version
    • Hook into Insulin pump to do it automatically
    • Replace the human do they same thing over-and-over again in the loop.
    • all portable
    • Person doesn’t have to wake up to adjust, petter sleep
  • OpenAPS
    • Open Source
    • Created a list of ways it could fail (battery fails, wires come out)
    • Focusing on safety
    • Limiting dosing ability in hardware and software
    • Failing back safely to standard device operation
  • Approx 1000 users
    • 9 million+ hours of DIY closed loop experience
    • Anonymized dataset available
  • Sample child user
    • Before: 4.5 manual interventions per day by parents
    • After: 0.7 per day
  • School Child ( 5 vs 6th Grade )
    • 420 visits to school nurse ( 2.3 /day ) – 66 visits for events
    • 5 visits with OpenAPS – 3 visits (Gym related)
  • Intel Edison Platform
    • Smaller than Raspberry Pi
    • But discontinued (looking for old ones to replace)
  • Going back to Pi, now smaller
    • Has built in display with “Explorer Hat”
  • Outsiders are building stuff because they can and the traditional companies are not meeting the need. Innovate with small solution and build it up
  • Ask what little things you can do for people with similar problems. This started with just “Make a louder alarm”.

Share 2019 – Tuesday – Session 3 – Docs Down Under

How to Avoid Meetings – Maia Sauren

Maia Sauren
  • What Distributed/International Teams involves
    • Lots of late meetings
    • Not up on the inclusive languages
    • We’d language quirks from ESL speakers
    • Taxonomy changes between fields
    • Stereotypes are incomplete
    • Ask Culture vs Guess Culture
    • Micro-cultures , down to schools, extra years.
    • When you make a private joke without context, someone is left out
    • What governance model wins (who do they raise barriers for)
    • It’s harder to change a relationship over the phone than maintain one
    • A relationship with a CoC is less fragile (shouldn’t be figured out in the fly during conflict)
    • “How do you want to have arguments?”
    • Set standards early and resolve swiftly
    • Normalise conflict resolution
    • Adulting: it’s for people who don’t want to cry even more later
    • What have you done this week you are proud of?

Disaster Recovery Book – Svetlana Marina

Svetlana Marina
  • Software Development Process
    • Speaning all time “firefighting” so no time to improve processes
  • Operation Support Model
  • Incident Model
    • Alert Raise
    • Initial Response
      • Runbook should have: Alert type -> Investigation help, exact queries into log search, links
    • Impact, escalations, SLA
      • Send email to stakeholders, keep their trust
      • Include “communication required” in runbook
    • Investigation and Damage control
      • Do what is required to fix problem, not fix the root cause
      • Message to stakeholders again
    • Through Investigation (depends on situation)
    • Fix root cause
    • Post Incident Review

The Bus Plan: Junior Staff Training – Andrew Jeffree

Andrew Jeffree
  • Staff coming into industries with lots of automation, but what when automation fails?
  • Staff only know about automation, can only run a playbook, don’t understand what it does
  • Disclaimer: Automation isn’t bad.
    • Config Managment, Custom tooling, CICD, Infrastructure as code
  • Buttons
    • Need to understand what they do
    • Sometimes they are not there
    • Can be Hard to implement a button that doesn’t exist
  • Do it the manual way
  • Training
    • Do something manually (install wordpress) and document
    • Expand what you have done
    • Tweak it
    • Break and fix it
    • Don’t be afraid to change something manually
  • Questions
    • Split for new people – 30% training , 70% work
    • Manual -> ansible -> puppet
    • Don’t want them to be stuck in the mindset that everthing we do is the best way

When Agile Doesn’t Work Anymore: Managing a Large Documentation Project – Lana Brindley

Lana Brindley
  • We all age
    • Sometimes old docs should just be binned
  • Old docs base, massive changes needed, but wanted to save it
    • Changes to toolchain too
  • Proof of concept
    • Define how big the job will be, what is involved
    • Get buy-in
  • Plan, Plan, Plan
    • Create a real timeline
    • Advertise your plan, tell everybody about it
    • Do a presentation to team for each phase
  • Research
    • Who is your audience?
    • No, really, who is your audience? – eg Sales and support may use docs more than customers
    • Lets the customers know that “somebody cares about these docs”
  • User/task analysis
    • Where do we need to focus work
    • What are the big tasks
  • Do the thing
    • You have to sit down and write it
    • You are breaking the agile system
    • You need to track your work and who doing what
    • eg this case, moving from big chapters to small topic-based docs.
    • Agree ahead of time on process, have buy-in
  • Review
    • What went on?
    • WHo not to stuff it up next time
    • Try not to blame people too much
  • Agile vs … something else
    • Sometimes Agile is not the best model
  • Tips for working outside Agile
    • Outreach
      • Especially product owners.
      • Get people on board
    • Track your work
      • Create Epics of sprints
      • Don’t go overboard
    • Should about it
      • outreach never stops
      • Present at sprint reviews
      • Brownbags

Share 2019 – Tuesday – Session 2 – Picking a community & Mycroft AI

Finding Your Tribe: Choosing open source Communities – Cintia Del Rio

  • When started out she couldn’t find info on picked what project to work on. Crowdsourced some opinions from others
  • You need to work out why you want to volenteer for a project.
  • 3 types of code on Github
    • Source Available
      • Backed by companies (without open source as their business model)
      • Core devs from company, roadmap controlled by them
      • Limited influence from externals
      • Most communication not on public channels
      • You will be seen as guest/outsider
    • On Person Band
      • Single core maintainer, working in spare time
      • Common even for very popular libraries and tools (lots of examples from node and java ecosystem)
      • Few resource
      • Conflicts might not be handled well
    • Communities
      • Communication Channells – forums, mailing list, chat
      • Multiple core devs
      • Github org
      • Code of conduct
  • Some things to check beforehand
    • Is it dead yet?
      • Communication channels, Commits, issues, pull requests – how old, recent updates
    • How aggressive is the community?
      • Look at how a clueless user is handled
      • Declined pull requests. HOW did they handle the decline
      • ” Is the mailing list/icr/slack, is it a trash fire? ”
      • “Can you please rule” – add “Can you please” in front of a comment and does it sound nice or still mean/sarcasm?
    • Is non-coding work valued
    • Communities with translations?
    • Look at photos from the conferences
    • Grammar mistakes and typos – how are the handled?
    • Newbie tags and Getting started docs
    • Cool languages tend to attract toxic people
    • Look for Jerks in leadership
    • Ask around
  • Does it spark joy? – If not let it go.

Intro to the Open Source Voice Stack By Kathy Reid

Kathy Reid
  • In the past we have taught spreadsheets etc. We need to teach the latest thing and that is now voice interfaces
  • Worked for Mycroft, mainly using that for demo
  • Voice stack
    • Wake Word
    • Speech 2 Text (utterance)
    • Action ( command )
    • Text to Speech (dialogue )
  • Request response life-Cycle
    • Wake word
    • Intent parser over utterance
  • When Kathy just started she was first woman and first Australian so few/no samples in database and had problems understanding her.
  • Text to Speech
    • Needs to have a well speaker speaking for 40-60 hours
    • Mimic Recording Studio – List of Phrases that people need to speak to train the output

Share 2019 – Tuesday – Session 1 – Docs Down Under Miniconf

Being Kind to 3am You – Katie McLaughlin

Katie McLaughlin
  • Not productive and operating at her best at 3am
  • But 3am’s will happen and they probably will be important
  • Essentials
    • You should have documentation, don’t keep it in your head since people are not available at 3am
    • Full doc management system might take a while
    • Must be Editable
      • Must be updateable at any time
    • Searchable
    • Have browser keywords that search confluence or github
    • Secure but discoverable by co-workers
  • Your Tools
    • Easy cache commands to use
    • Not dangerous
  • Stepping Up
    • Integrate your docs so it’ll be available and visible when people need it.
    • Alerts could link to docs for service
  • Post Mortem
    • List of commands you typed to fix it
  • Reoccurring Issues
    • Sometimes the quick fix is all you can do or is good enough. You can get back to sleep.
    • Maybe just log rotate to clean the disk. Or restart process once a week
    • Make you fix an ansible playbook you can just click
  • Learning
    • Learn new stuff so when you have chance you can do it write
  • Flag Changes
    • Handover changes to over to everyone else
  • So Empathy towards the other people (and they may show it back)
  • Audience
    • One guy gave anyone who go paged overnight $100 bill on their desk next day ( although he charged customers $150 )
    • From Fire Depts – Label everything, Have the docs come with the alert. Practice during the daylight.
    • Project IPXE – every single error message is a link to wiki page
    • Advice: Write down every command, everything you did, every output you saw. So useful for next day.

Making youself Redundant on Day One – Alexandra Perkins

Alexandra Perkins
  • Experiences
    • All Docs as facebook posts
    • All docs as comment codes
    • Word documents hidden in folders
  • Why you should document in your first weeks
    • Could you know the what the relevant questions for new people
    • You won’t remember it the first time you hear it
    • Easier for the next person
    • Inclusive and diverse workplace
  • What should you document
    • Document the stuff you find hard
    • Think about who else can use your docs
    • Stuff like: How to book leave, Who to ask about what topics, Info on workplace social events. Where lunch is.
  • How to document from the start
    • In wiki or Sharepoint
    • Word docs locally and copy it the official place once you have access
    • Saved support tickets
    • Notes to yourself on slack
    • Screenshots of slack conversations
    • Keep it simple, informal content is your friend. All the Memes!
    • Example Tutorial: “Send yourself an email and trace it though the logs”
  • Future Proofing
    • Create or Improve the place for Internal documentation
    • Everyone should be able to access and edit (regardless of technical expertise)
    • Must be searchable and editable so can be updated
    • Transfer all docs you did on your personal PC to company-wide documentation
    • Make others aware of the work you have done
    • Foster a culture of strong documentation.
      • Policy to document all newly announced changes
      • Have a rotation for the documentation person
    • Quality Internal Docs should be
      • Accessible
      • Editable
      • Searchable
      • Peer Reviewed

JIT Learning: It’s great until it isn’t – Tessa Bradbury

Tessa Bradbury
  • What should we learn?
    • There is a lot to learn
  • What is JIT Learning?
    • Write Code -> Hit an issue -> Define Problem -> Find a Solution
  • Assumption – You will ask the required questions (hit the issue)
    • Counter example: accessibility, you might not hit the problem yourself
  • Assumption – You can figure out the problem
    • Sometimes you can’t easily, you might not have the expereince and/or training
  • Assumption – You can find the solution
    • Sometimes you can’t find the solution on google
    • Sometimes you are not in Open Source, you can’t just read the code and the docs may be lacking
  • Assumption – You might not be sure the best way to write your fix
    • Best way to implement the code, if you should fix it in code
    • Or if your code has actually fixed the whole problem
  • Assumption – The benefit of getting it done now outweighs the cost of getting it wrong
    • Counter example – Security


January 21, 2019 2019 – Tuesday – Keynote: Rory Aronson


  • Had idea to automated small-scale gardening
  • Wrote up a proposal: FarmBot – 3d printer for growing your garden
    • Open Source
    • Impact > Money
    • Based on 3rd printer frame (with moving arms)
  • Created prototype 2015-2016
    • Add Web App
  • Crowdfunding and Video campaign
    • 58 Million views
    • $800k pre-sales, 300 orders
  • Created and shipped first version
  • But still open source

OPen Source Hardware

    • All the CAD models
    • Bill of materials
    • Everything Versioned
    • Mods and add-ons “for inspiration only” (not officially supported)
  • Open Source Community
    • Code of Conduct

Open Source Company

  • If you have a business you will have competitors
  • Competitors -> Collaborators
  • Lots of numbers online (profit, bills, etc), Stuff that would be on internal wiki (like how to handle orders or do taxes) at other company is public
  • Compensation formula
  • See “Buffer’s Transparency Dashboard


January 17, 2019

Installing NixOS on an ASUS EeePC 701

Those of you familiar with the ASUS EeePC 701 will be aware of it's limited memory (512M) and small internal drive (4G) which doesn't render it useless but does reduce it's modern usage.

I have two of these still in perfect working order and wanted to extend their functional lives. I have no need of any more routers and there are Raspberry Pi's everywhere here. I do have kids though and decided these would make great, ready made portable retro gaming consoles.

This blog post details how I installed NixOS and configured the 701's to overcome their short comings to extend their life.

At a Glance

  1. Download the latest NixOS minimal installer for 32-bit Intel/AMD.
  2. Boot off USB
  3. Set forcepae in Grub
  4. Connect networking
  5. Partition then format the SD card as / and the internal drive as swap
  6. Run the configuration builder
  7. Copy across nix files
  8. Install
  9. Enjoy

Process Detail

Have a copy of the NixOS manual handy. It will also be at [ALT]+[F8] once the installer has booted.


  1. Download the minimal installation CD, 32-bit Intel/AMD image and burn it to a USB stick.
  2. Insert the SD card and USB stick into the 701 (I used a spare 8G SD card I had handy).
  3. Power the 701 on and press [ESC], select the USB stick as the boot device,
  4. [TAB] at grub menu, append forcepae
  5. Continue the boot process until it reaches a root prompt
  6. If you have not connected Ethernet, connect to WiFi using wpa_supplicant, as per manual.

Partitioning and Formatting

With only a little deviation from section 2.2.2. Legacy Boot (MBR) we format the SD card to be the root disk and the internal drive as swap:

  • Create an MBR on the internal drive:

    # parted /dev/sda -- mklabel msdos

  • Create the root partition on the SD card:

    # parted /dev/sdc -- mkpart primary 1MiB 100%

  • Create the swap partition on the internal drive:

    # parted /dev/sda -- mkpart primary linux-swap 1MiB 100%

As per section 2.2.3 Formatting we now format the drives appropriately.

  • Format and label the / partition:

    # mkfs.ext4 -L nixos /dev/sdc1

  • Format the swap partion:

    # mkswap -L swap /dev/sda1

Installation and Configuration:

Now you can just continue to follow section 2.3 Installing and configure the system as you desire. I initially use a bare bones install and it can be read here. It's worth taking a brief look at if you're new to NixOS.

The section worth copying is related to how the kernel uses memory on this old i386 system:

  nixpkgs.config = {
    packageOverrides = pkgs: {
      stdenv = pkgs.stdenv // {
        platform = pkgs.stdenv.platform // {
          kernelExtraConfig = ''
            HIGHMEM64G? n  # 32-bit proc with > 4G RAM
            HIGHMEM4G y # 32-bit # proc # with # =< # 4G # RAM

Do not copy my import of retro-gaming.nix, that's a for after a successful install and a future blog post.

Once you've completed the install, the EeePC 701 ought to reboot into NixOS successfully.

January 15, 2019

Support for Raspberry Pi and Orange Pi GPIOs in Home Assistant


So, I’ve been off in the GPIO library salt mines for a while, but am now ready to circle back and document how to get GPIO inputs and outputs working in Home Assistant. This now works on both Raspberry Pi and OrangePi, assuming that my patch gets merged.

First off, let’s talk about GPIO outputs. This is something which has been working for a while on both platforms (a while being a week or so, assuming you’ve patched Home Assistant with my pull request, but you’re all doing that right?).

To configure an output in Home Assistant, you would add the following to configuration.yaml:

  board_family: orange_pi

 - platform: rpi_gpio
     PA7: LED

Where board_family can be either “raspberry_pi” or “orange_pi”. Note that for Raspberry Pis, the pin numbers are always numbers whereas for OrangePi we are using “SUNXI” numbering, which is of the form “PA7”.

The circuit for this LED is really simple:

A simple LED circuit

Now we have a switch we can control in Home Assistant:

Raspberry Pi LED switch in Home Assistant

GPIO inputs are similar. The configuration looks like this:

  board_family: orange_pi

 - platform: rpi_gpio
   invert_logic: true

With a circuit like this:

A circuit with a button in it

invert_logic set to true is required because our circuit sends the value of PA7 to ground when the button is pressed.

A push button being pressed in Home AssistantNoting that sensors look different to switches in Home Assistant, you can see the binary sensor at the top right of the image, with its history being displayed in the dialog box in the foreground.


January 14, 2019

Updated examples for OrangePi GPIOs


As part of working through adding OrangePi support to Home Assistant, Alastair and I decided to change to a different GPIO library for OrangePi to avoid the requirement for Home Assistant to have access to /dev/mem.

I just realised that I hadn’t posted updated examples of how to do GPIO output with the new library. So here’s a quick post about that.

Assuming that we have an LED on GPIO PA7, which is pin 29, then the code to blink the LED would look like this with the new library:

import OPi.GPIO as GPIO
import time

# Note that we use SUNXI mappings here because its way less confusing than
# board mappsings. For example, these are all the same pin:
# sunxi: PA7 (the label on the board)
# board: 29
# gpio:  7

GPIO.setup('PA7', GPIO.OUT)

while True:
    GPIO.output('PA7', GPIO.HIGH)
    GPIO.output('PA7', GPIO.LOW)

The most important thing there is the note about SUNXI pin mappings. I find the whole mapping scheme hugely confusing, unless you use SUNXI and then its all fine. So learn from my fail people!

What about input? Well, that’s not too bad either. Let’s assume that you have a button in a circuit like this:

A circuit with a button in it

The to read the button the polling way, you’d just do this:

import OPi.GPIO as GPIO
import time

GPIO.setup('PA7', GPIO.IN, pull_up_down=GPIO.PUD_DOWN)

while True:
    if GPIO.input('PA7') == GPIO.LOW:

Let’s pretend it didn’t take me ages to get that to work right because I had the circuit wrong, ok?

Now, we have self respect, so you wouldn’t actually poll like that. Instead you’d use edge detection, and end up with code like this:

import OPi.GPIO as GPIO
import time

GPIO.setup('PA7', GPIO.IN, pull_up_down=GPIO.PUD_DOWN)

def event_callback(channel):
    print('Event detected: %s' % GPIO.input('PA7'))
GPIO.add_event_detect('PA7', GPIO.BOTH, callback=event_callback, bouncetime=50)

while True:

So there ya go.


GPIO inputs on Raspberry Pi


Now that I have GPIO outputs working nicely for Home Assistant using either a Raspberry Pi or an Orange Pi, I want to get GPIO inputs working as well. Naively, that’s pretty easy to do in python on the Raspberry Pi:

import RPi.GPIO as GPIO
import time

GPIO.setup(17, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)

while True:
    if GPIO.input(17) == GPIO.HIGH:

That code is of course horrid. Its horrid because its polling the state of the button, and its quite likely that I can sneak a button press in during one of those sleeps and it will never be noticed. Instead we can use edge detection callbacks to be informed of button presses as they happen:

import RPi.GPIO as GPIO
import time

GPIO.setup(17, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)

def event_callback(channel):
    print('Event detected: %s' % GPIO.input(17))
GPIO.add_event_detect(17, GPIO.BOTH, callback=event_callback, bouncetime=50)

while True:

This second program provides helpful output like this:

pi@raspberrypi:~ $ python 
Event detected: 1
Event detected: 0

Which is me pressing the button once (it go high when pressed, and then low again when released). This is of course with a button wired to GPIO17 with a current limiting resistor between the button and the 3.3v rail.


Annual Penguin Picnic, January 19, 2019

Jan 19 2019 12:00
Jan 19 2019 16:00
Jan 19 2019 12:00
Jan 19 2019 16:00
Yarra Bank Reserve, Hawthorn

The Linux Users of Victoria Annual Penguin Picnic will be held on Saturday, January 19, starting at 12 noon at the Yarra Bank Reserve, Hawthorn.

LUV would like to acknowledge Infoxchange for the Richmond venue.

Linux Users of Victoria Inc., is a subcommitee of Linux Australia.

January 19, 2019 - 12:00

read more

Transport security for BGP, AKA BGP-STARTTLS, a proposal

Several days ago, inspired in part by an old work mail thread being resurrected I sent this image as a tweet captioned "The state of BGP transport security.":

The context of the image for those not familiar with it is this image about noSQL databases.

This triggered a bunch of discussion, with multiple people saying "so how would *you* do it", and we'll get to that (or for the TL;DR skip to the bottom), but first some background.

The tweet is a reference to the BGP protocol the Internet uses to exchange routing data between (and within) networks. This protocol (unless inside a separate container) is never encrypted, and can only be authenticated (in practice) by a TCP option known as TCP-MD5 (standardised in RFC2385). The BGP protocol itself has no native encryption or authentication support. Since routing data can often be inferred by the packets going across a link anyway, this has lead to this not being a priority to fix.

Transport authentication & encryption is a distinct issue from validation of the routing data transported by BGP, an area already being worked on by the various RPKI projects, eventually transport authentication may be able to benefit from some of the work done by those groups.

TCP-MD5 is quite limited, and while generally supported by all major BGP implementations it has one major limitation that makes it particularly annoying, in that it takes a single key, making key migration difficult (and in many otherwise sensible topologies, impossible without impact). Being a TCP option is also a pain, increasing fragility.

At the time of its introduction TCP-MD5 gave two main benefits the first was to have some basic authentication beyond the basic protocol (for which the closest element in the protocol is the validation of peer-as in the OPEN message, and a mismatch will helpfully tell you who the far side was looking for), plus making it harder to interfere with the TCP session, which on many of the TCP implementations of the day was easier than it should have been. Time, however has marched on, and protection against session interference from non-MITM is no longer needed, the major silent MITM case of Internet Exchanges using hubs is long obsolete, plus, in part due to the pain associated in changing keys many networks have a "default" key they will use when turning up a peering session, these keys are often so well known for major networks that they've often been shared on public mailing lists, eliminating what little security benefit TCP-MD5 still brings.

This has been known to be a problem for many years, and the proposed replacement TCP-AO (The TCP Authentication Option) was standardised in 2010 as RFC5925, however, to the best of my knowledge eight years later no mainstream BGP implementation supports it, and as it too is a TCP option, not only does it still has many of the downsides of MD5, but major OS kernels are much less likely to implement new ones (indeed, an OS TCP maintainer commenting such on the thread I mentioned above is what kicked off my thinking).

TCP, the wire format, is in practice unchangeable. This is one of the major reasons for QUIC, the TCP replacement protocol soon to be standardised as HTTP/3, so for universal deployment any proposal that requires changes to TCP is a non-starter.

Any solution must be implementable & deployable.
  • Implementable - BGP implementations must be able to implement it, and do so correctly, ideally with a minimum of effort.
  • Deployable - Networks need to be able to deploy it, when authentication issues occur error messages should be no worse than with standard BGP (this is an area most TCP-MD5 implementations fail at, of those I've used JunOS is a notable exception, Linux required kernel changes for it to even be *possible* to debug)

Ideally any security-critical code should already exist in a standardised form, with multiple widely-used implementations.

Fortunately for us, that exists in the form of TLS. IPSEC, while it exists, fails the deployable tests, as almost anyone who's ever had the misfortune of trying to get IPSEC working between different organisations using different implementations can attest, sure it can usually be made to work, but nowhere near as easily as TLS.

Discussions about the use of TLS for this purpose have happened before, but always quickly run into the problem of how certificates for this should be managed, and that is still an open problem, potentially the work on RPKI may eventually provide a solution here, but until that time we have a workable alternative in the form of TLS-PSK (first standardised in RFC4279), a set of TLS modes that allow the use of pre-shared keys instead of certificates (for those wondering, not only does this still exist in TLS1.3 it's in a better form). For a variety of reasons, not the least the lack of real-time clocks in many routers that may not be able to reach an NTP server until BGP is established, PSK modes are still more deployable than certificate verification today. One key benefit for TLS-PSK is it supports multiple keys to allow migration to a new key in a significantly less impactful manner.

The most obvious way to support BGP-in-TLS would simply be to listen on a new port (as is done for HTTPS for example), however there's a few reasons why I don't think such a method is deployable for BGP, primarily due to the need to update control-plane ACLs, a task that in large networks is often distant from the folk working with BGP, and in small networks may not be understood by any current staff (a situation not dissimilar to the state of TCP). Another option would simply be to use protocol multiplexing and do a TLS negotiation if a TLS hello is received, or unencrypted BGP for a BGP OPEN, this would violate the general "principal of least astonishment", and would be harder for operators to debug.

Instead I propose a design similar to that used by SMTP (where it is known as STARTTLS), during early protocol negotiation support for TLS is signalled using a zero-length capability in the BGP OPEN, the endpoints do a TLS negotiation, and then the base protocol continues inside the new TLS tunnel. Since this negotiation happens during the BGP OPEN, it does mean that other data included in the OPEN leaks. Primarily this is the ASN, but also the various other capabilities supported by the implementation (which could identify the implementation), I suggest that if TLS is required information in the initial OPEN not be validated, and standard reserved ASN be sent instead, and any other capabilities not strictly required not sent, with a fresh OPEN containing all normal information sent inside the TLS session.

Migration from TCP-MD5 is key point, however not one I can find any good answer for. Some implementations already allow TCP-MD5 to be optional, and that would allow an easy upgrade, however such support is rare, and unlikely to be more widely supported.

On that topic, allowing TLS to be optional in a consistent manner is particularly helpful, and something that I believe SHOULD be supported to allow cases like currently unauthenticated public peering sessions to be migrated to TLS with minimal impact. Allowing this does open the possibility of a downgrade attack, and make more plausible attacks causing state machine confusions (implementation believes it's talking on a TLS-secured session when it isn't).

What do we lose from TCP-MD5? Some performance, whilst this is not likely to be an issue for most situations, it is likely not an option for anyone still trying to run IPv4 full tables on a Cisco Catalyst 6500 with Sup720. We do also lose the TCP manipulation prevention aspects, however these days those are (IMO) of minimal value in practice. There's also the costs of implementations needing to include a TLS implementations, and whilst nearly every system will have one (at the very least for SSH) it may not already be linked to the routing protocol implementation.

Lastly, my apologies to anyone who has proposed this before, but my neither I nor my early reviewers were aware of such a proposal. Should such a proposal already exist, meeting the goals of implementable & deployable it may be sensible to pick that up instead.

The IETF is often said to work on "rough consensus and running code", for this proposal here's what I believe a minimal *actual* demonstration of consensus with code would be:
  • Two BGP implementations, not derived from the same source.
  • Using two TLS implementations, not derived from the same source.
  • Running on two kernels (at the very least, Linux & FreeBSD)

The TL;DR version:
  • Using a zero-length BGP capability in the BGP OPEN message implementations advertise their support for TLS
    • TLS version MUST be at least 1.3
    • If TLS is required, the AS field in the OPEN MAY be set to a well-known value to prevent information leakage, and other capabilities MAY be removed, however implementations MUST NOT require the TLS capability be the first, last or only capability in the OPEN
    • If TLS is optional, which MUST NOT be default behaviour), the OPEN MUST be (other than the capability) be the same as a session configured for no encryption
  • After the TCP client receives a conformation of TLS support from the TCP server's OPEN message, a TLS handshake begins
    • To make this deployable TLS-PSK MUST be supported, although exact configuration is TBD.
    • Authentication-only variants of TLS (ex RFC4785) REALLY SHOULD NOT be supported.
    • Standard certificate-based verification MAY be supported, and if supported MUST validate use client certificates, validating both. However, how roots of trust would work for this has not been investigated.
  • Once the TCP handshake completes the BGP state starts over with the client sending a new OPEN
    • Signalling the TLS capability in this OPEN is invalid and MUST be rejected
  • (From here, everything is unchanged from normal BGP)

Magic numbers for development:
  • Capability: (to be referred to as EXPERIMENTAL-STARTTLS) 219
  • ASN (for avoiding data leaks in OPEN messages): 123456
    • Yes this means also sending 4-byte capability. Every implementation that might possibly implement this already supports 4-byte ASNs.

The key words "MUST (BUT WE KNOW YOU WON'T)", "SHOULD CONSIDER", "REALLY SHOULD NOT", "OUGHT TO", "WOULD PROBABLY", "MAY WISH TO", "COULD", "POSSIBLE", and "MIGHT" in this document are to be interpreted as described in RFC 6919.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC2119.

Are Men the Victims?

A very famous blog post is Straight White Male: The Lowest Difficulty Setting There Is by John Scalzi [1]. In that post he clearly describes that life isn’t great for straight white men, but that there are many more opportunities for them.

Causes of Death

When this post is mentioned there are often objections, one common objection is that men have a lower life expectancy. The CIA World factbook (which I consider a very reliable source about such matters) says that the US life expectancy is 77.8 for males and 82.3 for females [2]. The country with the highest life expectancy is Monaco with 85.5 for males and 93.4 years for females [3]. The CDC in the US has a page with links to many summaries about causes of death [4]. The causes where men have higher rates in 2015 are heart disease (by 2.1%), cancer (by 1.7%), unintentional injuries (by 2.8%), and diabetes (by 0.4%). The difference in the death toll for heart disease, cancer, unintentional injuries, and diabetes accounts for 7% of total male deaths. The male top 10 lists of causes of death also includes suicide (2.5%) and chronic liver disease (1.9%) which aren’t even in the top 10 list for females (which means that they would each comprise less than 1.6% of the female death toll).

So the difference in life expectancy would be partly due to heart problems (which are related to stress and choices about healthy eating etc), unintentional injuries (risk seeking behaviour and work safety), cancer (the CDC reports that smoking is more popular among men than women [5] by 17.5% vs 13.5%), diabetes (linked to unhealthy food), chronic liver disease (alcohol), and suicide. Largely the difference seems to be due to psychological and sociological issues.

The American Psychological Association has for the first time published guidelines for treating men and boys [6]. It’s noteworthy that the APA states that in the past “psychology focused on men (particularly white men), to the exclusion of all others” and goes on to describe how men dominate the powerful and well paid jobs. But then states that “men commit 90 percent of homicides in the United States and represent 77 percent of homicide victims”. They then go on to say “thirteen years in the making, they draw on more than 40 years of research showing that traditional masculinity is psychologically harmful and that socializing boys to suppress their emotions causes damage that echoes both inwardly and outwardly”. The article then goes on to mention use of alcohol, tobacco, and unhealthy eating as correlated with “traditional” ideas about masculinity. One significant statement is “mental health professionals must also understand how power, privilege and sexism work both by conferring benefits to men and by trapping them in narrow roles”.

The news about the new APA guidelines focuses on the conservative reaction, the NYT has an article about this [7].

I think that there is clear evidence that more flexible ideas about gender etc are good for men’s health and directly connect to some of the major factors that affect male life expectancy. Such ideas are opposed by conservatives.

Risky Jobs

Another point that is raised is the higher rate of work accidents for men than women. In Australia it was illegal for women to work in underground mines (one of the more dangerous work environments) until the late 80’s (here’s an article about this and other issues related to women in the mining industry [8]).

I believe that people should be allowed to work at any job they are qualified for. I also believe that we need more occupational health and safety legislation to reduce the injuries and deaths at work. I don’t think that the fact that a group of (mostly male) politicians created laws to exclude women from jobs that are dangerous and well-paid while also not creating laws to mitigate the danger is my fault. I’ll vote against such politicians at every opportunity.

Military Service

Another point that is often raised is that men die in wars.

In WW1 women were only allowed to serve in the battlefield as nurses. Many women died doing that. Deaths in war has never been an exclusively male thing. Women in many countries are campaigning to be allowed to serve equally in the military (including in combat roles).

As far as I am aware the last war where developed countries had conscription was the Vietnam war. Since then military technology has developed to increasingly complex and powerful weapons systems with an increasing number of civilians and non-combat military personnel supporting each soldier who is directly involved in combat. So it doesn’t seem likely that conscription will be required for any developed country in the near future.

But not being directly involved in combat doesn’t make people safe. NPR has an interesting article about the psychological problems (potentially leading up to suicide) that drone operators and intelligence staff experience [9]. As an aside the article reference two women doing that work.

Who Is Ignoring These Things?

I’ve been accused of ignoring these problems, it’s a general pattern on the right to accuse people of ignoring these straight white male problems whenever there’s a discussion of problems that are related to not being a straight white man. I don’t think that I’m ignoring anything by failing to mention death rates due to unsafe workplaces in a discussion about the treatment of trans people. I try to stay on topic.

The New York Times article I cited shows that conservatives are the ones trying to ignore these problems. When the American Psychological Association gives guidelines on how to help men who suffer psychological problems (which presumably would reduce the suicide rate and bring male life expectancy closer to female life expectancy) they are attacked by Fox etc.

My electronic communication (blog posts, mailing list messages, etc) is mostly connected to the free software community, which is mostly male. The majority of people who read what I write are male. But it seems that the majority of positive feedback when I write about such issues is from women. I don’t think there is a problem of women or left wing commentators failing men. I think there is a problem of men and conservatives failing men.

What Can We Do?

I’m sure that there are many straight white men who see these things as problems but just don’t say anything about it. If you don’t want to go to the effort of writing a blog post then please consider signing your name to someone else’s. If you are known for your work (EG by being a well known programmer in the Linux community) then you could just comment “I agree” on a post like this and that makes a difference while also being really easy to do.

Another thing that would be good is if we could change the hard drinking culture that seems connected to computer conferences etc. Kara has an insightful article on Model View Culture about drinking and the IT industry [10]. I decided that drinking at Linux conferences had got out of hand when about 1/3 of the guys at my table at a conference dinner vomited.

Linux Conf Au (the most prestigious Linux conference) often has a Depression BoF which is really good. I hope they have one this year. As an aside I have problems with depression, anyone who needs someone to talk to about such things and would rather speak to me than attend a BoF is welcome to contact me by email (please take a failure to reply immediately as a sign that I’m behind on checking my email not anything else) or social media.

If you have any other ideas on how to improve things please make a comment here, or even better write a blog post and link to it in a comment.

January 09, 2019

Simple FOSS versus Complex Enterprise Software

As is often the case real IT operators in large organisations find themselves having to deal with "enterprise" software which has been imposed upon them. The decision to implement such software is usually determined by perceived business requirements (which is reasonable enough), but with little consideration of the operations and flexibility for new, or even assumed, needs.

For example, a certain large university which I have some familiarity with has decided to introduce an absolutely $awful_ticketing_system campus-wide. It serves as an excellent example because every step of the way there are just terrible things about this which illustrates how IT management decisions cause unnecessary problems for IT operators.

The beginning of this story comes with what initially seemed to be a simple problem. Emails that were sent to the ticketing system were being automatically assigned as an Incident, which had a tight Service Level Agreement period, as one would expect. The tickets that were being received, however, were not Incidents, but rather feature Requests. Changing this would be simple and easy (more on this in a moment), right?

Apparently not. I feel for the poor person who had to respond to me with this explanation.

Technically, an incoming mail to a mailbox can be converted into any type of ServiceNow ticket.

Well, this is good news.

But, out of all ticket types, this conversion is the most complicated for "request" type of tickets due to its three layered structure (REQ, RITM, CTASK). It has to have all three layers created for each request type of ticket. There is no out of the box or simple way of converting incoming mails to request tickets. Hence, it has not been implemented so far.

Wait, what? In well-known FOSS ticketing systems (e.g., OTRS, Request Tracker, Trac), the operator can set the queue or equivalent. Here, with this proprietary enterprise software, not even the administrators can do it in a simple manner. The solution being developed? Get rid of email tickets and use a web portal instead.

... the option of mailbox that creates tickets in Servicenow, was made available for use cases where users (such a externals, guests, students etc.) could not access forms and hence could not request relevant services. But the idea was always to use $awful_ticketing_system Portal as the access point to report any issues or request services, thereby avoiding the need to use mailboxes.

I'll leave it to others to think about how a web portal won't be associated with a mailbox or how this will allow operators to assign the ticket type. But don't spend too much time on it. Because this specific software and this specific example isn't the real issue at hand.

The first problem is that IT Managers don't listen carefully enough to IT Operators. An article by Jeff Ello, ten years old now, explains the many problems involved. The entire thing is worth reading, multiple times even, but to give a summary of a single paragraph from the article:

While everyone would like to work for a nice person who is always right, IT pros will prefer a jerk who is always right over a nice person who is always wrong. Wrong creates unnecessary work, impossible situations and major failures. Wrong is evil, and it must be defeated. Capacity for technical reasoning trumps all other professional factors, period.

There is one additional statement that needs to be added to this: This is not optional. In many other human-to-human roles, such as management, it is possible to reconstruct questions, find a compromise between competing agendas, and so forth. This is not possible with IT Operators (assuming they are honest and knowledgeable), not because they don't want to do it (whatever "it" is), but because it simply isn't possible. To repeat: technical reasoning trumps all other professional factors. When it comes to technical matters, IT Managers need to do more than just take advice from the Operators; they need to do what the Operators tell them. Otherwise, the wrong technical decisions will be made and that will cost time and money in the future.

The second problem, and yes, it cascades from the first, is that IT Managers have an erroneous propensity to choose enterprise software which is complex and easy rather than free and open source software which is simple and hard. The difference between the two has been well stated by Rich Hickey. Managers tend to choose software which is easy to use because they are typically not Operators themselves. They become especially enticed by software which is easy to use and feature-rich because they incorrectly perceive that will satisfy the requirements of (often dogmatic) business logic. When it is inevitably discovered that there are new business requirements the enterprise software needs to be somehow re-developed (at additional cost and time, as determined by the external body), or worked around. or try to find a new enterprise software product (and then experience the horror of vendor lock-in).

All of this is something that would make an experienced FOSS IT Operator's remaining grey hairs stand on end. In our world if something isn't working right, we fix it there and then, and if we can't we ask around because somebody else will know how. Collectively we have more knowledge than any of us individually can have. In the FOSS world, a product can be hard, in both senses of the word. It does have a steep learning curve, but it also is durable. It is however, also simple, that is, it isn't interleaved. Simplicity is a prerequisite for reliability said Edsger W. Dijkstra in 1975. Most FOSS (and yes, there are exceptions) is built on the UNIX philosophy: Design programs to do only a single thing, but to do it well, and to work together well with other programs.

The third problem, and yes, it cascades from the second, is that IT systems are often too heavily orientated towards singular solutions for the entire environment. Now initially I (erroneously) agreed with this approach. There was a time that I would have argued that a single ticketing system throughout an entire campus was a good idea as it lowered administrative and training costs and allowed for a single point from which a variety of metrics could be derived. My concern was that in selecting a single system that the wrong system might be chosen (and remember "wrong is evil and it must be defeated") or, one is terrified to say, the worst system might be selected (and usually it is, for the reasons previously stated).

The problem is with the conceptualisation. A singular system as a monolith will almost inevitably suffer the aforementioned issues of complexity, and if it's a closed-source enterprise product, there's nothing you can do about it. A single system, especially a cloud-based system, will suffer massive performance issues simply due to the physics of distance and physics is not optional. So rather than a single system, it really is worth using multiple systems according to what is most contextually appropriate and which can interact with other systems when needed. Again, getting back to these core computer science concepts of modularity, specialisation, and connectivity. The supposed gains in reducing administrative overhead are actually losses if the operator is unable to adapt to new systems, and that adaption is achieved through knowing how generic systems work together.

By way of conclusion, it is worth thinking in terms of stripped-down, minimalist, free and open-source programs which satisfy all functional requirements and can be modularly extended in a manner that is simple but durable. The philosophy of the team behind is an excellent approach. As for a practical example of a ticketing system, Joe Amstrong's example of the Erlang Ticketing System as a "minimal viable program" satisfies the criteria.

"Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand."
-- Archibald Putt, "Putt's Law and the Successful Technocrat", 2006

January 08, 2019

Erasing Persistent Storage Securely on Linux

Here are some notes on how to securely delete computer data in a way that makes it impractical for anybody to recover that data. This is an important thing to do before giving away (or throwing away) old disks.

Ideally though, it's better not to have to rely on secure erasure and start use full-disk encryption right from the start, for example, using LUKS. That way if the secure deletion fails for whatever reason, or can't be performed (e.g. the drive is dead), then it's not a big deal.

Rotating hard drives

With ATA or SCSI hard drives, DBAN seems to be the ideal solution.

  1. Burn it on CD,
  2. boot with it,
  3. and following the instructions.

Note that you should disconnect any drives you don't want to erase before booting with that CD.

This is probably the most trustworth method of wiping since it uses free and open source software to write to each sector of the drive several times. The methods that follow rely on proprietary software built into the firmware of the devices and so you have to trust that it is implemented properly and not backdoored.

ATA / SATA solid-state drives

Due to the nature of solid-state storage (i.e. the lifetime number of writes is limited), it's not a good idea to use DBAN for those. Instead, we must rely on the vendor's implementation of ATA Secure Erase.

First, set a password on the drive:

hdparm --user-master u --security-set-pass p /dev/sdX

and then issue a Secure Erase command:

hdparm --user-master u --security-erase-enhanced p /dev/sdX

NVMe solid-state drives

For SSDs using an NVMe connector, simply request a User Data Erase

nvme format -s1 /dev/nvme0n1

January 06, 2019

Photograph Your Work

One thing I should have learned before (but didn’t) and hope I’ve learned now is to photograph sysadmin work.

If you work as a sysadmin you probably have a good phone, if you are going to run ssh from a phone or use a phone to read docs while in a server room with connectivity problems you need a phone with a good screen. You will also want a phone that has current security support. Such a phone will have a reasonable amount of storage space, I doubt that you can get a phone with less than 32G of storage that has a decent screen and Android security support. Admittedly Apple has longer security support for iPhones than Google does for Nexus/Pixel phones so it might be possible to get an older iPhone with a decent screen and hardly any space (but that’s not the point here).

If you have 32G of storage on your phone then there’s no real possibility of using up your storage space by photographing a day’s work. You could probably take an unreasonable number of photos of a week’s work as well as a few videos and not use up much of that.

The first time I needed photos recently was about 9 months ago when I was replacing some network gear (new DSL modem and switch for a client). The network sockets in the rack weren’t labelled and I found it unreasonably difficult to discover where everything was (the tangle of cables made tracking them impossible). What I should have done is to photograph the cables before I started and then I would have known where to connect everything. A 12MP camera allows zooming in on photos to get details, so a couple of quick shots of that rack would have saved me a lot of time – and in the case where everything goes as planned taking a couple of photos isn’t going to delay things.

Last night there was a power failure in a server room that hosts a couple of my machines. When power came back on the air-conditioner didn’t start up and the end result was a server with one of it’s disks totally dead (maybe due to heat, maybe power failures, maybe it just wore out). For unknown reasons BTRFS wouldn’t allow me to replace the disk in the RAID-1 array so I needed to copy the data to a new disk and create a new mirror (taking a lot of my time and also giving downtime). While I was working on this the filesystem would only mount read-only so no records of the kernel errors were stored. If I had taken photos of the screen I would have records of this which might allow me to reproduce the problem and file a bug report. Now I have no records, I can’t reproduce it, and I have a risk that next time a disk dies in a BTRFS RAID-1 I’ll have the same problem. Also presumably random people all over the world will suffer needless pain because of this while lacking the skills to file a good bug report because I didn’t make good enough records to reproduce it.

Hopefully next time I’m in a situation like this I’ll think to take some photos instead of just rebooting and wiping the evidence.

As an aside I’ve been finding my phone camera useful for zooming in on serial numbers that I can’t read otherwise. I’ve got new glasses on order that will hopefully address this, but in the mean time it’s the only way I can read the fine print. Another good use of a phone camera is recording error messages that scroll past too quickly to read and aren’t logged. Some phones support slow motion video capture (up to 120fps or more) and even for phones that don’t you can use slow play (my favourite Android video player MX Player works well at 5% normal speed) to capture most messages that are too quick to read.

January 05, 2019

Audiobooks – December 2018

The Casebook of Sherlock Holmes by Sir Arthur Conan Doyle. Read by Stephen Fry

A bit weaker than the other volumes. The author tries mixing the style in places but several stories feel like repeats. Lacking excitement 7/10

Children of Dune by Frank Herbert

A good entry for the 2nd tier of Dune Books (behind 1, even with 2 & 4). Good mix of story, philosophy and politics. Plot a little plodding though.  8/10

Modern Romance: An Investigation by Aziz Ansari

Lots of hardish data and good advice for people looking to date online. Around 3-4 years old so fairly uptodate. Funny in a lot of places & some audiobook extra bits. 8/10

The Day of Battle: The War in Sicily and Italy, 1943-1944 by Rick Atkinson

Part 2 of a trilogy. I liked this a little more than the first edition but it is still hard to follow in the audiobook format without maps etc. Individual stories lift it up. 7/10

The End of Night: Searching for Natural Darkness in an Age of Artificial Light
by Paul Bogard

A book about how darkness is being lost for most people. How we are missing out on the stars, a good sleep and much else. I really liked this, writing good and topics varied. 8/10

President Carter: The White House Years by Stuart E. Eizenstat

Eizenstat was Chief Domestic Policy Advisor to Carter & took extensive notes everywhere. The book covers just above everything, all the highs and lows. Long but worth it. 7/10


Happy New Workbench

For the summer solstice I got a brand new second hand stainless steel bench top. Which by coincidence turns out to be the answer to the question "What do you get the man who has everything", apparently.

Brand new second hand stainless steel bench top

Turning to my cliché handbook, it turns out that when life gives you bench tops, you make workbenches. I needed a second standing desk for out on the deck and for hardware hacking, so under the watchful eye of a pair of local King Parrots I gathered some spare timber to cobble together a standing workbench for the bench top.

Stainless steel workbench upside down

Stainless steel workbench bare

Stainless steel workbench stage 1

There's a lot more work to be done on that desk but with stage 1 at least completed, it was time to install it on the deck, ready to hack 2019 away :-D

Stainless steel workbench in place

Yes, that is the Australian Sun reaching in, in the background to kill us all.

The astute amongst you will have also noticed the (empty) Lagavulin and Talisker bottles propping up my laptop so they're both physically and metaphorically supporting my work.

Looks like I've already put my hand up for some hacks on Termonad to be the first piece of work completed from the new deck desk, apart from this post of course :-D

So now that leaves me with an inside and an outside standing desk as well as a hammock to work from. Not sure what says about me, I'll leave that up to you.

LPCNet and Codec 2 Part 2

Since the last post I’ve trained LPCNet using quantised Codec 2 parameters. I’ve also modified the Codec 2 decoder (c2dec) to dump the received features to a disk file so I can use them to synthesise speech using LPCNet.

So now I can decode a Codec 2 bit stream using a LPCNet decoder:

Condition Sample
Original 8kHz sample Original
Codec 2 1300 encoder and Codec 2 decoder codec2_1300
Codec 2 1300 encoder and LPCNet decoder lpcnet_1300
Codec 2 2400 encoder and Codec 2 decoder codec2_2400
Codec 2 2400 encoder and LPCNet decoder lpcnet_2400
Codec 2 unquantised 6th order LSP and LPCNet decoder lpcnet_6lsps

I think it sounds pretty good, especially at 2400 bits/s. Even the lpcnet_1300 decoder sounds better than Codec 2 at 2400. LPCNet is much more natural, without the underwater sound of low rate vocoded speech.

Less Features

The Kleijn et al paper showed NN based synthesises can produce high quality speech from feature sets of legacy vocoders like Codec 2. This implies the legacy vocoders are sending a lot of extra information that is not normally used by the legacy decoder/synthesis algorithms. If high quality speech is not required, it could be argued we are sending “too much” information, and scope exists to reduce the bit rate by coarser quantisation or sending less features.

The lpcnet_6lsps sample uses just 6th order Linear Prediction, in the form of 6 Line Spectral Pairs (LSPs). A LPC order of 10 is common for 8 kHz sampled speech. There is no way 6th order LPC would work (i.e. provide intelligible speech) for any existing LPC based vocoder (CELP, MELP etc). This sample has some odd artefacts; but is intelligible, and also quite natural sounding compared to a vocoder.

Curiously, when the synthesis breaks down (e.g. “depth of the well”), it sounds to me like the pitch is halved. I can hear this on both the male and female segments.

Next Steps

I still have much to learn in this area, but the initial results are promising, especially at 2400 bit/s. The main difference between the Codec 2 feature set at 2400 and 1300 is the update rate, so it would be useful to explore those differences further with a series of tests.

This work is not far from being usable “over the air” as a FreeDV mode, and promises a significant jump in quality. Codec 2 1300 is the vocoder used for FreeDV 1600, so it may be possible to develop a drop in replacement for initial testing.

The “best” (in terms of quality at a given bit rate) encoder and feature set (especially under quantisation) is an open research question. It is amazing that it works with the Codec 2 bit stream at all – we should be able to do better with a custom encoder.

December 31, 2018

Part 1 of my “Selfies on Seven Continents” series

One of the defining aspects of my 2018 was travel. There was travel for fun, travel for work, and travel for adventure. This year, I was fortunate enough to visit all seven continents, so in honour of my selfie habits over on my photo blog, here are some of my favourite selfies from this year, by continent.

South America

South America was entirely for fun and adventure. My dad and I got to tourist our way around, before we visited Antarctica.


Leaving from Punta Arenas, we travelled with Aurora Expeditions south along the Antarctic Peninsula, crossing the Antarctic Circle, before returning north again. Visiting Antarctica was one of the most spectacular, other-worldly, and unforgettable experiences of my life. If you’ve ever considered it, I highly recommend taking the plunge! (Both figuratively and literally. 😉)

North America

Work dominated my travel to North America, but there was some time for fun and selfies.


Thanks to a long layover, I just managed to visit Asia, dropping in for dinner with some friends in Hong Kong.


Europe was mostly work travel, which meant a distinct lack of selfies, sadly. This is clearly a shortcoming in my selfie taking habits that I need to work on next year.


Home was a place for fun, family, and a little bit of introducing my kids to the excitement of travelling! 😁

(I wish I could fit into an economy seat like a three year old.)


Rounding out the year, my first trip to Africa! Just a little adventure to Kenya, seeing a few of the sights, but leaving plenty more for my next trip. 🤫

So, that’s been my year of travel. I flew 202,717km, visiting 28 cities in 12 different countries, maintaining an average speed of 23km/h over the year.

I’m looking forward to an exciting 2019, I hope your 2019 will be excellent, too. Happy New Year, everyone! 🎉

Book Review: Fear: Trump in the White House

by Bob Woodward

Fear: Trump in the White House

If you suffer from anxiety about what's happening in the world and the competence of those in power, this is probably not the book for you. Bob Woodward's book shows in startling detail just how inadequate, incompetent and immature not only the President of the USA is but also most of the team that he had/has around him.

I didn't takes notes as I read this book and it really needs it. There's so much going on that its difficult to know where to begin. I can understand why Bob wrote this in a pretty much chronological fashion. There's one quote that stands out from Dowd, Trump's personal lawyer, the man that I came away with feeling had Trump's best interests at heart more than anyone else:

But in the man and his presidency Dowd had seen the tragic flaw. In the political back-and-forth, the evasions, the denials, the tweeting, the obscuring, crying "Fake News," the indignation, Trump had one overriding problem that Dowd knew but could not bring himself to say to the president:

"You're a fucking liar."

The primary feeling I was left with after reading this book was arrested development. This phrase best describes for me the behaviour of most of the people in involved in this tale. They just are not mature adults with a steady hand.

It's a phrase I've also come to associate with Australian politicians like Tony Abbott, Peter Dutton, Pauline Hanson, Clive Palmer... I'll stop there, it could be long list and there are not many Australian politicians that I could leave off that list sadly.

The kind of people who put their own narrow interests or the interests of their base or more importantly, donors, ahead of clear eyed long term national interests. The public rightly becomes disengaged with politics which leads to our politicians being drawn from an increasingly shallow pool of staffers and cronies, which leads to increased public disengagement, which further shrinks the pool from which our politicians are drawn.

It's a vicious loop that Wolfgang Streek expounds upon quite well in How Will Capitalism End? : Essays On A Failing System , a highly recommended read that looks deeply into and explains the socio-political problems of the modern era.

The secondary feeling I'm left with is just how broken "western" democracy truly is (I've argued at times that we do not actually live in a genuine democracy, just because we get to vote).

Clearly, in "western" nations across the world, politicians are not rising into power based on any form of merit or competency. More and more politicians in the USA, UK and Australia are coming into power despite their clearly displayed inadequacy, in spite of their arrested development, they are wielding power and being voted in by a jaded and disengaged public.

I've got no clear answer for you, only more questions.

Increasing broader community participation in politics is one, albeit unlikely solution (although the recent success of socialists in the US mid-terms show it can be done).

Another is to completely change the nature of our Democracy so that it functions as advertised but you first need to remove those in power with vested interests in the status quo.

So, this was a book review. Fantastic, insightful and frightening. Well worth a read.

December 27, 2018

Restricting outgoing HTTP traffic in a web application using a squid proxy

I recently had to fix a Server-Side Request Forgery bug in Libravatar's OpenID support. In addition to enabling authentication on internal services whenever possible, I also forced all outgoing network requests from the Django web-application to go through a restrictive egress proxy.

OpenID logins are prone to SSRF

Server-Side Request Forgeries are vulnerabilities which allow attackers to issue arbitrary GET requests on the server side. Unlike a Cross-Site Request Forgery, SSRF requests do not include user credentials (e.g. cookies). On the other hand, since these requests are done by the server, they typically originate from inside the firewall.

This allows attackers to target internal resources and issue arbitrary GET requests to them. One could use this to leak information, especially when error reports include the request payload, tamper with the state of internal services or portscan an internal network.

OpenID 1.x logins are prone to these vulnerabilities because of the way they are initiated:

  1. Users visit a site's login page.
  2. They enter their OpenID URL in a text field.
  3. The server fetches the given URL to discover the OpenID endpoints.
  4. The server redirects the user to their OpenID provider to continue the rest of the login flow.

The third step is the potentially problematic one since it requires a server-side fetch.

Filtering URLs in the application is not enough

At first, I thought I would filter out undesirable URLs inside the application:

  • hostnames like localhost, or ::1
  • non-HTTP schemes like file or gopher
  • non-standard ports like 5432 or 11211

However this filtering is going to be very easy to bypass:

  1. Add a hostname in your DNS zone which resolves to
  2. Setup a redirect to a blacklisted URL such as file:///etc/passwd.

Applying the filter on the original URL is clearly not enough.

Install and configure a Squid proxy

In order to fully restrict outgoing OpenID requests from the web application, I used a Squid HTTP proxy.

First, install the package:

apt install squid3

and set the following in /etc/squid3/squid.conf:

acl to_localnet dst # RFC 1122 "this" network (LAN)
acl to_localnet dst            # RFC 1918 local private network (LAN)
acl to_localnet dst         # RFC 6598 shared address space (CGN)
acl to_localnet dst        # RFC 3927 link-local (directly plugged) machines
acl to_localnet dst         # RFC 1918 local private network (LAN)
acl to_localnet dst        # RFC 1918 local private network (LAN)
acl to_localnet dst fc00::/7              # RFC 4193 local private network range
acl to_localnet dst fe80::/10             # RFC 4291 link-local (directly plugged) machines

acl SSL_ports port 443
acl Safe_ports port 80
acl Safe_ports port 443

http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access deny manager
http_access deny to_localhost
http_access deny to_localnet
http_access allow localhost
http_access deny all


Ideally, I would like to use a whitelist approach to restrict requests to a small set of valid URLs, but in the case of OpenID, the set of valid URLs is not fixed. Therefore the only workable approach is a blacklist. The above snippet whitelists port numbers (80 and 443) and blacklists requests to localhost (a built-in squid acl variable which resolves to and ::1) as well as known local IP ranges.

Expose the proxy to Django in the WSGI configuration

In order to force all outgoing requests from Django to go through the proxy, I put the following in my WSGI application (/etc/libravatar/django.wsgi):

os.environ['ftp_proxy'] = ""
os.environ['http_proxy'] = ""
os.environ['https_proxy'] = ""

The whole thing seemed to work well in my limited testing. There is however a bug in urllib2 with proxying HTTPS URLs that include a port number, and there is an open issue in python-openid around proxies and OpenID.