Planet Linux Australia
Celebrating Australians & Kiwis in the Linux and Free/Open-Source community...

January 17, 2019

Installing NixOS on an ASUS EeePC 701

Those of you familiar with the ASUS EeePC 701 will be aware of it's limited memory (512M) and small internal drive (4G) which doesn't render it useless but does reduce it's modern usage.

I have two of these still in perfect working order and wanted to extend their functional lives. I have no need of any more routers and there are Raspberry Pi's everywhere here. I do have kids though and decided these would make great, ready made portable retro gaming consoles.

This blog post details how I installed NixOS and configured the 701's to overcome their short comings to extend their life.

At a Glance

  1. Download the latest NixOS minimal installer for 32-bit Intel/AMD.
  2. Boot off USB
  3. Set forcepae in Grub
  4. Connect networking
  5. Partition then format the SD card as / and the internal drive as swap
  6. Run the configuration builder
  7. Copy across nix files
  8. Install
  9. Enjoy

Process Detail

Have a copy of the NixOS manual handy. It will also be at [ALT]+[F8] once the installer has booted.

Booting

  1. Download the minimal installation CD, 32-bit Intel/AMD image and burn it to a USB stick.
  2. Insert the SD card and USB stick into the 701 (I used a spare 8G SD card I had handy).
  3. Power the 701 on and press [ESC], select the USB stick as the boot device,
  4. [TAB] at grub menu, append forcepae
  5. Continue the boot process until it reaches a root prompt
  6. If you have not connected Ethernet, connect to WiFi using wpa_supplicant, as per manual.

Partitioning and Formatting

With only a little deviation from section 2.2.2. Legacy Boot (MBR) we format the SD card to be the root disk and the internal drive as swap:

  • Create an MBR on the internal drive:

    # parted /dev/sda -- mklabel msdos

  • Create the root partition on the SD card:

    # parted /dev/sdc -- mkpart primary 1MiB 100%

  • Create the swap partition on the internal drive:

    # parted /dev/sda -- mkpart primary linux-swap 1MiB 100%

As per section 2.2.3 Formatting we now format the drives appropriately.

  • Format and label the / partition:

    # mkfs.ext4 -L nixos /dev/sdc1

  • Format the swap partion:

    # mkswap -L swap /dev/sda1

Installation and Configuration:

Now you can just continue to follow section 2.3 Installing and configure the system as you desire. I initially use a bare bones install and it can be read here. It's worth taking a brief look at if you're new to NixOS.

The section worth copying is related to how the kernel uses memory on this old i386 system:

  nixpkgs.config = {
    packageOverrides = pkgs: {
      stdenv = pkgs.stdenv // {
        platform = pkgs.stdenv.platform // {
          kernelExtraConfig = ''
            HIGHMEM64G? n  # 32-bit proc with > 4G RAM
            HIGHMEM4G y # 32-bit # proc # with # =< # 4G # RAM
          '';
        };
      };
    };
  };

Do not copy my import of retro-gaming.nix, that's a for after a successful install and a future blog post.

Once you've completed the install, the EeePC 701 ought to reboot into NixOS successfully.

January 15, 2019

Support for Raspberry Pi and Orange Pi GPIOs in Home Assistant

Share

So, I’ve been off in the GPIO library salt mines for a while, but am now ready to circle back and document how to get GPIO inputs and outputs working in Home Assistant. This now works on both Raspberry Pi and OrangePi, assuming that my patch gets merged.

First off, let’s talk about GPIO outputs. This is something which has been working for a while on both platforms (a while being a week or so, assuming you’ve patched Home Assistant with my pull request, but you’re all doing that right?).

To configure an output in Home Assistant, you would add the following to configuration.yaml:

rpi_gpio:
  board_family: orange_pi

switch:
 - platform: rpi_gpio
   ports:
     PA7: LED

Where board_family can be either “raspberry_pi” or “orange_pi”. Note that for Raspberry Pis, the pin numbers are always numbers whereas for OrangePi we are using “SUNXI” numbering, which is of the form “PA7”.

The circuit for this LED is really simple:

A simple LED circuit

Now we have a switch we can control in Home Assistant:

Raspberry Pi LED switch in Home Assistant

GPIO inputs are similar. The configuration looks like this:

rpi_gpio:
  board_family: orange_pi

binary_sensor:
 - platform: rpi_gpio
   invert_logic: true
   ports:
     PA7: PUSHYBUTTON

With a circuit like this:

A circuit with a button in it

invert_logic set to true is required because our circuit sends the value of PA7 to ground when the button is pressed.

A push button being pressed in Home AssistantNoting that sensors look different to switches in Home Assistant, you can see the binary sensor at the top right of the image, with its history being displayed in the dialog box in the foreground.

Share

January 14, 2019

Updated examples for OrangePi GPIOs

Share

As part of working through adding OrangePi support to Home Assistant, Alastair and I decided to change to a different GPIO library for OrangePi to avoid the requirement for Home Assistant to have access to /dev/mem.

I just realised that I hadn’t posted updated examples of how to do GPIO output with the new library. So here’s a quick post about that.

Assuming that we have an LED on GPIO PA7, which is pin 29, then the code to blink the LED would look like this with the new library:

import OPi.GPIO as GPIO
import time


# Note that we use SUNXI mappings here because its way less confusing than
# board mappsings. For example, these are all the same pin:
# sunxi: PA7 (the label on the board)
# board: 29
# gpio:  7

GPIO.setmode(GPIO.SUNXI)
GPIO.setwarnings(False)
GPIO.setup('PA7', GPIO.OUT)

while True:
    GPIO.output('PA7', GPIO.HIGH)
    time.sleep(1)
    GPIO.output('PA7', GPIO.LOW)
    time.sleep(1)

The most important thing there is the note about SUNXI pin mappings. I find the whole mapping scheme hugely confusing, unless you use SUNXI and then its all fine. So learn from my fail people!

What about input? Well, that’s not too bad either. Let’s assume that you have a button in a circuit like this:

A circuit with a button in it

The to read the button the polling way, you’d just do this:

import OPi.GPIO as GPIO
import time

GPIO.setmode(GPIO.SUNXI)
GPIO.setwarnings(False)
GPIO.setup('PA7', GPIO.IN, pull_up_down=GPIO.PUD_DOWN)

while True:
    print('Reading...')
    if GPIO.input('PA7') == GPIO.LOW:
        print('Pressed')
    else:
        print('Released')
    time.sleep(1)

Let’s pretend it didn’t take me ages to get that to work right because I had the circuit wrong, ok?

Now, we have self respect, so you wouldn’t actually poll like that. Instead you’d use edge detection, and end up with code like this:

import OPi.GPIO as GPIO
import time

GPIO.setmode(GPIO.SUNXI)
GPIO.setwarnings(False)
GPIO.setup('PA7', GPIO.IN, pull_up_down=GPIO.PUD_DOWN)

def event_callback(channel):
    print('Event detected: %s' % GPIO.input('PA7'))
    
GPIO.add_event_detect('PA7', GPIO.BOTH, callback=event_callback, bouncetime=50)

while True:
    time.sleep(1)

So there ya go.

Share

GPIO inputs on Raspberry Pi

Share

Now that I have GPIO outputs working nicely for Home Assistant using either a Raspberry Pi or an Orange Pi, I want to get GPIO inputs working as well. Naively, that’s pretty easy to do in python on the Raspberry Pi:

import RPi.GPIO as GPIO
import time

GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)
GPIO.setup(17, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)

while True:
    print('Reading...')
    if GPIO.input(17) == GPIO.HIGH:
        print('Pressed')
    else:
        print('Released')
    time.sleep(1)

That code is of course horrid. Its horrid because its polling the state of the button, and its quite likely that I can sneak a button press in during one of those sleeps and it will never be noticed. Instead we can use edge detection callbacks to be informed of button presses as they happen:

import RPi.GPIO as GPIO
import time

GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)
GPIO.setup(17, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)

def event_callback(channel):
    print('Event detected: %s' % GPIO.input(17))
    
GPIO.add_event_detect(17, GPIO.BOTH, callback=event_callback, bouncetime=50)

while True:
    time.sleep(1)

This second program provides helpful output like this:

pi@raspberrypi:~ $ python gpio_button_edge_detect.py 
Event detected: 1
Event detected: 0

Which is me pressing the button once (it go high when pressed, and then low again when released). This is of course with a button wired to GPIO17 with a current limiting resistor between the button and the 3.3v rail.

Share

Annual Penguin Picnic, January 19, 2019

Jan 19 2019 12:00
Jan 19 2019 16:00
Jan 19 2019 12:00
Jan 19 2019 16:00
Location: 
Yarra Bank Reserve, Hawthorn

The Linux Users of Victoria Annual Penguin Picnic will be held on Saturday, January 19, starting at 12 noon at the Yarra Bank Reserve, Hawthorn.

LUV would like to acknowledge Infoxchange for the Richmond venue.

Linux Users of Victoria Inc., is a subcommitee of Linux Australia.

January 19, 2019 - 12:00

read more

Transport security for BGP, AKA BGP-STARTTLS, a proposal

Several days ago, inspired in part by an old work mail thread being resurrected I sent this image as a tweet captioned "The state of BGP transport security.":



The context of the image for those not familiar with it is this image about noSQL databases.

This triggered a bunch of discussion, with multiple people saying "so how would *you* do it", and we'll get to that (or for the TL;DR skip to the bottom), but first some background.

The tweet is a reference to the BGP protocol the Internet uses to exchange routing data between (and within) networks. This protocol (unless inside a separate container) is never encrypted, and can only be authenticated (in practice) by a TCP option known as TCP-MD5 (standardised in RFC2385). The BGP protocol itself has no native encryption or authentication support. Since routing data can often be inferred by the packets going across a link anyway, this has lead to this not being a priority to fix.

Transport authentication & encryption is a distinct issue from validation of the routing data transported by BGP, an area already being worked on by the various RPKI projects, eventually transport authentication may be able to benefit from some of the work done by those groups.

TCP-MD5 is quite limited, and while generally supported by all major BGP implementations it has one major limitation that makes it particularly annoying, in that it takes a single key, making key migration difficult (and in many otherwise sensible topologies, impossible without impact). Being a TCP option is also a pain, increasing fragility.

At the time of its introduction TCP-MD5 gave two main benefits the first was to have some basic authentication beyond the basic protocol (for which the closest element in the protocol is the validation of peer-as in the OPEN message, and a mismatch will helpfully tell you who the far side was looking for), plus making it harder to interfere with the TCP session, which on many of the TCP implementations of the day was easier than it should have been. Time, however has marched on, and protection against session interference from non-MITM is no longer needed, the major silent MITM case of Internet Exchanges using hubs is long obsolete, plus, in part due to the pain associated in changing keys many networks have a "default" key they will use when turning up a peering session, these keys are often so well known for major networks that they've often been shared on public mailing lists, eliminating what little security benefit TCP-MD5 still brings.

This has been known to be a problem for many years, and the proposed replacement TCP-AO (The TCP Authentication Option) was standardised in 2010 as RFC5925, however, to the best of my knowledge eight years later no mainstream BGP implementation supports it, and as it too is a TCP option, not only does it still has many of the downsides of MD5, but major OS kernels are much less likely to implement new ones (indeed, an OS TCP maintainer commenting such on the thread I mentioned above is what kicked off my thinking).

TCP, the wire format, is in practice unchangeable. This is one of the major reasons for QUIC, the TCP replacement protocol soon to be standardised as HTTP/3, so for universal deployment any proposal that requires changes to TCP is a non-starter.

Any solution must be implementable & deployable.
  • Implementable - BGP implementations must be able to implement it, and do so correctly, ideally with a minimum of effort.
  • Deployable - Networks need to be able to deploy it, when authentication issues occur error messages should be no worse than with standard BGP (this is an area most TCP-MD5 implementations fail at, of those I've used JunOS is a notable exception, Linux required kernel changes for it to even be *possible* to debug)


Ideally any security-critical code should already exist in a standardised form, with multiple widely-used implementations.

Fortunately for us, that exists in the form of TLS. IPSEC, while it exists, fails the deployable tests, as almost anyone who's ever had the misfortune of trying to get IPSEC working between different organisations using different implementations can attest, sure it can usually be made to work, but nowhere near as easily as TLS.

Discussions about the use of TLS for this purpose have happened before, but always quickly run into the problem of how certificates for this should be managed, and that is still an open problem, potentially the work on RPKI may eventually provide a solution here, but until that time we have a workable alternative in the form of TLS-PSK (first standardised in RFC4279), a set of TLS modes that allow the use of pre-shared keys instead of certificates (for those wondering, not only does this still exist in TLS1.3 it's in a better form). For a variety of reasons, not the least the lack of real-time clocks in many routers that may not be able to reach an NTP server until BGP is established, PSK modes are still more deployable than certificate verification today. One key benefit for TLS-PSK is it supports multiple keys to allow migration to a new key in a significantly less impactful manner.

The most obvious way to support BGP-in-TLS would simply be to listen on a new port (as is done for HTTPS for example), however there's a few reasons why I don't think such a method is deployable for BGP, primarily due to the need to update control-plane ACLs, a task that in large networks is often distant from the folk working with BGP, and in small networks may not be understood by any current staff (a situation not dissimilar to the state of TCP). Another option would simply be to use protocol multiplexing and do a TLS negotiation if a TLS hello is received, or unencrypted BGP for a BGP OPEN, this would violate the general "principal of least astonishment", and would be harder for operators to debug.

Instead I propose a design similar to that used by SMTP (where it is known as STARTTLS), during early protocol negotiation support for TLS is signalled using a zero-length capability in the BGP OPEN, the endpoints do a TLS negotiation, and then the base protocol continues inside the new TLS tunnel. Since this negotiation happens during the BGP OPEN, it does mean that other data included in the OPEN leaks. Primarily this is the ASN, but also the various other capabilities supported by the implementation (which could identify the implementation), I suggest that if TLS is required information in the initial OPEN not be validated, and standard reserved ASN be sent instead, and any other capabilities not strictly required not sent, with a fresh OPEN containing all normal information sent inside the TLS session.

Migration from TCP-MD5 is key point, however not one I can find any good answer for. Some implementations already allow TCP-MD5 to be optional, and that would allow an easy upgrade, however such support is rare, and unlikely to be more widely supported.

On that topic, allowing TLS to be optional in a consistent manner is particularly helpful, and something that I believe SHOULD be supported to allow cases like currently unauthenticated public peering sessions to be migrated to TLS with minimal impact. Allowing this does open the possibility of a downgrade attack, and make more plausible attacks causing state machine confusions (implementation believes it's talking on a TLS-secured session when it isn't).

What do we lose from TCP-MD5? Some performance, whilst this is not likely to be an issue for most situations, it is likely not an option for anyone still trying to run IPv4 full tables on a Cisco Catalyst 6500 with Sup720. We do also lose the TCP manipulation prevention aspects, however these days those are (IMO) of minimal value in practice. There's also the costs of implementations needing to include a TLS implementations, and whilst nearly every system will have one (at the very least for SSH) it may not already be linked to the routing protocol implementation.

Lastly, my apologies to anyone who has proposed this before, but my neither I nor my early reviewers were aware of such a proposal. Should such a proposal already exist, meeting the goals of implementable & deployable it may be sensible to pick that up instead.

The IETF is often said to work on "rough consensus and running code", for this proposal here's what I believe a minimal *actual* demonstration of consensus with code would be:
  • Two BGP implementations, not derived from the same source.
  • Using two TLS implementations, not derived from the same source.
  • Running on two kernels (at the very least, Linux & FreeBSD)


The TL;DR version:
  • Using a zero-length BGP capability in the BGP OPEN message implementations advertise their support for TLS
    • TLS version MUST be at least 1.3
    • If TLS is required, the AS field in the OPEN MAY be set to a well-known value to prevent information leakage, and other capabilities MAY be removed, however implementations MUST NOT require the TLS capability be the first, last or only capability in the OPEN
    • If TLS is optional, which MUST NOT be default behaviour), the OPEN MUST be (other than the capability) be the same as a session configured for no encryption
  • After the TCP client receives a conformation of TLS support from the TCP server's OPEN message, a TLS handshake begins
    • To make this deployable TLS-PSK MUST be supported, although exact configuration is TBD.
    • Authentication-only variants of TLS (ex RFC4785) REALLY SHOULD NOT be supported.
    • Standard certificate-based verification MAY be supported, and if supported MUST validate use client certificates, validating both. However, how roots of trust would work for this has not been investigated.
  • Once the TCP handshake completes the BGP state starts over with the client sending a new OPEN
    • Signalling the TLS capability in this OPEN is invalid and MUST be rejected
  • (From here, everything is unchanged from normal BGP)


Magic numbers for development:
  • Capability: (to be referred to as EXPERIMENTAL-STARTTLS) 219
  • ASN (for avoiding data leaks in OPEN messages): 123456
    • Yes this means also sending 4-byte capability. Every implementation that might possibly implement this already supports 4-byte ASNs.


The key words "MUST (BUT WE KNOW YOU WON'T)", "SHOULD CONSIDER", "REALLY SHOULD NOT", "OUGHT TO", "WOULD PROBABLY", "MAY WISH TO", "COULD", "POSSIBLE", and "MIGHT" in this document are to be interpreted as described in RFC 6919.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC2119.

Are Men the Victims?

A very famous blog post is Straight White Male: The Lowest Difficulty Setting There Is by John Scalzi [1]. In that post he clearly describes that life isn’t great for straight white men, but that there are many more opportunities for them.

Causes of Death

When this post is mentioned there are often objections, one common objection is that men have a lower life expectancy. The CIA World factbook (which I consider a very reliable source about such matters) says that the US life expectancy is 77.8 for males and 82.3 for females [2]. The country with the highest life expectancy is Monaco with 85.5 for males and 93.4 years for females [3]. The CDC in the US has a page with links to many summaries about causes of death [4]. The causes where men have higher rates in 2015 are heart disease (by 2.1%), cancer (by 1.7%), unintentional injuries (by 2.8%), and diabetes (by 0.4%). The difference in the death toll for heart disease, cancer, unintentional injuries, and diabetes accounts for 7% of total male deaths. The male top 10 lists of causes of death also includes suicide (2.5%) and chronic liver disease (1.9%) which aren’t even in the top 10 list for females (which means that they would each comprise less than 1.6% of the female death toll).

So the difference in life expectancy would be partly due to heart problems (which are related to stress and choices about healthy eating etc), unintentional injuries (risk seeking behaviour and work safety), cancer (the CDC reports that smoking is more popular among men than women [5] by 17.5% vs 13.5%), diabetes (linked to unhealthy food), chronic liver disease (alcohol), and suicide. Largely the difference seems to be due to psychological and sociological issues.

The American Psychological Association has for the first time published guidelines for treating men and boys [6]. It’s noteworthy that the APA states that in the past “psychology focused on men (particularly white men), to the exclusion of all others” and goes on to describe how men dominate the powerful and well paid jobs. But then states that “men commit 90 percent of homicides in the United States and represent 77 percent of homicide victims”. They then go on to say “thirteen years in the making, they draw on more than 40 years of research showing that traditional masculinity is psychologically harmful and that socializing boys to suppress their emotions causes damage that echoes both inwardly and outwardly”. The article then goes on to mention use of alcohol, tobacco, and unhealthy eating as correlated with “traditional” ideas about masculinity. One significant statement is “mental health professionals must also understand how power, privilege and sexism work both by conferring benefits to men and by trapping them in narrow roles”.

The news about the new APA guidelines focuses on the conservative reaction, the NYT has an article about this [7].

I think that there is clear evidence that more flexible ideas about gender etc are good for men’s health and directly connect to some of the major factors that affect male life expectancy. Such ideas are opposed by conservatives.

Risky Jobs

Another point that is raised is the higher rate of work accidents for men than women. In Australia it was illegal for women to work in underground mines (one of the more dangerous work environments) until the late 80’s (here’s an article about this and other issues related to women in the mining industry [8]).

I believe that people should be allowed to work at any job they are qualified for. I also believe that we need more occupational health and safety legislation to reduce the injuries and deaths at work. I don’t think that the fact that a group of (mostly male) politicians created laws to exclude women from jobs that are dangerous and well-paid while also not creating laws to mitigate the danger is my fault. I’ll vote against such politicians at every opportunity.

Military Service

Another point that is often raised is that men die in wars.

In WW1 women were only allowed to serve in the battlefield as nurses. Many women died doing that. Deaths in war has never been an exclusively male thing. Women in many countries are campaigning to be allowed to serve equally in the military (including in combat roles).

As far as I am aware the last war where developed countries had conscription was the Vietnam war. Since then military technology has developed to increasingly complex and powerful weapons systems with an increasing number of civilians and non-combat military personnel supporting each soldier who is directly involved in combat. So it doesn’t seem likely that conscription will be required for any developed country in the near future.

But not being directly involved in combat doesn’t make people safe. NPR has an interesting article about the psychological problems (potentially leading up to suicide) that drone operators and intelligence staff experience [9]. As an aside the article reference two women doing that work.

Who Is Ignoring These Things?

I’ve been accused of ignoring these problems, it’s a general pattern on the right to accuse people of ignoring these straight white male problems whenever there’s a discussion of problems that are related to not being a straight white man. I don’t think that I’m ignoring anything by failing to mention death rates due to unsafe workplaces in a discussion about the treatment of trans people. I try to stay on topic.

The New York Times article I cited shows that conservatives are the ones trying to ignore these problems. When the American Psychological Association gives guidelines on how to help men who suffer psychological problems (which presumably would reduce the suicide rate and bring male life expectancy closer to female life expectancy) they are attacked by Fox etc.

My electronic communication (blog posts, mailing list messages, etc) is mostly connected to the free software community, which is mostly male. The majority of people who read what I write are male. But it seems that the majority of positive feedback when I write about such issues is from women. I don’t think there is a problem of women or left wing commentators failing men. I think there is a problem of men and conservatives failing men.

What Can We Do?

I’m sure that there are many straight white men who see these things as problems but just don’t say anything about it. If you don’t want to go to the effort of writing a blog post then please consider signing your name to someone else’s. If you are known for your work (EG by being a well known programmer in the Linux community) then you could just comment “I agree” on a post like this and that makes a difference while also being really easy to do.

Another thing that would be good is if we could change the hard drinking culture that seems connected to computer conferences etc. Kara has an insightful article on Model View Culture about drinking and the IT industry [10]. I decided that drinking at Linux conferences had got out of hand when about 1/3 of the guys at my table at a conference dinner vomited.

Linux Conf Au (the most prestigious Linux conference) often has a Depression BoF which is really good. I hope they have one this year. As an aside I have problems with depression, anyone who needs someone to talk to about such things and would rather speak to me than attend a BoF is welcome to contact me by email (please take a failure to reply immediately as a sign that I’m behind on checking my email not anything else) or social media.

If you have any other ideas on how to improve things please make a comment here, or even better write a blog post and link to it in a comment.

January 10, 2019

Further adventures in Home Assistant OrangePi GPIO

Share

Its funny how a single sentence can change your course. In the last post about this work, I said:

We also need to run hass as root,  because OrangePi GPIO support requires access to /dev/mem for reasons I haven’t dug into just yet.

That’s turned out to be (reasonably) a pretty big sticking point upstream. Access to /dev/mem gives you a whole bunch of access to the machine that Home Assistant probably shouldn’t have.

Alastair went off spelunking because he’s more patient than me and found yet another OrangePi GPIO library. I think we’re up to three or four of these at the moment, but this is the first one we’ve found which supports the sysfs interface to GPIO pins. That’s exciting because it removes our run-as-root requirement. Its unexciting in that the sysfs interface has been deprecated by the kernel, but will remain supported for a while.

I think people would be within their rights to conclude that the state of GPIO libraries for OrangePi is a bit of a dumpster fire right now.

Anyways, the point of this post is mostly to write down how to use the sysfs interface to GPIO pins so that I can remember it later, I’ll take more about this new library and if it meets our needs in a later post.

The first step is to determine what pin number the GPIO pin is. On the OrangePi these are labelled with names like “PA7”, which is the 7th bit in the “A” GPIO register. To convert that into a pin number as used by sysfs you do this:

def pin_number(letter, digit):
    return (ord(letter) - ord('A')) * 32 + digit

So, pin_number(‘A’, 7) for PA7 is just … 7.

Note that I now recommend that people use SUNXI pin mapping, as its much less confusing. You can read more about alternative pin mappings in this post of worked OrangePi GPIO examples.

Now we can enable the pin, set it to output, and then blink the LED to prove it works:

# cd /sys/class/gpio
# echo 7 > export
# cd gpio7
# echo "out" > direction
# echo 1 > value
# sleep 1
# echo 0 > value

The next step? To make sure that the new GPIO library supports sysfs access to GPIOs on the OrangePi Prime.

Share

January 09, 2019

Simple FOSS versus Complex Enterprise Software

As is often the case real IT operators in large organisations find themselves having to deal with "enterprise" software which has been imposed upon them. The decision to implement such software is usually determined by perceived business requirements (which is reasonable enough), but with little consideration of the operations and flexibility for new, or even assumed, needs.

For example, a certain large university which I have some familiarity with has decided to introduce an absolutely $awful_ticketing_system campus-wide. It serves as an excellent example because every step of the way there are just terrible things about this which illustrates how IT management decisions cause unnecessary problems for IT operators.

The beginning of this story comes with what initially seemed to be a simple problem. Emails that were sent to the ticketing system were being automatically assigned as an Incident, which had a tight Service Level Agreement period, as one would expect. The tickets that were being received, however, were not Incidents, but rather feature Requests. Changing this would be simple and easy (more on this in a moment), right?

Apparently not. I feel for the poor person who had to respond to me with this explanation.

Technically, an incoming mail to a mailbox can be converted into any type of ServiceNow ticket.

Well, this is good news.

But, out of all ticket types, this conversion is the most complicated for "request" type of tickets due to its three layered structure (REQ, RITM, CTASK). It has to have all three layers created for each request type of ticket. There is no out of the box or simple way of converting incoming mails to request tickets. Hence, it has not been implemented so far.

Wait, what? In well-known FOSS ticketing systems (e.g., OTRS, Request Tracker, Trac), the operator can set the queue or equivalent. Here, with this proprietary enterprise software, not even the administrators can do it in a simple manner. The solution being developed? Get rid of email tickets and use a web portal instead.

... the option of mailbox that creates tickets in Servicenow, was made available for use cases where users (such a externals, guests, students etc.) could not access forms and hence could not request relevant services. But the idea was always to use $awful_ticketing_system Portal as the access point to report any issues or request services, thereby avoiding the need to use mailboxes.

I'll leave it to others to think about how a web portal won't be associated with a mailbox or how this will allow operators to assign the ticket type. But don't spend too much time on it. Because this specific software and this specific example isn't the real issue at hand.

The first problem is that IT Managers don't listen carefully enough to IT Operators. An article by Jeff Ello, ten years old now, explains the many problems involved. The entire thing is worth reading, multiple times even, but to give a summary of a single paragraph from the article:

While everyone would like to work for a nice person who is always right, IT pros will prefer a jerk who is always right over a nice person who is always wrong. Wrong creates unnecessary work, impossible situations and major failures. Wrong is evil, and it must be defeated. Capacity for technical reasoning trumps all other professional factors, period.

There is one additional statement that needs to be added to this: This is not optional. In many other human-to-human roles, such as management, it is possible to reconstruct questions, find a compromise between competing agendas, and so forth. This is not possible with IT Operators (assuming they are honest and knowledgeable), not because they don't want to do it (whatever "it" is), but because it simply isn't possible. To repeat: technical reasoning trumps all other professional factors. When it comes to technical matters, IT Managers need to do more than just take advice from the Operators; they need to do what the Operators tell them. Otherwise, the wrong technical decisions will be made and that will cost time and money in the future.

The second problem, and yes, it cascades from the first, is that IT Managers have an erroneous propensity to choose enterprise software which is complex and easy rather than free and open source software which is simple and hard. The difference between the two has been well stated by Rich Hickey. Managers tend to choose software which is easy to use because they are typically not Operators themselves. They become especially enticed by software which is easy to use and feature-rich because they incorrectly perceive that will satisfy the requirements of (often dogmatic) business logic. When it is inevitably discovered that there are new business requirements the enterprise software needs to be somehow re-developed (at additional cost and time, as determined by the external body), or worked around. or try to find a new enterprise software product (and then experience the horror of vendor lock-in).

All of this is something that would make an experienced FOSS IT Operator's remaining grey hairs stand on end. In our world if something isn't working right, we fix it there and then, and if we can't we ask around because somebody else will know how. Collectively we have more knowledge than any of us individually can have. In the FOSS world, a product can be hard, in both senses of the word. It does have a steep learning curve, but it also is durable. It is however, also simple, that is, it isn't interleaved. Simplicity is a prerequisite for reliability said Edsger W. Dijkstra in 1975. Most FOSS (and yes, there are exceptions) is built on the UNIX philosophy: Design programs to do only a single thing, but to do it well, and to work together well with other programs.

The third problem, and yes, it cascades from the second, is that IT systems are often too heavily orientated towards singular solutions for the entire environment. Now initially I (erroneously) agreed with this approach. There was a time that I would have argued that a single ticketing system throughout an entire campus was a good idea as it lowered administrative and training costs and allowed for a single point from which a variety of metrics could be derived. My concern was that in selecting a single system that the wrong system might be chosen (and remember "wrong is evil and it must be defeated") or, one is terrified to say, the worst system might be selected (and usually it is, for the reasons previously stated).

The problem is with the conceptualisation. A singular system as a monolith will almost inevitably suffer the aforementioned issues of complexity, and if it's a closed-source enterprise product, there's nothing you can do about it. A single system, especially a cloud-based system, will suffer massive performance issues simply due to the physics of distance and physics is not optional. So rather than a single system, it really is worth using multiple systems according to what is most contextually appropriate and which can interact with other systems when needed. Again, getting back to these core computer science concepts of modularity, specialisation, and connectivity. The supposed gains in reducing administrative overhead are actually losses if the operator is unable to adapt to new systems, and that adaption is achieved through knowing how generic systems work together.

By way of conclusion, it is worth thinking in terms of stripped-down, minimalist, free and open-source programs which satisfy all functional requirements and can be modularly extended in a manner that is simple but durable. The philosophy of the team behind suckless.org is an excellent approach. As for a practical example of a ticketing system, Joe Amstrong's example of the Erlang Ticketing System as a "minimal viable program" satisfies the criteria.

"Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand."
-- Archibald Putt, "Putt's Law and the Successful Technocrat", 2006

January 08, 2019

Erasing Persistent Storage Securely on Linux

Here are some notes on how to securely delete computer data in a way that makes it impractical for anybody to recover that data. This is an important thing to do before giving away (or throwing away) old disks.

Ideally though, it's better not to have to rely on secure erasure and start use full-disk encryption right from the start, for example, using LUKS. That way if the secure deletion fails for whatever reason, or can't be performed (e.g. the drive is dead), then it's not a big deal.

Rotating hard drives

With ATA or SCSI hard drives, DBAN seems to be the ideal solution.

  1. Burn it on CD,
  2. boot with it,
  3. and following the instructions.

Note that you should disconnect any drives you don't want to erase before booting with that CD.

This is probably the most trustworth method of wiping since it uses free and open source software to write to each sector of the drive several times. The methods that follow rely on proprietary software built into the firmware of the devices and so you have to trust that it is implemented properly and not backdoored.

ATA / SATA solid-state drives

Due to the nature of solid-state storage (i.e. the lifetime number of writes is limited), it's not a good idea to use DBAN for those. Instead, we must rely on the vendor's implementation of ATA Secure Erase.

First, set a password on the drive:

hdparm --user-master u --security-set-pass p /dev/sdX

and then issue a Secure Erase command:

hdparm --user-master u --security-erase-enhanced p /dev/sdX

NVMe solid-state drives

For SSDs using an NVMe connector, simply request a User Data Erase

nvme format -s1 /dev/nvme0n1

January 06, 2019

Photograph Your Work

One thing I should have learned before (but didn’t) and hope I’ve learned now is to photograph sysadmin work.

If you work as a sysadmin you probably have a good phone, if you are going to run ssh from a phone or use a phone to read docs while in a server room with connectivity problems you need a phone with a good screen. You will also want a phone that has current security support. Such a phone will have a reasonable amount of storage space, I doubt that you can get a phone with less than 32G of storage that has a decent screen and Android security support. Admittedly Apple has longer security support for iPhones than Google does for Nexus/Pixel phones so it might be possible to get an older iPhone with a decent screen and hardly any space (but that’s not the point here).

If you have 32G of storage on your phone then there’s no real possibility of using up your storage space by photographing a day’s work. You could probably take an unreasonable number of photos of a week’s work as well as a few videos and not use up much of that.

The first time I needed photos recently was about 9 months ago when I was replacing some network gear (new DSL modem and switch for a client). The network sockets in the rack weren’t labelled and I found it unreasonably difficult to discover where everything was (the tangle of cables made tracking them impossible). What I should have done is to photograph the cables before I started and then I would have known where to connect everything. A 12MP camera allows zooming in on photos to get details, so a couple of quick shots of that rack would have saved me a lot of time – and in the case where everything goes as planned taking a couple of photos isn’t going to delay things.

Last night there was a power failure in a server room that hosts a couple of my machines. When power came back on the air-conditioner didn’t start up and the end result was a server with one of it’s disks totally dead (maybe due to heat, maybe power failures, maybe it just wore out). For unknown reasons BTRFS wouldn’t allow me to replace the disk in the RAID-1 array so I needed to copy the data to a new disk and create a new mirror (taking a lot of my time and also giving downtime). While I was working on this the filesystem would only mount read-only so no records of the kernel errors were stored. If I had taken photos of the screen I would have records of this which might allow me to reproduce the problem and file a bug report. Now I have no records, I can’t reproduce it, and I have a risk that next time a disk dies in a BTRFS RAID-1 I’ll have the same problem. Also presumably random people all over the world will suffer needless pain because of this while lacking the skills to file a good bug report because I didn’t make good enough records to reproduce it.

Hopefully next time I’m in a situation like this I’ll think to take some photos instead of just rebooting and wiping the evidence.

As an aside I’ve been finding my phone camera useful for zooming in on serial numbers that I can’t read otherwise. I’ve got new glasses on order that will hopefully address this, but in the mean time it’s the only way I can read the fine print. Another good use of a phone camera is recording error messages that scroll past too quickly to read and aren’t logged. Some phones support slow motion video capture (up to 120fps or more) and even for phones that don’t you can use slow play (my favourite Android video player MX Player works well at 5% normal speed) to capture most messages that are too quick to read.

January 05, 2019

Audiobooks – December 2018

The Casebook of Sherlock Holmes by Sir Arthur Conan Doyle. Read by Stephen Fry

A bit weaker than the other volumes. The author tries mixing the style in places but several stories feel like repeats. Lacking excitement 7/10

Children of Dune by Frank Herbert

A good entry for the 2nd tier of Dune Books (behind 1, even with 2 & 4). Good mix of story, philosophy and politics. Plot a little plodding though.  8/10

Modern Romance: An Investigation by Aziz Ansari

Lots of hardish data and good advice for people looking to date online. Around 3-4 years old so fairly uptodate. Funny in a lot of places & some audiobook extra bits. 8/10

The Day of Battle: The War in Sicily and Italy, 1943-1944 by Rick Atkinson

Part 2 of a trilogy. I liked this a little more than the first edition but it is still hard to follow in the audiobook format without maps etc. Individual stories lift it up. 7/10

The End of Night: Searching for Natural Darkness in an Age of Artificial Light
by Paul Bogard

A book about how darkness is being lost for most people. How we are missing out on the stars, a good sleep and much else. I really liked this, writing good and topics varied. 8/10

President Carter: The White House Years by Stuart E. Eizenstat

Eizenstat was Chief Domestic Policy Advisor to Carter & took extensive notes everywhere. The book covers just above everything, all the highs and lows. Long but worth it. 7/10

Share

Happy New Workbench

For the summer solstice I got a brand new second hand stainless steel bench top. Which by coincidence turns out to be the answer to the question "What do you get the man who has everything", apparently.

Brand new second hand stainless steel bench top

Turning to my cliché handbook, it turns out that when life gives you bench tops, you make workbenches. I needed a second standing desk for out on the deck and for hardware hacking, so under the watchful eye of a pair of local King Parrots I gathered some spare timber to cobble together a standing workbench for the bench top.

Stainless steel workbench upside down

Stainless steel workbench bare

Stainless steel workbench stage 1

There's a lot more work to be done on that desk but with stage 1 at least completed, it was time to install it on the deck, ready to hack 2019 away :-D

Stainless steel workbench in place

Yes, that is the Australian Sun reaching in, in the background to kill us all.

The astute amongst you will have also noticed the (empty) Lagavulin and Talisker bottles propping up my laptop so they're both physically and metaphorically supporting my work.

Looks like I've already put my hand up for some hacks on Termonad to be the first piece of work completed from the new deck desk, apart from this post of course :-D

So now that leaves me with an inside and an outside standing desk as well as a hammock to work from. Not sure what says about me, I'll leave that up to you.

LPCNet and Codec 2 Part 2

Since the last post I’ve trained LPCNet using quantised Codec 2 parameters. I’ve also modified the Codec 2 decoder (c2dec) to dump the received features to a disk file so I can use them to synthesise speech using LPCNet.

So now I can decode a Codec 2 bit stream using a LPCNet decoder:

Condition Sample
Original 8kHz sample Original
Codec 2 1300 encoder and Codec 2 decoder codec2_1300
Codec 2 1300 encoder and LPCNet decoder lpcnet_1300
Codec 2 2400 encoder and Codec 2 decoder codec2_2400
Codec 2 2400 encoder and LPCNet decoder lpcnet_2400
Codec 2 unquantised 6th order LSP and LPCNet decoder lpcnet_6lsps

I think it sounds pretty good, especially at 2400 bits/s. Even the lpcnet_1300 decoder sounds better than Codec 2 at 2400. LPCNet is much more natural, without the underwater sound of low rate vocoded speech.

Less Features

The Kleijn et al paper showed NN based synthesises can produce high quality speech from feature sets of legacy vocoders like Codec 2. This implies the legacy vocoders are sending a lot of extra information that is not normally used by the legacy decoder/synthesis algorithms. If high quality speech is not required, it could be argued we are sending “too much” information, and scope exists to reduce the bit rate by coarser quantisation or sending less features.

The lpcnet_6lsps sample uses just 6th order Linear Prediction, in the form of 6 Line Spectral Pairs (LSPs). A LPC order of 10 is common for 8 kHz sampled speech. There is no way 6th order LPC would work (i.e. provide intelligible speech) for any existing LPC based vocoder (CELP, MELP etc). This sample has some odd artefacts; but is intelligible, and also quite natural sounding compared to a vocoder.

Curiously, when the synthesis breaks down (e.g. “depth of the well”), it sounds to me like the pitch is halved. I can hear this on both the male and female segments.

Next Steps

I still have much to learn in this area, but the initial results are promising, especially at 2400 bit/s. The main difference between the Codec 2 feature set at 2400 and 1300 is the update rate, so it would be useful to explore those differences further with a series of tests.

This work is not far from being usable “over the air” as a FreeDV mode, and promises a significant jump in quality. Codec 2 1300 is the vocoder used for FreeDV 1600, so it may be possible to develop a drop in replacement for initial testing.

The “best” (in terms of quality at a given bit rate) encoder and feature set (especially under quantisation) is an open research question. It is amazing that it works with the Codec 2 bit stream at all – we should be able to do better with a custom encoder.

January 03, 2019

Adventures in Home Assistant Raspberry Pi GPIO

Share

Alastair D’Silva is running what looks to be a very well prepared home automation tutorial at LCA2019 based on Home Assistant. I offered to have a hack on the support for GPIO pins on OrangePi boards in Home Assistant because it sounded interesting for a vacation week. The only catch being that I’d never done anything with GPIO pins at all on either Raspberry Pi or Orange Pi.

A simple LED circuit for a Raspberry PiThe first step seemed to be to get GPIO working at all on a Raspberry Pi (which is currently supported out of the box with Home Assistant). This online tutorial has a simple example of a circuit and the associated python code to blink a LED on a Raspberry Pi, so off I went to build that circuit.

The circuit has a LED with a 330 ohm pull up resistor on GPIO pin 18 on the board. The sample python code on the page above just blinks that LED, which I used to make sure that the circuit as working as intended.

To configure the GPIO pin as a switch in Home Assistant, I added the following to configuration.yaml (noting that the empty rpi_gpio entry isn’t strictly required, but will be later):

rpi_gpio:

switch:
 - platform: rpi_gpio
   ports:
     18: LED

Which leaves me with something like this in the web UI:

Raspberry Pi LED switch in Home Assistant

It even works!

I’ve lied to you a little bit above, for which I apologise. I’ve also been working on helping Alastair with adding Orange Pi to the rpi_gpio component in Home Assistant, as the tutorial is based on Orange Pi Primes, with a custom home automation shield installed. Now that I have a sample configuration that works for Raspberry Pi and a test circuit, its time to make sure that Orange Pi works correctly too.

Home Assistant doesn’t currently have any support for Orange Pi GPIOs. The first approach I took was to forward port this ancient patch which adds Orange Pis as a new component beside Raspberry Pis. That port is available here, but in the end I decided it would be nicer to just have the existing Raspberry Pi component also support Orange Pis, instead of duplicating a whole bunch of code and adding some confusion.

(It should be noted that there are downsides to this new approach — the code is more complicated this way, and Raspberry Pi owners need to download the Orange Pi GPIO library even though they’ll never use it. That said, I see these downsides as relatively minor).

GPIO pin mapping for an Orange PiA small hitch however. Orange Pi names the GPIO ports in a quite different way from how Raspberry Pi does, and this took some time to get used to. The mapping of GPIO pins was a little hard to find, so I’ll include it here (the image to the left). A second hitch was that I needed a linux image for the board. I’ve used Armbian Stretch, as the hass.io image is quite locked down (no ssh to the base OS for example).

Based on the pin image instead of the Pin 18 from the previous example, I moved to what is labelled on the tutorial shield as “PA7”, and which is referred to in code as Pin 29.

The code for blinking is a bit different from the example linked above, so here is a tweaked version:

import OPi.GPIO as GPIO
import time


GPIO.setboard(GPIO.PRIME)
GPIO.setmode(GPIO.BOARD)
GPIO.setwarnings(False)
GPIO.setup(29, GPIO.OUT)

while True:
    GPIO.output(29, GPIO.HIGH)
    time.sleep(1)
    GPIO.output(29, GPIO.LOW)
    time.sleep(1)

Note here that we need to specify what board we’re on (in this case a Prime), and we set the mode differently than the linked example.

So now let’s be over achievers and get things working in Home Assistant too! We need a configuration.yaml which includes something like this:

rpi_gpio:
  board_family: orange_pi
  board: prime

switch:
 - platform: rpi_gpio
   ports:
     29: LED

Note the additional config in the rpi_gpio entry. We also need to run hass as root,  because OrangePi GPIO support requires access to /dev/mem for reasons I haven’t dug into just yet.

OrangePi GPIO support currently requires a patch to Home Assistant, which you can find at a github branch.

Share

December 31, 2018

Part 1 of my “Selfies on Seven Continents” series

One of the defining aspects of my 2018 was travel. There was travel for fun, travel for work, and travel for adventure. This year, I was fortunate enough to visit all seven continents, so in honour of my selfie habits over on my photo blog, here are some of my favourite selfies from this year, by continent.

South America

South America was entirely for fun and adventure. My dad and I got to tourist our way around, before we visited Antarctica.

Antarctica

Leaving from Punta Arenas, we travelled with Aurora Expeditions south along the Antarctic Peninsula, crossing the Antarctic Circle, before returning north again. Visiting Antarctica was one of the most spectacular, other-worldly, and unforgettable experiences of my life. If you’ve ever considered it, I highly recommend taking the plunge! (Both figuratively and literally. 😉)

North America

Work dominated my travel to North America, but there was some time for fun and selfies.

Asia

Thanks to a long layover, I just managed to visit Asia, dropping in for dinner with some friends in Hong Kong.

Europe

Europe was mostly work travel, which meant a distinct lack of selfies, sadly. This is clearly a shortcoming in my selfie taking habits that I need to work on next year.

Australia

Home was a place for fun, family, and a little bit of introducing my kids to the excitement of travelling! 😁

(I wish I could fit into an economy seat like a three year old.)

Africa

Rounding out the year, my first trip to Africa! Just a little adventure to Kenya, seeing a few of the sights, but leaving plenty more for my next trip. 🤫


So, that’s been my year of travel. I flew 202,717km, visiting 28 cities in 12 different countries, maintaining an average speed of 23km/h over the year.

I’m looking forward to an exciting 2019, I hope your 2019 will be excellent, too. Happy New Year, everyone! 🎉

Book Review: Fear: Trump in the White House

by Bob Woodward

Fear: Trump in the White House

If you suffer from anxiety about what's happening in the world and the competence of those in power, this is probably not the book for you. Bob Woodward's book shows in startling detail just how inadequate, incompetent and immature not only the President of the USA is but also most of the team that he had/has around him.

I didn't takes notes as I read this book and it really needs it. There's so much going on that its difficult to know where to begin. I can understand why Bob wrote this in a pretty much chronological fashion. There's one quote that stands out from Dowd, Trump's personal lawyer, the man that I came away with feeling had Trump's best interests at heart more than anyone else:

But in the man and his presidency Dowd had seen the tragic flaw. In the political back-and-forth, the evasions, the denials, the tweeting, the obscuring, crying "Fake News," the indignation, Trump had one overriding problem that Dowd knew but could not bring himself to say to the president:

"You're a fucking liar."

The primary feeling I was left with after reading this book was arrested development. This phrase best describes for me the behaviour of most of the people in involved in this tale. They just are not mature adults with a steady hand.

It's a phrase I've also come to associate with Australian politicians like Tony Abbott, Peter Dutton, Pauline Hanson, Clive Palmer... I'll stop there, it could be long list and there are not many Australian politicians that I could leave off that list sadly.

The kind of people who put their own narrow interests or the interests of their base or more importantly, donors, ahead of clear eyed long term national interests. The public rightly becomes disengaged with politics which leads to our politicians being drawn from an increasingly shallow pool of staffers and cronies, which leads to increased public disengagement, which further shrinks the pool from which our politicians are drawn.

It's a vicious loop that Wolfgang Streek expounds upon quite well in How Will Capitalism End? : Essays On A Failing System , a highly recommended read that looks deeply into and explains the socio-political problems of the modern era.

The secondary feeling I'm left with is just how broken "western" democracy truly is (I've argued at times that we do not actually live in a genuine democracy, just because we get to vote).

Clearly, in "western" nations across the world, politicians are not rising into power based on any form of merit or competency. More and more politicians in the USA, UK and Australia are coming into power despite their clearly displayed inadequacy, in spite of their arrested development, they are wielding power and being voted in by a jaded and disengaged public.

I've got no clear answer for you, only more questions.

Increasing broader community participation in politics is one, albeit unlikely solution (although the recent success of socialists in the US mid-terms show it can be done).

Another is to completely change the nature of our Democracy so that it functions as advertised but you first need to remove those in power with vested interests in the status quo.

So, this was a book review. Fantastic, insightful and frightening. Well worth a read.

December 27, 2018

Restricting outgoing HTTP traffic in a web application using a squid proxy

I recently had to fix a Server-Side Request Forgery bug in Libravatar's OpenID support. In addition to enabling authentication on internal services whenever possible, I also forced all outgoing network requests from the Django web-application to go through a restrictive egress proxy.

OpenID logins are prone to SSRF

Server-Side Request Forgeries are vulnerabilities which allow attackers to issue arbitrary GET requests on the server side. Unlike a Cross-Site Request Forgery, SSRF requests do not include user credentials (e.g. cookies). On the other hand, since these requests are done by the server, they typically originate from inside the firewall.

This allows attackers to target internal resources and issue arbitrary GET requests to them. One could use this to leak information, especially when error reports include the request payload, tamper with the state of internal services or portscan an internal network.

OpenID 1.x logins are prone to these vulnerabilities because of the way they are initiated:

  1. Users visit a site's login page.
  2. They enter their OpenID URL in a text field.
  3. The server fetches the given URL to discover the OpenID endpoints.
  4. The server redirects the user to their OpenID provider to continue the rest of the login flow.

The third step is the potentially problematic one since it requires a server-side fetch.

Filtering URLs in the application is not enough

At first, I thought I would filter out undesirable URLs inside the application:

  • hostnames like localhost, 127.0.0.1 or ::1
  • non-HTTP schemes like file or gopher
  • non-standard ports like 5432 or 11211

However this filtering is going to be very easy to bypass:

  1. Add a hostname in your DNS zone which resolves to 127.0.0.1.
  2. Setup a redirect to a blacklisted URL such as file:///etc/passwd.

Applying the filter on the original URL is clearly not enough.

Install and configure a Squid proxy

In order to fully restrict outgoing OpenID requests from the web application, I used a Squid HTTP proxy.

First, install the package:

apt install squid3

and set the following in /etc/squid3/squid.conf:

acl to_localnet dst 0.0.0.1-0.255.255.255 # RFC 1122 "this" network (LAN)
acl to_localnet dst 10.0.0.0/8            # RFC 1918 local private network (LAN)
acl to_localnet dst 100.64.0.0/10         # RFC 6598 shared address space (CGN)
acl to_localnet dst 169.254.0.0/16        # RFC 3927 link-local (directly plugged) machines
acl to_localnet dst 172.16.0.0/12         # RFC 1918 local private network (LAN)
acl to_localnet dst 192.168.0.0/16        # RFC 1918 local private network (LAN)
acl to_localnet dst fc00::/7              # RFC 4193 local private network range
acl to_localnet dst fe80::/10             # RFC 4291 link-local (directly plugged) machines

acl SSL_ports port 443
acl Safe_ports port 80
acl Safe_ports port 443
acl CONNECT method CONNECT

http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access deny manager
http_access deny to_localhost
http_access deny to_localnet
http_access allow localhost
http_access deny all

http_port 127.0.0.1:3128

Ideally, I would like to use a whitelist approach to restrict requests to a small set of valid URLs, but in the case of OpenID, the set of valid URLs is not fixed. Therefore the only workable approach is a blacklist. The above snippet whitelists port numbers (80 and 443) and blacklists requests to localhost (a built-in squid acl variable which resolves to 127.0.0.1 and ::1) as well as known local IP ranges.

Expose the proxy to Django in the WSGI configuration

In order to force all outgoing requests from Django to go through the proxy, I put the following in my WSGI application (/etc/libravatar/django.wsgi):

os.environ['ftp_proxy'] = "http://127.0.0.1:3128"
os.environ['http_proxy'] = "http://127.0.0.1:3128"
os.environ['https_proxy'] = "http://127.0.0.1:3128"

The whole thing seemed to work well in my limited testing. There is however a bug in urllib2 with proxying HTTPS URLs that include a port number, and there is an open issue in python-openid around proxies and OpenID.

December 26, 2018

Manage and tweak Fedora with Ansible (and apply Korora settings by default)

Korora Project is a Linux distro I created over 13 years ago, which (since 2010) takes Fedora and applies dozens of tweaks in an effort to make it more usable “out of the box” for every day users.

Even with one or two others helping, it has been a lot of work so I’ve taken a break from the project for the last year to focus on other things. There has been no release of Korora since and so lately I’ve been running stock Fedora 29 Workstation (GNOME) on my laptop.

I enjoy the Korora defaults though and given that my family also runs Korora, I wanted a way to be able to move them to stock Fedora while keeping the same packages as well as the look and feel.

So, I created a Korora Ansible Role (it’s also on Ansible Galaxy) to apply the same Korora tweaks for stock Fedora Workstation (GNOME) plus an example playbook which uses it.

I tried to make it flexible by using variables so that users can change default package lists and settings for each machine, as required.

Running it on your local machine is pretty trivial, there’s a shell script with a sample inventory for localhost.

$ git clone --recursive https://github.com/csmart/korora-ansible ~/korora-ansible
$ cd ~/korora-ansible
$ ./run.sh

The end result is something like this!

Fedora after running Korora Ansible Role.

Customising it for particular machines is pretty easy thanks to Ansible, especially when using a YAML inventory.

For example, the following inventory for localhost at ~/korora-ansible/inventory/hosts-custom.yml will override defaults and add a few extras.

  • Remove totem
  • Install kodi
  • Change GNOME favourites (links in the dash)
  • Install nginx webserver, enable it and open the firewall for web traffic
  • Install Adapta theme and set GTK and GNOME shell themes
  • Set fonts for the system
  • You can do more!

cd ~/korora-ansible
cat > ./inventory/hosts-custom.yml<<EOF
all:
  hosts:
    localhost:
      ansible_connection: local
      korora_packages_custom:
        remove:
          - totem
      korora_shell_favourites:
        install:
          - kodi
        apps:
          - firefox.desktop
          - vlc.desktop
          - kodi.desktop
          - shotwell.desktop
          - org.gnome.Nautilus.desktop
          - org.gnome.Software.desktop
          - org.gnome.Terminal.desktop
      korora_services_custom:
        install:
          - nginx
        enable:
          - nginx
      korora_firewall_custom:
        allow:
          - http
      korora_gtk_theme:
        name: Adapta
        install:
          - adapta-gtk-theme
      korora_shell_theme:
        name: Adapta
      korora_desktop_font:
        name: "Droid Sans Regular 11"
        install:
          - google-droid-sans-fonts
      korora_desktop_document_font:
        name: "Droid Sans Regular 11"
      korora_desktop_titlebar_font:
        name: "Droid Sans Bold 11"
      korora_terminal_font:
        name: "Droid Sans Mono Regular 11"
        install:
          - google-droid-sans-mono-fonts
  vars:
    ansible_python_interpreter: /usr/bin/python3
EOF

You would run the playbook with that custom inventory file by passing it in as the first argument.

$ ./run.sh ./inventory/hosts-custom.yml

The Consuming Fire

Share

Another fast run read from Mr Scalzi, this book is the sequel to The Collapsing Empire. I think this book is actually better than the first, which I guess is fair given the first had to set the universe up. I particularly like the twist about two thirds of the way through this one, and I think the universe has a lot of potential to be really interesting in future books. Mr Scalzi remains on my I-buy-everything-he-does list.

I wish he’d write another book in the Old Man’s War universe.






The Consuming Fire Book Cover




The Consuming Fire





John Scalzi





Tor




October 18, 2018




336

The Consuming Fire by John Scalzi is the dazzling follow-up to The Collapsing Empire - a space opera in a universe on the brink of destruction. The Interdependency, humanity's interstellar empire, is on the verge of collapse. The Flow, the extra-dimensional pathway between the stars, is disappearing, leaving planets stranded. Billions of lives will be lost - unless desperate measures can be taken. Emperox Grayland II, the leader of the Interdependency, is ready to take those measures. But it's not that easy. There are those who believe the collapse of the Flow is a myth - or an opportunity for them to ascend to power. While Grayland prepares for disaster, others prepare for civil war. A war that will take place in the halls of power, the markets of business and the altars of worship as much as between spaceships. Nothing about this power struggle will be simple or easy . . . and all of human civilization is at stake.

Share

December 24, 2018

Book Review: Frankenstein or The modern Prometheus (uncensored 1818 edition)

by Mary Wollstonecraft Shelley

Frankenstein or The modern Prometheus: The 1818 Text

This is a book I've had a steadfast disinterest in my entire life. Horror, old horror and a subject that has been covered ad nauseum in pop culture - it simply held no interest.

That was until I heard Radio National's The Science Show's special Frankenstein - It’s Alive!!! on the 1818 edition being re-released for the 200th anniversary of it's publication. That program shifted this book from "pulp fiction" to "must read" in my mind. This episode of The Science Show is worth a listen on it's own and is one of my favourite pod casts.

Yes, the first version is in 1818, and there were several more of that first edition, hardly altered at all. And then there's the edition in 1831 which she'd really seriously revised, and she'd really changed the impact of it.

... And really the changes that were made there were because some critics and some readers obviously were rather shocked by the science, the interest in the life issue. They thought the whole thing was dynamite really, they thought the public shouldn't really be given this plot and they shouldn't really be encouraged to think about it, so that each version of it is really a censored version of it after that. The novel gets rather more like the stage plays. She realised that that was the way critics anyway and the respectable classes wanted the plot to be interpreted.

This piqued my interest in the novel for the first time.

One of Mary's personal anxieties that Fiona mentions had been the loss of her first-born child, born prematurely and living for only a few days. She wrote in her journal:

Dreamt that my little baby came to life again, that it had only been cold and that we rubbed it before the fire and it lived.

and this added a personal connection for me between the author and the material. One of my favourite authors, Philip K. Dick described himself as a philosopher who used novels to explore his philosophies. I'd come to view this book in a similar light.

Thus armed, I convinced the local library of the importance of having this particular edition in their collection and waited patiently for it to arrive.

This edition proved to be a rewarding and surprising read. The antiquated story telling devices take a little getting used to, along with the spelling of many words and words that have dropped from modern usage in the 200 years since it was first published.

This original edition, as the podcast expounded upon, is nothing like what we've become accustomed to in modern pop culture but is a much more textured, thought provoking read. It may have been horror to an audience in 1818 but for me, the author is clearly exploring interesting philosophical question and that's where my attention was firmly fixed.

I highly recommend reading this original edition of this work. It's a surprisingly short, fast paced and thought provoking read.

I was mistaken to have avoided it for so long.

December 22, 2018

Linux Security Summit Europe 2018 Wrap-up

The inaugural Linux Security Summit Europe (LSS-EU) was held in October, in Edinburgh, UK.

For 2018, the LSS program committee decided to add a new event in Europe, with the aim of fostering Linux security community engagement beyond North America. There are many Linux security developers and users in Europe who may not be able to obtain funding to travel to North America for the conference each year. The lead organizer and MC for LSS EU is Elena Reshetova, of Intel Finland.

This was my first LSS as a speaker, as I’ve always been the MC for the North American events. I provided a brief overview of the Linux kernel security subsystem.

Sub-maintainers of kernel security projects presented updates on their respective areas, and there were also several referred presentations.

Slides may be found here, while videos of all talks are available via this youtube playlist.

There are photos, too!

The event overall seemed very successful, with around 150 attendees. We expect to continue now to have both NA and EU LSS events each year, although there are some scheduling challenges for 2019, with several LF events happening closely together. From 2020 on, it seems we will have 4-5 months separation between the EU and NA events, which will work much better for all involved.

 

HomeAssistant configuration

Share

I’ve recently been playing with HomeAssistant, which is quite cool. Its not perfect — for example it broke recently for me without any debug logs indicating problems because it didn’t want to terminate SSL any more, but its better than anything else I’ve seen so far.

Along the way its been super handy to be able to refer to other people’s HomeAssistant configurations to see how they got things working. So in that spirit, here’s my current configuration with all of the secrets pulled out. Its not the most complicated config, but it does do some things which took me a while to get working. Some examples:

  • The Roomba runs when no one is home, and let’s me know when its bin is full.
  • A custom component to track when events last occurred so that I can rate limit things like how often the Roomba runs when no one is home.
  • I detect when my wired doorbell goes off, and play a “ding dong” MP3 in the office yurt out the back so I know when someone is visiting.
  • …and probably other things.

I intend to write up interesting things as I think of them, but we’ll see how we go with that.

Share

December 21, 2018

Observer Pattern in Javascript Asynchronous Generators

One of the more recent additions to the Javascript specification is the asynchronous generator protocol. This is an especially useful pattern when you want to consume data off a socket/serial port etc., because it lets you do something like so:

for await (const buffer of readable) {
    await writable.write(buffer);
}

Which is pretty cool, but not a huge improvement on the `pipe` functionality already exposed in Node streams.

Where it really shines is the ability to also yield observations, allowing you to build an observer pattern:

async * download(writable) {
  await this.open();

  try {
    const readable = this.readSectors(...);
    let counter = 0;

    for await (const buffer of readable) {
      const buffer = SECTOR.parse(buffer);
      await writable.write(buffer);

      counter++;
      yield counter;
    }

  } finally {
    await this.close();
  }
}

The primary advantage is the flatness makes our exit handling very obvious. Similarly in readSectors it flattens the entry and exit of the read mode.

Those building React/Redux apps probably want to get those observations into their state. This is relatively easily achieved in redux-saga through the eventChannel API.

function asyncToChannel(generator) {
  return eventChannel(emit => {

    // Set up a promise that iterates the async iterator and emits
    // events on the channel from it.
    (async () => {
      for await (const elem of generator) {
        emit(elem);
      }

      emit(STOP_ITERATION);
    })();

    return () => {
      generator.return();
    };
  });
}

// Saga triggered on DOWNLOAD_BEGIN
function* downloadSaga(action) {
  const writable = ...
  const channel = asyncToChannel(action.data.download(writable));

  // Consume the channel into Redux actions
  while (true) {
    const progress = yield take(channel);

    if (progress === STOP_ITERATION) break;

    yield put(downloadProgress(action.data, progress));
  }

  console.debug("Download complete");
  yield put(downloadComplete(action.data));
}

Tricky to work out, but much more easily read than callback hell.

December 18, 2018

IPMI: Initiating Better Overrides

On platforms that support it Petitboot can interact with the inband IPMI interface to pull information from the BMC. One particularly useful example of this is the "Get System Boot Options" command which we use to implement boot "overrides". By setting parameter 5 of the command a user can remotely force Petitboot to boot from only one class of device or disable autoboot completely. This is great for automation or debug purposes, but since it can only specify device types like "disk" or "network" it can't be used to boot from specific devices.

Introducing..

The Boot Initiator Mailbox

Alexander Amelkin pointed out that the "Get System Boot Options" command also specifies parameter 7, "Boot Initiator Mailbox". This parameter just defines a region of vendor-defined data that can be used to influence the booting behaviour of the system. The parameter description specifies that a BMC must support at least 80 bytes of data in that mailbox so as Alex pointed out we could easily use it to set a partition UUID. But why stop there? Let's go further and use the mailbox to provide an alterate "petitboot,bootdevs=.." parameter and let a user set a full substitute boot order!

The Mailbox Format

Parameter 7 has two fields, 1 byte for the "set selector", and up to 16 bytes of "block data". The spec sets the minimum amount of data to support at 80 bytes, which means a BMC must support at least 5 of these 16-byte "blocks" which can be individually accessed via the set selector. Aside from the first 3 bytes which must be an IANA ID number, the rest of the data is defined by us.

So if we want to set an alternate Petitboot boot order such as "network, usb, disk", the format of the mailbox would be:

Block # |       Block Data              |
-----------------------------------------
0       |2|0|0|p|e|t|i|t|b|o|o|t|,|b|o|o|
1       |t|d|e|v|s|=|n|e|t|w|o|r|k| |u|s|
2       |b| |d|i|s|k| | | | | | | | | | |
3       | | | | | | | | | | | | | | | | |
4       | | | | | | | | | | | | | | | | |

Where the string is null-terminated, 2,0,0 is the IBM IANA ID, and the contents of any remaining data is not important. The ipmi-mailbox-config.py script constructs and sends the required IPMI commands from a given parameter string to make this easier, eg:

./utils/ipmi-mailbox-config.py -b bmc-ip -u user -p pass -m 5 \
        -c "petitboot,bootdevs=uuid:c6e4c4f9-a9a2-4c30-b0db-6fa00f433b3b"

Active Mailbox Override


That is basically all there is to it. Setting a boot order this way overrides the existing order from NVRAM if there is one. Parameter 7 doesn't have a 'persistent' flag so the contents need to either be manually cleared from the BMC or cleared via the "Clear" button in the System Configuration screen.

From the machines I've been able to test on at least AMI BMCs support the mailbox, and hopefully OpenBMC will be able to add it to their IPMI implementation. This is supported in Petitboot as of v1.10.0 so go ahead and try it out!

December 17, 2018

pwnedkeys: who has the keys to *your* kingdom?

pwnedkeys.com logo

I am extremely pleased to announce the public release of pwnedkeys.com – a database of compromised asymmetric encryption keys. I hope this will become the go-to resource for anyone interested in avoiding the re-use of known-insecure keys. If you have a need, or a desire, to check whether a key you’re using, or being asked to accept, is potentially in the hands of an adversary, I would encourage you to take a look.

Pwnage... EVERYWHERE

By now, most people in the IT industry are aware of the potential weaknesses of passwords, especially short or re-used passwords. Using a password which is too short (or, more technically, with “insufficient entropy”) leaves us open to brute force attacks, while re-using the same password on multiple sites invites a credential stuffing attack.

It is rare, however, that anyone thinks about the “quality” of RSA or ECC keys that we use with the same degree of caution. There are so many possible keys, all of which are “high quality” (and thus not subject to “brute force”), that we don’t imagine that anyone could ever compromise a private key except by actually taking a copy of it off our hard drives.

There is a unique risk with the use of asymmetric cryptography, though. Every time you want someone to encrypt something to you, or verify a signature you’ve created, you need to tell them your public key. While someone can’t calculate your private key from your public key, the public key does have enough information in it to be able to identify your private key, if someone ever comes across it.

So what?

smashed window

The risk here is that, in many cases, a public key truly is public. Every time your browser connects to a HTTPS-protected website, the web server sends a copy of the site’s public key (embedded in the SSL certificate). Similarly, when you connect to an SSH server, you get the server’s public key as part of the connection process. Some services provide a way for anyone to query a user’s public keys.

Once someone has your public key, it can act like an “index” into a database of private keys that they might already have. This is only a problem, of course, if someone happens to have your private key in their stash. The bad news is that there are a lot of private keys already out there, that have either been compromised by various means (accident or malice), or perhaps generated by a weak RNG.

When you’re generating keys, you usually don’t have to worry. The chances of accidentally generating a key that someone else already has is as close to zero as makes no difference. Where you need to be worried is when you’re accepting public keys from other people. Unlike a “weak” password, you can’t tell a known-compromised key just by looking at it. Even if you saw the private key, it would look just as secure as any other key. You cannot know whether a public key you’re being asked to accept is associated with a known-compromised private key. Or you couldn’t, until pwnedkeys.com came along.

The solution!

The purpose of pwnedkeys.com is to try and collect every private key that’s ever gotten “out there” into the public, and warn people off using them ever again. Don’t think that people don’t re-use these compromised keys, either. One of the “Debian weak keys” was used in an SSL certificate that was issued in 2016, some eight years after the vulnerability was made public!

My hope is that pwnedkeys.com will come to be seen as a worthwhile resource for anyone who accepts public keys, and wants to know that they’re not signing themselves up for a security breach in the future.

Lessons from Canada and France: FWD50 2018 and SIIViM

A couple of weeks ago I had a whirlwind trip to Canada, France and back again, in 6 days! I spoke at the FWD50 conference in Ottawa, Canada, which is an optimistic and inspiring event focused on the next 50 days, weeks and years of society, with a special focus on transforming our public sectors for the 21st century. Then I went to Nevers, France for SIIViM, a regional Governments event exploring digital government, open data, open source and smart cities. At both events I shared my lessons and work, as well as met with folk from the Canadian, regional French, US and Taiwanese Governments (amongst others). I also met with OECD, industry and open source folk and came back with new ideas, connections and opportunities to collaborate for our ambitious human-centred digital government transformation work in NSW. Many thanks to the FWD50 organisers and ADULLACT (a French Free Software non-profit organisation) for bringing me over and providing the opportunity to learn and share my experiences.

My contributions

I gave several speeches in my personal professional capacity (meaning I was not formally representing any of my employers past or present) which may be of interest:

Insights from Canada

In between the three presentations I gave, I got to catch up with a range of wonderful people to talk about transforming and improving public sectors.

I spoke to the Canadian School of Public Service:

  • The Canadian Government is creating a Digital Academy to develop better digital acumen across the public sector, better digital leaders, and a community that is ongoing, engaged and mutually supportive. Check out this video on the value of the Canadian Digital Academy.
  • There was strong interest in innovation of public management, including AI in regulation making.
  • They are building a modern policy capability, a tiger team approach, to support policy modernisation and transformation across government.

I visited the Canadian Digital Service and had a great chat with some of the people there, as well as a tour of their new office. It was great to see how much has been achieved in the last year and to exchange stories and lessons on trying to drive change in government. A big thank you to Sean Boot who coordinated the visit and showed me around, great to catch up Sean!

  • We spoke about the challenges faced when under pressure to deliver services whilst trying to transform government, and the need to balance foundational work (like reusable components, standards, capability uplift, modular architecture) with service redesign or improvements.
  • We spoke about legislation as code and the New Zealand entitlements engine we developed as an example of reusable rules for more integrated service delivery. I recommended the Canadian dev team chat to the NZ dev team about OpenFisca Aotearoa.
  • We spoke about emerging tech and how we can prepare public sectors for change, as well as the challenges and benefits of product vs service vs system design.
  • I heard about several great Canadian projects including one helping veterans get access to services and entitlements.
  • We also talked about GCcollab, the open source all of government collaboration suite which is being heavily used across agencies, particularly by policy folk.

I also got to catch up with some folk from the Canadian Treasury Board Secretariat to talk about open government, digital transformation, funding approaches, policy innovation and more. Thanks very much Thom Kearney who is always doing interesting things and connecting people to do interesting things :)

I managed to also get a little time to chat to Michael Karlin, who is driving the ethical AI and algorithmic transparency work in the Canadian Government. It was great to hear where the work is up to and find opportunities to collaborate.

I also met a lot of non-Canadians at the conference, a few takeaways were:

  • Audrey Tang, Digital Minister for Taiwan – Audrey was, as usual, wonderfully inspiring. Her talk pushed the audience to think much bigger and bolder about radical transparency, citizen empowerment and an engaged State. Audrey shared some great pamphlets with me in 8 languages that showed how open government in Taiwan works, which includes issues raised by citizens, prioritised by government, consulted on openly, and fixed collaboratively with citizens. Audrey also shared how they do public consultations in the local language of an area and then transcribe to Mandarin for accessibility. I love this idea and want to consider how we could do multi-lingual consultations better in Australia.
  • I caught up with the always extraordinary Audrey Lobo-Pulo who is a brilliant data scientist and advocate for Opening Government Models. Audrey introduced me to Natalie Evans Harris who had worked in the office of the US Chief Technology Officer and had a lot of expertise around digitising public services.

Insights from France

The SIIViM conference itself was fascinating. A lot of focus on open data, “Smart Cities”, IoT, Virtual Reality, and autonomous cars.

Whilst there we got into a discussion about digital asset valuation and how software/data may be measured as an asset,but is usually not valued as a public asset. Often when data is valued as an asset it quickly leads to cost recovery activities or asset depreciation which can get tricky when we are talking about foundational datasets that could be available as digital public infrastructure for digital society.

When in Paris, I was delighted to meet up with Alex Roberts from the OECD (formerly of DesignGov and Public Sector Innovation Network fame) and Jamie, to talk about innovation in government. We talked about the new OECD Declaration of Public Innovation Alex has developed which beautifully frames the four different innovation types as being across two spectrums of certainty/uncertainty, and directed/undirected, which nicely frames the different forms of innovation efforts I’ve seen over the years. Great work Alex! There is also a report on innovation in the innovation in the Canadian Government worth reading. Perhaps OECD could come to NSW Government next? ;)

I also met with Roberto Di Cosmo who founded the Software Heritage initiative, which is like a super archive for software repositories that stores the code in a uniform data model for the purpose of analysis, science and posterity. Roberto has been involved in the French Free and Open Source Software community for a long time and he told me about the French Government investment in Open Source with 200m euros invested in 10 years (40% public money and 60% private investment). Fascinating and it explains why so many great French Government technologies are Open Source!

I got to catch up with the excellent Matti Schneider, who worked with my team in New Zealand for a few weeks on OpenFisca. I highly recommend Matti’s talk about the French State Incubator (a public sector innovation lab) or another talk on turning legislation into code from New Zealand. Matti kindly gave me a short tour of central Paris from a historical context, and I got to hear about the three Parisian revolutions and see significant landmarks along the way. Fascinating, and as always, there are lessons relevant to the present moment.

To wrap it all up, Patrick introduced me to Mark from the US National Archives who shared some thoughts about https://www.lockss.org/ and the importance of ensuring validity of historic digital archives. I also met Margaret from ICANN who talked about the personal empowerment of staff to make good decisions and to engage in stopping things that are wrong, unfair or inconsistent with the mission. She encouraged me to be humble about evidence and realistic about change being inevitable.

Useful links:

December 14, 2018

ARM NEON Optimisation

I’ve been trying to optimise NEON DSP code on a Raspberry Pi. Using the intrinsics I managed to get a speed increase of about 3 times over vanilla C with just a few hours work. However the results are still significantly slower than the theoretical speed of the machine, which is 4 multiply-acculumates (8 float operations) per cycle. On a 1.2 GHz core that’s 9.6 GFLOPs.

Since then I’ve been looking at ARM manuals, Googling, and trying various ad-hoc ideas. There is a lack of working, fully optimised code examples, and I can’t find any data on cycle times and latency information for the Cortex A53 device used for the Rpi. The number of ARM devices and families is bewildering, and trying to find information in a series of thousand-page manuals daunting.

Fortunately the same NEON assembler seems to work (i.e. it assembles cleanly and you get the right results) on many ARM machines. It’s just unclear how fast it runs and why.

To get a handle on the problem I wrote a series of simple floating point dot product programs, and attempted to optimise them. Each program runs through a total of 1E9 dot product points, using an inner and outer loop. I made the inner loop pretty small (1000 floats) to try to avoid cache miss issues. Here are the results, using cycle counts measured with “perf”:

Program Test Theory cycles/loop Measured cycles/loop GFLOPS
dot1 Dot product no memory reads 1 4 1.2*8/4 = 1.2
dot2 Dot product no memory reads unrolled 1 1 1.2*8/1 = 9.6
dot3 Dot product with memory reads 3 9.6 1.2*8/9.6 = 1
dot4 Dot product with memory reads assembler 3 6.1 1.2*8/6.1 = 1.6
dotne10 Dot product with memory reads Ne10 3 11 1.2*8/11 = 0.87

Cycles/loop is how many cycles are executed for one iteration of the inner loop. The last column assumes a 1.2 GHz clock, and 8 floating point ops for every NEON vector multiply-accumulate (vmul.f32) instruction (a multiply, an add, 4 floats per vector processed in parallel).

The only real success I had was dot2, but that’s an unrealistic example as it doesn’t read memory in the inner loop. I guessed that the latencies in the NEON pipeline meant an unrolled loop would work better.

Assuming (as I can’t find any data on instruction timing) two cycles for the memory reads, and one for the multiply-accumulate, I was hoping at 3 cycles for dot3 and dot4. Maybe even better if there is some dual issue magic going on. Best I can do is 6 cycles.

I’d rather have enough information to “engineer” the system than have to rely on guesses. I’ve worked on many similar DSP optimisation projects in the past which have had data sheets and worked examples as a starting point.

Here is the neon-dot source code on GitLab. If you can make the code run faster – please send me a patch! The output looks something like:

$ make test
sum: 4e+09 FLOPS: 8e+09
sum: 4e+09 FLOPS: 8e+09
sum: 4.03116e+09 target cycles: 1e+09 FLOPS: 8e+09
sum: 4.03116e+09 target cycles: 1e+09 FLOPS: 8e+09
FLOPS: 4e+09
grep cycles dot_log.txt
     4,002,420,630      cycles:u    
     1,000,606,020      cycles:u    
     9,150,727,368      cycles:u
     6,361,410,330      cycles:u
    11,047,080,010      cycles:u

The dotne10 program requires the Ne10 library. There’s a bit of floating point round off in some of the program outputs (adding 1.0 to a big number), that’s not really a bug.

Some resources I did find useful:

  1. tterribe NEON tutorial. I’m not sure if the A53 has the same cycle timings as the Cortex-A discussed in this document.
  2. ARM docs, I looked at D0487 ARMv8 Arch Ref Manual, DDI500 Cortex A53 TRM, DDI502 Cortex A53 FPU TRM, which both reference the DEN013 ARM Cortex-A Series Programmer’s Guide. Couldn’t find any instruction cycle timing in any of them, but section 20.2 of DEN013 had some general tips.
  3. Linux perf was useful for cycle counts, and in record/report mode may help visualise pipeline stalls (but I’m unclear if that’s what I’m seeing due to my limited understanding).

Immersion: A FOSS VR MMORPG MOOC

Rationale

"Immersion" is an innovative digital product that combines multiple established existing technologies and processes to provide a new product that fills gaps in the higher education and gaming market.

The existing technologies are Massive Multi-player Online Roleplaying Games (MMORPGs), Virtual Reality (VR), Massive Open Online Courses (MOOC). Using Free and Open Source Software (FOSS) for development the product is a highly interactive VR-enabled MMORPG that uses authentic historical and mythological settings for an educational purpose, but through the immersive and escapist computer gaming.

"Immersion" seeks to address several major issues relating to both Massive Open Online Courses (MOOCs) and popular Massive Multi-player Online Roleplaying Games (MMORPGs) in both a technological and business sense. MOOCs have a high drop-out rate due to their passive or non-engaging interactions. Likewise, in an increasingly globalised world where cross-cultural understanding is considered of increasing importance, enrolments in history, archaeology, and cultural anthropology are declining (Bookings, 2018; Jaschik, 2016) . Both MOOCs and these liberal arts courses lack an immersive, interactive, and enjoyable quality (Tauber, 2013).

On the technical side MMORPGs are extremely popular with millions of players and provide a high degree of immersive enjoyment as well as significant commercial success but as usually lacking educational value or multicultural understanding. In comparison the various VR systems over-promise in terms of environment, but are typically limited to single player and prone to a loss of verisimilitude, and lack sophisticated game-play (Riveria, 2018).

In most cases, FOSS development tends to be backed more by shared or bespoke business or academic needs, and thus open-source game engines for MMORPGs and VRs have seen only modest development. Attempts to develop monopolistic profits through proprietary software actually limits development as it requires replicated effort and in the case of software patents, prohibits development (Klemens, 2005), and is counter to the marginal cost of production. The incentive and financing for "Immersion" is via hardware and educational qualifications.

(313 words, excluding references)

Description


Why just read about the French revolution when you can actually be there, and be Danton?

Danton file still

The rationale incorporates the idea of the MOOC/MMORPG/VR educational game, produced and delivered via open-source software. Ideas, however, are not innovations, even if they are a necessary first step towards an innovation. An innovation requires the idea, but also the implementation and the addition of value (Kastelle and Steen, 2011), where both engineering and management come into the equation.

MOOC example

As with most innovations precursor technologies already exist. Open-source virtual reality software and hardware have been developed such as OSVR's Hacker Development Kit (Robertson, 2018), as do open-source MMORPGs (Olivetti, 2017). As can be expected there are numerous history and mythology MOOCs. All the components, in varying forms, exist to provide "gamified" historical and mythic MOOCs as a VR MMORPG. The project and product objective is bringing these together. This, of course, suggests a number of challenges and barriers to entry, such as capital and expertise requirements and artificial scarcity in information goods.

The use of FOSS is relatively weak in game development, however quite strong in educational content and technology (Weller, 2018). As an educational game the use of free software provides a leveraged foundation for both community and academic development. This contributes to reduced start-up costs and funding opportunities. The incorporation of the development (software engineering, client-server architecture, video game development courses) and the content (history, anthropology, literature, religion courses) into the formal curriculum provides both encourages enrolments and financing.

"Immersion" would require a sponsoring university who (a) already has established courses in history and mythology as well, (b) video game and video hardware development, and are prepared to engage in such a multidisciplinary project, (c) a desire to promote assessment criteria based on real-world practice.

World of Warcraft VR

From the historic-mythic disciplines it would require experts to identify key historical and mythic narratives and generate an authentic experience of the lifeworld of characters, using insights from social phenomenology and historical hermeneutics. From the programming angle it would require expertise in vector graphics, especially for the VR elements, and client-server architectures such as used in MMORPGs. From a game-play experience, because reality doesn't follow narrative logic, a certain degree of incremental challenges and importance will be required, which would also match with the graduation of student knowledge and understanding of the setting (e.g., in an ancient Hellenic immersion, such as The Trial and Death of Socrates, some understanding of Ancient Greek would be advantageous).

Layered Programming

One advantage of adopting a FOSS model is that the product can be distributed in layered stage models according to capability or desire, rather than a large binary object for a specific type of game (Richards, 2015). This better fits actually programming implementation and allows for structured elaboration as well as satisfying the often-overlooked matter of accessibility. At the very basic system, it will be able to operate as a single-person offline text-based story. This can be elaborated to include either the graphic elements or the online multiplayer elements (multi-user shared environment, MUSH), then both, and then with VR levels. At each layer of the programming model the incorporation of educational gaming of this sort within the RPG (roleplaying game) milieu from a lived experience model is fairly unique (Hafer, 2017). The game engine can be ported to incorporate the magic turning points in history that are desired, or the great mythic stories (e.g., the French and American revolutions, the founding of the Pharaonic Kingdom of Egypt by Menes, the story of the Christian Gospels, the enlightenment of Siddhārtha Gautama, etc).

(603 words, excluding references)

Credits

Gérard Depardieu image still from the film Danton (1983) film, directed by Andrzej Wajda, distributed by Gaumont.

World of Warcraft image from the article You Can Now Download A World of Warcraft (WoW) VR Mod, 2017.

MOOC example by Athanassios Jimoyiannis

Layered software image from "Software Architecture Patterns" by Mark Richards (2015).

Video of the reconstruction of Babylon made for the Mesopotamia exhibition of the Royal Ontario Museum and British Museum at ROM, Toronto. Video introduction and text made with Blender. Soundtrack, "Rhythm of the Heat" by Peter Gabriel, 1982. Composition made with OpenShot Video Editor.

References

Brookins, Julia. (2018). Enrollment Declines Continue: AHA Survey Again Shows Fewer Undergraduates in History Courses. Perspectives on History
https://www.historians.org/publications-and-directories/perspectives-on-...

Catropa, Dayna. (2013). Big (MOOC) Data. Inside Higher Ed
https://www.insidehighered.com/blogs/stratedgy/big-mooc-data

Hafer, T. J.. (2017). The most historically accurate PC games. PC Gamer
https://www.pcgamer.com/historical-games/

Jaschik, Scott. (2016). The Shrinking Humanities Major. Inside HigherEd
https://www.insidehighered.com/news/2016/03/14/study-shows-87-decline-hu...

Kastelle, Tim., Steen, John. (2011). Ideas are not innovations. Prometheus: Critical Studies in Innovation. 29:2, 199-205

Koutsodimou, Konstantina., Jimoyiannis, Athanassios. (2015). MOOCs for teacher professional development: investigating views and perceptions of the participants. 8th International Conference of Education, Research and Innovation - ICERI2015.

Klemens, Ben. (2005). Software Patents Don't Compute: No clear boundary between math and software exists. IEEE Spectrum. Institute of Electrical and Electronics.
https://spectrum.ieee.org/computing/software/software-patents-dont-compute/

Olivetti, Justin. (2017). The Game Archaeologist: Where are all of the open-source MMOs?. Massively Overpowered
https://massivelyop.com/2017/07/08/the-game-archaeologist-where-are-all-...

Richards, Mark. (2015). Software Architecture Patterns. O'Reilly Media.

Riveria, Joshua. (2018). Is VR Gaming Over Before It Even Started?. GQ
https://www.gq.com/story/is-vr-gaming-over-before-it-even-started

Robertson, Adi. (2018). OSVR's virtual reality headset is for making the weird VR of the future. The Verge
https://www.theverge.com/2016/6/16/11952016/osvr-hdk-2-vr-headset-hands-...

Tauber, Todd. (2013). The dirty little secret of online learning: Students are bored and dropping out. Quartz
https://qz.com/65408/the-dirty-little-secret-of-online-learning-students...

Weller, Martin., Jordan, Katy., DeVries, Irwin., Rolfe, Viv. (2018). Mapping the open education landscape: citation network analysis of historical open and distance education research. Open Praxis. 10 (2). 109-126. International Council for Open and Distance Education.

December 12, 2018

Python 2.7.x with GCC 8.x and EasyBuild

An attempted build of Python-2.7.13 with GCC-8.2.0 led to an unexpected error where the build failed to generation of POSIX vars. This is kind of important and unsurprisingly, others on in the Python community have noticed it as well both this year, and in a directly related matter from late 2016, with a recommended patchfile provided on the Python-Dev mailing list.

My own contribution to this process is to incorporate the hard and insightful work done by those linked above, with developing an EasyBuild recipe for Python 2.7.x with GCC 8.x. At the moment, EasyBuild does not offer support for GCC8.x, however for those who have their own similar installation processes to the default, the following build script and pathfile might be useful.

Python-2.7.13-GCC-8.2.0-bare.eb


# Built with EasyBuild version 3.7.1 on 2018-12-12_14-27-32
name = 'Python'
version = '2.7.13'
versionsuffix = '-bare'
homepage = 'http://python.org/'
description = "Python is a programming language that lets you work more quickly and integrate your systems more effectively."
toolchain = {'name': 'GCC', 'version': '8.2.0'}
source_urls = ['http://www.python.org/ftp/%(namelower)s/%(version)s/']
sources = [SOURCE_TGZ]
# python needs bzip2 to build the bz2 package
dependencies = [
('bzip2', '1.0.6'),
('zlib', '1.2.11'),
('libreadline', '7.0'),
('ncurses', '6.1'),
('SQLite', '3.25.3'),
('Tk', '8.6.9'),
# ('OpenSSL', '1.0.1k'), # OS dependency should be preferred if the os version is more recent then this version, it's
# nice to have an up to date openssl for security reasons
]
patches = [
'GCC8-Python2.7.x.patch'
]
osdependencies = [('openssl-devel', 'libssl-dev')]
# bare installation: only pip & setuptools included.
exts_list = [
('setuptools', '36.7.2', {
'source_urls': ['https://pypi.python.org/packages/source/s/setuptools/'],
}),
('pip', '9.0.1', {
'source_urls': ['https://pypi.python.org/packages/source/p/pip/'],
})
]
moduleclass = 'lang'

GCC8-Python2.7.x.patch

The patch file is the same as provided in the Python-Dev link, above.


diff Include/objimpl.h Include/objimpl.h
index 55e83eced6..aa906144dc 100644
--- a/Include/objimpl.h
+++ b/Include/objimpl.h
@@ -248,6 +248,18 @@ PyAPI_FUNC(PyVarObject *) _PyObject_GC_Resize(PyVarObject *
, Py_ssize_t);
/* for source compatibility with 2.2 */
#define _PyObject_GC_Del PyObject_GC_Del
+/* Former over-aligned definition of PyGC_Head, used to compute the
+ size of the padding for the new version below. */
+union _gc_head;
+union _gc_head_old {
+ struct {
+ union _gc_head *gc_next;
+ union _gc_head *gc_prev;
+ Py_ssize_t gc_refs;
+ } gc;
+ long double dummy;
+};
+
+ struct {
+ union _gc_head *gc_next;
+ union _gc_head *gc_prev;
+ Py_ssize_t gc_refs;
+ } gc;
+ long double dummy;
+};
+
/* GC information is stored BEFORE the object structure. */
typedef union _gc_head {
struct {
@@ -255,7 +267,8 @@ typedef union _gc_head {
union _gc_head *gc_prev;
Py_ssize_t gc_refs;
} gc;
- long double dummy; /* force worst-case alignment */
+ double dummy; /* force worst-case alignment */
+ char dummy_padding[sizeof(union _gc_head_old)];
} PyGC_Head;
extern PyGC_Head *_PyGC_generation0;

GCC-8.2.0.eb


# Built with EasyBuild version 3.6.2 on 2018-10-24_10-03-02
easyblock = 'Bundle'
name = 'GCC'
version = '8.2.0'
binutilsver = '2.30'
#versionsuffix = '-%s' % binutilsver
homepage = 'http://gcc.gnu.org/'
description = """The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Java, and Ada, as well as libraries for these languages (libstdc++, libgcj,...)."""
toolchain = {'name': 'dummy', 'version': ''}
dependencies = [
('GCCcore', version),
# binutils built on top of GCCcore, which was built on top of (dummy-built) binutils
('binutils', binutilsver, '', ('GCCcore', version)),
]
altroot = 'GCCcore'
altversion = 'GCCcore'
# this bundle serves as a compiler-only toolchain, so it should be marked as compiler (important for HMNS)
moduleclass = 'compiler'

Falsehoods Programmers Believe About Pagination

The world needs it, so I may as well write it.

  • The number of items on a page is fixed for all time.
  • The number of items on a page is fixed for one user.
  • The number of items on a page is fixed for one result set.
  • The pages are only browsed in one direction.
  • No item will be added to the result set during retrieval.
  • No item will be removed from the result set during retrieval.
  • Item sort order is stable.
  • Only one page of results will be retrieved at one time.
  • Pages will be retrieved in order.
  • Pages will be retrieved in a timely manner.
  • No problem will result from two different users seeing different pagination of the same items at about the same time. (From @ronburk)

December 10, 2018

UNDP 2018: Evidence based vs experimentation based policy

Recently I have a remote talk to a UNDP event about Evidence based versus experimentation based policy. Below are the notes.
  • We invented all of this, and we can reinvent it. We can co-create a better future for everyone, if we choose. But if we settle for making things just a bit better, a bit more sustainable, a bit anything, then we will fundamentally fail the world because change and complexity is growing exponentially, and we need an exponential response to keep up.
  • There is a dramatic shift in paradigm from control to enablement, from being a king in a castle to a node in a network, which assumes a more collaborative approach to governance.
  • Evidence based approaches are great to identify issues, but we need experimentation based approaches, equitably co-designed with communities, so create sustainable and effective solutions. Evidence based solutions often are normative rather than transformative.
  • We need both evidence and experimentation based policy making, combined with system thinking and public engagement to make a real difference.
  • Digital transformation is often mistaken for meaning the digitisation of or service design led improvement of services, but digital transformation means creating institutions that are fit for purpose for the 21st century, from policy, regulation, services, public engagement, a full rethink and redesign of our social, economic and political systems.
  • History in implementation, and we realised that it was the disconnect between policy and implementation, the idea of policy as separate to implementation is undermining the possibility of meeting the policy intent through implementation.
  • Measurement ends up being limited to the context of function rather than outcomes.
  • Urgently need to reform how we do policy, regulation and legislation, to embrace an outcomes based approach, to bring design thinking and system design into the process from the start, from policy development in the first instance.
  • Working in the open is essential to getting both the demand and supply of evidence based policy, and working openly also means engaging in the shared design of policy and services with the communities we serve, to draw on the experience, expertise and values of the communities.
  • Public Values Management
  • Evidence based AND experimentation based policy.
  • Examples:
    • Service Innovation Lab – NZ
      • Service design and delivery – rapid prototyping is trusted for service design
      • Applying design thinking to regulation and policy
      • Legislation as code – rapid testing of policy and legislation, Holidays Act, it is critical if we want to have a chance of ensuring traceable, accountable and trusted decision making by public sectors as we see more automated decision making with the adoption of AI and ML grow.
      • Simultaneous legislation and implementation, to ensure implementation has a chance of meeting the original policy intent.
    • Taiwan – Uber case study, civic deliberation
    • Their Future Matters – data driven insights and outcomes mapping and then co-design of solutions, co-design with Aboriginal NGOs
    • 50 year optimistic future – to collaboratively design what a contextual, cultural and values driven “good” looks like for a society, so we can reverse engineer what we need to put in place to get us there.
  • Final point – if we want people to trust our policies, services and legislation, we need to do open government data, models, traceable and accountable decision making, and representative and transparent public participation in policy.
  • Links:

 

Chaos Monkeys

Share

A very well written tale of a Wall Street quant who left during the GFC to adventure in startup land and ended up at Facebook attempting to solve their monetization problems for an indifferent employer. Martinez must have been stomping around Mountain View because his description of the environment and what its like to work inside a Silicon Valley company ring very true to me.

A good read.






Chaos Monkeys Book Cover




Chaos Monkeys





Antonio Garcia Martinez





Business & Economics




Harper Paperbacks




July 24, 2018




320



The instant New York Times bestseller, now available in paperback and featuring a new afterword from the author--the insider's guide to the Facebook/Cambridge Analytica scandal, the inner workings of the tech world, and who really runs Silicon Valley “Incisive.... The most fun business book I have read this year.... Clearly there will be people who hate this book — which is probably one of the things that makes it such a great read.” — Andrew Ross Sorkin, New York Times Imagine a chimpanzee rampaging through a datacenter powering everything from Google to Facebook. Infrastructure engineers use a software version of this “chaos monkey” to test online services’ robustness—their ability to survive random failure and correct mistakes before they actually occur. Tech entrepreneurs are society’s chaos monkeys. One of Silicon Valley’s most audacious chaos monkeys is Antonio García Martínez. After stints on Wall Street and as CEO of his own startup, García Martínez joined Facebook’s nascent advertising team. Forced out in the wake of an internal product war over the future of the company’s monetization strategy, García Martínez eventually landed at rival Twitter. In Chaos Monkeys, this gleeful contrarian unravels the chaotic evolution of social media and online marketing and reveals how it is invading our lives and shaping our future.

Share

December 09, 2018

Donations 2018

Each year I do the majority of my Charity donations in early December (just after my birthday) spread over a few days (so as not to get my credit card suspended).

I also blog about it to hopefully inspire others. See: 2017, 2016, 2015

All amounts this year are in $US

My main donations was to Givewell (to allocate to projects as they prioritize). I’m happy that they are are making efficient uses of donations.

I gave some money to the Software Conservancy to allocate across the projects (mostly open source software) they support and also to Mozilla to support the Firefox browser (which I use) and other projects.

Next were three advocacy and infrastructure projects.

and finally I gave some money to a couple of outlets whose content I consume. Signum University produce various education material around science-fiction, fantasy and medieval literature. In my case I’m following their lectures on Youtube about the Lord of the Rings. The West Wing Weekly is a podcast doing a episode-by-episode review of the TV series The West Wing.

 

Share

December 08, 2018

School-wide Understanding Our World® implementations

Are you considering implementing our integrated HASS+Science program, but getting a tad confused by the pricing?  Our subscription model didn’t not provide a So nowstraightforward calculation for a whole school or year-level.  However, it generally works out to $4.40 (inc.GST) per student.  So now we’re providing this as an option directly: implement our integrated HASS+Science program […]

December 06, 2018

Codec 2 and TWELP

DSP Innovations have recently published comparisons of Codec 2 with their TWELP codec at 2400 and 600 bit/s.

Along with some spirited rhetoric, they have published some TWELP 600 samples (including source). The comparison, especially in the 600 bit/s range, is very useful to my work.

I’ve extracted a random subset of the 600 bit/s a_eng.wav samples, broken up into a small chunks to make them easier to compare. Have a listen, and see what you think:

Sample Source MELP 600e Codec 2 700c TWELP 600
1 Listen Listen Listen Listen
2 Listen Listen Listen Listen
3 Listen Listen Listen Listen
4 Listen Listen Listen Listen
5 Listen Listen Listen Listen
6 Listen Listen Listen Listen

The samples do have quite a bit of background noise. The usual approach for noisy samples is to use a noise suppression algorithm first, e.g. we use the Speex noise suppression in FreeDV. However it’s also a test of the codecs robustness to background noise, so I didn’t perform any noise suppression for the Codec 2 samples.

Comparison

I am broadly in agreement with their results. Using the samples provided, the TWELP codec appears to be comparable to MELP 2400, with Codec 2 2400 a little behind both. This is consistent with other Codec 2 versus MELP/AMBE comparisons at 2400 bits/s. That’s not a rate I have been focussing on, most of my work has been directed at lower rates required for HF Digital voice.

I think – for these samples – their 600 bit/s codec also does better than Codec 2 700C, but not by a huge margin. Their results support our previous findings that Codec 2 is as good as (or even a little better) than MELP 600e. It does depend on the samples used, as I will explain below.

DSP Innovations have done some fine work in handling non-speech signals, a common weakness with speech codecs in this range.

Technology Claims

As to claims of superior technology, and “30 year old technology”:

  1. MELP 2400 was developed in the 1990’s, and DSP Innovations own results show similar speech quality, especially at 2400 bits/s.
  2. AMBE is in widespread use, and uses a very similar harmonic sinusoidal model to Codec 2.
  3. The fundamental work on speech compression was done in the 1970s and 80’s, and much of what we use today (e.g. in your mobile phone) is based on incremental advances on that.
  4. As any reader of this blog will know, Codec 2 has been under continual development for the past decade. I haven’t finished, still plenty of “DSP Innovation” to come!

While a fine piece of engineering, TWELP isn’t in a class of it’s own – it’s still a communications quality speech codec in the MELP/AMBE/Codec 2 quality range. They have not released any details of their algorithms, so they cannot be evaluated objectively by peer review.

PESQ and Perceptual evaluation of speech quality

DSP Innovations makes extensive use of the PESQ measure, for both this study and for comparisons to other competitors.

Speech quality is notoriously hard to estimate. The best way is through controlled subjective testing but this is expensive and time consuming. A utility to accurately estimate fine differences in speech quality would be a wonderful research tool. However in my experience (and the speech coding R&D community in general), such a tool does not exist.

The problem is even worse for speech codecs beneath 4 kbit/s, as they distort the signal so significantly.

The P.862 standard acknowledges these limits, and explicitly states in Table 3 “Factors, technologies and applications for which PESQ has not currently been validated … CELP and hybrid codecs < 4 kbit/s". The standard they are quoting does not support use of PESQ for their tests.

PESQ is designed for phone networks, and much higher bit rate codecs. In section 2 of the standard they present best-case correlation results of +/- 0.5 MOS points (note on a scale of 1-5, this is +/- 10% error). That’s when it is used for speech codecs > 4 kbit/s that it is designed for.

So DSP Innovations statements like “Superiority of the TWELP 2400 and MELPe 2400 over CODEC2 2400 is on average 0.443 and 0.324 PESQ appropriately” are unlikely to be statistically valid.

The PESQ algorithm (Figure 4a of the standard) throws away all phase information, keeping just the FFT power spectrum. This means it cannot evaluate aspects of the speech signal that are very important for speech quality. For example PESQ could not tell the difference between voiced speech (like a vowel) an unvoiced (like a consonant) with the same power spectrum.

DSP Innovations haven’t shown any error bars or standard deviations on their results. Even the best subjective tests will have error bars wider than the PESQ results DSP Innovations are claiming as significant.

I do sympathise with them. This isn’t a huge market, they are a small company, and subjective testing is expensive. Numbers look good on a commercial web site from a marketing sense. However I suggest you disregard the PESQ numbers.

Speech Samples

Speech codecs tend to work well with some samples and fall over with others. It is natural to present the best examples of your product. DSP Innovations chose what speech material they would present in their evaluation of Codec 2. I have asked them to give me the same courtesy and code speech samples of my choice using TWELP. I have received no response to my request.

Support and Porting

An open source codec can be ported to another machine in seconds (rather than months that DSP Innovations quote) with a cross compiler. At no cost.

Having the source code makes minor problems easy to fix yourself. We have a community that can answer many questions. For tougher issues; well I’m available for paid support – just like DSP Innovations.

Also …. well open source is just plain cool. As a reminder, here are the reasons I started Codec 2, nearly 10 years ago.

To quote myself:

A free codec helps a large amount of people and promotes development and innovation. A closed codec helps a small number people make money at the expense of stifled business and technical development for the majority.

Reading Further

Open Source Low Rate Speech Codec Part 1, the post that started Codec 2.
P.862 PESQ standard.
CODEC2 vs TWELP on 2400 bps. DSP Innovations evaluate Codec 2, MELP, and TWELP at 2400 bits/s.
CODEC2 vs TWELP on 700 bps. DSP Innovations evaluate Codec 2, MELP, and TWELP at 600 (ish) bits/s.
AMBE+2 and MELPe 600 Compared to Codec 2. An earlier comparison, using samples from DSP Innovations.

Entrepreneurs’ Mental Health and Well-being Survey

Jamie Pride has partnered with Swinburne University and Dr Bronwyn Eager to conduct the largest mental health and well-being survey of Australian entrepreneurs and founders. This survey will take approx 5 minutes to complete. Can you also please spread the word and share this via your networks!

Getting current and relevant Australian data is extremely important! The findings of this study will contribute to the literature on mental health and well-being in entrepreneurs, and that this will potentially lead to future improvements in the prevention and treatment of psychological distress.

Jamie is extremely passionate about this cause! Your help is greatly appreciated.

Configuring Solarized Colours in Termonad

I'm currently using Termonad as my terminal of choice. What is Termonad?

Termonad is a terminal emulator configurable in Haskell. It is extremely customizable and provides hooks to modify the default behavior. It can be thought of as the "XMonad" of terminal emulators.

As a long time Xmonad user, this is a rather appealing description as well as a fairly lofty and worthy goal. It's also a niche not currently filled, one that I'm pretty happy to see being filled. By default, Termonad looks like this.

Default Termonad palette

Which is pretty standard as far as terminal defaults go but a long way from the eye-soothing grace of the Solarized palette which I essentially won't work without.

The Solarized dark output looks like this (Haskell in vim):

Solarized (dark) Termonad palette

This is the function controlling the dark palette:

    solarizedDark1 :: Vec N8 (Colour Double)
    solarizedDark1 =
         sRGB24   0  43  54 -- base03, background
      :* sRGB24 220  50  47 -- red
      :* sRGB24 133 153   0 -- green
      :* sRGB24 181 137   0 -- yellow
      :* sRGB24  38 139 210 -- blue
      :* sRGB24 211  54 130 -- magenta
      :* sRGB24  42 161 152 -- cyan
      :* sRGB24 238 232 213 -- base2
      :* EmptyVec

    solarizedDark2 :: Vec N8 (Colour Double)
    solarizedDark2 =
         sRGB24   7  54  66 -- base02, background highlights
      :* sRGB24 203  75  22 -- orange
      :* sRGB24  88 110 117 -- base01, comments / secondary text
      :* sRGB24 131 148 150 -- base0, body text / default code / primary content
      :* sRGB24 147 161 161 -- base1, optional emphasised content
      :* sRGB24 108 113 196 -- violet
      :* sRGB24 101 123 131 -- base00
      :* sRGB24 253 246 227 -- base3
      :* EmptyVec

The Solarized light output looks like this (Haskell in vim):

Solarized (light) Termonad palette

This is the function controlling the light palette:

    solarizedLight1 :: Vec N8 (Colour Double)
    solarizedLight1 =
         sRGB24 238 232 213 -- base2, background highlights
      :* sRGB24 220  50  47 -- red
      :* sRGB24 133 153   0 -- green
      :* sRGB24 181 137   0 -- yellow
      :* sRGB24  38 139 210 -- blue
      :* sRGB24 211  54 130 -- magenta
      :* sRGB24  42 161 152 -- cyan
      :* sRGB24   7  54  66 -- base02
      :* EmptyVec

    solarizedLight2 :: Vec N8 (Colour Double)
    solarizedLight2 =
         sRGB24 253 246 227 -- base3, background
      :* sRGB24 203  75  22 -- orange
      :* sRGB24 147 161 161 -- base1, comments / secondary text
      :* sRGB24 101 123 131 -- base00, body text / default code / primary content
      :* sRGB24  88 110 117 -- base01, optional emphasised content
      :* sRGB24 108 113 196 -- violet
      :* sRGB24 131 148 150 -- base0
      :* sRGB24   0  43  54 -- base03
      :* EmptyVec

You can see how I've applied these options in this commit.

Update: My complete example is now included in the termonad examples.

December 05, 2018

Audiobooks – November 2018

The Vanity Fair Diaries 1983-1992 by Tina Brown

Well written although I forgot who was who at times. The author came over very real and it is interesting to feel what has/hasn’t changed since the 1980s. 7/10

His Last Bow and The Valley of Fear by Sir Arthur Conan Doyle. Read by Stephen Fry

The Valley of Fear is solid. The short stories are not among my favorites but everything is well produced 7/10

First Man: The Life of Neil A. Armstrong by James R. Hansen

I read this prompted by the movie. Unlike the movie covers his family, early and post-moon life and has a lot more detail everywhere. Not overly long however 8/10

Don’t Make Me Pull Over! : An Informal History of the Family Road Trip by Richard Ratay

Nice combination of the author’s childhood experiences in the early-70s along with a history of the hotel, highway and related topics. 8/10

Giants’ Star by James P. Hogan

3rd book in the trilogy. Worth reading if you read and liked the first two. 6/10

U.S.S. Seawolf: Submarine Raider of the Pacific by Joseph Eckberg

First person account of a crew-member of a US Sub before & during the first year (up to Jan 1943) of US involvement in WW2. Published during the war and solely sourced for one person, so missing some details due to wartime censorship and lack of reference to other sources. Engaging though. 8/10

Mind of the Raven: Investigations and Adventures with Wolf-Birds by Bernd Heinrich

I didn’t like these quite as much as “Summer World” and “Winter World” since 100% ravens got a bit much but still it was well written & got me interested in the birds. 7/10

The Greater Journey: Americans in Paris by David McCullough

Covering American visitors (mostly artists, writers and doctors) to Paris mainly from 1830 to 1900. Covering how they lived and how Paris influenced them along with some history of the city. 9/10

Share

December 04, 2018

Finding Bugs in an Aercus WS3083

Aercus WS3083

While we're not drought declared, there's been precious little rain over the last 6 months or so, which made it hard to work out if I really had a problem or not.

The rainfall had been light and infrequent while each time the weather station recorded 0mm rainfall, which didn't seem unreasonable but was dubious nonetheless.

All the sensors appeared to be working OK. I re-seated, reconnected and reset everything to make sure that the system was connected and working fine but with no change in the result.

I eventually was driven to digging out the manual gauge which recorded 18mm the next time it rained while the weather station recorded 0mm.

No choice this time but to get up the ladder and disassemble the rain gauge.

I removed the cover and everything passed an initial eye balling.

Aercus WS3083 rain gauge

The Aercus WS3083 uses a lever arm to measure and discard collected rain. Being a simple primate, I attempted toggled it with my index finger, instantly noted it wasn't moving freely before a massive grasshopper|cricket flew out from under the arm and into my face. It had made a nice home under the arm, preventing it from dipping down and counting rain.

Unsurprisingly it moved freely after that and has been accurately measuring rain since.

Postscript: I think the chickens are still upset at me for failing to catch the grasshopper|cricket and feed it to them.

December 03, 2018

LUV December 2018 Main Meeting: Linux holiday gift ideas

Dec 4 2018 18:30
Dec 4 2018 19:30
Dec 4 2018 18:30
Dec 4 2018 19:30
Location: 
Kathleen Syme Library, 251 Faraday Street Carlton VIC 3053

PLEASE NOTE ONE HOUR DURATION

6:30 PM to 7:30 PM Tuesday, December 4, 2018
Training Room, Kathleen Syme Library, 251 Faraday Street Carlton VIC 3053

Speakers:

  • Andrew Chalmers
  • Andrew Pam

 

Many of us like to go for dinner nearby after the meeting, typically at Brunetti's or Trotters Bistro in Lygon St.  Please let us know if you'd like to join us!

Linux Users of Victoria is a subcommittee of Linux Australia.

December 4, 2018 - 18:30

read more

November 30, 2018

Artemis

Share

Its been ages since I’ve read a book in a couple of days, let alone stayed up late when I really shouldn’t in order to finish a book. Artemis is the book which broke me out of that rut — this is a fun, clever, light read. Its quite different when compared to The Martian, but I think that’s good. Weir has attempted to do something new instead of just playing on his previous successes.

An excellent book, and Mr Weir is solidly landing on my buy-everything-he-writes list.






Artemis Book Cover




Artemis





Andy Weir





Fiction




Del Rey




November 13, 2017




384



She grew up on the moon, of course she has a dark side... Jazz Bashara is a criminal. Well, sort of. Life on Artemis, the first and only city on the moon, is tough if you're not a rich tourist or an eccentric billionaire. So smuggling in the occasional harmless bit of contraband barely counts, right? Not when you've got debts to pay and your job as a porter barely covers the rent. Everything changes when Jazz sees the chance to commit the perfect crime, with a reward too lucrative to turn down. But pulling off the impossible is just the start of Jazz's problems, as she learns that she's stepped square into a conspiracy for control of Artemis itself - and that now, her only chance at survival lies in a gambit even more unlikely than the first.

Share

November 29, 2018

Turmoil

Share

A very readable set of essays from Robyn Williams, the broadcaster of the Australian Science Show, not the comedian. Covering the state of modern science, journalism, the ABC, and whether modern democracy is doomed in an approachable and very readable form. I enjoyed this book greatly. A good Sunday morning and vacation read if you’re into approachable non-fiction.

Turmoil Book Cover Turmoil
Robyn Williams
Memoir
Newsouth Press

Robyn Williams, presenter of The Science Show on ABC Radio, reveals all in Turmoil, a searingly honest and often blackly funny reflection on his life, friends, the people he loves and loathes, and a multi-faceted career that includes over forty years on radio. Robyn writes frankly about everything, from performing with Monty Python, his impressions of fellow scientists Richard Dawkins and David Attenborough, and his unique insights on climate change and the recent devaluing of science, to frugality and being treated for bowel cancer.

Share

November 22, 2018

Dark Doodad

It's been a while since I did a blog, so after twiddling the way the front page of the site displays, it's time to post a new one. The attached photo is of my favourite dark nebula, "The Dark Doodad". What looks like a long thin nebula is apparently a sheet over 40 light years wide that we happen to be seeing edge-on. On the left you can see a few dark tendrils that are par of the coal sack nebula. The Dark Doodad This is one of the first images created from a stack of subs I took using AstroDSLR. Each exposure is 2 minutes and I stacked 20 of them. My polar alignment was pretty decent, I think!

November 21, 2018

OpenPOWER Summit Europe 2018: A Software Developer's Introduction to OpenCAPI

Last month, I was in Amsterdam at OpenPOWER Summit Europe. It was great to see so much interest in OpenPOWER, with a particularly strong contingent of researchers sharing how they're exploiting the unique advantages of OpenPOWER platforms, and a number of OpenPOWER hardware partners announcing products.

(It was also my first time visiting Europe, so I had a lot of fun exploring Amsterdam, taking a few days off in Vienna, then meeting some of my IBM Linux Technology Centre colleagues in Toulouse. I also now appreciate just what ~50 hours on planes does to you!)

One particular area which got a lot of attention at the Summit was OpenCAPI, an open coherent high-performance bus interface designed for accelerators, which is supported on POWER9. We had plenty of talks about OpenCAPI and the interesting work that is already happening with OpenCAPI accelerators.

I was invited to present on the Linux Technology Centre's work on enabling OpenCAPI from the software side. In this talk, I outline the OpenCAPI software stack and how you can interface with an OpenCAPI device through the ocxl kernel driver and the libocxl userspace library.

My slides are available, though you'll want to watch the presentation for context.

Apart from myself, the OzLabs team were well represented at the Summit:

Unfortunately none of their videos are up yet, but they'll be there over the next few weeks. Keep an eye on the Summit website and the Summit YouTube playlist, where you'll find all the rest of the Summit content.

If you've got any questions about OpenCAPI feel free to leave a comment!

November 20, 2018

LPCNet meets Codec 2

The previous post described my attempts to come up to speed with NN based speech synthesis, with the kind help of Jean-Marc Valin and his LPCNet system.

As an exercise, I have adapted LPCNet to use Codec 2 features, and have managed to synthesise high quality speech at a sample rate of 8kHz. Here are the output speech samples:

Sample original LPCNet Codec 2
cq_ref Listen Listen
hts1a Listen Listen
hts2a Listen Listen
mmt1 Listen Listen
morig Listen Listen
speech_orig Listen Listen

I’m happy with all of the samples except cq_ref. That sample has a lot of low freq energy (like the pitch fundamental) which may not have been well represented in the training database. mmt1 has some artefacts, but this system already does better than any other low rate codec on this sample.

This is not quite a quantised speech codec, as I used unquantised Codec 2 parameters (10 Line Spectral Pairs, pitch, energy, and a binary voicing flag). However it does show how LPCNet (and indeed NN synthesis in general) can be trained to use different sets of input features, and the system I have built is close to an open source version of the Codec 2/NN system presented by Kleijn et al.

Why 8kHz rather than the higher quality 16 kHz? Well LPCNet requires a set of Linear Prediction Coefficients (LPCs). The LPCs dumped by Codec 2 are sampled at 8kHz. It’s possible, but not straight forward, to resample the LPC spectra at 16 kHz, but I chose to avoid that step for now.

Training

My initial attempts led to good quality speech using samples from within the training database, but poor quality on speech samples (like the venerable hts1a) from outside the training database. In Machine Learning land, this suggests “not enough training data”. So I dug up an old TIMIT speech sample database, and did a bunch of filtering on my speech samples to simulate what I have seen from microphones in my Codec 2 adventures. It’s all described in gory detail here (Training Tips section). Then, much to my surprise, it worked! Clean, good quality speech from all sorts of samples.

Further Work

  • Add code to generate 16 kHz LPCs from 8 kHz LPCs and try for 16 kHz synthesised speech
  • Use quantised Codec 2 parameters from say Codec 2 2400 or 1300 and see how it sounds.
  • Help Jean-Marc convert LPCNet to C and get it running in real time on commodity hardware.
  • Make a real world, over the air contact using NN based speech synthesis and FreeDV.
  • A computationally large part of the LPCNet (and indeed any *Net speech synthesis system) is dedicated to handling periodic pitch information. The harmonic sinusoidal model used in Codec 2 can remove this information and hence much of the CPU load. So a dramatic further reduction in the number of weights (and hence CPU load) is possible, although this may result in some quality reduction. Another way of looking at this (as highlighted by Jean-Marc’s LPCNet paper) is “how do we model the excitation” in source/filter type speech systems.
  • The Kleijn et al paper had the remarkable result that we can synthesise high quality speech from low bit rate Codec 2 features. What is quality trade off between the bit rate of the features and the speech quality? How coarsely can we quantise the speech features and still get high quality speech? How much of the quality is due to the NN, and how much the speech features?

Reading Further

Jean Marc’s blog post on LPCNet, including links to LPCNet source code and his ICASSP 2019 paper.
WaveNet and Codec 2
Source Code for my Codec 2 version of LPCNet

LPCNet – Open Source Neural Net Speech Synthesis

Jean-Marc Valin has been working on Neural Network (NN) based speech synthesis in his project called LPCNet. It has similar speech quality to Wavenet, but is based on an architecture called WaveRNN, and includes many new innovations.

Jean-Marc’s work is aimed at reducing the synthesis CPU load down to the level of a modern CPU, for example a mobile phone or Raspberry Pi, and he has made significant progress in that direction.

As well as being useful for his research – this code is a working, open source reference system for Neural Net (NN) based synthesis projects. He has also written an ICASSP 2019 paper on LPCNet, which explains many of the finer details of NN speech synthesis. Fantastic resources for other people coming up to speed in NN synthesis. Well done Jean-Marc!

Over the past few weeks Jean-Marc has kindly answered many NN-noob questions from me. I have used the answers to comment his code and add to his README. There are still many aspects of how this code works that I do not understand. However I can drive his software well enough to synthesise high quality speech:


The first sample was from inside the training database, the second outside.

The network is driven by some speech codec like parameters, but it’s not actually running as a speech codec at present. However it’s a great starting point for high quality speech (de)coding, or indeed speech synthesis.

How I trained

My GTX1060 GPU isn’t quite up to spec, so for training I had to reduce the batch_size to 16, and run for 60 epochs. I used the TSP speech database discussed in the LPCNet README, and followed Jean-Marc’s suggestion of resampling it twice (once at +5% Fs, once at -5% Fs), to get 3x the training data. It took 14 hours for me to train. Synthesis runs 10 times slower than real time on my GPU, however much of this is overhead. If the Keras code was ported to C – it would be close to real time on a modern laptop/phone CPU.

References

Jean Marc’s blog post on LPCNet, including links to LPCNet source code and his ICASSP 2019 paper.
WaveNet and Codec 2
WaveRNN
FFTNet, some good figures that helped me get my head around the idea of sampling a probability distribution.

November 11, 2018

LUV November 2018 Workshop: Introduction to shell programming

Nov 17 2018 12:30
Nov 17 2018 16:30
Nov 17 2018 12:30
Nov 17 2018 16:30
Location: 
Infoxchange, 33 Elizabeth St. Richmond

Introduction to shell programming

A lot of things in Linux are done on the command line. This talk starts with a very basic knowledge of the command line and describes how to leverage this into a powerful programming system. Transform those snips of commands into powerful self documenting scripts to super charge your productivity.

The meeting will be held at Infoxchange, 33 Elizabeth St. Richmond 3121.  Late arrivals please call (0421) 775 358 for access to the venue.

LUV would like to acknowledge Infoxchange for the venue.

Linux Users of Victoria is a subcommittee of Linux Australia.

November 17, 2018 - 12:30

read more

November 06, 2018

DevOpsDaysNZ 2018 – Day 2 – Session 4

Allen Geer, Amanda Baker – Continuously Testing govt.nz

  • Various .govt.nz sites
  • All Silverstripe and Common Web Platform
  • Many sites out of date, no automated testing, no test metrics, manual testing
  • Micro-waterfall agile
  • Specification by example (prod owner, Devops, QA)  created Gherkin tests
  • Standardised on CircleCI
  • Visualised – Spec by example
  • Prioritised feature tests
  • Ghirkinse
  • Test at start of dev process. Bake Quality in at the start
  • Visualise and display metrics, people could then improve.
  • Path to automation isn’t binary
  • Involve everyone in the team
  • Automation only works if humanised

Jules Clements – Configuration Pipeline : Ruling the One Ring

  • Desired state
  • I didn’t quite understand what he was saying

Nigel Charman – Keep Calm and Carry On Organising

  • 71 Conferences worldwide this year
  • NZ following the rules
  • Lots of help from people
  • Stuff stuff stuff

Jessica DeVita – Retrospecting our Retrospectives

  • Works on Azure DevOps
  • Post-mortems
  • What does it mean to have robust systems and resilience? Is resilience even a property? It just Is. When we fly on planes, we’re trusting machines and automation. Even planes require regular reboots to avoid catastrophic failures, and we just trust that it happen
  • CEO after a million dollar outage said “Can you get me a million dollars of learning out of this?”
  • After US Navy had accidents caused by slept deprivation switched to new watch structure
  • Postmortems are not magic, they don’t automatically make things change
  • http://stella.report
  • We dedicate a lot of time to to below the line, looking at the technology. Not a lot of conversation about above-the-line things like mental models.
  • Resilience is above the line
  • Catching the Apache SNAFU
  • The Ironies of Automation – Lisanne Bainbridge
  • Well facilitated debriefings support recalibration of mental models
  • US Forest Service – Learning Review – Blame discourages people speaking up about problems
  • We never know where the accident boundary is, only when we have crossed it.
    • SRE, Chaos Engineer and Human Factors help hadle
  • In postmortems please be mindful of judging timelines without context. Saying something happened in a short or long period of time is damanging
  • Ask “what made it hard to get that team on the phone?” , “What were you trying to achieve”
  • Etsy Debriefing Guide – lots of important questions.
  • “Moving post shallow incident data” – Adaptive Capacity Labs
  • Safety is a characteristics of Systems and not of their components
  • Ask people about their history, ask every person about what they do and how they got there because that is what shapes your culture as an organisation

Share

DevOpsDaysNZ 2018 – Day 2 – Session 3

Kubernetes

I’ll fill this in later.

Observability

  • Honeycomb, Sumologic. Use AI to look at what happened at same time and magically correlate
  • Expensive or hard to send all logs as volumes go up
  • What is the logging is wrong or missing?
  • Metrics
    • Export in prometheus format
    • Read RED and USE paper
    • Create a company schema with half a dozen metrics that all services expose
  • Had and event or transaction ID that flows across all the microservices sorry logs can be correlated
  • Non technical solutions
    • Refer to previous incident logs
    • Part of deliverables for product is SLA stats which require logs etc
  • Testing logs
    • Make sure certain events produce a log
  • Chaos Monkey

ANZ Drivetrain

  • Change control cares about
    • Avaiability
    • Risk
    • Dependencies
    • Rollback
  • But the team doing the change knows about these all
  • Saw tools out there that seem very opinated
  • Drivetrain
    • Automated Checklist
    • Work with Change people to create checklist
    • Pipeline talks to drivetrain and tells it what has been down
    • Slack messages sent for manual changes (they login to app to approve)
  • Looked at some other tools (eg chef automate, udeploy )
    • Forced team to work in a certain pattern
  • But use ServiceNow tool as official corporate standard
    • Looking at making DriveTrail fill in ServiceNow forms
  • People worried about stages in tool often didn’t realise the existing process had same limitations
  • Risk assessed at the Story and Feature level. Not release level
  • Not suitable for products that due huge released every few months with a massive number of changes.

 

 

Share

DevOpsDaysNZ 2018 – Day 2 – Session 2

Interesting article I read today

Why Doctors Hate their Computers by Atul Gawande

Mrinal Mukherjee – A DevOps Confessional

  • Not about accidents, it is about Planned Blunders that people are doing in DevOps
  • One Track DevOps
    • From Infrastructure background
    • Job going into places, automated the low hanging fruit, easy wins
    • Column of tools on resume
    • Started becoming the bottleneck, his team was the only one who knew how the infrastructure worked.
    • Not able to “DevOps” a company since only able to fix the infrastructure, not able to fix testing etc so not dilvering everything that company expected
    • If you are the only person who understands the infrastructure you are the only one blamed when it goes wrong
    • Fixes
      • Need to take all team on a journey
      • But need to have right expectations set
      • Need to do learning in areas where you have gaps
      • DevOps is not about individual glory, Devops is about delivering value
      • HR needs to make sure they don’t reward the wrong thing
  • MVP-Driven Devops
    • Mostly working on Greenfields products that need to be delivered quickly
    • MVP = Maximum Technical Debt
    • MVP = Delays later and Security audits = Your name attached to the problem
    • Minimum Standard of Engineering
      • Test cases, Documentation, Modular
      • Peer review
    • Evolve architecture, not re-architect
  • Judgemental Devops
    • That team sucks, they are holding things up, playing a different game from us
    • Laughing at other teams
    • Consequence – Stubbornness from the other team
    • Empathy
      • Find out why things are they way they are
    • Collaborate to find common ground and improve
    • Design my system to I plan to work within constraints of the other team

Share

November 05, 2018

DevOpsDaysNZ 2018 – Day 2 – Session 1

Alison Polton-Simon – The DevOps Experiments: Reflections From a Scaling Startup

  • Software engineer at Angaza, previously Thoughtworks, “Organizational Anthropologist”
  • Angaza
    • Enable sales of life-changing products (eg solar chargers, water pumps, cook stoves in 3rd world countries)
    • Originally did hardware, changed to software company doing pay-as-you-go of existing devices
    • ~50 people. Kenya and SF, 50% engineering
    • No dedicated Ops
    • Innovate with empathy, Maximise impact
    • Model is to provide software tools to activate/deactivate products sold to peopel with low credit-scores. Plus out software around the activity like reports for distributors.
  • Reliability
    • Platform is business critical
    • Outages disrupt real people (households without light, farmers without irrigation)
    • Buildkite, Grafana, Zendesk
  • Constraints
    • Operate in 30+ countries
    • Low connectivity, 2G networks best case
    • Rural and peri-urban areas
    • Team growing by 50% in 2018 (2 eng teams in Kenya + 1 QA)
    • Most customers in timezone where day = SF night
  • Eras of experimentation
    • Ad Hoc
    • Tributes (sacrifice yourself for the stake of the team)
    • Collectives (multiple teams)
    • Product teams
  • ad Hoc – 5 eng
    • 1 eng team
    • Ops by day – you broke, you fix
    • Ops by night – Pagerduty rotation
    • Paged on all backend exception, 3 pages = amnesty
    • Good
      • Small but senior
      • JIT maturity
      • Everyone sitting next to each other
    • Bad
      • Each incident higherly disruptive
      • prioritized necessity over sustainability
  • Tribute – 5-12 eng
    • One person protecting team from interuptions
    • Introduced support person and triage
    • Expanded PD rotation
    • Good
      • More sustainable
      • Blue-Green deploys
      • Clustered workloads
    • Not
      • Headcount != horizontal scaling
      • Hard to hire
      • Customer service declined
  • Collectives 13-20 engs
    • Support and Ops teams – Ops staffed with devs
    • Other teams build roadmaps and requests
    • Teams rotate quarterly – helps onboarding
    • Good
      • Cross train ppl
      • Allow for focus
      • allowed ppl to get depth
    • Bad
      • Teams don’t op what they built
      • Quarter flies by quickly
      • Context switch is costly
      • Still a juggling act
      • 1m ramping up, 6w going okay, 2w winging down
  • Product teams  21 -? eng
    • 5 teams, 2 in Nairobi
    • Teams allighned with business virticals, KPIs
    • Dev, own and maintain services
    • Per-team tributes
    • No [Dev]Ops team
    • Intended goals
      • Independent teams
      • own what build
      • Support biz KPIs
      • cross team coordination
    • Expected Chellenges
      • Ownership without responsbility
      • Global knowledge sharing
      • Return to tribute system (2w out of the workflow)
  • Next
    • Keep growing team
    • Working groups
    • Eventual SRE
    • 24h global coverage
  • Case a “Constitution” of values that everybody who is hired signs
  • Takeaways
    • Maximise impact
      • Dependable tools over fashionable ones
      • Prefer industry-std tech
      • But get creative when necessary
    • Define what reliability means for your system
    • Evolve with Empathy
      • Don’t be dogmatic without structure
      • Serve your customers and your team
      • Adapt when necessary
      • Talk to people
      • Be explicit as to the tradeoffs you are making

Anthony Borton – Four lessons learnt from Microsoft’s DevOps Transformation

  • Microsoft starting in 1975
  • 93k odd engineers at Microsoft
    • 78k deployments per day
    • 2m commits per month
    • 4.4 builds/month
    • 500 million tests/day
  • 2018 State of Devops reports looks at Elite performers in the space
  • TFS – Team Foundation Server
    • Move product to the cloud
    • Moved on-prem to one instance
    • Each account had it’s own DB (broke stuff at 11k DBs)
  • 4 lessons
    • Customer focussed
      • Listen to customers, uservoice.com
      • Lots of team members keep eye on it
      • Stackoverflow
      • Embed with customers
      • Feedback inside product
      • Have to listen in all the channels
    • Failure is an opportunity to learn
    • Definition of done
      • Live in prod, collecting telemetary that examines hypotheses that it was created to prove
    • “For those of you who don’t know who Encarta is, look it up on Wikipedia”
  • Team Structure
    • Combined engineering = devs + testing
      • Some testers left team or organisation
    • Feature team
      • Physical team rooms
      • Cross discipline
      • 10-12 people
      • self managing
      • Clear charter and goals
      • Intact for 12-18 months
    • Sticky note exercise, people pick which teams they would like to join (first 3 choices)
      • 20% choose to change
      • 100% get the choice
  • New constants and requirements
    • Problems
      • Tests took too long – 22h to 2days
      • Tests failed frequently – On 60% passed 100%
      • Quality signal unreliable in master
    • Publish VSTS Quality vision
      • Sorted by exteranl dependancies
      • Unit tests
        • L0 – in-memory unit tests
        • L1 – More with SQL
      • Functional Tests
        • L2 – Functional tests against testable service deployment
        • L3 – Restricted class integration tests that run against production
      • 83k L0 tests run agains all pulls very fast
  • Deploy to rings of users
    • Ring 0 – Internal Only
    • Ring 1 – Small Datacentre 1-1.5m accounts in Brazil (same TZ as US)
    • Ring 2 – Public accounts, medium-large data centre
    • Ring 3 – Large internal accounts
    • Ring 4-5 – everyone else
    • Takes about a week for normal releases.
    • Binaries go out and then the database changes
    • Delays of minutes (up to 75) during the deploys to allow bugs to manafest
    • Some customers have access to feature flags
    • Customers who are risk tolerant can opt in to early deploys. Allows them to get faster feedback from people who are able to provide it
  • More features delivered in 2016 than previous 4 years. 50% more in 2017

 

Share

DevOpsDaysNZ 2018 – Day 1 – Session 4

Everett Toews – A Trip Down CI/CD Lane

I missed most of this talk. Sounded Good.

Jeff Smith – Creating Shared Contexts

  • Ideas and viewpoints are different from diff people
  • Happens in organisation, need to make sure everybody is on the same page
  • Build a shared context via conversations
  • Info exchange
  • Communications tools
  • Context Tools
  • X/Y Problem
  • Data can bridge conversations. Shared reality.
  • Use the same tools as other teams so you know what they are doing
  • Give the context to your requests, ask for them and it will automatically happen eventually.

Peter Sellars – 2018: A Build Engineers Odyssey

  • Hungry, Humble and Smart

Katrina Clokie – Testing in DevOps for Engineers

  • We can already write, so how hard can it be to write a novel?
  • Hopefully some of you are doing testing already
  • Problem is that people overestimate their testing skills, not interested in finding out anything else.
  • The testing you are doing now might be with one tool, in one spot. You are probably finding stuff but missing other things
  • Why important
    • Testing is part of you role
    • In Devops testing goes though Operations as well
    • Testing is DevOps is like air, it is all around you in every role
    • Roles of testers is to tech people to breath continuously and naturally.
  • Change the questions that you ask
    • How do you know that something is okay? What questions are you asking of your product?
    • Oracles are the ways that we recognise a problem
    • Devs ask: “Does it work how we think it should?”
    • Ops ask: “Does it work how it usually works?”
    • Devs on claims, Ops on history
    • Does it work like our competors, does it meet it’s purpose without harmful side effects, doesn’t it meet legal requirements, Does it work like similar services.
    • HICCUPPS – Testing without a Map – Michael Bolton, 2005
    • How do we compare to what other people are doing?  ( Not just a BA’s job , cause the customer will be asking a question and so should you)
    • Flip the Oracle, compare them against other things not just the usual.
    • Audit – Continuous compliance, Always think about if it works like the standards say it should.
    • These are things that the business is asking. If you ask then gain confidence of business
  • Look for Answers in Other Places
    • Number of tests: UI <  Service < Unit
    • The Test Pyramid as a bug catcher. Catch the Simple bugs first and then the subtle ones
  • Testing mesh
    • Unit tests – fine mesh
    • Intergration – Bigger/Fewer tests but cover more
    • Next few layers: End to End, Alerting, Monitoring, Logging. Each stops different types of bugs
    • Conversation should be “Where do we put our mesh?”, “How far can this bug fall?”.
    • If another layer will pick the bug up do we need a test.
  • Use Monitoring as testing
    • Push risk really late, no in all cases but can often work
  • A/B testing
    • Ops needs to monitor
    • Dev needs framework to role out and put in different options
  • Chaos Engineering
    • Start with something small, communicate well and do during daylight hours.
    • Yours customers are testing in production all the time, so why arn’t you too?
  • https://leanpub.com/testingindevops

 

Share

DevOpsDaysNZ 2018 – Day 1 – Session 3

Open Space 1 – Prod Support, who’s responsible

  • Problem that Ops doesn’t know products, devs can’t fix, product support owners not technical enough
  • Xero have embedded Ops and dev in teams. Each person oncall maybe 2 weeks in 20
  • Customer support team does everything?
  • “Ops have big graphs on screens, BI have a couple of BI stats on screens, Devs have …youtube videos”
  • Tiers support vs Product team vs Product support team
  • Tiered support
    • Single point of entry
    • lower paid person can handle easy stuff
    • Context across multiple apps
  • Product Team
    • Buck stops with someone
    • More likely to be able to action
    • Ownership of issues
    • Everyone must be enabled to do stuff
    • Everyone needs to be upskilled
  • Prod Support
    • Big skilled can fix anything team
    • Devs not keen
    • Even the best teams don’t know everything

Open Space 2 – DevOps at NZ Scale

  • Devops team, 3rd silo
    • Sometimes they are the new team doing cool stuff
    • One model is evangelism team
  • Do you want devops culture or do you just want somebody to look after your pipeline?
  • Companies often don’t know what they want to hire
  • Companies get some benefit with the tools (pipelines, agile)  but not the culture. But to get the whole benefit they need to adopt everything.
  • The Way of Ways article by John Cutler

Open Space 3 – Responding Quickly

I was taking notes on the board.

Share

DevOpsDaysNZ 2018 – Day 1 – Session 2

Mark Simpson, Carlie Osborne – Transforming the Bank: pipelines not paperwork

  • Change really is possible even in the least likely places
  • Big and risk adverse
    • Lots of paperwork and progress, very slow
  • Needed to change – In the beginning – 18 months ago
    • 6 months talking about how we could change things
  • Looked for a pilot project – Internet Banking team – ~80 people – Go-money platform
    • Big monolith, 1m lines of code
    • New release every 6 weeks
    • 10 weeks for feature from start to production
    • Release on midnight on a Friday night, 4-5 hours outage, 20-25 people.
    • Customer complaints about outage at midnight, move to 2am Sunday morning
  • Change to release every single week
    • Has to be middle of the day, no outage
    • How do we do this?
  • Took whole Internet banking team for 12 weeks to create process, did nothing else.
  • What we didn’t do
    • Didn’t replatform, no time
  • What we did
    • Jenkins – Created a single Pipeline, from commit to master all the way to projection
    • Got rid of selenium tests
    • Switched to cypress.io
      • Just tested 5 key customer journeys
    • Drivetrain – Internal App
      • Wanted to empower the teams, but lots of limits within industry/regulations
      • Centralise decision making
      • Lightweight Rules engine, checks that all the requirements have been done by the team before going to the next stage.
    • Cannery Deployments
      • Two versions running, ability to switch users to one or other
  • Learning to Break things down into small chunks
  • Change Process
    • Lots of random rules, eg mandatory standdown times
    • New change process for teams using Drivetrain, certified process no each release
  • Lots of times spent talking to people
    • Had to get lots of signoffs
  • Result
    • Successful
    • 16 weeks rather than 12
    • 28 releases in less than 6 months (vs approx 4 previously)
    • 95% less toil for each release
  • Small not Big changes
    • Now takes just 4-5 weeks to cycle though a feature
    • Don’t like saying MVP. Pitch is as quickly delivering a bit of value
    • and iterating
    • 2 week pilot, not iterations -> 8 week pilot, 4 iterations
    • Solution at start -> Solution derived over time
  • Sooner, not later
    • Previously
      • Risk, operations people not engaged until too late
      • Dev team disengaged from getting things into production
    • Now
      • Everybody engaged earlier
  • Other teams adopting similar approach

Ryan McCarvill – Fighting fires with DevOps

  • Lots of information coming into a firetruck, displayed on dashboard
  • Old System was 8-degit codes
  • Rugged server in each each truck
    • UPS
    • Raspberry Pi
    • Storage
    • Lots of different networking
  • Requirements
    • Redundant Comms
    • Realtime
    • Offline Mpas
    • Offline documentation, site reports, photos, building info
    • Offline Hazzards
    • Allow firefighters to update
    • Track appliance and firefighter status
    • Be a hub for an incident
    • Needs to be very secure
  • Stack on the Truck
    • Ansible, git, docker, .netcode, redis, 20 micoservices
  • What happens if update fails?
  • More than 1000 trucks, might be offline for months at a time
  • How to keep secure
  • AND iterate quickly
  • Pipeline
    • Online update when truck is at home
    • Don’t update if moving
    • Blue/Green updates
    • Health probes
  • Visual Studio Team Services -> Azure cont registry
  • Playbooks on git , ansible pull,
  • Nginx in front of blue/green
  • Built – there were problems
    • Some overheating
    • Server in truck taken out of scope, lost offline strategy
    • No money or options to buy new solution
  • MVP requirements
    • Lots of gigs of data, made some so only online
    • But many gigs still needed online
    • Create virtual firetruck in the sky, worked for online
    • Still had communication device – 1 core, minimum storage, locked down Linux
  • Put a USB stick in the back device and updated it
    • Can’t use a lot of resources or will inpact comms
    • Hazard search
      • Java/python app, no much impact on system
      • Re-wrote in rust, low impact and worked
      • Changed push to rsync and bash
  • Lessons
    • Automation gots us flexability to change
    • Automation gave us flexability to grow
    • Creativity can solve any problem
    • You can solve new problems with old technology
    • Sometimes the only way to get buy in is to just do it.

Share

November 04, 2018

DevOpsDaysNZ 2018 – Day 1 – Session 1

Jeff Smith – Moving from Ops to DevOps: Centro’s Journey to the Promiseland

  • Everyone’s transformation will look a little different
  • Tools are important but not the main problem (dev vs Ops)
  • Hiring a DevOps Manager
    • Just sounds like a normal manager
    • Change it to “Director of Production Operations”
  • A “DevOps Manager” is the 3rd silo on top of Dev and Ops
  • What did peopel say was wrong when he got there?
    • Paternalistic Ops view
      • Devs had no rights on instances
      • Devs no prod access
      • Devs could not create alerts
  • Fix to reduce Ops load
    • Devs get root to instances, but access to easily destroy and recreate if they broke it
    • Devs get access to common safe tasks, required automation and tools (which Ops also adopted)
    • Migrated to datadog – Single tool for all monitoring that anyone could access/update.
    • Shared info about infrastructure. Docs, lunch and learns. Pairing.
  • Expanding the scope of Ops
    • Included in the training and dev environment, CICD. Customers are internal and external
    • Used same code to build every environment
    • Offering Operation expertise
      • Don’t assume the people who write the software know the best way to run it
  • Behaviour can impact performance
    • See book “Turn the Ship around”
    • Participate in Developer rituals – Standups, Retros
    • Start with “Yes.. But” instead of “No” for requests. Assume you can but make it safe
    • Can you give me some context. Do just do the request, get the full picture.
  • Metrics to Track
    • Planned vs unplanned work
    • What are you doing lots of times.
  • What we talk about?
    • Don’t allow your ops dept to be a nanny
    • Remove nanny state but maintain operation safety
    • Monitor how your language impacts behavour
    • Monitor and track the type of work you are doing

François Conil – Monitoring that cares (the end of user based monitoring)

  • User Based monitoring (when people who are affected let you know it is down)
  • Why are we not getting alerts?
    • We are are not measuring the right thing
    • Just ignore the dashboard (always orange or red)
    • Just don’t understand the system
  • First Steps
    • Accept that things are not fine
    • Decide what you need to be measuring, who needs to know, etc. First principals
    • A little help goes a long way ( need a team with complementary strengths)
  • Actionable Alerts
    • Something Broken, User affected, I am the best person to fix, I need to fix immediately
    • Unless all 4 apply then nobody should be woken up.
      • Measured: Take to QA or performance engineers to find out the baseline
      • User affected: If nobody is affected do we care? Do people even work nights? How do you gather feedback?
      • Best person to fix: Should ops guys who doesn’t understand it be the first person to page?
      • Do it need to be fixed? – Backup environment, Too much detail in the alerts, Don’t alert on everything that is broken, just the one causing the problem
  • Fix the cause of the alerts that are happening the most often
  • You need time to get things done
    • Talk to people
    • Find time for fixes
  • You need money to get things done
    • How much is the current situation costing the company?
    • Tech-Debt Friday

Share

November 02, 2018

LUV November 2018 Main Meeting: Computerbank / Designing Open Democracy

Nov 7 2018 18:30
Nov 7 2018 20:30
Nov 7 2018 18:30
Nov 7 2018 20:30
Location: 
Kathleen Syme Library, 251 Faraday Street Carlton VIC 3053

PLEASE NOTE CHANGE OF DAY DUE TO MELBOURNE CUP HOLIDAY

6:30 PM to 8:30 PM Wednesday, November 7, 2018
Training Room, Kathleen Syme Library, 251 Faraday Street Carlton VIC 3053

Speakers:

Many of us like to go for dinner nearby after the meeting, typically at Brunetti's or Trotters Bistro in Lygon St.  Please let us know if you'd like to join us!

Linux Users of Victoria is a subcommittee of Linux Australia.

November 7, 2018 - 18:30

read more

November 01, 2018

Lean data in practice

Mozilla has been promoting the idea of lean data for a while. It's about recognizing both that data is valuable and that it is a dangerous thing to hold on to. Following these lean data principles forces you to clarify the questions you want to answer and think hard about the minimal set of information you need to answer these questions.

Out of these general principles came the Firefox data collection guidelines. These are the guidelines that every team must follow when they want to collect data about our users and that are enforced through the data stewardship program.

As one of the data steward for Firefox, I have reviewed hundreds of data collection requests and can attest to the fact that Mozilla does follow the lean data principles it promotes. Mozillians are already aware of the problems with collecting large amounts of data, but the Firefox data review process provides an additional opportunity for an outsider to question the necessity of each piece of data. In my experience, this system is quite effective at reducing the data footprint of Firefox.

What does lean data look like in practice? Here are a few examples of changes that were made to restrict the data collected by Firefox to what is truly needed:

  • Collecting a user's country is not particularly identifying in the case of large countries likes the USA, but it can be when it comes to very small island nations. How many Firefox users are there in Niue? Hard to know, but it's definitely less than the number of Firefox users in Germany. After I raised that issue, the team decided to put all of the small countries into a single "other" bucket.

  • Similarly, cities generally have enough users to be non-identifying. However, some municipalities are quite small and can lead to the same problems. There are lots of Firefox users in Portland, Oregon for example, but probably not that many in Portland, Arkansas or Portland, Pennsylvania. If you want to tell the Oregonian Portlanders apart, it might be sufficient to bucket Portland users into "Oregon" and "not Oregon", instead of recording both the city and the state.

  • When collecting window sizes and other pixel-based measurements, it's easier to collect the exact value. However, that exact value could be stable for a while and create a temporary fingerprint for a user. In most cases, teams wanting to collect this kind of data have agreed to round the value in order to increase the number of users in each "bucket" without affecting their ability to answer their underlying questions.

  • Firefox occasionally runs studies which involve collecting specific URLs that users have consented to share with us (e.g. "this site crashes my Firefox"). In most cases though, the full URL is not needed and so I have often been able to get teams to restrict the collection to the hostname, or to at least remove the query string, which could include username and passwords on badly-designed websites.

  • When making use of Google Analytics, it may not be necessary to collect everything it supports by default. For example, my suggestion to trim the referrers was implemented by one of the teams using Google Analytics since while it would have been an interesting data point, it wasn't necessary to answer the questions they had in mind.

Some of these might sound like small wins, but to me they are a sign that the process is working. In most cases, requests are very easy to approve because developers have already done the hard work of data minimization. In a few cases, by asking questions and getting familiar with the problem, the data steward can point out opportunities for further reductions in data collection that the team may have missed.

October 29, 2018

Installing Vidyo on Ubuntu 18.04

Following these instructions as well as the comments in there, I was able to get Vidyo, the proprietary videoconferencing system that Mozilla uses internally, to work on Ubuntu 18.04 (Bionic Beaver). The same instructions should work on recent versions of Debian too.

Installing dependencies

First of all, install all of the package dependencies:

sudo apt install libqt4-designer libqt4-opengl libqt4-svg libqtgui4 libqtwebkit4 sni-qt overlay-scrollbar-gtk2 libcanberra-gtk-module

Then, ensure you have a system tray application running. This should be the case for most desktop environments.

Building a custom Vidyo package

Download version 3.6.3 from the CERN Vidyo Portal but don't expect to be able to install it right away.

You need to first hack the package in order to remove obsolete dependencies.

Once that's done, install the resulting package:

sudo dpkg -i vidyodesktop-custom.deb

Packaging fixes and configuration

There are a few more things to fix before it's ready to be used.

First, fix the ownership on the main executable:

sudo chown root:root /usr/bin/VidyoDesktop

Then disable autostart since you don't probably don't want to keep the client running all of the time (and listening on the network) given it hasn't received any updates in a long time and has apparently been abandoned by Vidyo:

sudo rm /etc/xdg/autostart/VidyoDesktop.desktop

Remove any old configs in your home directory that could interfere with this version:

rm -rf ~/.vidyo ~/.config/Vidyo

Finally, launch VidyoDesktop and go into the settings to check "Always use VidyoProxy".

October 28, 2018

Audiobooks – October 2018

How to Rig an Election by Nic Cheeseman & Brian Klaas

The authors take experiences in various countries (mostly recent 3rd-world examples) as to how elections are rigged. Some advice for reducing it. 8/10

The Hound of the Baskervilles by Sir Arthur Conan Doyle. Read by Stephen Fry

Once again great reading by Fry and a great story. Works very well with all Holmes and Watson action and no giant backstory. 8/10

Our Oriental Heritage: Story of Civilization Series, Book 1 by Will Durant

Covers the early history of Egypt, the Middle East, India, China and Japan. In some cases up to the 20th Century. The book cover arts, religion and philosophy as well Kings and dates. This was written in the 1930s so has some stuff that has been superseded and out of date attitudes to race and religion in places. It long (50 hours) with another 11 volumes still to go but it is pretty good if you don’t mind these problems. 7/10

Chasing New Horizons: Inside the Epic First Mission to Pluto by Alan Stern & David Grinspoon

Stern was one of the originators and principal investigator of the mission so lots of firsthand details about all stages of the project from first conception though various versions.

Share

October 27, 2018

Six years and 9 months...

The Drupal Association

Six years and 9 months... is a relatively long time. Not as long as some things, longer than others. Relative. As is everything.

But Six years and 9 months is the length of time I've been on the board of the Drupal Association.

I was elected to serve on the board by the community in February 2012, and then nominated to serve for another two terms. That second term expires on 31 October. My original candidate statement makes somewhat nostalgic reading now... and it's now that I wonder, what I achieved. If anything?

But that's the wrong question. There's nothing useful to be gained in trying to answer it.

Instead - I want to reflect on what I learned.

I learned something from everyone at that table. Honestly, I never really lost my sense of imposter syndrome, and I'm freely and gleefully willing to admit that.

Cary Gordon - we shared a passion for DrupalCon. That show grew into the incredible event it is because of seeds you sewed. And your experience running big shows, and supporting small community libraries seemed to be the perfect mix for fueling what Drupal needed.

Steve Purkiss - we were elected together! Your passion for cooperatives, for Drupal, and for getting on with it, and making things happen was infectious! Thank you for standing with me in those weird first few months of being in this weird new place, called the board of the Drupal Association!

Pedro Cambra - I wish I'd heed the lesson you taught me more often. Listen carefully. Speak only when there's something important to say, or to make the case for a perspective that's being missed. But also good humour. And Thank you for helping make the election process better, and helping the DA "own" the mechanics.

Morten - brother. I can't even find the words to say. Your passion for Drupal, for theming, and for our community always inspired me. I miss your energy.

Angie "webchick" Byron - mate! I still can't fathom how you did what you do so effortlessly! Well, I know it's not effortless, but you make it look that way. Your ability to cut through noise, sort things out, get things done, and inspire the Drupal masses to greatness is breathtaking.

Matthew Saunders - you made me appreciate the importance of governance from a different perspective. Thank you for the work you did to strengthen our board processes.

Addison Berry - Sorry Addi - this is a bit shameful, but it was the mezcal, tequila and bourbon lessons that really stuck.

Danese Cooper - I was so grateful for your deep wisdom of Open Source, and the twists and turns of the path it's followed over such a long time. Your eye to pragmatism over zealotry, but steadfast in the important principles.

Shyamala Rajaram - Oh Shyamala! I can't believe we only first met at DrupalCon Mumbai, or perhaps it was only the first time, this time! Thank you for teaching us all how important it is for us to be in India, and embrace our global community.

Ryan Szrama - you stepped onto the board at such a tough moment, but you stepped up into the role of community elected Director, and helped make sense out of what was happening. Sorry not to see you in Drupal Europe.

Rob Gill - Running. I didn't learn this. Sorry.

Tiffany Farriss - You're formidable! You taught me the importance of having principles, and sticking to them. And then using them to build a foundation in the bedrock. You do this with such style, and grace, and good humour. I'm so thankful I've had this time with you.

Jeff Walpole - You made me question my assumptions all the time! You made me laugh, and you gave me excellent bourbon. You always had a way of bringing us back to the real world when we waded too deep into the weeds.

Vesa Palmu - So many things - but the one that still resonates, is we should all celebrate failure. We should create ritual around it, and formalise the lessons failure teaches. We all learn so much more from mistakes, than from successes.

Sameer Verna - For a time, we were the only linux users at the table, and then I defected back to MacOS - I still feel a bit guilty about this, I admit. You championed Free Software at every step - but also, so often, guided us through the strategic mumbo jumbo, to get to the point we needed to.

Steve Francia - "It's not as bad as you all seem to think it is" I don't know why, but I hear this mantra, spoken with your voice, whenever I think of you. Thank you for your Keynote in Nashville, and for everything.

Mike Lamb - I've not yet put into practice the lesson I need to learn from you. To switch off. To really go home, and be home, and switch off the world. I need me some of that, after all of this. Thank you so much for all you've done, but more for your positive, real world perspective. Ta!

Annie - I missed your presence in Germany so much - I feel like I've still got so much to learn from you. You bridged the worlds of digital and marketing, and brought much needed perspective to our thinking. Twas an honour to serve with you.

Audra - With you too, I feel like I was only beginning to get into the groove of the wisdom you're bringing to the table. I hope our paths continue to cross, so I can keep learning!

Baddy Sonja Breidert - A powerful lesson - as volunteers, we have to account for the time, passion and energy we borrow from the rest of our lives, when we give it to Drupal. And Drupal needs to properly recognise it too.

Ingo Rübe - You taught me how to have courage to bring big ideas to the table, and show grace in letting them go.

Michel van Velde - You taught me to interrogate my assumptions, with fun, with good humour, and honest intention of doing good.

George Matthes - You taught me the power of questioning the received wisdom from history. You reminded me of the importance of bringing fresh eyes to every challenge.

Adam Goodman - a simple, but important lesson. That leadership is about caring for people.

Suzanne Dergacheva - newly elected, and about to start your term - I had too little chance to learn from you at the board table, but I already learned that you can teach the whole community kindness by giving them carnations! #DrupalThanks to you too. And power to your arms as you take the oars as a community elected director, and help row us forward!

And to all the staff who've served over the years, your dedication to this organisation and community it serves is incredible. You've all made a difference, together, to all of us. Special mentions for four of you...

Kris - from Munich to Vienna - my constant companion, and my dive bar adventure buddy. Til next time there is cheese...

Holly - Inspiring me to knit! Or, more accurately, to wish I could knit better than I can. To knit with conviction! It's a metaphor for so much, but also very very literally. Also I miss you.

Steph - Your vibrant enthusiasm, and commitment to DrupalCon always inspired me. Your advice on food trucks in Portland nourished me.

Megan - where to start? I'd never finish. Kindness, compassion, steely focus, commercial reality, "operational excellence", and cactus margaritas.

I save my penultimate words for Dries... Thank you for having faith in me. Thank you for creating Drupal, and for sharing it with all of us. Also, thank you sharing many interesting kinds of Gin!

These final words are for Tim - as you take the reins of this crazy sleigh ride into the future - I feel like I'm leaving just before the party is really about to kick off.

Go you good thing.

Good bye, so long, and thanks for all the fish.

The DA does amazing work.
If you rely on Drupal, you rely on them.

Please consider becoming a member, or a supporting partner.

October 26, 2018

Iterating on Merge Proposals

Developing new WordPress features as plugins has been a wonderfully valuable process for all sorts of features to come into being, from the MP6 Dashboard Redesign, to oEmbed endpoints, and including multiple Customiser enhancements over the years. Thanks to the flexibility that this model offers, folks have been able to iterate rapidly on a wide range of features, touching just about every part of WordPress.

The “Features as Plugins” idea was first introduced during the WordPress 3.7 development cycle, during which the features were merged after a short discussion during a core chat: it was only in the WordPress 3.8 cycle that the idea of a merge proposal post (called “Present Your Feature” back then) came into being. It was envisioned as a way to consult with WordPress leaders, key contributors, and the wider WordPress community on the readiness of this feature to be released. Ultimately, WordPress leaders would make a decision on whether the feature was right for WordPress, and the release lead would decide if it was ready for that release.

Since then, most feature plugins have published some form of merge proposal post before they were ultimately merged into WordPress, and they’ve nearly all benefited to some degree from this process.

The merge proposal process has worked well for smaller features, but it struggled with larger changes.

The REST API is a great example of where the merge proposal process didn’t work. The REST API was a significant change, and trying to communicate the scope of that change within the bounds of a single merge proposal post didn’t really do it justice. It was impossible to convey everything that was changing, how it all worked together, and what it meant for WordPress.

I’d go so far as to say that the shortcomings of the merge proposal process are at least partially responsible for why the REST API hasn’t seen the level of adoption we’d hoped for. It’s managed to gain a moderate amount of popularity with WordPress development agencies, and a handful of plugins use it in some ways, but it never really entered into mainstream usage in the ways it could’ve.

In a project that prides itself on being willing to try new ideas, the merge proposal process has remained largely static for many years.

Gutenberg is the first opportunity since the REST API was merged where we can examine the shortcomings of the merge proposal process, and see how we can apply the original intent of it to the Gutenberg project’s scope and long term vision.

Merge Consultation

Over the last six months, Gutenberg project leads have been consulting with teams across the WordPress project. Helping them get involved when they didn’t have any Gutenberg experience, explaining how their focus fit into the vision for Gutenberg, and listening to feedback on where things needed to be improved. In many circumstances, this consultation process has been quite successful: the WordPress Media and REST API teams are great examples of that. Both teams have got up to speed on the Gutenberg project, and have provided their valuable experience to make it even better.

That’s not to say it’s been entirely successful. There’s been a lot of discussion about Gutenberg and Accessibility recently, much of it boils down to what Joe Dolson summarised as being “too little, too late”. He’s correct, the Accessibility team should’ve been consulted more closely, much earlier in the process, and that’s a mistake I expect to see rectified as the Gutenberg project moves into its next phase after WordPress 5.0. While Gutenberg has always aimed to prioritise accessibility, both providing tools to make the block editor more accessible, as well as encouraging authors to publish accessible content, there are still areas where we can improve.

While there’s much to be discussed following WordPress 5.0, we can already see now that different teams needed to be consulted at different points during the project. Where Gutenberg has aimed to consult with teams earlier than a previous feature plugin would’ve, we need to push that further, ensuring that teams are empowered to get involved earlier still in the process.

All feature plugins in the future, great and small, will benefit from this iteration.

Creating a framework for more fluid feedback over the entire lifecycle of a feature project is beneficial for everyone. WordPress teams can ensure that their feedback is taken on board at the right time, project leads gain experience across the broad range of teams that work on WordPress, and projects themselves are able to produce a better resulting feature.

They important thing to remember throughout all of this is that everything is an experiment. We can try an approach, discover the weaknesses, and iterate. We’re all only human, we all make mistakes, but every mistake is an opportunity to ensure the same mistake can’t happen again. Sometimes that means changing the software, and sometimes that means changing the processes that help build the software. Either way, we’re always able to iterate further, and make WordPress fun for everyone. 🙂

October 18, 2018

Performance Improvements with GPUs for Marine Biodiversity: A Cross-Tasman Collaboration

Identifying probable dispersal routes and for marine populations is a data and processing intensive task of which traditional high performance computing systems are suitable, even for single-threaded applications. Whilst processing dependencies between the datasets exist, a large level of independence between sets allows for use of job arrays to significantly improve processing time. Identification of bottle-necks within the code base suitable for GPU optimisation however had led to additional performance improvements which can be coupled with the existing benefits from job arrays. This small example offers an example of how to optimise single-threaded applications suitable for GPU architectures for significant performance improvements. Further development is suggested with the expansion of the GPU capability of the University of Melbourne’s “Spartan” HPC system.

A presentation to EResearchAustralasia 2018.

October 16, 2018

Children in Singapore will no longer be ranked by exam results. Here’s why | World Economic Forum

https://www.weforum.org/agenda/2018/10/singapore-has-abolished-school-exam-rankings-here-s-why The island nation is changing its educational focus to encourage school children to develop the life skills they will need when they enter the world of work.

October 12, 2018

International HPC Certification Program

The HPC community has always considered the training of new and existing HPC practitioners to be of high importance to its growth. The significance of training will increase even further in the era of Exascale when HPC encompasses even more scientic disciplines. This diversification of HPC practitioners challenges the traditional training approaches, which are not able to satisfy the specific needs of users, often coming from non-traditionally HPC disciplines and only interested in learning a particular set of skills. HPC centres are struggling to identify and overcome the gaps in users' knowledge. How should we support prospective and existing users who are not aware of their own knowledge gaps? We are working towards the establishment of an International HPC Certification program that would clearly categorize, define and examine them similarly to a school curriculum. Ultimately, we aim for the certificates to be recognized and respected by the HPC community and industry.

International HPC Certification Program, International Supercomputing Conference, Frankfurt, June, 2018

Julian Kunkel (University of Reading), Kai Himstedt (Universität Hamburg), Weronika Filinger (University of Edinburgh), Jean-Thomas Acquaviva (DDN), William Jalby (Université de Versailles Saint-Quentin), Lev Lafayette (University of Melbourne)

October 10, 2018

Audiobooks – September 2018

Lone Star: A History of Texas and the Texans by T. R. Fehrenbach

About 80% of the 40 hour book covers the period 1820-1880. Huge amounts of detail during then but skips over the rest quickly. Great stories though. 8/10

That’s Not English – Britishisms, Americanisms, and What Our English Says About Us by Erin Moore

A series of short chapters (usually one per word) about how the English language is used differently in England from the US. Fun light read. 7/10

The Complacent Class: The Self-Defeating Quest for the American Dream by Tyler Cowen

How American culture (and I’d extend that to countries like NZ) has stopped innovating and gone the safe route in most areas. Main thesis is that pressure is building up and things may break hard. Interesting 8/10

A History of Britain, Volume 2 : The British Wars 1603 – 1776 by Simon Schama

Covering the Civil War, Glorious Revolution and bits of the early empire and American revolution. A nice overview. 7/10

I Find Your Lack of Faith Disturbing: Star Wars and the Triumph of Geek Culture by A. D. Jameson

A personal account of the author’s journey though Geekdom (mainly of the Sci-Fi sort) mixed in with a bit of analysis of how the material is deeper than critics usually credit. 7/10

Share

October 08, 2018

Open Source Firmware Conference 2018

I recently had the pleasure of attending the 2018 Open Source Firmware Conference in Erlangen, Germany. Compared to other more general conferences I've attended in the past, the laser focus of OSFC on firmware and especially firmware security was fascinating. Seeing developers from across the world coming together to discuss how they are improving their corner of the stack was great, and I've walked away with plenty of new knowledge and ideas (and several kilos of German food and drink..).


What was especially exciting though is that I had the chance to talk about my work on Petitboot and what's happened from the POWER8 launch until now. If you're interested in that, or seeing how I talk after 36 hours of travel, check it out here:


OSFC have made all the talks from the first two days available in a playlist on Youtube
If you're after a few suggestions there was, in no particular order:

Ryan O'Leary giving an update on Linuxboot - also known as NERF, Google's approach to a Linux bootloader all written in Go.

Subrate Banik talking about porting Coreboot on top of Intel's FSP

Ron Minnich describing his work with "rompayloads" on Coreboot

Vadmin Bendebury describing Google's "Secure Microcontroller" Chip

Facebook presenting their use of Linuxboot and "systemboot"

And heaps more, check out the full playlist!

Not The Best Customer Service (laptop.com.au)

You would think with a website like laptop.com.au you would be sitting on a gold mine of opportunity. It would take real effort not to turn such a domain advantage into a real advantage, to become the country's specialist and expert provider of laptops. But alas, some effort is required in this regard and it involves what, in my considered opinion, is not doing the right thing. I leave you, gentle reader, to form your own opinion on the matter from the facts provided.

In mid-August 2018 I purchased a laptop from said provider. I didn't require anything fancy, but it did need to be light and small. The Lenovo Yoga 710-11ISK for $699 seemed to fit the bill. The dispatch notice was sent on August 14, and on August 21st I received the item and noticed that there were a few things wrong. Firstly, the processor was nowhere near as powerful as advertised (and no surprise there - they're advertising the bust-speed of a water-cooled processor, not an air-cooled small laptop). Further, the system came with half of the advertised 8GB of RAM.

With the discrepancy pointed out they offered what I considered a paltry sum of $100 - which would be quite insufficient for the loss of performance, and it was not the kind of system that could be upgraded with ease. Remarkably they made the claim "We would offer to swap over, however, it's expensive to ship back and forth and we don't have another in stock at this time". I asked that if this was the case why they were still advertising the supposedly absent model on their website (and, at the time of writing, October 8), it is apparently still available. I pointed out that their own terms and conditions stated: "A refund, repair, exchange or credit is available if on arrival the goods are advised faulty or the model or the specifications are incorrect", which was certainly the case here.

Receiving no reply several days later I had to contact them again. The screen on the system had completely ceased to function. I demanded that the refund the cost of the laptop plus postage, as per their own terms and conditions. The system was faulty and the specifications are incorrect. They offered to replace the machine. I told them I preferred a refund, as I now had reasonable doubts about their quality control, as per Victorian consumer law.

I sent the laptop, express post with signature, and waited. A week later I had to contact them again and provided the Australia Post tracking record to show that it had been delivered (but not collected). It was at the point that, instead of providing a refund as I had requested, they sent a second laptop, completely contrary to my wishes. They responded that they had "replaced machine with original spec that u ordered. Like new condition" and that "We are obliged under consumer law to provide a refund within 30 days of purchase" (any delays were due to their inaction). At that point a case was opened at the Commonwealth Bank (it was paid via credit card), and Consumer Affair Victoria.

But it gets better. They sent the wrong laptop again. This time with a completely different processor, and significantly heavier and larger. It was pointed out to them that they have sent the wrong machine, twice, and the second time contrary to my requests. It was pointed out to them that all they had to do was provide a refund as requested for the machine and my postage costs. It was pointed out that it is my fault that you sent the wrong machine and that was their responsibility. It was pointed out that that it was not my fault that they sent a second, wrong, machine, contrary to my request, and that, again, their responsibility. Indeed, they could benefit by having someone look at their business processes and quality assurance - because there has been many years of this retailer showing less than optimal customer service.

At this point, they buckled and agreed to provide a full refund if I sent the second laptop back - which I have done and will update this 'blog post as the story unfolds.

Now whilst some of you gentle readers may think that surely it couldn't have been that bad, and surely there's another side to this story. So it is in the public interest and in the principle of disclosure and transparency, that I provide a full set of the correspondence as a text file attached. You can make up your own mind.

October 03, 2018

WordPress 5.0 Needs You!

Yesterday, we started the WordPress 5.0 release cycle with an announcement post.

It’s a very exciting time to be involved in WordPress, and if you want to help make it the best, now’s an excellent opportunity to jump right in.

A critical goal of this release cycle is transparency.

As a member of the WordPress 5.0 leadership team, the best way for me to do my job is to get feedback from the wider WordPress community as early, and as quickly as possible. I think I speak for everyone on the leadership team when I say that we all feel the same on this. We want everyone to be able to participate, which will require some cooperation from everyone in the wider WordPress community.

The release post was published as soon as it was written, we wanted to get it out quickly, so everyone could be aware of what’s going on. Publishing quickly does mean that we’re still writing the more detailed posts about scope, and timeline, and processes. Instead of publishing a completed plan all at once, we intentionally want to include everyone from the start, and evolve plans as we get feedback.

With no other context, the WordPress 5.0 timeline of “release candidate in about a month” would be very short, which is why we’ve waited until Gutenberg had proved itself before setting a timeline. As we mentioned in the post, WordPress 5.0 will be “WordPress 4.9.8 + Gutenberg”. The Gutenberg plugin is running on nearly 500k sites, and WordPress 4.9.8 is running on millions of sites. For comparison, it’s considered a well tested major version if we see 20k installs before the final release date. Gutenberg is a bigger change than we’ve done in the past, so should be held to a higher standard, and I think we can agree that 500k sites is a pretty good test base: it arguably meets, or even exceeds that standard.

We can have a release candidate ready in a month.

The Gutenberg core team are currently focussed on finishing off the last few features. The Gutenberg plugin has evolved exceedingly quickly thanks to their work, it’s moved so much faster than anything we’ve done in WordPress previously. As we transition to bug fixing, you should expect to see the same rapid improvement.

The block editor’s backwards compatibility with the classic editor is important, of course, and the Classic Editor plugin is a part of that: if you have a site that doesn’t yet work with the block editor, please go ahead and install the plugin. I’d be happy to see the Classic Editor plugin getting 10 million or more installs, if people need it. That would both show a clear need for the classic interface to be maintained for a long time, and because it’s the official WordPress plugin for doing it, we can ensure that it’s maintained for as long as it’s needed. This isn’t a new scenario to the WordPress core team, we’ve been backporting security fixes to WordPress 3.7 for years. We’re never going to leave site owners out in the cold there, and exactly the same attitude applies to the Classic Editor plugin.

The broader Gutenberg project is a massive change, and WordPress is a big ship to turn.

It’s going to take years to make this transition, and it’s okay if WordPress 5.0 isn’t everything for everyone. There’ll be a WordPress 5.1, and 5.2, and 5.3, and so on, the block editor will continue to evolve to work for more and more people.

My role in WordPress 5.0 is to “generally shepherd the merge”. I’ve built or guided some of the most complex changes we’ve made in Core in recent years, and they’ve all been successful. I don’t intend to change that record, WordPress 5.0 will only be released when I’m as confident in it as I was for all of those previous projects.

Right now, I’m asking everyone in the WordPress community for a little bit of trust, that we’re all working with the best interests of WordPress at heart. I’m also asking for a little bit of patience, we’re only human, we can only type so fast, and we do need to sleep every now and then. 😉

WordPress 5.0 isn’t the finish line, it’s the starter pistol.

This is a marathon, not a sprint, and the goal is to set WordPress up for the next 15 years of evolution. This can only happen one step at a time though, and the best way to get there will be by working together. We can have disagreements, we can have different priorities, and we can still come together to create the future of WordPress.

October 02, 2018

CNC made close up lens filter holder

Close up filters attach to the end of a camera lens and allow you to take photos closer to the subject than you normally would have been able to do. This is very handy for electronics and other work as you can get clear images of circuit boards and other small detail. I recently got a collection of 3 such filters which didn't come with any sort of real holder, the container they shipped in was not really designed for longer term use.


The above is the starting design for a filter holder cut in layers from walnut and stacked together to create the enclosure. The inside is shown below where the outer diameter can hold the 80mm black ring and the inner circles are 70mm and are there to keep the filters from touching each other. Close up filters can be quite fish eyed looking with a substantial curve to the lens on the filter, so a gap is needed to keep each filter away from the next one. A little felt is used to cushion the filter from the walnut itself which adds roughly 1.5mm to the design so the felt layer all have space to live as well.



The bottom has little feet which extend slightly beyond the tangent of the circle so they both make good contact with the ground and there is no rocking. Using two very cheap hinges works well in this design to try to minimize the sideways movement (slop) in the hinges themselves. A small leather strap will finish the enclosure off allowing it to be secured closed.

It is wonderful to be able to turn something like this around. I can only imagine what the world looks like from the perspective of somebody who is used to machining with 5 axis CNC.



September 28, 2018

Helping Migrants to Australia

The end of the school year is fast approaching with the third term either over or about to end and the start of the fourth term looming ahead. There never seems to be enough time in the last term with making sure students have met all their learning outcomes for the year and with final […]

Codec 2 2200 Candidate D

Every time I start working on Deep Learning and Codec 2 I get side tracked! This time, I started developing a reference codec that could be used to explore machine learning, however the reference codec was sounding pretty good early in it’s development so I have pushed it through to a fully quantised state. For lack of a better name it’s called candidate D, as that’s where I am up to in a series of codec prototypes.

The previous Codec 2 2200 post described candidate C. That also evolved from a “quick effort” to develop a reference codec to explore my deep learning ideas.

Learning about Vector Quantisation

This time, I explored Vector Quantisation (VQ) of spectral magnitude samples. I feel my VQ skills are weak, so did a bit of reading. I really enjoy learning, especially in areas I have been fooling around for a while but never really understood. It’s a special feeling when the theory clicks into place with the practical.

So I have these vectors of K=40 spectral magnitude samples, that I want to quantise. To get a feel for the data I started out by looking at smaller 2 and 3 dimensional vectors. Forty dimensions is a bit much to handle, so I started out by plotting smaller slices. Here are 2D and 3D scatter plots of adjacent samples in the vector:


The data is highly correlated, almost a straight line relationship. An example of a 2-bit, 2D vector quantiser for this data might be the points (0,0) (20,20) (30,30) (40,40). Consider representing the same data with two 1D (scalar) quantisers over the same 2 bit range (0,20,30,40). This would take 4 bits in total, and be wasteful as it would represent points that would never occur, such as (60,0).

[1] helped me understand the relationship between covariance and VQ, using 2D vectors. For Candidate D I extended this to K=40 dimensions, the number of samples I am using for the spectral magnitudes. Then [2] (thirty year old!) paper how the DCT relates to vector quantisation and the eigenvector/value rotations described in [1]. I vaguely remember snoring my way through eigen-thingies at math lectures in University!

My VQ work to date has used minimum Mean Square Error (MSE) to train and match vectors. I have been uncomfortable with MSE matching for a while, as I have observed poor choices in matching vectors to speech. For example if the target vector falls off sharply at high frequencies (say a LPF at 3500 Hz), the VQ will try to select a vector that matches that fall off, and ignore smaller, more perceptually important features like formants.

VQs are often trained to minimise the average error. They tend to cluster VQ points closer to those samples that are more likely to occur. However I have found that beneath a certain threshold, we can’t hear the quantisation error. In Codec 2 it’s hard to hear any distortion when spectral magnitudes are quantised to 6 dB steps. This suggest that we are wasting bits with fine quantiser steps, and there may be better ways to design VQs, for example a uniform grid of points that covers a few standard deviations of data on the scatter plots above.

I like the idea of uniform quantisation across vector dimensions and the concepts I learnt during this work allowed me to do just that. The DCT effectively lets me use scalar quantisation of each vector element, so I can easily choose any quantiser shape I like.

Spectral Quantiser

Candidate D uses a similar design and bit allocation to Candidate C. Candidate D uses K=40 resampling of the spectral magnitudes, to help preserve narrow high frequency formants that are present for low pitch speakers like hts1a. The DCT of the rate K vectors is computed, and quantised using a Huffman code.

There are not enough bits to quantise all of the coefficients, so we stop when we run out of bits, typically after 15 or 20 (out of a total of 40) DCTs. On each frame the algorithm tries direct or differential quantisation, and chooses the method with the lowest error.

Results

I have a couple of small databases that I use for listening tests (about 15 samples in total). I feel Candidate D is better than Codec 2 1300, and also Codec 2 2400 for most (but not all) samples.

In particular, Candidate D handles samples with lots of low frequency energy better, e.g. cq_ref and kristoff in the table below.

Sample 1300 2400 2200 D
cq_ref Listen Listen Listen
kristoff Listen Listen Listen
me Listen Listen Listen
vk5local_1 Listen Listen Listen
ebs Listen Listen Listen

For a high quality FreeDV mode I want to improve speech quality over FreeDV 1600 (which uses Codec 2 1300 plus some FEC bits), and provide better robustness to different speakers and recording conditions. As you can hear – there is a significant jump in quality between the 1300 bit/s codec and candidate D. Implemented as a FreeDV mode, it would compare well with SSB at high SNRs.

Next Steps

There are many aspects of Candidate D that could be explored:

  • Wideband audio, like the work from last year.
  • Back to my original aim of exploring deep learning with Codec 2.
  • Computing the DCT coefficients from the rate L (time varying) magnitude samples.
  • Better time/freq quantisation using a 2D DCT rather than the simple difference in time scheme used for Candidate D.
  • Porting to C and developing a real time FreeDV 2200 mode.

The current candidate D 2200 codec is implemented in Octave, so porting to C is required before it is usable for real world applications, plus some more C to integrate with FreeDV.

If anyone would like to help, please let me know. It’s fairly straight forward C coding, I have already done the DSP. You’ll learn a lot, and be part of the open source future of digital radio.

Reading Further

[1] A geometric interpretation of the covariance matrix, really helped me understand what was going on with VQ in 2 dimensions, which can then be extended to larger dimensions.

[2] Vector Quantization in Speech Coding, Makhoul et al.

[3 Codec 2 Wideband, previous DCT based Codec 2 Work.

September 27, 2018

2018 Linux Security Summit North America: Wrapup

The 2018 Linux Security Summit North America (LSS-NA) was held last month in Vancouver, BC.

Attendance continued to grow this year, with a record of 220+ attendees.  Our room was upgraded as a result, with spectacular views.

LSS-NA 2018 Vancouver BC

Linux Security Summit NA 2018, Vancouver,BC

We also had many great proposals and the schedule ended up being a very tight fit.  We’ve asked for an extra day for LSS-NA next year — here’s hoping.

Slides of all presentations are available here: https://events.linuxfoundation.org/events/linux-security-summit-north-america-2018/program/slides/

Videos may be found in this youtube playlist.

Once again, as is typical, the conference was focused around development, somewhat uniquely in the world of security conferences.  It’s interesting to see more attention seemingly being paid to the lower parts of the stack: secure booting, firmware, and hardware roots of trust, as well as the continued efforts in hardening the kernel.

LWN provided some excellent coverage of LSS-NA:

Paul Moore has a brief writeup here.

Thanks to everyone involved in the event for 2018: the speakers, attendees, the program committee, the sponsors, and the organizing team at the Linux Foundation.  LSS-NA would not be possible without all of you!

Our interwoven ancestry

In 2008 a new group of human ancestors – the Denisovans, were defined on the basis of a single finger knuckle (phalanx) bone discovered in Denisova cave in the Altai mountains of Siberia. A molar tooth, found at Denisova cave earlier (in 2000) was determined to be of the same group. Since then extensive work […]

September 24, 2018

Straight White Guy Discovers Diversity and Inclusion Problem in Open Source

This is a bit of strange post for me to write, it’s a topic I’m quite inexperienced in. I’ll warn you straight up: there’s going to be a lot of talking about my thought processes, going off on tangents, and a bit of over-explaining myself for good measure. Think of it something like high school math, where you had to “show your work”, demonstrating how you arrived at the answer. 20 years later, it turns out there really is a practical use for high school math. 😉


I’m Gary. I come from a middle-class, white, Australian family. My parents both worked, but also had the time to encourage me to do well in school. By way of doing well in school, I was able to get into a good university, I could support myself on a part time job, because I only had to pay my rent and bar tab. There I met many friends, who’ve helped me along the way. From that, I’ve worked a series of well paid tech jobs, allowing me to have savings, and travel, and live in a comfortable house in the suburbs.

I’ve learned that it’s important for me to acknowledge the privileges that helped me get here. As a “straight white male”, I recognise that a few of my privileges gave me a significant boost that many people aren’t afforded. This is backed up by the data, too. Men are paid more than women. White women are paid more than black women. LGBT people are more likely to suffer workplace bullying. The list goes on and on.

Some of you may’ve heard the term “privilege” before, and found it off-putting. If that’s you, here’s an interesting analogy, take a moment to read it (and if the title bugs you, please ignore it for a moment, we’ll get to that), then come back.

Welcome back! So, are you a straight white male? Did that post title make you feel a bit uncomfortable at being stereotyped? That’s okay, I had a very similar reaction when I first came across the “straight white male” stereotype. I worked hard to get to where I am, trivialising it as being something I only got because of how I was born hurts. The thing is, this is something that many people who aren’t “straight white males” experience all the time. I have a huge amount of respect for people who have to deal with that on daily basis, but are still able to be absolute bosses at their job.

My message to my dudes here is: don’t sweat it. A little bit of a joke at your expense is okay, and I find it helps me see things from another person’s perspective, in what can be a light-hearted, friendly manner.

Diversity Makes WordPress Better

My job is to build WordPress, which is used by just shy of a third of the internet. That’s a lot of different people, building a lot of different sites, for a lot of different purposes. I can draw on my experiences to imagine all of those use cases, but ultimately, this is a place where my privilege limits me. Every time I’ve worked on a more diverse team, however, I’m exposed to a wider array of experiences, which makes the things we build together better.

Of course, I’m not even close to being the first person to recognise how diversity can improve WordPress, and I have to acknowledge the efforts of many folks across the community. The WordPress Community team are doing wonderful work helping folks gain confidence with speaking at WordPress events. WordCamps have had a Code of Conduct for some time, and the Community team are working creating a Code of Conduct for the entire WordPress project. The Design team have built up excellent processes and resources to help folks get up to speed with helping design WordPress. The Core Development team run regular meetings for new developers to learn how to write code for WordPress.

We Can Do Better. I Can Do Better.

As much as I’d love it to be, the WordPress community isn’t perfect. We have our share of problems, and while I do believe that everyone in our community is fundamentally good, we don’t always do our best. Sometimes we’re not as welcoming, or considerate, as we could be. Sometimes we don’t take the time to consider the perspectives of others. Sometimes it’s just a bunch of tech-dude-bros beating their chests. 🙃

Nobody wins when we’re coming from a place of inequality.

So, this post is one of my first steps in recognising there’s a real problem, and learning about how I can help make things better. I’m not claiming to know the answers, I barely know where to start. But I’m hoping that my voice, added to the many that have come before me, and the countless that will come after, will help bring about the changes we need.

September 22, 2018

Scared Weird Frozen Guy

Share

The true life story of a kid from Bribie Island (I’ve been there!) running a marathon in Antartica, via being a touring musical comedian, doing things like this:

This book is an interesting and light read, and came kindly recommended by Michael Carden, who pretty much insisted I take the book off him at a cafe. I don’t regret reading it and would recommend it to people looking for a light autobiography for a rainy (and perhaps cold) evening or two.

Oh, and the Scared Weird Little Guys of course are responsible for this gem…

This book is highly recommended and now I really want to go for a run.

Scared Weird Frozen Guy Book Cover Scared Weird Frozen Guy
Rusty Berther
Comedians
2012
325

After 20 incredible years as part of a musical comedy duo, Scared Weird Little Guy, Rusty Berther found himself running a marathon in Antarctica. What drove him to this? In this hilarious and honest account of his life as a Scared Weird Little Guy, and his long journey attempting an extreme physical and mental challenge at the bottom of the world, Rusty examines where he started from, and where he just might be going to.

Share

September 20, 2018

Words Have Meanings

As a follow-up to my post with Suggestions for Trump Supporters [1] I notice that many people seem to have private definitions of words that they like to use.

There are some situations where the use of a word is contentious and different groups of people have different meanings. One example that is known to most people involved with computers is “hacker”. That means “criminal” according to mainstream media and often “someone who experiments with computers” to those of us who like experimenting with computers. There is ongoing discussion about whether we should try and reclaim the word for it’s original use or whether we should just accept that’s a lost cause. But generally based on context it’s clear which meaning is intended. There is also some overlap between the definitions, some people who like to experiment with computers conduct experiments with computers they aren’t permitted to use. Some people who are career computer criminals started out experimenting with computers for fun.

But some times words are misused in ways that fail to convey any useful ideas and just obscure the real issues. One example is the people who claim to be left-wing Libertarians. Murray Rothbard (AKA “Mr Libertarian”) boasted about “stealing” the word Libertarian from the left [2]. Murray won that battle, they should get over it and move on. When anyone talks about “Libertarianism” nowadays they are talking about the extreme right. Claiming to be a left-wing Libertarian doesn’t add any value to any discussion apart from demonstrating the fact that the person who makes such a claim is one who gives hipsters a bad name. The first time penny-farthings were fashionable the word “libertarian” was associated with left-wing politics. Trying to have a sensible discussion about politics while using a word in the opposite way to almost everyone else is about as productive as trying to actually travel somewhere by penny-farthing.

Another example is the word “communist” which according to many Americans seems to mean “any person or country I don’t like”. It’s often invoked as a magical incantation that’s supposed to automatically win an argument. One recent example I saw was someone claiming that “Russia has always been communist” and rejecting any evidence to the contrary. If someone was to say “Russia has always been a shit country” then there’s plenty of evidence to support that claim (Tsarist, communist, and fascist Russia have all been shit in various ways). But no definition of “communism” seems to have any correlation with modern Russia. I never discovered what that person meant by claiming that Russia is communist, they refused to make any comment about Russian politics and just kept repeating that it’s communist. If they said “Russia has always been shit” then it would be a clear statement, people can agree or disagree with that but everyone knows what is meant.

The standard response to pointing out that someone is using a definition of a word that is either significantly different to most of the world (or simply inexplicable) is to say “that’s just semantics”. If someone’s “contribution” to a political discussion is restricted to criticising people who confuse “their” and “there” then it might be reasonable to say “that’s just semantics”. But pointing out that someone’s writing has no meaning because they choose not to use words in the way others will understand them is not just semantics. When someone claims that Russia is communist and Americans should reject the Republican party because of their Russian connection it’s not even wrong. The same applies when someone claims that Nazis are “leftist”.

Generally the aim of a political debate is to convince people that your cause is better than other causes. To achieve that aim you have to state your cause in language that can be understood by everyone in the discussion. Would the person who called Russia “communist” be more or less happy if Russia had common ownership of the means of production and an absence of social classes? I guess I’ll never know, and that’s their failure at debating politics.

September 17, 2018

LUV October 2018 Workshop: CoCalc and Nectar

Oct 20 2018 12:30
Oct 20 2018 16:30
Oct 20 2018 12:30
Oct 20 2018 16:30
Location: 
Infoxchange, 33 Elizabeth St. Richmond

CoCalc and Nectar

Paul Leopardi will give a talk and live demo of the cloud services that he is currently using, CoCalc to host Jupyter notebooks, Python code, and similar and the Nectar research cloud service to host a PostgreSQL database and a Plotly Dash dashboard to disseminate his research results:

The meeting will be held at Infoxchange, 33 Elizabeth St. Richmond 3121.  Late arrivals please call (0421) 775 358 for access to the venue.

LUV would like to acknowledge Infoxchange for the venue.

Linux Users of Victoria is a subcommittee of Linux Australia.

October 20, 2018 - 12:30

read more

LUV October 2018 Main Meeting: Non-alphabetic languages / Haskell

Oct 2 2018 18:30
Oct 2 2018 20:30
Oct 2 2018 18:30
Oct 2 2018 20:30
Location: 
Kathleen Syme Library, 251 Faraday Street Carlton VIC 3053

PLEASE NOTE RETURN TO ORIGINAL START TIME

6:30 PM to 8:30 PM Tuesday, October 2, 2018
Training Room, Kathleen Syme Library, 251 Faraday Street Carlton VIC 3053

Speakers:

  • Wen Lin, Using Linux/FOSS in non-alphabetic languages like Chinese
  • Shannon Pace, Haskell

Using Linux/FOSS in non-alphabetic languages like Chinese

Wen will discuss different input methods on a keyboard first designed only for alphabetic-base languages.

Many of us like to go for dinner nearby after the meeting, typically at Brunetti's or Trotters Bistro in Lygon St.  Please let us know if you'd like to join us!

Linux Users of Victoria is a subcommittee of Linux Australia.

October 2, 2018 - 18:30

read more

September 16, 2018

The Mission: Democratise Publishing

It’s exciting to see the Drupal Gutenberg project getting under way, it makes me proud of the work we’ve done ensuring the flexibility of the underlying Gutenberg architecture. One of the primary philosophies of Gutenberg’s technical architecture is platform agnosticism, and we can see the practical effects of this practice coming to fruition across a variety of projects.

Yoast are creating new features for the block editor, as well as porting existing features, which they’re able to reuse in the classic editor.

Outside of WordPress Core, the Automattic teams who work on Calypso have been busy adding Gutenberg support, in order to make the block editor interface available on WordPress.com. Gutenberg and Calypso are large JavaScript applications, built with strong opinions on design direction and technical architecture, and having significant component overlap. That these two projects can function together at all is something of an obscure engineering feat that’s both difficult and overwhelming to appreciate.

If we reached the limit of Gutenberg’s platform agnosticism here, it would still be a successful project.

But that’s not where the ultimate goals of the Gutenberg project stand. From early experiments in running the block editor as a standalone application, to being able to compile it into a native mobile component, and now seeing it running on Drupal, Gutenberg’s technical goals have always included a radical level of platform agnosticism.

Better Together

Inside the WordPress world, significant effort and focus has been on ensuring backwards compatibility with existing WordPress sites, plugins, and practices. Given that WordPress is such a hugely popular platform, it’s exceedingly important to ensure this is done right. With Gutenberg expanding outside of the WordPress world, however, we’re seeing different focuses and priorities arise.

The Gutenberg Cloud service is a fascinating extension being built as part of the Drupal Gutenberg project, for example. It provides a method for new blocks to be shared and discovered, the sample hero block sets a clear tone of providing practical components that can be rapidly put together into a full site. While we’ve certainly seen similar services appear for the various site builder plugins, this is the first one (that I’m aware of, at least) build specifically for Gutenberg.

By making the Gutenberg experience available for everyone, regardless of their technical proficiency, experience, or even preferred platform, we pave the way for a better future for all.

Democratising Publishing

You might be able to guess where this is going. 😉

WordPress’ mission is to “democratise publishing”. It isn’t to “be the most popular CMS”, or to “run on old versions of PHP”, though it’s easy to think that might be the case on the surface. That these statements are true is simply a side effect of the broader principle: All people, regardless of who they are or where they come from, should be able to publish their content as part of a free and open web.

The WordPress mission is not to “democratise publishing with WordPress”.

WordPress has many advantages that make it so popular, but hoarding those to ourselves doesn’t help the open web, it just creates more silos. The open web is the only platform on which publishing can be democratised, so it makes sense for Gutenberg to work anywhere on the open web, not just inside WordPress. Drupal isn’t a competitor here, we’re all working towards the same goal, the different paths we’ve taken have made the open web stronger as a whole.

Much as the block editor has been the first practical implementation of the Gutenberg architecture, WordPress is simply the first practical integration of the block editor into a CMS. The Gutenberg project will expand into site customisation and theming next, and while there’s no requirement that Drupal make use of these, I’d be very interested to see what they came up with if they did. Bringing together our many years of experience in tackling these complex problems can only make the end result better.

I know I’m looking forward to all of us working together for the betterment of the open web.

September 14, 2018

Porting a LDPC Decoder to a STM32 Microcontroller

A few months ago, FreeDV 700D was released. In that post, I asked for volunteers to help port 700D to the STM32 microcontroller used for the SM1000. Don Reid, W7DMR stepped up – and has been doing a fantastic job porting modules of C code from the x86 to the STM32.

Here is a guest post from Don, explaining how he has managed to get a powerful LDPC decoder running on the STM32.

LDPC for the STM32

The 700D mode and its LDPC function were developed and used on desktop (x86) platforms. The LDPC decoder is implemented in the mpdecode_core.c source file.

We’d like to run the decoder on the SM1000 platform which has an STM32F4 processor. This requires the following changes:

  • The code used doubles in several places, while the stm32 has only single precision floating point hardware.
  • It was thought that the memory used might be too much for a system with just 192k bytes of RAM.
  • There are 2 LDPC codes currently supported, HRA_112_112 used in 700D and, H2064_516_sparse used for Balloon Telemetry. While only the 700D configuration needed to work on the STM32 platform, any changes made to the mainstream code needed to work with the H2064_516_sparse code.

Testing

Before making changes it was important to have a well defined test process to validate new versions. This allowed each change to be validated as it was made. Without this the final debugging would likely have been very difficult.

The ldpc_enc utility can generate standard test frames and the ldpc_dec utility receive the frames and measure bit errors. So errors can be detected directly and BER computed. ldpc_enc can also output soft decision symbols to emulate what the modem would receive and pass into the LDPC decoder. A new utility ldpc_noise was written to add AWGN to the sample values between the above utilities. here is a sample run:

$ ./ldpc_enc /dev/zero - --sd --code HRA_112_112 --testframes 100 | ./ldpc_noise - - 1 | ./ldpc_dec - /dev/null --code HRA_112_112 --sd --testframes
single sided NodB = 1.000000, No = 1.258925
code: HRA_112_112
code: HRA_112_112
Nframes: 100
CodeLength: 224 offset: 0
measured double sided (real) noise power: 0.640595
total iters 3934
Raw Tbits..: 22400 Terr: 2405 BER: 0.107
Coded Tbits: 11200 Terr: 134 BER: 0.012

ldpc_noise is passed a “No” (N-zero) level of 1dB, Eb=0, so Eb/No = -1, and we get a 10% raw BER, and 1% after LDPC decoding. This is a typical operating point for 700D.

A shell script (ldpc_check) combines several runs of these utilities, checks the results, and provides a final pass/fail indication.

All changes were made to new copies of the source files (named *_test*) so that current users of codec2-dev were not disrupted, and so that the behaviour could be compared to the “released” version.

Unused Functions

The code contained several functions which are not used anywhere in the FreeDV/Codec2 system. Removing these made it easier to see the code that was used and allowed the removal of some variables and record elements to reduce the memory used.

First Compiles

The first attempt at compiling for the stm32 platform showed that the the code required more memory than was available on the processor. The STM32F405 used in the SM1000 system has 128k bytes of main RAM.

The largest single item was the DecodedBits array which was used to saved the results for each iteration, using 32 bit integers, one per decoded bit.

    int *DecodedBits = calloc( max_iter*CodeLength, sizeof( int ) );

This used almost 90k bytes!

The decode function used for FreeDV (SumProducts) used only the last decoded set. So the code was changed to save only one pass of values, using 8 bit integers. This reduced the ~90k bytes to just 224 bytes!

The FreeDV 700D mode requires on LDPC decode every 160ms. At this point the code compiled and ran but was too slow – using around 25ms per iteration, or 300 – 2500ms per frame!

C/V Nodes

The two main data structures of the LDPC decoder are c_nodes and v_nodes. Each is an array where each node contains additional arrays. In the original code these structures used over 17k bytes for the HRA_112_112 code.

Some of the elements of the c and v nodes (index, socket) are indexes into these arrays. Changing these from 32 bit to 16 bit integers and changing the sign element into a 8 bit char saved about 6k bytes.

The next problem was the run time. Each 700D frame must be fully processed in 160 ms and the decoder was taking several times this long. The CPU load was traced to the phi0() function, which was calling two maths library functions. After optimising the phi0 function (see below) the largest use of time was the index computations of the nested loops which accessed these c and v node structures.

With each node having separate arrays for index, socket, sign, and message these indexes had to be computed separately. By changing the node structures to hold an array of sub-nodes instead this index computation time was significantly reduced. An additional benefit was about a 4x reduction in the number of memory blocks allocated. Each allocation block includes additional memory used by malloc() and free() so reducing the number of blocks reduces memory use and possible heap fragmentation.

Additional time was saved by only calculating the degree elements of the c and v nodes at start-up rather than for every frame. That data is kept in memory that is statically allocated when the decoder is initialized. This costs some memory but saves time.

This still left the code calling malloc several hundred times for each frame and then freeing that memory later. This sort of memory allocation activity has been known to cause troubles in some embedded systems and is usually avoided. However the LDPC decoder needed too much memory to allow it to be statically allocated at startup and not shared with other parts of the code.

Instead of allocating an array of sub-nodes for each c or v node, a single array of bytes is passed in from the parent. The initialization function which calculates the degree elements of the nodes also counts up the memory space needed and reports this to its caller. When the decoder is called for a frame, the node’s pointers are set to use the space of this array.

Other arrays that the decoder needs were added to this to further reduce the number of separate allocation blocks.

This leaves the decisions of how to allocate and share this memory up to a higher level of the code. The plan is to continue to use malloc() and free() at a higher level initially. Further testing can be done to look for memory leakage and optimise overall memory usage on the STM32.

PHI0

There is a non linear function named “phi0” which is called inside several levels of nested loops within the decoder. The basic operation is:

   phi0(x) = ln( (e^x + 1) / (e^x - 1) )

The original code used double precision exp() and log(), even though the input, output, and intermediate values are all floats. This was probably an oversight. Changing to the single single precision versions expf() and logf() provided some improvements, but not enough to meet our CPU load goal.

The original code used piecewise approximation for some input values. This was extended to cover the full range of inputs. The code was also structured differently to make it faster. The original code had a sequence of if () else if () else if () … This can take a long time when there are many steps in the approximation. Instead two ranges of input values are covered with linear steps that is implemented with table lookups.

The third range of inputs in non linear and is handled by a binary tree of comparisons to reduce the number of levels. All of this code is implemented in a separate file to allow the original or optimised version of phi0 to be used.

The ranges of inputs are:

             x >= 10      result always 0
      10   > x >=  5      steps of 1/2
       5   > x >= 1/16    steps of 1/16
    1/16   > x >= 1/4096  use 1/32, 1/64, 1/128, .., 1/4096
    1/4096 > x            result always 10

The range of values that will appear as inputs to phi0() can be represented with as fixed point value stored in a 32 bit integer. By converting to this format at the beginning of the function the code for all of the comparisons and lookups is reduced and uses shifts and integer operations. The step levels use powers of 2 which let the table index computations use shifts and make the fraction constants of the comparisons simple ones that the ARM instruction set can create efficiently.

Misc

Two of the configuration values are scale factors that get multiplied inside the nested loops. These values are 1.0 in both of the current configurations so that floating point multiply was removed.

Results

The optimised LDPC decoder produces the same output BER as the original.

The optimised decoder uses 12k of heap at init time and needs another 12k of heap at run time. The original decoder just used heap at run time, that was returned after each call. We have traded off the use of static heap to clean up the many small heap allocations and reduce execution time. It is probably possible to reduce the static space further perhaps at the cost of longer run times.

The maximum time to decode a frame using 100 iterations is 60.3 ms and the median time is 8.8 ms, far below our budget of 160ms!

Future Possibilities

The remaining floating point computations in the decoder are addition and subtraction so the values could be represented with fix point values to eliminate the floating point operations.

Some values which are computed from the configuration (degree, index, socket) are constants and could be generated at compile time using a utility called by cmake. However this might actually slow down the operation as the index computations might become slower.

The index and socket elements of C and V nodes could be pointers instead of indexes into arrays.

Experiments would be required to ensure these changes actually speed up the decoder.

Bio

Don got his first amateur license in high school but was soon distracted with getting an engineering degree (BSEE, Univ. of Washington), then family and life. He started his IC design career with the CPU for the HP-41C calculator. Then came ICs for printers and cameras, work on IC design tools, and some firmware for embedded systems. Exposure to ARES public service lead to a new amateur license – W7DMR and active involvement with ARES. He recently retired after 42 years and wanted to find an open project that combined radio, embedded systems and DSP.

Don lives in Corvallis, Oregon, USA a small city with the state technical university and several high tech companies.

Open Source Projects and Volunteers

Hi it’s David back again ….

Open source projects like FreeDV and Codec 2 rely on volunteers to make them happen. The typical pattern is people get excited, start some work, then drift away after a few weeks. Gold is the volunteer that consistently works week in, week out until their particular project is done. The number of hours/week doesn’t matter – it’s the consistency that is really helpful to the projects. I have a few contributors/testers/users of FreeDV in this category and I appreciate you all deeply – thank you.

If you would like to help out, please contact me. You’ll learn a lot and get to work towards an open source future for HF radio.

If you can’t help out technically, but would like to support this work, please consider Patreon or PayPal.

Reading Further

LDPC using Octave and the CML library. Our LDPC decoder comes from Coded Modulation Library (CML), which was originally used to support Matlab/Octave simulations.

Horus 37 – High Speed SSTV Images. The CML LDPC decoder was converted to a regular C library, and used for sending images from High Altitude Balloons.

Steve Ports an OFDM modem from Octave to C. Steve is another volunteer who put in a fine effort on the C coding of the OFDM modem. He recently modified the modem to handle high bit rates for voice and HF data applications.

Rick Barnich KA8BMA did a fantastic job of designing the SM1000 hardware. Leading edge, HF digital voice hardware, designed by volunteers.

September 12, 2018

Tony K2MO Tests FreeDV

Tony, K2MO, has recently published some fine videos of FreeDV 1600, 700C, and 700D passing through simulated HF channels. The results are quite interesting.

This video shows the 700C mode having the ability to decode with 50% of it’s carriers removed:

This 700C modem sends two copies of the tx signal at high and low frequencies, a form of diversity to help overcome selective fading. These are the combined at the receiver.

Tony’s next video shows three FreeDV modes passing through a selective fading HF channel simulation:

This particular channel has slow fading, a notch gradually creeps across the spectrum.

Tony originally started testing to determine which FreeDV mode worked best on NVIS paths. He used path parameters based on VOACAP prediction models which show the relative time delay and signal power for the each propagation mode i.e., F1, F2:

Note the long delay paths (5ms). The CCIR NVIS path model also suggests a path delay of 7ms. That much delay puts the F-layer at 1000 km (well out into space), which is a bit of a puzzle.

This video shows the results of the VOCAP NVIS path:

In this case 700C does better than 700D. The 700C modem (COHPSK) is a parallel tone design, which is more robust to long multipath delays. The OFDM modem used for 700D is configured for multipath delays of up to 2ms, but tends to fall over after that as the “O” for Orthogonal assumption breaks down. It can be configured for longer delays, at a small cost in low SNR performance.

The OFDM modem gives much tighter packing for carriers, which allows us to include enough bits for powerful FEC, and have a very narrow RF bandwidth compared to 700C. FreeDV 700D has the ability to perform interleaving (Tools-Options “FreeDV 700 Options”), which is a form of time diversity. This feature is not widely used at present, but simulations suggest it is worth up to 4dB.

It would be interesting to combine frequency diversity, LDPC, and OFDM in a wider bandwidth signal. If anyone is interested in doing a little C coding to try this let me know.

I’ve actually seen long delay on NVIS paths in the “real world”. Here is a 40M 700D contact between myself and Mark, VK5QI, who is about 40km away from me. Note at times there are notches on the waterfall 200Hz apart, indicating a round trip path delay of 1500km:

Reading Further

Modems for HF Digital Voice Part 1
, explaining the frequency diversity used in 700C
Testing FreeDV 700C, shows how to use some built in test features like noise insertion and interfering carriers.
FreeDV 700D
FreeDV User Guide, including new 700D features like interleaving

September 11, 2018

Thinkpad X1 Carbon Gen 6

In February I reviewed a Thinkpad X1 Carbon Gen 1 [1] that I bought on Ebay.

I have just been supplied the 6th Generation of the Thinkpad X1 Carbon for work, which would have cost about $1500 more than I want to pay for my own gear. ;)

The first thing to note is that it has USB-C for charging. The charger continues the trend towards smaller and lighter chargers and also allows me to charge my phone from the same charger so it’s one less charger to carry. The X1 Carbon comes with a 65W charger, but when I got a second charger it was only 45W but was also smaller and lighter.

The laptop itself is also slightly smaller in every dimension than my Gen 1 version as well as being noticeably lighter.

One thing I noticed is that the KDE power applet disappears when battery is full – maybe due to my history of buying refurbished laptops I haven’t had a battery report itself as full before.

Disabling the touch pad in the BIOS doesn’t work. This is annoying, there are 2 devices for mouse type input so I need to configure Xorg to only read from the Trackpoint.

The labels on the lid are upside down from the perspective of the person using it (but right way up for people sitting opposite them). This looks nice for observers, but means that you tend to put your laptop the wrong way around on your desk a lot before you get used to it. It is also fancier than the older model, the red LED on the cover for the dot in the I in Thinkpad is one of the minor fancy features.

As the new case is thinner than the old one (which was thin compared to most other laptops) it’s difficult to open. You can’t easily get your fingers under the lid to lift it up.

One really annoying design choice was to have a proprietary Ethernet socket with a special dongle. If the dongle is lost or damaged it will probably be expensive to replace. An extra USB socket and a USB Ethernet device would be much more useful.

The next deficiency is that it has one USB-C/DisplayPort/Thunderbolt port and 2 USB 3.1 ports. USB-C is going to be used for everything in the near future and a laptop with only a single USB-C port will be as annoying then as one with a single USB 2/3 port would be right now. Making a small laptop requires some engineering trade-offs and I can understand them limiting the number of USB 3.1 ports to save space. But having two or more USB-C ports wouldn’t have taken much space – it would take no extra space to have a USB-C port in place of the proprietary Ethernet port. It also has only a HDMI port for display, the USB-C/Thunderbolt/DisplayPort port is likely to be used for some USB-C device when you want an external display. The Lenovo advertising says “So you get Thunderbolt, USB-C, and DisplayPort all rolled into one”, but really you get “a choice of one of Thunderbolt, USB-C, or DisplayPort at any time”. How annoying would it be to disconnect your monitor because you want to read a USB-C storage device?

As an aside this might work out OK if you can have a DisplayPort monitor that also acts as a USB-C hub on the same cable. But if so requiring a monitor that isn’t even on sale now to make my laptop work properly isn’t a good strategy.

One problem I have is that resume from suspend requires holding down power button. I’m not sure if it’s hardware or software issue. But suspend on lid close works correctly and also suspend on inactivity when running on battery power. The X1 Carbon Gen 1 that I own doesn’t suspend on lid close or inactivity (due to a Linux configuration issue). So I have one laptop that won’t suspend correctly and one that won’t resume correctly.

The CPU is an i5-8250U which rates 7,678 according to cpubenchmark.net [2]. That’s 92% faster than the i7 in my personal Thinkpad and more importantly I’m likely to actually get that performance without having the CPU overheat and slow down, that said I got a thermal warning during the Debian install process which is a bad sign. It’s also only 114% faster than the CPU in the Thinkpad T420 I bought in 2013. The model I got doesn’t have the fastest possible CPU, but I think that the T420 didn’t either. A 114% increase in CPU speed over 5 years is a long way from the factor of 4 or more that Moore’s law would have predicted.

The keyboard has the stupid positions for the PgUp and PgDn keys I noted on my last review. It’s still annoying and slows me down, but I am starting to get used to it.

The display is FullHD, it’s nice to have a laptop with the same resolution as my phone. It also has a slider to cover the built in camera which MIGHT also cause the microphone to be disconnected. It’s nice that hardware manufacturers are noticing that some customers care about privacy.

The storage is NVMe. That’s a nice feature, although being only 240G may be a problem for some uses.

Conclusion

Definitely a nice laptop if someone else is paying.

The fact that it had cooling issues from the first install is a concern. Laptops have always had problems with cooling and when a laptop has cooling problems before getting any dust inside it’s probably going to perform poorly in a few years.

Lenovo has gone too far trying to make it thin and light. I’d rather have the same laptop but slightly thicker, with a built-in Ethernet port, more USB ports, and a larger battery.

ABC iview and the ‘Australia tax’

Unless you have been living in a cave, it is probable that you heard about a federal parliamentary inquiry into IT pricing (somewhat aptly entitled “At what cost? — IT pricing and the Australia tax”) which reported that, amongst other things, online geo-blocking can as much as double pricing for IT products in what is blatant price discrimination. Not only do Australians pay, on average, 42% more than US’ians for Adobe products, and 66% more for Microsoft products, but music (such as the iTunes Store), video games, and e-books (e.

iPads as in-flight entertainment

I’m writing this whilst sitting on a Qantas flight from Perth to Sydney, heading home after attending the fantastic linux.conf.au 2014. The plane is a Boeing 767, and unlike most flights I have been on in the last decade, this one has no in-flight entertainment system built into the backs of seats. Instead, every passenger is issued with an Apple iPad (located in the back seat pocket), fitted with what appears to be a fairly robust leather jacket emblazoned with the words “SECURITY DEVICE ATTACHED” (presumably to discourage theft).

You reap what you sow

So the ABC has broken iview for all non–Chrome Linux users. How so? Because the ABC moved iview to use a streaming format supported only by the latest versions of Adobe Flash (e.g. version 11.7, which is available on Windows and OS X), but Adobe have ceased Linux support for Flash as of version 11.2 (for reasons I don’t yet understand, some users report that the older Flash 10.3 continues to work with iview).

Turning out the lights

We put an unbelievable amount of data in the hands of third parties. In plain text. Traceable right back to you with mimimal effort to do so. For me, giving my data to the likes of Google, Apple, Microsoft, and the rest of the crowd, has always been a tradeoff: convenience vs. privacy. Sometimes, privacy wins. Most of the time, convenience wins. My iPhone reports in to Apple. My Android phone reports in to Google.

We do not tolerate bugs; they are of the devil

I was just reading an article entitled “Nine traits of the veteran network admin”, and this point really struck a chord with me: Veteran network admin trait No. 7: We do not tolerate bugs; they are of the devil On occasion, conventional troubleshooting or building new networks run into an unexplainable blocking issue. After poring over configurations, sketching out connections, routes, and forwarding tables, and running debugs, one is brought no closer to solving the problem.

SPA525G with ASA 9.1.x

At work, we have a staff member who has a Cisco SPA525G phone at his home that has built-in AnyConnect VPN support. Over the weekend, I updated our Cisco ASA firewall (which sits in front of our UC500 phone system) from version 8.4.7 to 9.1.3 and the phone broke with the odd error “Failed to obtain WebVPN cookie”. Turns out the fix was very simple. Just update the firmware on the SPA525G to the latest version.

Floppy drive music

Some time in 2013 I set up a rig to play music with a set of floppy drives. At linux.conf.au 2015 in Auckland I gave a brief lightning talk about this, and here is a set of photos and some demo music to accompany. The hardware consists of six 3.5″ floppy drives connected to a LeoStick (Arduino) via custom vero board that connects the direction and step pins (18 and 20, respectively) as well as permanently grounding the select pin A (14).

Configuring Windows for stable IPv6 addressing

By default, Windows will use randomised IPv6 addresses, rather than using stable EUI-64 addresses derived from the MAC address. This is great for privacy, but not so great for servers that need a stable address. If you run an Exchange mail server, or need to be able to access the server remotely, you will want a stable IPv6 address assigned. You may think this is possible simply by editing the NIC and setting a manual IPv6 address.

One week with the Nexus 5

My ageing Motorola Milestone finally received a kick to the bucket last week when my shiny new Nexus 5 phone arrived. Though fantastic by 2009 standards, the Milestone could only officially run Android 2.2, and 2.3 with the help of an unofficial CyanogenMod port. Having been end-of-lifed for some time now, and barely being able to render a complex web page without running out of memory, it was time for me to move on.

Restore ASA 5500 configuration from blank slate

The Cisco ASA 5500 series (e.g. 5505, 5510) firewalls have a fairly nice GUI interface called ASDM. It can sometimes be a pain, but it could be a lot worse than it is. One of the nice things ASDM does it let you save a .zip file backup of your entire ASA configuration. It includes your startup-configuration, VPN secrets, AnyConnect image bundles, and all those other little niceties. But when you set up an ASA from scratch to restore from said .

September 09, 2018

Fail2ban

I’ve recently setup fail2ban [1] on a bunch of my servers. It’s purpose is to ban IP addresses associated with password guessing – or whatever other criteria for badness you configure. It supports Linux, OpenBSD [2] and probably most Unix type OSs too. I run Debian so I’ve been using the Debian packages of fail2ban.

The first thing to note is that it is very easy to install and configure (for the common cases at least). For a long time installing it had been on my todo list but I didn’t make the time to do it, after installing it I realised that I should have done it years ago, it was so easy.

Generally to configure it you just create a file under /etc/fail2ban/jail.d with the settings you want, any settings that are different from the defaults will override them. For example if you have a system running dovecot on the default ports and sshd on port 999 then you could put the following in /etc/fail2ban/jail.d/local.conf:

[dovecot]
enabled = true

[sshd]
port = 999

By default the Debian package of fail2ban only protects sshd.

When fail2ban is running on Linux the command “iptables -L -n -v|grep f2b” will show the rules that match inbound traffic and the names of the chains they direct traffic to. To see if fail2ban has acted to protect a service you can run a command like “iptables -L f2b-sshd -n” to see the iptables rules.

The fail2ban entries in the INPUT table go before other rules, so it should work with any custom iptables rules you have configured as long as either fail2ban is the last thing to be started or your custom rules don’t flush old entries.

There are hooks for sending email notifications etc, that seems excessive to me but it’s always good to have options to extend a program.

In the past I’ve tried using kernel rate limiting to minimise hostile activity. That didn’t work well as there are legitimate end users who do strange things (like a user who setup their web-cam to email them every time it took a photo).

Conclusion

Fail2ban has some good features. I don’t think it will do much good at stopping account compromise as anything that is easily guessed could be guessed using many IP addresses and anything that has a good password can’t be guessed without taking many years of brute-force attacks while also causing enough noise in the logs to be noticed. What it does do is get rid of some of the noise in log files which makes it easier to find and fix problems. To me the main benefit is to improve the signal to noise ratio of my log files.

September 08, 2018

Google and Certbot (Letsencrypt)

Like most people I use Certbot AKA Letsencrypt to create SSL certificates for my sites. It’s a great service, very easy to use and it generally works well.

Recently the server running www.coker.com.au among other domains couldn’t get a certbot certificate renewed, here’s the error message:

Failed authorization procedure. mail.gw90.de (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: "mail.gw90.de" was considered an unsafe domain by a third-party API, listen.gw90.de (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: "listen.gw90.de" was considered an unsafe domain by a third-party API

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: mail.gw90.de
   Type:   unauthorized
   Detail: "mail.gw90.de" was considered an unsafe domain by a third-
   party API

   Domain: listen.gw90.de
   Type:   unauthorized
   Detail: "listen.gw90.de" was considered an unsafe domain by a
   third-party API

It turns out that Google Safebrowsing had listed those two sites. Visit https://listen.gw90.de/ or https://mail.gw90.de/ today (and maybe for some weeks or months in the future) using Google Chrome (or any other browser that uses the Google Safebrowsing database) and it will tell you the site is “Dangerous” and probably refuse to let you in.

One thing to note is that neither of those sites has any real content, I only set them up in Apache to get SSL certificates that are used for other purposes (like mail transfer as the name suggests). If Google had listed my blog as a “Dangerous” site I wouldn’t be so surprised, WordPress has had more than a few security issues in the past and it’s not implausible that someone could have compromised it and made it serve up hostile content without me noticing. But the two sites in question have a DocumentRoot that is owned by root and was (until a few days ago) entirely empty, now they have a index.html that just says “This site is empty”. It’s theoretically possible that someone could have exploited a RCE bug in Apache to make it serve up content that isn’t in the DocumentRoot, but that seems unlikely (why waste an Apache 0day on one of the less important of my personal sites). It is possible that the virtual machine in question was compromised (a VM on that server has been compromised before [1]) but it seems unlikely that they would host bad things on those web sites if they did.

Now it could be that some other hostname under that domain had something inappropriate (I haven’t yet investigated all possibilities). But if so Google’s algorithm has a couple of significant problems, firstly if they are blacklisting sites related to one that had an issue then it would probably make more sense to blacklist by IP address (which means including some coker.com.au entries on the same IP). In the case of a compromised server it seems more likely to have multiple bad sites on one IP than multiple bad subdomains on different IPs (given that none of the hostnames in question have changed IP address recently and Google of course knows this). The next issue is that extending blacklisting doesn’t make sense unless there is evidence of hostile intent. I’m pretty sure that Google won’t blacklist all of ibm.com when (not if) a server in that domain gets compromised. I guess they have different policies for sites of different scale.

Both I and a friend have reported the sites in question to Google as not being harmful, but that hasn’t changed anything yet. I’m very disappointed in Google, listing sites, not providing any reason why (it could be a hostname under that domain was compromised and if so it’s not fixed yet BECAUSE GOOGLE DIDN’T REPORT A PROBLEM), and not removing the listing when it’s totally obvious there’s no basis for it.

While it makes sense for certbot to not issue SSL certificates to bad sites. It seems that they haven’t chosen a great service for determining which sites are bad.

Anyway the end result was that some of my sites had an expired SSL certificate for a day. I decided not to renew certificates before they expired to give Google a better chance of noticing their mistake and then I was busy at the time they expired. Now presumably as the sites in question have an invalid SSL certificate it will be even harder to convince anyone that they are not hostile.

September 07, 2018

A floating shelf for tablets

The choice of replacing the small marble table entirely or trying to "work around" it with walnut. The lower walnut tabletop is about 44cm by 55cm and is just low enough to give easy access to slide laptop(s) under the main table top. The top floating shelf is wide enough to happily accommodate two ipad sized tablets. The top shelf and lower tabletop are attached to the backing by steel brackets which cut through to the back through four CNC created mortises.


Cutting the mortises was interesting, I had to drop back to using a 1/2 inch cutting bit in order to service the 45mm depth of the timber. The back panel was held down with machining clamps but toggles would have done the trick, it was just what was on hand at the time. I cut the mortises through from the back using an upcut bit and the front turned out very clean without any blow out. You could probably cut yourself on the finish it was so clean.

The upcut doesn't make a difference in this job but it is always good to plan and see the outcomes for the next time when the cut will be exposed. The fine grain of walnut is great to work with CNC, though most of my bits are upcut for metal work.

I will likely move on to adding a head rest to the eames chair next. But that is a story for another day.

September 04, 2018

New Developments in Supercomputing

Over the past 33 years the International Super Computing conference in Germany has become one of the world's major computing events with the bi-annual announcement of the Top500 systems, which continues to be dominated in entirety by Linux systems. In June this year over 3,500 people attended ISC with a programme of tutorials, workshops and miniconferences, poster sessions, student competitions, a vast vendor hall, and numerous other events.

This presentation gives an overview of ISC and makes an attempt to cover many of the new developments and directions in supercomputing including new systems. metrics measurement, machine learning, and HPC education. In addition, the presentation will also feature material from the HPC Advisory Council conference in Fremantle held in August.

Kubernetes Fundamentals: Setting up nginx ingress

Share

I’m doing the Linux Foundation Kubernetes Fundamentals course at the moment, and I was very disappointed in the chapter on Ingress Controllers. To be honest it feels like an after thought — there is no lab, and the provided examples don’t work if you re-type them into Kubernetes (you can’t cut and paste of course, just to add to the fun).

I found this super annoying, so I thought I’d write up my own notes on how to get nginx working as an Ingress Controller on Kubernetes.

First off, the nginx project has excellent installation resources online at github. The only wart with their instructions is that they changed the labels used on the pods for the ingress controller, which means the validation steps in the document don’t work until that is fixed. That is reported in a github issue and there was a proposed fix that didn’t have an associated issue that pre-dates the creation of the issue.

The basic process, assuming a baremetal Kubernetes install, is this:

$ NGINX_GITHUB="https://raw.githubusercontent.com/kubernetes/ingress-nginx"
$ kubectl apply -f $NGINX_GITHUB/master/deploy/mandatory.yaml
$ kubectl apply -f $NGINX_GITHUB/master/deploy/provider/baremetal/service-nodeport.yaml

Wait for the pods to fetch their images, and then check if the pods are healthy:

$ kubectl get pods -n ingress-nginx
NAME                                       READY     STATUS    RESTARTS   AGE
default-http-backend-6586bc58b6-tn7l5      1/1       Running   0          20h
nginx-ingress-controller-79b7c66ff-m8nxc   1/1       Running   0          20h

That bit is mostly explained by the Linux Foundation course. Well, he links to the github page at least and then you just read the docs. The bit that isn’t well explained is how to setup ingress for a pod. This is partially because kubectl doesn’t have a command line to do this yet — you have to POST an API request to get it done instead.

First, let’s create a target deployment and service:

$ kubectl run ghost --image=ghost
deployment.apps/ghost created
$ kubectl expose deployments ghost --port=2368
service/ghost exposed

The YAML to create an ingress for this new “ghost” service looks like this:

$ cat sample_ingress.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ghost
spec:
  rules:
  - host: ghost.10.244.2.13.nip.io
    http:
      paths:
      - path: /
        backend:
          serviceName: ghost
          servicePort: 2368

Where 10.244.2.13 is the IP that my CNI assigned to the nginx ingress controller. You can look that up with a describe of the nginx ingress controller pod:

$ kubectl describe pod nginx-ingress-controller-79b7c66ff-m8nxc -n ingress-nginx | grep IP
IP:                 10.244.2.13

Now we can create the ingress entry for this ghost deployment:

$ kubectl apply -f sample_ingress.yaml 
ingress.extensions/ghost created

This causes the nginx configuration to get re-created inside the nginx pod by magix pixies. Now, assuming we have a route from our desktop to 10.244.2.13, we can just go to http://ghost.10.244.2.13.nip.io in a browser and you should be greeted by the default front page for the ghost installation (which turns out to be a publishing platform, who knew?).

To cleanup the ingress, you can use the normal “get”, “describe”, and “delete” verbs that you use for other things in kubectl, with the object type of “ingress”.

Share

September 03, 2018

Audiobooks – August 2018

Homo Deus: A Brief History of Tomorrow by Yuval Noah Harari

An interesting listen. Covers both history of humanity and then extrapolates ways things might go in the future. Many plausible ideas (although no doubt some huge misses). 8/10

Higher: A Historic Race to the Sky and the Making of a City by Neal Bascomb

The architects, owners & workers behind the Manhattan Trust Building, the Chrysler Building and the Empire State Building all being built New York at the end of the roaring 20s. Fascinating and well done. 9/10

The Invention Of Childhood by Hugh Cunningham

The story of British childhood from the year 1000 to the present. Lots of quotes (by actors) from primary sources such as letters (which is less distracting than sometimes). 8/10

The Sign of Four by Arthur Conan Doyle – Read by Stephen Fry

Very well done reading by Fry. Story excellent of course. 8/10

My Happy Days in Hollywood: A Memoir by Garry Marshall

Memoir by writer, producer (Happy Days, etc) and director (Pretty Woman, The Princess Diaries, etc). Great stories mostly about the positive side of the business. Very inspiring 8/10

Napoleon by J. Christopher Herold

A biography of Napoleon with a fair amount about the history of the rest of Europe during the period thrown in. A fairly short (11 hours) book but some but not exhausting detail. 7/10

Storm in a Teacup: The Physics of Everyday Life by Helen Czerski

A good popular science book linking everyday situations and objects with bigger concepts (eg coffee stains to blood tests). A fun listen. 7/10

All These Worlds Are Yours: The Scientific Search for Alien Life by Jon Willis

The author reviews recent developments in the search for life and suggests places it might be found and how missions to search them (he gives himself a $4 billion budget) should be prioritised. 8/10

Ready Player One by Ernest Cline

I’m right in the middle of the demographic for most of the references here so I really enjoyed it. Good voicing by Wil Wheaton too. Story is straightforward but all pretty fun. 8/10

Share