Planet Linux Australia
Celebrating Australians & Kiwis in the Linux and Free/Open-Source community...

March 30, 2017

Staying safe during Cyclone Debbie

Cyclone Debbie seen from the International Space StationCyclone Debbie as seen from the ISS

We hope all teachers and students are safe in the areas of Queensland and New South Wales affected by the cyclone weather! We understand that many state schools (any South of Agnes Water to Northern New South Wales) are closed today, the radar shows a very large rain front coming through. Near Brisbane it’s been raining for many hours already, and the wind is now picking up as well. It’s good to be inside, although things are starting to feel moist (reminding Arjen of when he lived in Cairns).

Why not take this opportunity to replace dry old teaching materials using coupon code DEBBIE for 25% discount on any Understanding our World™ unit. This special Cyclone Debbie offer ends Sunday 2nd April.

Did you know that, in the Understanding Our World units, Year 5 students did work on Natural Disasters during this term!

Also, do take a peek at the Open Source Earth Wind Patterns site at NullSchool – using live data to create a moving image. All open. Beautiful.

March 29, 2017

Hello world!

Welcome to WordPress. This is your first post. Edit or delete it, then start writing!

Life in Yemen, Blogger2Book BASH script, and More

- main reasons why it's been in the news is because of recent conflict. A lot of them are saying this is a proxy war between Saudi Arabia and Iran who effectively represent different sects within Islam. Over and over again feels like people in the Middle East just want some dignity and respect from the West? Shortages across the board (food, medicine, petrol, etc...) Yemen people saying Saudi

March 28, 2017

Evaluating CephFS on Power

Methodology

To evaluate CephFS, we will create a ppc64le virtual machine, with sufficient space to compile the software, as well as 3 sparse 1TB disks to create the object store.

We will then build & install the Ceph packages, after adding the PowerPC optimisiations to the code. This is done, as ceph-deploy will fetch prebuilt packages that do not have the performance patches if the packages are not installed.

Finally, we will use the ceph-deploy to deploy the instance. We will ceph-deploy via pip, to avoid file conflicts with the packages that we built.

For more information on what each command does, visit the following tutorial, upon which which this is based: http://palmerville.github.io/2016/04/30/single-node-ceph-install.html

Virtual Machine Config

Create a virtual machine with at least the following: - 16GB of memory - 16 CPUs - 64GB disk for the root filesystem - 3 x 1TB for the Ceph object store - Ubuntu 16.04 default install (only use the 64GB disk, leave the others unpartitioned)

Initial config

  • Enable ssh
    sudo apt install openssh-server
    sudo apt update
    sudo apt upgrade
    sudo reboot
  • Install build tools
    sudo apt install git debhelper

Build Ceph

    mkdir $HOME/src
    cd $HOME/src
    git clone --recursive https://github.com/ceph/ceph.git  # This may take a while
    cd ceph
    git checkout master
    git submodule update --force --init --recursive
  • Cherry-pick the Power performance patches:
    git remote add kestrels https://github.com/kestrels/ceph.git
    git fetch --all
    git cherry-pick 59bed55a676ebbe3ad97d8ec005c2088553e4e11
  • Install prerequisites
    ./install-deps.sh
    sudo apt install python-requests python-flask resource-agents curl python-cherrypy python3-pip python-django python-dateutil python-djangorestframework
    sudo pip3 install ceph-deploy
    cd $HOME/src/ceph
    sudo dpkg-buildpackage -J$(nproc) # This will take a couple of hours (16 cpus)
  • Install the packages (note that python3-ceph-argparse will fail, but is safe to ignore)
    cd $HOME/src
    sudo dpkg -i *.deb

Create the ceph-deploy user

    sudo adduser ceph-deploy
    echo "ceph-deploy ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph-deploy
    sudo chmod 0440 /etc/sudoers.d/ceph-deploy

Configure the ceph-deploy user environment

    su - ceph-deploy
    ssh-keygen
    node=`hostname`
    ssh-copy-id ceph-deploy@$node
    mkdir $HOME/ceph-cluster
    cd $HOME/ceph-cluster
    ceph-deploy new $node # If this fails, remove the bogus 127.0.1.1 entry from /etc/hosts
    echo 'osd pool default size = 2' >> ceph.conf
    echo 'osd crush chooseleaf type = 0' >> ceph.conf

Complete the Ceph deployment

    ceph-deploy install $node
    ceph-deploy mon create-initial
    drives="vda vdb vdc"  # the 1TB drives - check that these are correct for your system
    for drive in $drives; do ceph-deploy disk zap $node:$drive; ceph-deploy osd prepare $node:$drive; done
    for drive in $drives; do ceph-deploy osd activate $node:/dev/${drive}1; done
    ceph-deploy admin $node
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring
    ceph -s # Check the state of the cluster

Configure CephFS

    ceph-deploy mds create $node
    ceph osd pool create cephfs_data 128
    ceph osd pool create cephfs_metadata 128
    ceph fs new cephfs cephfs_metadata cephfs_data
    sudo systemctl status ceph\*.service ceph\*.target # Ensure the ceph-osd, ceph-mon & ceph-mds daemons are running
    sudo mkdir /mnt/cephfs
    key=`grep key ~/ceph-cluster/ceph.client.admin.keyring | cut -d ' ' -f 3`
    sudo mount -t ceph $node:6789:/ /mnt/cephfs -o name=admin,secret=$key

References

  1. http://docs.ceph.com/docs/master/install/clone-source/
  2. http://docs.ceph.com/docs/master/install/build-ceph/
  3. http://palmerville.github.io/2016/04/30/single-node-ceph-install.html

Making Views in Drupal 7 and Drupal 8

This talk was written for DrupalGov Canberra 2017.

Download a PDF of the slides.

Or view below on slideshare.

 

Busyness: A Modern Health Crisis | LinkedIn

Benjamin Cardullo writes about an issue that we really have to take (more) seriously.  Particularly with mobile devices enabling us to be “connected” 24/7, being busy (or available) all of that time is not a good thing at all.

How do we measure professional success? Is it by the location of our office or the size of our paycheck? Is it measured by the dimensions of our home or the speed of our car? Ten years ago, those would have been the most prominent answers; however, today when someone is really pulling out the big guns, when they really want to show you how important they are, they’ll tell you all about their busy day and how they never had a moment to themselves.

Read the full article: https://www.linkedin.com/pulse/busyness-modern-health-crisis-benjamin-cardullo

This Week in HASS – term 1, week 9

The last week of our first unit – time to wrap up, round off, finish up any work not yet done and to perhaps get a preliminary taste of what’s to come in future units. Easter holidays are just around the corner. Our youngest students are having a final discussion about celebrations; slightly older students are finishing off their quest for Aunt Madge, by looking at landmarks and the older students are considering democracy in Australia, compared to its early beginnings in Ancient Greece.

Foundation to Year 3

Foundation/Prep (units F.1 and F.6) students are finishing off their discussions about celebrations, just in time for the Easter holidays, by looking at celebrations around the world. Teachers may wish to focus on how other countries celebrate Easter, with passion plays, processions and special meals. Students in Years 1 (unit 1.1), 2 (unit 2.1) and 3 (unit 3.1) are finishing off their Aunt Madge activity, looking at landmarks in Australia and around the world. There is the option for teachers to concentrate on Australian landmarks in this lesson, setting the stage for some local history studies in the next unit, next term.

Years 3 to 6

Ancient Greek pottery with votes scratched into the surface

Older students in Years 3 (unit 3.5), 4 (unit 4.1), 5 (unit 5.1) and 6 (unit 6.1) start looking ahead and laying the foundations for later studies on the Australian system of government and democracy, by comparing democracy as it arose in Ancient Greece, with the modern Australian democratic system. Our word for democracy comes from the Ancient Greek words demos (people) and kratia (power). Students move on from their discussion of Eratosthenes to looking at the Ancient Greek democratic system, which was to lay the groundwork for modern democratic systems around the world. Discussing Ancient Greek democracy leads students to consider the rights and responsibilities of being a citizen, at both the local and international levels. Students also consider who could and could not vote and what this meant for different groups. They can also touch on the ancient practise of ostracism, which can lead to ethical debates around fair election practises. By considering these fundamental concepts, students are better able to relate the ideas around modern democracy to their own lives.

 

March 26, 2017

AMBE+2 and MELPe 600 Compared to Codec 2

Yesterday I was chatting on the #freedv IRC channel, and a good question was asked: how close is Codec 2 to AMBE+2 ? Turns out – reasonably close. I also discovered, much to my surprise, that Codec 2 700C is better than MELPe 600!

Samples

Original AMBE+2 3000 AMBE+ 2400 Codec 2 3200 Codec 2 2400
Listen Listen Listen Listen Listen
Listen Listen Listen Listen Listen
Listen Listen Listen Listen Listen
Listen Listen Listen Listen Listen
Listen Listen Listen Listen Listen
Listen Listen Listen Listen Listen
Listen Listen Listen Listen Listen
Listen Listen Listen Listen Listen
Listen Listen Listen Listen Listen
Original MELPe 600 Codec 2 700C
Listen Listen Listen
Listen Listen Listen
Listen Listen Listen
Listen Listen Listen
Listen Listen Listen
Listen Listen Listen
Listen Listen Listen
Listen Listen Listen
Listen Listen Listen

Here are all the samples in one big tar ball.

Discussion

I don’t have a AMBE or MELPe codec handy so I used the samples from the DVSI and DSP Innovations web sites. I passed the original “DAMA” speech samples found on these sites through Codec 2 (codec2-dev SVN revision 3053) at various bit rates. Turns out the DAMA samples were the same for the AMBE and MELPe samples which was handy.

These particular samples are “kind” to codecs – I consistently get good results with them when I test with Codec 2. I’m guessing they also allow other codecs to be favorably demonstrated. During Codec 2 development I make a point of using “pathological” samples such as hts1a, cg_ref, kristoff, mmt1 that tend to break Codec 2. Some samples of AMBE and MELP using my samples on the Codec 2 page.

I usually listen to samples through a laptop speaker, as I figure it’s close to the “use case” of a PTT radio. Small speakers do mask codec artifacts, making them sound better. I also tried a powered loud speaker with the samples above. Through the loudspeaker I can hear AMBE reproducing the pitch fundamental – a bass note that can be heard on some males (e.g. 7), whereas Codec 2 is filtering that out.

I feel AMBE is a little better, Codec 2 is a bit clicky or impulsive (e.g. on sample 1). However it’s not far behind. In a digital radio application, with a small speaker and some acoustic noise about – I feel the casual listener wouldn’t discern much difference. Try replaying these samples through your smart-phone’s browser at an airport and let me know if you can tell them apart!

On the other hand, I think Codec 2 700C sounds better than MELPe 600. Codec 2 700C is more natural. To my ear MELPe has very coarse quantisation of the pitch, hence the “Mr Roboto” sing-song pitch jumps. The 700C level is a bit low, an artifact/bug to do with the post filter. Must fix that some time. As a bonus Codec 2 700C also has lower algorithmic delay, around 40ms compared to MELPe 600’s 90ms.

Curiously, Codec 2 uses just 1 voicing bit which means either voiced or unvoiced excitation in each frame. xMBE’s claim to fame (and indeed MELP) over simpler vocoders is the use of mixed excitation. Some of the spectrum is voiced (regular pitch harmonics), some unvoiced (noise like). This suggests the benefits of mixed excitation need to be re-examined.

I haven’t finished developing Codec 2. In particular Codec 2 700C is very much a “first pass”. We’ve had a big breakthrough this year with 700C and development will continue, with benefits trickling up to other modes.

However the 1300, 2400, 3200 modes have been stable for years and will continue to be supported.

Next Steps

Here is the blog post that kicked off Codec 2 – way back in 2009. Here is a video of my linux.conf.au 2012 Codec 2 talk that explains the motivations, IP issues around codecs, and a little about how Codec 2 works (slides here).

What I spoke about then is still true. Codec patents and license fees are a useless tax on business and stifle innovation. Proprietary codecs borrow as much as 95% of their algorithms from the public domain – which are then sold back to you. I have shown that open source codecs can meet and even exceed the performance of closed source codecs.

Wikipedia suggests that AMBE license fees range from USD$100k to USD$1M. For “one license fee” we can improve Codec 2 so it matches AMBE+2 in quality at 2400 and 3000 bit/s. The results will be released under the LGPL for anyone to use, modify, improve, and inspect at zero cost. Forever.

Maybe we should crowd source such a project?

Command Lines

This is how I generated the Codec 2 wave files:

~/codec2-dev/build_linux//src/c2enc 3200 9.wav - | ~/codec2-dev/build_linux/src/c2dec 3200 - - | sox -t raw -r 8000 -s -2 - 9_codec2_3200.wav

Links

DVSI AMBE sample page

DSP Innovations, MELPe samples. Can anyone provide me with TWELP samples from these guys? I couldn’t find any on the web that includes the input, uncoded source samples.

Trying an OpenSTEM unit without a subscription

We have received quite a few requests for this option, so we’ve made it possible. As we understand it, in many cases an individual teacher wants to try our materials (often on behalf of the school, as a trial) but the teacher has to fund this from their classroom budget, so we appreciate they need to limit their initial outlay.

While purchasing units with an active subscription still works out cheaper (we haven’t changed that pricing), we have tweaked our online store to now also allow the purchase of individual unit bundles, from as little as $49.50 (inc.GST) for the Understanding Our World™ HASS+Science program units. That’s a complete term bundle with teacher handbook, student workbook, assessment guide, model answers and curriculum mapping, as well as all the base resource PDFs needed for that unit! After purchase, the PDF materials can be downloaded from the site (optionally many files together in a ZIP).

We’d love to welcome you as a new customer! From experience we know that you’ll love our materials. The exact pricing difference (between subscription and non-subscription) depends on the type of bundle (term unit, year bundle, or multi-year bundle) and is indicated per item.

Try OpenSTEM today! Browse our teacher unit bundles (Foundation Year to Year 6).

This includes units for Digital Technologies, the Ginger Beer Science project, as well as for our popular Understanding Our World™ HASS+Science program.

March 24, 2017

Linux Security Summit 2017: CFP Announcement

LSS logo

The 2017 Linux Security Summit CFP (Call for Participation) is now open!

See the announcement here.

The summit this year will be held in Los Angeles, USA on 14-15 September. It will be co-located with the Open Source Summit (formerly LinuxCon), and the Linux Plumbers Conference. We’ll follow essentially the same format as the 2016 event (you can find the recap here).

The CFP closes on June 5th, 2017.

March 23, 2017

Erasure Coding for Programmers, Part 2

We left part 1 having explored GF(2^8) and RAID 6, and asking the question "what does all this have to do with Erasure Codes?"

Basically, the thinking goes "RAID 6 is cool, but what if, instead of two parity disks, we had an arbitrary number of parity disks?"

How would we do that? Well, let's introduce our new best friend: Coding Theory!

Say we want to transmit some data across an error-prone medium. We don't know where the errors might occur, so we add some extra information to allow us to detect and possibly correct for errors. This is a code. Codes are a largish field of engineering, but rather than show off my knowledge about systematic linear block codes, let's press on.

Today, our error-prone medium is an array of inexpensive disks. Now we make this really nice assumption about disks, namely that they are either perfectly reliable or completely missing. In other words, we consider that a disk will either be present or 'erased'. We come up with 'erasure codes' that are able to reconstruct data when it is known to be missing. (This is a slightly different problem to being able to verify and correct data that might or might not be subtly corrupted. Disks also have to deal with this problem, but it is not something erasure codes address!)

The particular code we use is a Reed-Solomon code. The specific details are unimportant, but there's a really good graphical outline of the broad concepts in sections 1 and 3 of the Jerasure paper/manual. (Don't go on to section 4.)

That should give you some background on how this works at a pretty basic mathematical level. Implementation is a matter of mapping that maths (matrix multiplication) onto hardware primitives, and making it go fast.

Scope

I'm deliberately not covering some pretty vast areas of what would be required to write your own erasure coding library from scratch. I'm not going to talk about how to compose the matricies, how to invert them, or anything like that. I'm not sure how that would be a helpful exercise - ISA-L and jerasure already exist and do that for you.

What I want to cover is an efficient implementation of the some algorithms, once you have the matricies nailed down.

I'm also going to assume your library already provides a generic multiplication function in GF(2^8). That's required to construct the matrices, so it's a pretty safe assumption.

The beginnings of an API

Let's make this a bit more concrete.

This will be heavily based on the ISA-L API but you probably want to plug into ISA-L anyway, so that shouldn't be a problem.

What I want to do is build up from very basic algorithmic components into something useful.

The first thing we want to do is to be able to is Galois Field multiplication of an entire region of bytes by an arbitrary constant.

We basically want gf_vect_mul(size_t len, <something representing the constant>, unsigned char * src, unsigned char * dest)

Simple and slow approach

The simplest way is to do something like this:

void gf_vect_mul_simple(size_t len, unsigned char c, unsigned char * src, unsigned char * dest) {

    size_t i;
    for (i=0; i<len; i++) {
        dest[i] = gf_mul(c, src[i]);
    }
}

That does multiplication element by element using the library's supplied gf_mul function, which - as the name suggests - does GF(2^8) multiplication of a scalar by a scalar.

This works. The problem is that it is very, painfully, slow - in the order of a few hundred megabytes per second.

Going faster

How can we make this faster?

There are a few things we can try: if you want to explore a whole range of different ways to do this, check out the gf-complete project. I'm going to assume we want to skip right to the end and know what is the fastest we've found.

Cast your mind back to the RAID 6 paper (PDF). I talked about in part 1. That had a way of doing an efficient multiplication in GF(2^8) using vector instructions.

To refresh your memory, we split the multiplication into two parts - low bits and high bits, looked them up separately in a lookup table, and joined them with XOR. We then discovered that on modern Power chips, we could do that in one instruction with vpermxor.

So, a very simple way to do this would be:

  • generate the table for a
  • for each 16-byte chunk of our input:
    • load the input
    • do the vpermxor with the table
    • save it out

Generating the tables is reasonably straight-forward, in theory. Recall that the tables are a * {{00},{01},...,{0f}} and a * {{00},{10},..,{f0}} - a couple of loops in C will generate them without difficulty. ISA-L has a function to do this, as does gf-complete in split-table mode, so I won't repeat them here.

So, let's recast our function to take the tables as an input rather than the constant a. Assume we're provided the two tables concatenated into one 32-byte chunk. That would give us:

void gf_vect_mul_v2(size_t len, unsigned char * table, unsigned char * src, unsigned char * dest)

Here's how you would do it in C:

void gf_vect_mul_v2(size_t len, unsigned char * table, unsigned char * src, unsigned char * dest) {
        vector unsigned char tbl1, tbl2, in, out;
        size_t i;

        /* Assume table, src, dest are aligned and len is a multiple of 16 */

        tbl1 = vec_ld(16, table);
        tbl2 = vec_ld(0, table);
        for (i=0; i<len; i+=16) {
            in = vec_ld(i, (unsigned char *)src);
            __asm__("vpermxor %0, %1, %2, %3" : "=v"(out) : "v"(tbl1), "v"(tbl2), "v"(in)
            vec_st(out, i, (unsigned char *)dest);
        }
}

There's a few quirks to iron out - making sure the table is laid out in the vector register in the way you expect, etc, but that generally works and is quite fast - my Power 8 VM does about 17-18 GB/s with non-cache-contained data with this implementation.

We can go a bit faster by doing larger chunks at a time:

    for (i=0; i<vlen; i+=64) {
            in1 = vec_ld(i, (unsigned char *)src);
            in2 = vec_ld(i+16, (unsigned char *)src);
            in3 = vec_ld(i+32, (unsigned char *)src);
            in4 = vec_ld(i+48, (unsigned char *)src);
            __asm__("vpermxor %0, %1, %2, %3" : "=v"(out1) : "v"(tbl1), "v"(tbl2), "v"(in1));
            __asm__("vpermxor %0, %1, %2, %3" : "=v"(out2) : "v"(tbl1), "v"(tbl2), "v"(in2));
            __asm__("vpermxor %0, %1, %2, %3" : "=v"(out3) : "v"(tbl1), "v"(tbl2), "v"(in3));
            __asm__("vpermxor %0, %1, %2, %3" : "=v"(out4) : "v"(tbl1), "v"(tbl2), "v"(in4));
            vec_st(out1, i, (unsigned char *)dest);
            vec_st(out2, i+16, (unsigned char *)dest);
            vec_st(out3, i+32, (unsigned char *)dest);
            vec_st(out4, i+48, (unsigned char *)dest);
    }

This goes at about 23.5 GB/s.

We can go one step further and do the core loop in assembler - that means we control the instruction layout and so on. I tried this: it turns out that for the basic vector multiply loop, if we turn off ASLR and pin to a particular CPU, we can see a improvement of a few percent (and a decrease in variability) over C code.

Building from vector multiplication

Once you're comfortable with the core vector multiplication, you can start to build more interesting routines.

A particularly useful one on Power turned out to be the multiply and add routine: like gf_vect_mul, except that rather than overwriting the output, it loads the output and xors the product in. This is a simple extension of the gf_vect_mul function so is left as an exercise to the reader.

The next step would be to start building erasure coding proper. Recall that to get an element of our output, we take a dot product: we take the corresponding input element of each disk, multiply it with the corresponding GF(2^8) coding matrix element and sum all those products. So all we need now is a dot product algorithm.

One approach is the conventional dot product:

  • for each element
    • zero accumulator
    • for each source
      • load input[source][element]
      • do GF(2^8) multiplication
      • xor into accumulator
    • save accumulator to output[element]

The other approach is multiply and add:

  • for each source
    • for each element
      • load input[source][element]
      • do GF(2^8) multiplication
      • load output[element]
      • xor in product
      • save output[element]

The dot product approach has the advantage of fewer writes. The multiply and add approach has the advantage of better cache/prefetch performance. The approach you ultimately go with will probably depend on the characteristics of your machine and the length of data you are dealing with.

For what it's worth, ISA-L ships with only the first approach in x86 assembler, and Jerasure leans heavily towards the second approach.

Once you have a vector dot product sorted, you can build a full erasure coding setup: build your tables with your library, then do a dot product to generate each of your outputs!

In ISA-L, this is implemented something like this:

/*
 * ec_encode_data_simple(length of each data input, number of inputs,
 *                       number of outputs, pre-generated GF(2^8) tables,
 *                       input data pointers, output code pointers)
 */
void ec_encode_data_simple(int len, int k, int rows, unsigned char *g_tbls,
                           unsigned char **data, unsigned char **coding)
{
        while (rows) {
                gf_vect_dot_prod(len, k, g_tbls, data, *coding);
                g_tbls += k * 32;
                coding++;
                rows--;
        }
}

Going faster still

Eagle eyed readers will notice that however we generate an output, we have to read all the input elements. This means that if we're doing a code with 10 data disks and 4 coding disks, we have to read each of the 10 inputs 4 times.

We could do better if we could calculate multiple outputs for each pass through the inputs. This is a little fiddly to implement, but does lead to a speed improvement.

ISA-L is an excellent example here. Intel goes up to 6 outputs at once: the number of outputs you can do is only limited by how many vector registers you have to put the various operands and results in.

Tips and tricks

  • Benchmarking is tricky. I do the following on a bare-metal, idle machine, with ASLR off and pinned to an arbitrary hardware thread. (Code is for the fish shell)

    for x in (seq 1 50)
        setarch ppc64le -R taskset -c 24 erasure_code/gf_vect_mul_perf
    end | awk '/MB/ {sum+=$13} END {print sum/50, "MB/s"}'
    
  • Debugging is tricky; the more you can do in C and the less you do in assembly, the easier your life will be.

  • Vector code is notoriously alignment-sensitive - if you can't figure out why something is wrong, check alignment. (Pro-tip: ISA-L does not guarantee the alignment of the gftbls parameter, and many of the tests supply an unaligned table from the stack. For testing __attribute__((aligned(16))) is your friend!)

  • Related: GCC is moving towards assignment over vector intrinsics, at least on Power:

    vector unsigned char a;
    unsigned char * data;
    // good, also handles word-aligned data with VSX
    a = *(vector unsigned char *)data;
    // bad, requires special handling of non-16-byte aligned data
    a = vec_ld(0, (unsigned char *) data);
    

Conclusion

Hopefully by this point you're equipped to figure out how your erasure coding library of choice works, and write your own optimised implementation (or maintain an implementation written by someone else).

I've referred to a number of resources throughout this series:

If you want to go deeper, I also read the following and found them quite helpful in understanding Galois Fields and Reed-Solomon coding:

For a more rigorous mathematical approach to rings and fields, a university mathematics course may be of interest. For more on coding theory, a university course in electronics engineering may be helpful.

Guess the Artefact!

Today we are announcing a new challenge for our readers – Guess the Artefact! We post pictures of an artefact and you can guess what it is. The text will slowly reveal the answer, through a process of examination and deduction – see if you can guess what it is, before the end. We are starting this challenge with an item from our year 6 Archaeological Dig workshop. Year 6 (unit 6.3) students concentrate on Federation in their Australian History segment – so that’s your first clue! Study the image and then start reading the text below.

OpenSTEM archaeological dig artefact (C) 2016 OpenSTEM Pty Ltd

Our first question is what is it? Study the image and see if you can work out what it might be – it’s an dirty, damaged piece of paper. It seems to be old. Does it have a date? Ah yes, there are 3 dates – 23, 24 and 25 October, 1889, so we deduce that it must be old, dating to the end of the 19th century. We will file the exact date for later consideration. We also note references to railways. The layout of the information suggests a train ticket. So we have a late 19th century train ticket!

Now why do we have this train ticket and whose train ticket might it have been? The ticket is First Class, so this is someone who could afford to travel in style. Where were they going? The railways mentioned are Queensland Railways, Great Northern Railway, New South Wales Railways and the stops are Brisbane, Wallangara, Tenterfield and Sydney. Now we need to do some research. Queensland Railways and New South Wales Railways seem self-evident, but what is Great Northern Railway? A brief hunt reveals several possible candidates: 1) a contemporary rail operator in Victoria; 2) a line in Queensland connecting Mt Isa and Townsville and 3) an old, now unused railway in New South Wales. We can reject option 1) immediately. Option 2) is the right state, but the towns seem unrelated. That leaves option 3), which seems most likely. Looking into the NSW option in more detail we note that it ran between Sydney and Brisbane, with a stop at Wallangara to change gauge – Bingo!

Wallangara Railway Station

More research reveals that the line reached Wallangara in 1888, the year before this ticket was issued. Only after 1888 was it possible to travel from Brisbane to Sydney by rail, albeit with a compulsory stop at Wallangara. We note also that the ticket contains a meal voucher for dinner at the Railway Refreshment Rooms in Wallangara. Presumably passengers overnighted in Wallangara before continuing on to Sydney on a different train and rail gauge. Checking the dates on the ticket, we can see evidence of an overnight stop, as the next leg continues from Wallangara on the next day (24 Oct 1889). However, next we come to some important information. From Wallangara, the next leg of the journey represented by this ticket was only as far as Tenterfield. Looking on a map, we note that Tenterfield is only about 25 km away – hardly a day’s train ride, more like an hour or two at the most (steam trains averaged about 24 km/hr at the time). From this we deduce that the ticket holder wanted to stop at Tenterfield and continue their journey on the next day.

We know that we’re studying Australian Federation history, so the name Tenterfield should start to a ring a bell – what happened in Tenterfield in 1889 that was relevant to Australian Federation history? The answer, of course, is that Henry Parkes delivered his Tenterfield Oration there, and the date? 24 October, 1889! If we look into the background, we quickly discover that Henry Parkes was on his way from Brisbane back to Sydney, when he stopped in Tenterfield. He had been seeking support for Federation from the government of the colony of Queensland. He broke his journey in Tenterfield, a town representative of those towns closer to the capital of another colony than their own, which would benefit from the free trade arrangements flowing from Federation. Parkes even discussed the issue of different rail gauges as something that would be solved by Federation! We can therefore surmise that this ticket may well be the ticket of Henry Parkes, documenting his journey from Brisbane to Sydney in October, 1889, during which he stopped and delivered the Tenterfield Oration!

This artefact is therefore relevant as a source for anyone studying Federation history – as well as giving us a more personal insight into the travels of Henry Parkes in 1889, it allows us to consider aspects of life at the time:

  • the building of railway connections across Australia, in a time before motor cars were in regular use;
  • the issue of different size railway gauges in the different colonies and what practical challenges that posed for a long distance rail network;
  • the ways in which people travelled and the speed with which they could cross large distances;
  • what rail connections would have meant for small, rural towns, to mention just a few.
  • Why might the railway companies have provided meal vouchers?

These are all sidelines of inquiry, which students may be interested to pursue, and which might help them to engage with the subject matter in more detail.

In our Archaeological Dig Workshops, we not only engage students in the processes and physical activities of the dig, but we provide opportunities for them to use the artefacts to practise deduction, reasoning and research – true inquiry-based learning, imitating real-world processes and far more engaging and empowering than more traditional bookwork.

March 22, 2017

LUV Main April 2017 Meeting: SageMath / Simultaneous multithreading

Apr 4 2017 18:30
Apr 4 2017 20:30
Apr 4 2017 18:30
Apr 4 2017 20:30
Location: 
The Dan O'Connell Hotel, 225 Canning Street, Carlton VIC 3053

PLEASE NOTE NEW LOCATION

Tuesday, April 4, 2017
6:30 PM to 8:30 PM
The Dan O'Connell Hotel
225 Canning Street, Carlton VIC 3053

Speakers:

• Adetokunbo "Xero" Arogbonlo, SageMath
• Stewart Smith, Simultaneous multithreading

The Dan O'Connell Hotel, 225 Canning Street, Carlton VIC 3053

Food and drinks will be available on premises.

Before and/or after each meeting those who are interested are welcome to join other members for dinner.

Linux Users of Victoria Inc., is an incorporated association, registration number A0040056C.

April 4, 2017 - 18:30

read more

LUV Beginners April Meeting: TBD

Apr 15 2017 12:30
Apr 15 2017 16:30
Apr 15 2017 12:30
Apr 15 2017 16:30
Location: 
Infoxchange, 33 Elizabeth St. Richmond

Meeting topic to be announced.


There will also be the usual casual hands-on workshop, Linux installation, configuration and assistance and advice. Bring your laptop if you need help with a particular issue. This will now occur BEFORE the talks from 12:30 to 14:00. The talks will commence at 14:00 (2pm) so there is time for people to have lunch nearby.

The meeting will be held at Infoxchange, 33 Elizabeth St. Richmond 3121 (enter via the garage on Jonas St.) Late arrivals, please call (0421) 775 358 for access to the venue.

LUV would like to acknowledge Infoxchange for the venue.

Linux Users of Victoria Inc., is an incorporated association, registration number A0040056C.

April 15, 2017 - 12:30

Flir ONE Issues

FLIR ONE for iOS or Android with solid orange power light

Troubleshooting steps when the FLIR ONE has a solid red/orange power light that will not turn to blinking green:

  • Perform a hard reset on the FLIR ONE by holding the power button down for 30 seconds.
  • Let the battery drain overnight and try charging it again (with another charger if possible) for a whole hour.

March 20, 2017

Life in Brazil, Random Stuff, and More

- history seems reminiscent of other Latin American nations. Mix of European colonisation and local tribes. Was obviously used for it's natural resources but it's clear that it's economy has diversified since then. That said, clear issues with corruption and wealth inequality brazil history Brazil export treemap by product (2014) from Harvard Atlas of Economic Complexity When the Portuguese

Pre-Cogs and Prophets 10, Random Stuff, and More

- a clear continuation of my other posts on pre-cogs/prophets: http://dtbnguyen.blogspot.com/2017/03/prophetsgenesisterraforming-mars-seek.html http://dtbnguyen.blogspot.com/2017/03/prophetspre-cogsstargate-program-8.html http://dtbnguyen.blogspot.com/2017/02/life-in-india-prophetspre-cogsstargate_82.html http://dtbnguyen.blogspot.com/2017/02/life-in-iran-examining-prophetspre-cogs.html

This Week in HASS – term 1, week 8

As we move into the final weeks of term, and the Easter holiday draws closer, our youngest students are looking at different kinds of celebrations in Australia. Students in years 1 to 3 are looking at their global family and students in years 3 to 6 are chasing Aunt Madge around the world, being introduced to Eratosthenes and examining Shadows and Light.

Foundation to Year 3

Our standalone Foundation/Prep students (Unit F.1) are studying celebrations in Australia and thinking about which is their favourite. It may well be Easter with its bunnies and chocolate eggs, which lies just around the corner now! They also get a chance to consider whether we should add any extra celebrations into our calendar in Australia. Those Foundation/Prep students in an integrated class with Year 1 students (Unit F.5), as well as Year 1 (Unit 1.1), 2 (Unit 2.1) and 3 (Unit 3.1) students are investigating where they, and other family members, were born and finding these places on the world map. Students are also examining features of the world map – including the different continents, North and South Poles, the equator and the oceans. Students also get a chance to undertake the Aunt Madge’s Suitcase Activity, in which they follow Aunt Madge around the world, learning about different countries and landmarks, as they go. Aunt Madge’s Suitcase is extremely popular with students of all ages – as it can easily be adapted to cover material at different depths. The activity encourages students to interact with the world map, whilst learning to recognise major natural and cultural landmarks in Australia and around the world.

Years 3 to 6

Aunt Madge

Students in Year 3 (Unit 3.5), who are integrated with Year 4, as well as the Year 4 (Unit 4.1), 5 (Unit 5.1) and 6 (Unit 6.1) students, have moved on to a new set of activities this week. The older students approach the Aunt Madge’s Suitcase Activity in more depth, deriving what items Aunt Madge has packed in her suitcase to match the different climates which she is visiting, as well as delving into each landmark visited in more detail. These landmarks are both natural and cultural and, although several are in Australia, examples are given from around the world, allowing teachers to choose their particular focus each time the activity is undertaken. As well as following Aunt Madge, students are introduced to Eratosthenes. Known as the ‘Father of Geography’, Eratosthenes also calculated the circumference of the Earth. There is an option for teachers to overlap with parts of the Maths curriculum here. Eratosthenes also studied the planets and used shadows and sunlight for his calculations, which provides the link for the Science activities – Shadows and Light, Sundials and Planets of the Solar System.

Next week is the last week of our first term units. By now students have completed the bulk of their work for the term, and teachers are able to assess most of the HASS areas already.

 

March 19, 2017

Erasure Coding for Programmers, Part 1

Erasure coding is an increasingly popular storage technology - allowing the same level of fault tolerance as replication with a significantly reduced storage footprint.

Increasingly, erasure coding is available 'out of the box' on storage solutions such as Ceph and OpenStack Swift. Normally, you'd just pull in a library like ISA-L or jerasure, and set some config options, and you'd be done.

This post is not about that. This post is about how I went from knowing nothing about erasure coding to writing POWER optimised routines to make it go fast. (These are in the process of being polished for upstream at the moment.) If you want to understand how erasure coding works under the hood - and in particular if you're interested in writing optimised routines to make it run quickly in your platform - this is for you.

What are erasure codes anyway?

I think the easiest way to begin thinking about erasure codes is "RAID 6 on steroids". RAID 6 allows you to have up to 255 data disks and 2 parity disks (called P and Q), thus allowing you to tolerate the failure of up to 2 arbitrary disks without data loss.

Erasure codes allow you to have k data disks and m 'parity' or coding disks. You then have a total of m + k disks, and you can tolerate the failure of up to m without losing data.

The downside of erasure coding is that computing what to put on those parity disks is CPU intensive. Lets look at what we put on them.

RAID 6

RAID 6 is the easiest way to get started on understanding erasure codes for a number of reasons. H Peter Anvin's paper on RAID 6 in the Linux kernel is an excellent start, but does dive in a bit quickly to the underlying mathematics. So before reading that, read on!

Rings and Fields

As programmers we're pretty comfortable with modular arithmetic - the idea that if you have:

unsigned char a = 255;
a++;

the new value of a will be 0, not 256.

This is an example of an algebraic structure called a ring.

Rings obey certain laws. For our purposes, we'll consider the following incomplete and somewhat simplified list:

  • There is an addition operation.
  • There is an additive identity (normally called 0), such that 'a + 0 = a'.
  • Every element has an additive inverse, that is, for every element 'a', there is an element -a such that 'a + (-a) = 0'
  • There is a multiplication operation.
  • There is a multiplicative identity (normally called 1), such that 'a * 1 = a'.

These operations aren't necessarily addition or multiplication as we might expect from the integers or real numbers. For example, in our modular arithmetic example, we have 'wrap around'. (There are also certain rules the addition and multiplication rules must satisfy - we are glossing over them here.)

One thing a ring doesn't have a 'multiplicative inverse'. The multiplicative inverse of some non-zero element of the ring (call it a), is the value b such that a * b = 1. (Often instead of b we write 'a^-1', but that looks bad in plain text, so we shall stick to b for now.)

We do have some inverses in 'mod 256': the inverse of 3 is 171 as 3 * 171 = 513, and 513 = 1 mod 256, but there is no b such that 2 * b = 1 mod 256.

If every non-zero element of our ring had a multiplicative inverse, we would have what is called a field.

Now, let's look at a the integers modulo 2, that is, 0 and 1.

We have this for addition:

+ 0 1
0 0 1
1 1 0

Eagle-eyed readers will notice that this is the same as XOR.

For multiplication:

* 0 1
0 0 0
1 0 1

As we said, a field is a ring where every non-zero element has a multiplicative inverse. As we can see, the integers modulo 2 shown above is a field: it's a ring, and 1 is its own multiplicative inverse.

So this is all well and good, but you can't really do very much in a field with 2 elements. This is sad, so we make bigger fields. For this application, we consider the Galois Field with 256 elements - GF(2^8). This field has some surprising and useful properties.

Remember how we said that integers modulo 256 weren't a field because they didn't have multiplicative inverses? I also just said that GF(2^8) also has 256 elements, but is a field - i.e., it does have inverses! How does that work?

Consider an element in GF(2^8). There are 2 ways to look at an element in GF(2^8). The first is to consider it as an 8-bit number. So, for example, let's take 100. We can express that as as an 8 bit binary number: 0b01100100.

We can write that more explicitly as a sum of powers of 2:

0 * 2^7 + 1 * 2^6 + 1 * 2^5 + 0 * 2^4 + 0 * 2^3 + 1 * 2^2 + 0 * 2 + 0 * 1
= 2^6 + 2^5 + 2^2

Now the other way we can look at elements in GF(2^8) is to replace the '2's with 'x's, and consider them as polynomials. Each of our bits then represents the coefficient of a term of a polynomial, that is:

0 x^7 + 1 x^6 + 1 x^5 + 0 x^4 + 0 x^3 + 1 x^2 + 0 x + 0 * 1

or more simply

x^6 + x^5 + x^2

Now, and this is important: each of the coefficients are elements of the integers modulo 2: x + x = 2x = 0 as 2 mod 2 = 0. There is no concept of 'carrying' in this addition.

Let's try: what's 100 + 79 in GF(2^8)?

100 = 0b01100100 => x^6 + x^5 +       x^2
 79 = 0b01001111 => x^6 +       x^3 + x^2 + x + 1

100 + 79         =>   0 + x^5 + x^3 +   0 + x + 1
                 =    0b00101011 = 43

So, 100 + 79 = 43 in GF(2^8)

You may notice we could have done that much more efficiently: we can add numbers in GF(2^8) by just XORing their binary representations together. Subtraction, amusingly, is the same as addition: 0 + x = x = 0 - x, as -1 is congruent to 1 modulo 2.

So at this point you might be wanting to explore a few additions yourself. Fortuantely there's a lovely tool that will allow you to do that:

sudo apt install gf-complete-tools
gf_add $A $B 8

This will give you A + B in GF(2^8).

> gf_add 100 79 8
43

Excellent!

So, hold on to your hats, as this is where things get really weird. In modular arithmetic example, we considered the elements of our ring to be numbers, and we performed our addition and multiplication modulo 256. In GF(2^8), we consider our elements as polynomials and we perform our addition and multiplication modulo a polynomial. There is one conventional polynomial used in applications:

0x11d => 0b1 0001 1101 => x^8 + x^4 + x^3 + x^2 + 1

It is possible to use other polynomials if they satisfy particular requirements, but for our applications we don't need to worry as we will always use 0x11d. I am not going to attempt to explain anything about this polynomial - take it as an article of faith.

So when we multiply two numbers, we multiply their polynomial representations. Then, to find out what that is modulo 0x11d, we do polynomial long division by 0x11d, and take the remainder.

Some examples will help.

Let's multiply 100 by 3.

100 = 0b01100100 => x^6 + x^5 + x^2
  3 = 0b00000011 => x + 1

(x^6 + x^5 + x^2)(x + 1) = x^7 + x^6 + x^3 + x^6 + x^5 + x^2
                         = x^7 + x^5 + x^3 + x^2

Notice that some of the terms have disappeared: x^6 + x^6 = 0.

The degree (the largest power of a term) is 7. 7 is less than the degree of 0x11d, which is 8, so we don't need to do anything: the remainder modulo 0x11d is simply x^7 + x^5 + x^3 + x^2.

In binary form, that is 0b10101100 = 172, so 100 * 3 = 172 in GF(2^8).

Fortunately gf-complete-tools also allows us to check multiplications:

> gf_mult 100 3 8
172

Excellent!

Now let's see what happens if we multiply by a larger number. Let's multiply 100 by 5.

100 = 0b01100100 => x^6 + x^5 + x^2
  5 = 0b00000101 => x^2 + 1

(x^6 + x^5 + x^2)(x^2 + 1) = x^8 + x^7 + x^4 + x^6 + x^5 + x^2
                           = x^8 + x^7 + x^6 + x^5 + x^4 + x^2

Here we have an x^8 term, so we have a degree of 8. This means will get a different remainder when we divide by our polynomial. We do this with polynomial long division, which you will hopefully remember if you did some solid algebra in high school.

                              1
                           ---------------------------------------------
x^8 + x^4 + x^3 + x^2 + 1 | x^8 + x^7 + x^6 + x^5 + x^4       + x^2
                          - x^8                   + x^4 + x^3 + x^2 + 1
                            -------------------------------------------
                          =       x^7 + x^6 + x^5       + x^3       + 1

So we have that our original polynomial (x^8 + x^4 + x^3 + x^2 + 1) is congruent to (x^7 + x^6 + x^5 + x^3 + 1) modulo the polynomial 0x11d. Looking at the binary representation of that new polynomial, we have 0b11101001 = 233.

Sure enough:

> gf_mult 100 5 8
233

Just to solidify the polynomial long division a bit, let's try a slightly larger example, 100 * 9:

100 = 0b01100100 => x^6 + x^5 + x^2
  9 = 0b00001001 => x^3 + 1

(x^6 + x^5 + x^2)(x^3 + 1) = x^9 + x^8 + x^5 + x^6 + x^5 + x^2
                           = x^9 + x^8 + x^6 + x^2

Doing long division to reduce our result:

                              x
                           -----------------------------------
x^8 + x^4 + x^3 + x^2 + 1 | x^9 + x^8       + x^6                   + x^2
                          - x^9                   + x^5 + x^4 + x^3       + x
                            -------------------------------------------------
                          =       x^8       + x^6 + x^5 + x^4 + x^3 + x^2 + x

We still have a polynomial of degree 8, so we can do another step:

                              x +   1
                           -----------------------------------
x^8 + x^4 + x^3 + x^2 + 1 | x^9 + x^8       + x^6                   + x^2
                          - x^9                   + x^5 + x^4 + x^3       + x
                            -------------------------------------------------
                          =       x^8       + x^6 + x^5 + x^4 + x^3 + x^2 + x
                          -       x^8                   + x^4 + x^3 + x^2     + 1
                                  -----------------------------------------------
                          =                   x^6 + x^5                   + x + 1

We now have a polynomial of degree less than 8 that is congruent to our original polynomial modulo 0x11d, and the binary form is 0x01100011 = 99.

> gf_mult 100 9 8
99

This process can be done more efficiently, of course - but understanding what is going on will make you much more comfortable with what is going on!

I will not try to convince you that all multiplicative inverses exist in this magic shadow land of GF(2^8), but it's important for the rest of the algorithms to work that they do exist. Trust me on this.

Back to RAID 6

Equipped with this knowledge, you are ready to take on RAID6 in the kernel (PDF) sections 1 - 2.

Pause when you get to section 3 - this snippet is a bit magic and benefits from some explanation:

Multiplication by {02} for a single byte can be implemeted using the C code:

uint8_t c, cc;
cc = (c << 1) ^ ((c & 0x80) ? 0x1d : 0);

How does this work? Well:

Say you have a binary number 0bNMMM MMMM. Mutiplication by 2 gives you 0bNMMMMMMM0, which is 9 bits. Now, there are two cases to consider.

If your leading bit (N) is 0, your product doesn't have an x^8 term, so we don't need to reduce it modulo the irreducible polynomial.

If your leading bit is 1 however, your product is x^8 + something, which does need to be reduced. Fortunately, because we took an 8 bit number and multiplied it by 2, the largest term is x^8, so we only need to reduce it once. So we xor our number with our polynomial to subtract it.

We implement this by letting the top bit overflow out and then xoring the lower 8 bits with the low 8 bits of the polynomial (0x1d)

So, back to the original statement:

(c << 1) ^ ((c & 0x80) ? 0x1d : 0)
    |          |          |     |
    > multiply by 2       |     |
               |          |     |
               > is the high bit set - will the product have an x^8 term?
                          |     |
                          > if so, reduce by the polynomial
                                |
                                > otherwise, leave alone

Hopefully that makes sense.

Key points

It's critical you understand the section on Altivec (the vperm stuff), so let's cover it in a bit more detail.

Say you want to do A * V, where A is a constant and V is an 8-bit variable. We can express V as V_a + V_b, where V_a is the top 4 bits of V, and V_b is the bottom 4 bits. A * V = A * V_a + A * V_b

We can then make lookup tables for multiplication by A.

If we did this in the most obvious way, we would need a 256 entry lookup table. But by splitting things into the top and bottom halves, we can reduce that to two 16 entry tables. For example, say A = 02.

V_a A * V_a
00 00
01 02
02 04
... ...
0f 1e
V_b A * V_b
00 00
10 20
20 40
... ...
f0 fd

We then use vperm to look up entries in these tables and vxor to combine our results.

So - and this is a key point - for each A value we wish to multiply by, we need to generate a new lookup table.

So if we wanted A = 03:

V_a A * V_a
00 00
01 03
02 06
... ...
0f 11
V_b A * V_b
00 00
10 30
20 60
... ...
f0 0d

One final thing is that Power8 adds a vpermxor instruction, so we can reduce the entire 4 instruction sequence in the paper:

vsrb v1, v0, v14
vperm v2, v12, v12, v0
vperm v1, v13, v13, v1
vxor v1, v2, v1

to 1 vpermxor:

vpermxor v1, v12, v13, v0

Isn't POWER grand?

OK, but how does this relate to erasure codes?

I'm glad you asked.

Galois Field arithmetic, and its application in RAID 6 is the basis for erasure coding. (It's also the basis for CRCs - two for the price of one!)

But, that's all to come in part 2, which will definitely be published before 7 April!

Many thanks to Sarah Axtens who reviewed the mathematical content of this post and suggested significant improvements. All errors and gross oversimplifications remain my own. Thanks also to the OzLabs crew for their feedback and comments.

March 18, 2017

Codec 2 700C and Short LDPC Codes

In the last blog post I evaluated FreeDV 700C over the air. This week I’ve been simulating the use of short LDPC FEC codes with Codec 2 700C over AWGN and HF channels.

In my HF Digital Voice work to date I have shied away from FEC:

  1. We didn’t have the bandwidth for the extra bits required for FEC.
  2. Modern, high performance codes tend to have large block sizes (1000’s of bits) which leads to large latency (several seconds) when applied to low bit rate speech.
  3. The error rates we are interested in (e.g. 10% raw, 1% after FEC decoder) are unusual – many codes don’t work well.

However with Codec 2 pushed down to 700 bit/s we now have enough bandwidth for a rate 1/2 code inside a standard 2kHz SSB channel. Over coffee a few weeks ago, Bill VK5DSP offered to develop some short LDPC codes for me specifically for this application. He sent me an Octave simulation of rate 1/2 and 2/3 codes of length 112 and 56 bits. Codec 2 700C has 28 bit frames so this corresponds to 4 or 2 Codec 2 700C frames, which would introduce a latencies of between 80 to 160ms – quite acceptable for Push To Talk (PTT) radio.

I re-factored Bill’s simulation code to produce ldpc_short.m. This measures BER and PER for Bill’s short LDPC codes, and also plots curves for theoretical, HF multipath channels, a Golay (24,12) code, and the current diversity scheme used in FreeDV 700C.

To check my results I compared the Golay BER and ideal HF multipath (Rayleigh Fading) channel curves to other peoples work. Always a good idea to spot check a few values and make sure they are sensible. I took a simple approach to get results in a reasonable amount of coding time (about 1 day of work in this case). This simulation runs at the symbol rate, and assumes ideal synchronisation. My other modem work (i.e experience) lets me move back and forth between this sort of simulation and real world modems, for example accounting for synchronisation losses.

Error Distribution and Packet Error Rate

I had an idea that Packet Error Rate (PER) might be important. Without FEC, bit errors are scattered randomly about. At our target 1% BER, many frames will have 1 or 2 bit errors. As discussed in the last post Codec 2 700C is sensitive to bit errors as “every bit counts”. For example one bit error in the Vector Quantiser (VQ) index (a big look up table) can throw the speech spectrum right off.

However a LDPC decoder will tend to correct all errors in a codeword, or “die trying” (i.e. fail badly). So an average output BER of say 1% will consist of a bunch of perfect frames, plus a completely trashed one every now and again. Digital voice works better with this style of error pattern than a few random errors in each codec packet. So for a given BER, a system that delivers a lower PER is better for our application. I’ve guesstimated a 10% PER target for intelligible low bit rate speech. Lets see how that works out…..

Results

Here are the BER and PER curves for an AWGN channel:

Here are the same curves for HF (multipath fading) channel:

I’ve included a Golay (24,12) block code (hard decision) and uncoded PSK for comparison to the AWGN curves, and the diversity system on the HF curves. The HF channel is modelled as two paths with 1Hz Doppler spread and a 1ms delay.

The best LDPC code reaches the 1% BER/10% PER point at 2dB Eb/No (AWGN) and 6dB (HF multipath). Comparing BER, the coding gain is 2.5 and 3dB (AWGN and HF). Comparing PER, the coding gain is 3 and 5dB (AWGN and HF).

Here is a plot of the error pattern over time using the LDPC code on a HF channel at Eb/No of 6dB:

Note the errors are confined to short bursts – isolated packets where the decoder fails. Even though the average BER is 1%, most of the speech is error free. This is a very nice error distribution for digital speech.

Speech Samples

Here are some speech samples, comparing the current diversity scheme used for FreeDV 700C to LDPC, for AWGN and LDPC channels. These were simulated by extracting the error pattern from the simulation then inserting these errors in a Codec 2 700C bit stream (see command lines section below).

AWGN Eb/No 2dB Diversity LDPC
HF Eb/No 6dB Diversity LDPC

Next Steps

These results are very encouraging and suggest a gain of 2 to 5dB over FreeDV 700C, and better error distribution (lower PER). Next step is to develop FreeDV 700D – a real world implementation using the 112 data-bit rate 1/2 LDPC code. This will require 4 frames of buffering, and some sort of synchronisation to determine the 112 bit frame boundaries. Fortunately much of the C code for these LDPC codes already exists, as it was developed for the Wenet High Altitude Balloon work.

If most frames at the decoder input are now error free, we can consider more efficient (but less robust) techniques for Codec 2, such as prediction (delta coding). This will decrease the codec bit rate for a given speech quality. We could then choose to reduce our bit rate (making the system more robust for a given channel SNR), or raise speech quality while maintaining the same bit rate.

Command Lines

Generating the decoded speech, first run the Octave ldpc_short simulation to generate “error pattern file”, then subject the Codec 2 700C bit stream to these error patterns.

octave:67> ldpc_short
$ ./c2enc 700C ../../raw/ve9qrp_10s.raw - | ./insert_errors - - ../../octave/awgn_2dB_ldpc.err 28 | ./c2dec 700C - - | aplay -f S16_LE -

The simulation generate .eps files as direct generation of PNG leads to font size issues. Converting EPS to PNG without transparent background:

mogrify -resize 700x600 -density 300 -flatten -format png *.eps

However I still feel the images are a bit fuzzy, especially the text. Any ideas? Here’s the eps file if some one would like to try to get a nicer PNG conversion for me! The EPS file looks great at any scaling when I render it using the Ubuntu document viewer.

Update: A friend of mine (Erich) has suggested using GIMP for the conversion. This does seem to work well and has options for text and line anti-aliasing. It would be nice to be able to generate nice PNGs directly from Octave – my best approach so far is to capture screen shots.

Links

LowSNR site Bill VK5DSP writes about his experiments in low SNR communications.

Wenet High Altitude Balloon SSDV System developed with Mark VK5QI and BIll VK5DSP that uses LDPC codes.

LPDC using Octave and the CML library

FreeDV 700C

Codec 2 700C

St Patrick’s Day 2017 – and a free resource on Irish in Australia

Happy St Patrick’s day!

Slane AbbeyAnd “we have a resource on that” – that is, on the Irish in Australia and the major contributions they made since the very beginning of the colonies. You can get that lovely 5 page resource PDF for free if you check out using coupon code TRYARESOURCE. It’s an option we’ve recently put in place so anyone can grab one resource of their choice to see if they like our materials and assess their quality.

StPatric statue - Slane AbbeyView of Tara from Slane AbbeyBack to St Patrick, we were briefly in Ireland last year and near Dublin we drove past a ruin at the top of a hill that piqued our interest, so we stopped and had a look. It turned out to be Slane Abbey, the site where it is believed in 433 AD, the first Christian missionary to Ireland, later known as St Patrick, lit a large (Easter) celebration fire (on the Hill of Slane). With this action he (unwittingly?) contravened orders by King Laoghaire at nearby Tara. The landscape photo past the Celtic cross shows the view towards Tara. Ireland is a beautiful country, with a rich history.

Slane Abbey - info plaquePhotos by Arjen Lentz & Dr Claire Reeler

March 15, 2017

Am I a Neanderthal?

Early reconstruction of NeanderthalEarly reconstruction of Neanderthal

The whole question of how Neanderthals are related to us (modern humans) has been controversial ever since the first Neanderthal bones were found in Germany in the 19th century. Belonging to an elderly, arthritic individual (a good example of how well Neanderthals cared for each other in social groups), the bones were reconstructed to show a stooping individual, with a more ape-like gait, leading to Neanderthals being described as the “Missing Link” between apes and humans, and given the epithet “ape-man”.

Who were the Neanderthals?

Modern reconstruction – Smithsonian Museum of Natural History

Neanderthals lived in the lands surrounding the Mediterranean Sea, and as far east as the Altai Mountains in Central Asia, between about 250,000 and about 30,000 years ago. They were a form of ancient human with certain physical characteristics – many of which probably helped them cope with the cold of Ice Ages. Neanderthals evolved out of an earlier ancestorHomo erectus, possibly through another species – Homo heidelbergensis. They had a larger brain than modern humans, but it was shaped slightly differently, with less development in the prefrontal cortex, which allows critical thinking and problem-solving, and larger development at the back of the skull, and in areas associated with memory in our brains. It is possible that Neanderthals had excellent memory, but poor analytical skills. They were probably not good at innovation – a skill which became vital as the Ice Age ended and the global climate warmed, sea levels rose and plant and animal habitats changed.

Neanderthals were stockier than modern humans, with shorter arms and legs, and probably stronger and all-round tougher. They had a larger rib cage, and probably bigger lungs, a bigger nose, larger eyes and little to no chin. Most of these adaptations would have helped them in Ice Age Europe and Asia – a more compact body stayed warmer more easily and was tough enough to cope with a harsh environment. Large lungs helped oxygenate the blood and there is evidence that they had more blood supply to the face – so probably had warm, ruddy cheeks. The large nose warmed up the air they breathed, before it reached their lungs, reducing the likelihood of contracting pneumonia. Neanderthals are known to have had the same range of hair colours as modern humans and fair skin, red hair and freckles may have been more common.

They made stone tools, especially those of the type called Mousterian, constructed simple dwellings and boats, made and used fire, including for cooking their food, and looked after each other in social groups. Evidence of skeletons with extensive injuries occurring well before death, shows that these individuals must have been cared for, not only whilst recovering from their injuries, but also afterwards, when they would probably not have been able to obtain food themselves. Whether or not Neanderthals intentionally buried their dead is an area of hot controversy. It was once thought that they buried their dead with flowers in the grave, but the pollen was found to have been introduced accidentally. However, claims of intentional burial are still debated from other sites.

What Happened to the Neanderthals?

Abrigo do Lagar Velho

Anatomically modern humans emerged from Africa about 100,000 years ago. Recent studies of human genetics suggests that modern humans had many episodes of mixing with various lineages of human ancestors around the planet. Modern humans moved into Asia and Europe during the Ice Age, expanding further as the Ice Age ended. Modern humans overlapped with Neanderthals for about 60,000 years, before the Neanderthals disappeared. It is thought that a combination of factors led to the decline of Neanderthals. Firstly, the arrival of modern humans, followed by the end of the Ice Age, brought about a series of challenges which Neanderthals might have been unable to adapt to, as quickly as necessary. Modern humans have more problem solving and innovation capability, which might have meant that they were able to out-compete Neanderthals in a changing environment. The longest held theory is that out ancestors wiped out the Neanderthals in the first genocide in (pre)history. A find of Neanderthals in a group, across a range of ages, some from the same family group, who all died at the same time, is one of the sites, which might support this theory, although we don’t actually know who (or what) killed the group. Cut marks on their bones show that they were killed by something using stone tools. Finally, there is more and more evidence of what are called “transitional specimens”. These are individuals who have physical characteristics of both groups, and must represent inter-breeding. An example is the 4 year old child from the site of Abrigo do Lagar Velho in Portugal, which seems to have a combination of modern and Neanderthal features. The discovery of Neanderthals genes in many modern people living today is also proof that we must have interbred with Neanderthals in the past. It is thought that the genes were mixed several times, in several parts of the world.

Am I a Neanderthal?

So how do we know if we have Neanderthals genes? Neanderthal genes have some physical characteristics, but also other attributes that we can’t see. In terms of physical characteristics, Neanderthal aspects to the skull include brow ridges (ridges of bone above the eyes, under the eyebrows); a bump on the back of the head – called an occipital chignon, or bun, because it looks like a ‘bun’ hairstyle, built into the bone; a long skull (like Captain Jean-Lu Picard from Star Trek – actor Patrick Stewart); a small, or non-existent chin; a large nose; a large jaw with lots of space for wisdom teeth; wide fingers and thumbs; thick, straight hair; large eyes; red hair, fair skin and freckles! The last may seem a little surprising, but it appears that the genes for these characteristics came from Neanderthals – who had a wide range of hair colours, fair skin and, occasionally, freckles. Increased blood flow to the face also would have given Neanderthals lovely rosy cheeks!

Less obvious characteristics include resistance to certain diseases – parts of our immune systems, especially with reference to European and Asian diseases; less positively, an increased risk of other diseases, such as type 2 diabetes. Certain genes linked to depression are present, but ‘switched off’ in Neanderthals. The way that these genes link to depression, and their role in the lifestyles of early people (where they may have had benefits that are no longer relevant) are future topics for research and may help us understand more about ourselves.

Neanderthals genes are present in modern populations from Europe, Asia, Northern Africa, Australia and Oceania. So, depending on which parts of the world our ancestry is from, we may have up to 4% of our genetics from long-dead Neanderthal ancestors!

March 14, 2017

Japanese House in Python for Minecraft

I have kids that I'm teaching to hack. They started of on Scratch (which is excellent) and are ready to move up to Python. They also happen to be mad Minecraft fans, so now they're making their way through Adventures in Minecraft.

As I used Scratch when they were, I'm also hacking in Python & Minecraft as they are. It helps if I hit the bumps and hurdles before they do, as well as have a sound handle on the problems they're solving.

Now I've branched out from the tutorial and I'm just having fun with it and leaving behind code the kids can use, hack whatever. This code is in my minecraft-tools repo (for want of a better name). It's just a collection of random tools I've written for Minecraft aren't quite up to being their own thing. I expect this will mostly be a collection of python programs to construct things inside Minecraft via CanaryMod and CanaryRaspberryJuicePlugin.

The first bit of code to be shaken out of the tree is japanese_house.py which produces a Minecraft interpretation of a classic Japanese house. Presently it only produces the single configuration that is little more than an empty shell.

Japanese House (day) Japanese House (night)

I intend to add an interior fit out plus a whole bunch of optional configurations you can set at run time but for now it is what it is, as I'm going to move onto writing geodesic domes and transport | teleport rings (as per the Expanse, which will lead to eventually coding a TARDIS, that will you know, be actually bigger on the inside ;-)

Testing FreeDV 700C

Since releasing FreeDV 700C I’ve been “instrumenting” the FreeDV GUI program – adding some code to perform various tests of the 700C waveform, especially over the air.

With the kind help of Gerhard OE3GBB, Mark VK5QI, and Peter VK5APR, I have collected some samples and performed some tests. The goals of this work were:

  1. Compare 700C Over the Air (OTA) to simulation on an AWGN channel.
  2. Compare 700C OTA to SSB on an AWGN channel.

Instrumentation

Here is a screen shot of the latest FreeDV GUI Options screen:

I’ve added some features to the top three rows:

Test Frames Send a payload of known test bits rather than vocoder bits
Channel Noise Simulate a channel using AWGN noise
SNR SNR of AWGN noise
Attn Carrier Attenuate just one carrier
Carrier The 700C carrier (1-14) to attenuate
Simulated Interference Tone Enable an interfering sine wave of specified frequency and amplitude
Clipping Enable clipping of 700C tx waveform, to increase RMS power
Diversity Combine for plots Scatter and Test Frame plots use combined (7 carrier) information.

To explore these options it is useful to run in full duplex mode (Tools-PTT Half Duplex unchecked) and using a loopback sound device:

  $ sudo modprobe snd-aloop

More information on loopback in the FreeDV GUI README.

Clipping the 700C tx waveform reduces the Peak to Average Power ratio (PAPR) which may be result in a higher average power over the channel. However clipping distorts the waveform and add some “shoulders (i.e. noise) to the spectrum adjacent to the 700C waveform:

Several users have noticed this distortion. At this stage I’m unsure if clipping is useful or not.

The Diversity Combine option is useful to explore each of the 14 carriers separately before they are combined into 7 carriers.

Many of these options were designed to explore tx filtering. I have long wondered if any of the FreeDV carriers were receiving less power than others, for example due to ripple or a low pass response from the crystal filter. A low power carrier would have a high bit error rate, adversely affecting overall performance. Plotting the scatter diagram or bit error rate on a carrier by carrier basis can measure the effect of tx filtering – if it exists.

Some of the features above – like attenuating a single carrier – were designed to “test the test”. Much of the work I do on FreeDV (and indeed other projects) involves carefully developing software and writing “code to test the code”. For example to build the experiments described in this blog post I worked several hours day for several weeks. Not glamorous, but where the real labour lies in R&D. Careful, meticulous testing and experimentation. One percent inspiration … then code, test, test.

Comparing Analog SSB to Digital Voice

One of my goals is to develop a HF DV system that is competitive with analog SSB. So we need a way to compare analog and DV at the same SNR. So I came with the idea of a wave files of analog SSB and DV which have the same average (RMS) power. If these are fed into a SSB transmitter, then they will be received at the same SNR. I added 10 seconds of a 1000Hz sine wave at the start for good measure – this could be used to measure the actual SNR.

I developed two files:

  1. sine_analog_700c
  2. sine_analog_testframes700c

The first has the same voice signal in analog and 700C, the second uses test frames instead of encoded voice.

Interfering Carriers

One feature described above simulates an interfering carrier (like a birdie), something I have seen on the air. Here is a plot of a carrier in the middle of one of the 700C carriers, but about 10dB higher:

The upper RH plot is a rolling plot of bit errors for each carrier. You can see one carrier is really messed up – lots of bit errors. The average bit error rate is about 1%, which is where FreeDV 700C starts to become difficult to understand. These bit errors would not be randomly distributed, but would affect one part of the codec all the time. For example the pitch might be consistently wrong, or part of the speech spectrum. I found that as long as the interfering carrier is below the FreeDV carrier, the effect on bit error rate is negligible.

Take away: The tx station must tune away from any interfering carriers that poke above the FreeDV signal carriers. Placing the interfering tones between FreeDV carriers is another possibility, e.g. a 50Hz shift of the tx signal.

Results – Transmit Filtering

Simulation results suggest 700C should produce reasonable results near 0db SNR. So that’s the SNR I’m shooting for in Over The Air (OTA) testing.

Mark VK5QI sent me several minutes of test frames so I could determine if there were any carriers with dramatically different bit error rates, which would indicate the presence of some tx filtering. Here is the histogram of BERs for each carrier for Mark’s signal, which was at about 3dB SNR:

There is one bar for each I and Q QPSK bit of the 14 carriers – 28 bars total (note Diversity combination was off). After running for a few minutes, we can see a range of 5E-2 and 8E-2 (5 to 8%). In terms of AWGN modem performance, this is only about 1dB difference in SNR or Eb/No, as per the BER versus Eb/No graphs in this post on the COHPSK modem used for 700C. One carrier being pinned at say 20% BER, or a slope of increasing BER with carrier frequency – would have meant tx filtering trouble.

Peter VK5APR, sent me a high SNR signal (he lives just 4 km away). Initially I could see a X shaped scatter diagram, a possible sign of tx filtering. However this ended up being some amplitude pumping due to Fast AGC on my radio. After I disabled fast AGC, I could see a scatter diagram with 4 clear dots, and no X-shape. Check.

I performed an additional test using my IC7200 as a transmitter, and a HackRF as a receiver. The HackRF has no crystal filter and a very flat response, so any distortion would be due to the IC7200 transmit filtering. Once again – 4 clean dots on the scatter diagram and no X-shape.

So I am happy to conclude that transmit filtering does not seem to be a problem, at least of the radios tested. All performance issues are therefore likely to be caused by me and my algorithms!

Results – Low SNR testing

Peter, VK5APR, configured his station to send the analog/700C equi-power test wave files described above at very low power, such that the received SNR at my station was about 0dB. As we are so close it was reasonable to assume the channel was AWGN, indeed we could see no sign of NVIS fading and the band (40M) was devoid of DX at the 12 noon test time.

Here is the rx signal I received, and the same file run through the 700C decoder. Neither the SSB or the decoded 700C audio are pretty. However it’s fair to say we could just get a message through on both modes and that 700C is holding it’s own next to SSB. The results are close to my simulations which was the purpose of this test.

You can decode the off air signal yourself if you download the first file and replay it through the FreeDV GUI program using “Tools – Start/Stop Play File from Radio”.

Discussion

While setting up these tests, Peter and I conversed comfortably for some time over FreeDV 700C at a high SNR. This proved to me that for our audience (experienced users of HF radio) – FreeDV 700C can be used for conversational contacts. Given the 700C codec is really just a first pass – that’s a fine result.

However it’s a near thing – the 700C codec adds a lot of distortion just compressing the speech. It’s pretty bad even if the SNR is high. The comments on the Codec 2 700C blog post indicate many lay-people can’t understand speech compressed by 700C. Add any bit errors (due to low SNR or fading) and it quickly becomes hard to understand – even for experienced users. This makes 700C very sensitive to bit errors as the SNR drops. But hey – every one of those 28 bits/frame counts at 700 bit/s so it’s not surprising.

In contrast, SSB scales a bit better with SNR. However even at high SNRs, that annoying hiss is always there – which is very fatiguing. Peter and I really noticed that hiss after a few minutes back on SSB. Yuck.

SSB gets a lot of it’s low SNR “punch” from making effective use of peak power. Here is a plot of the received SSB:

It’s all noise except for the speech peaks, where the “peak SNR” is much higher than 0dB. Our brains are adept at picking out words from those peaks, integrating the received phonetic symbols (mainly vowel energy) in our squishy biological receive filters. It’s a pity we didn’t evolve to detect coherent PSK. A curse on your evolution!

In contrast – 700C allocates just as much power to the silence between words as the most important parts of speech. This suggests we could do a better job at tailoring the codec and modem to peak power, e.g. allocating more power to parts of the speech that really matter. I had a pass at Time Variable Quantisation a few years ago. A variable rate codec might be called for, tightly integrated to the modem to pack more bits/power into perceptually important parts of speech.

The results above assumed equal average power for SSB and FreeDV 700C. It’s unclear if this happens in the real world. For example we may need to “back off” FreeDV drive further than SSB; SSB may use a compressor; and the PAs we are using are generally designed for PEP rather than average power operation.

Next Steps

I’m fairly happy with the baseline COHPSK modem, it seems to be hanging on OK as long as there aren’t any co-channel birdies. The 700C codec works better than expected, has plenty of room for improvement – but it’s sensitive to bit errors. So I’m inclined to try some FEC next. Aim for error free 700C at 0dB, which I think will be superior to SSB. I’ll swap out the diversity for FEC. This will increase the raw BER, but allow me to run a serious rate 0.5 code. I’ll start just with an AWGN channel, then tackle fading channels.

Links

FreeDV 700C
Codec 2 700C

Setting up a basic keystone for Swift + Keystone dev work

As a Swift developer, most of the development works in a Swift All In One (SAIO) environment. This environment simulates a mulinode swift cluster on one box. All the SAIO documentation points to using tempauth for authentication. Why?

Because most the time authentication isn’t the things we are working on. Swift has many moving parts, and so tempauth, which only exists for testing swift and is configured in the proxy.conf file works great.

However, there are times you need to debug or test keystone + swift integration. In this case, we tend to build up a devstack for keystone component. But if all we need is keystone, then can we just throw one up on a SAIO?… yes. So this is how I do it.

Firstly, I’m going to be assuming you have SAIO already setup. If not go do that first. not that it really matters, as we only configure the SAIO keystone component at the end. But I will be making keystone listen on localhost, so if you are doing this on anther machine, you’ll have to change that.

Further, this will set up a keystone server in the form you’d expect from a real deploy (setting up the admin and public interfaces).

 

Step 1 – Get the source, install and start keystone

Clone the sourcecode:
cd $HOME
git clone https://github.com/openstack/keystone.git

Setup a virtualenv (optional):
mkdir -p ~/venv/keystone
virtualenv ~/venv/keystone
source ~/venv/keystone/bin/activate

Install keystone:
cd $HOME/keystone
pip install -r requirements.txt
pip install -e .
cp etc/keystone.conf.sample etc/keystone.conf

Note: We are running the services from the source so config exists in source etc.

 

The fernet keys seems to assume a full /etc path, so we’ll create it. Maybe I should update this to put all config there but for now meh:
sudo mkdir -p /etc/keystone/fernet-keys/
sudo chown $USER -R /etc/keystone/

Setup the database and fernet:
keystone-manage db_sync
keystone-manage fernet_setup

Finally we can start keystone. Keystone is a wsgi application and so needs a server to pass it requests. The current keystone developer documentation seems to recommend uwsgi, so lets do that.

 

First we need uwsgi and the python plugin, one a debian/ubuntu system you:
sudo apt-get install uwsgi uwsgi-plugin-python

Then we can start keystone, by starting the admin and public wsgi servers:
uwsgi --http 127.0.0.1:35357 --wsgi-file $(which keystone-wsgi-admin) &
uwsgi --http 127.0.0.1:5000 --wsgi-file $(which keystone-wsgi-public) &

Note: Here I am just backgrounding them, you could run then in tmux or screen, or setup uwsgi to run them all the time. But that’s out of scope for this.

 

Now a netstat should show that keystone is listening on port 35357 and 5000:
$ netstat -ntlp | egrep '35357|5000'
tcp 0 0 127.0.0.1:5000 0.0.0.0:* LISTEN 26916/uwsgi
tcp 0 0 127.0.0.1:35357 0.0.0.0:* LISTEN 26841/uwsgi

Step 2 – Setting up keystone for swift

Now that we have keystone started, its time to configure it. Firstly you need the openstack client to configure it so:
pip install python-openstackclient

Next we’ll use all keystone defaults, so we only need to pick an admin password. For the sake of this how-to I’ll pick the developer documentation example of `s3cr3t`. Be sure to change this. So we can do a basic keystone bootstrap with:
keystone-manage bootstrap --bootstrap-password s3cr3t

Now we just need to set up some openstack env variables so we can use the openstack client to finish the setup. To make it easy to access I’ll dump them into a file you can source. But feel free to dump these in your bashrc or whatever:
cat > ~/keystone.env <<EOF
export OS_USERNAME=admin
export OS_PASSWORD=s3cr3t
export OS_PROJECT_NAME=admin
export OS_USER_DOMAIN_ID=default
export OS_PROJECT_DOMAIN_ID=default
export OS_IDENTITY_API_VERSION=3
export OS_AUTH_URL=http://localhost:5000/v3
EOF


source ~/keystone.env

 

Great, now  we can finish configuring keystone. Let’s first setup a service project (tennent) for our Swift cluster:
openstack project create service

Create a user for the cluster to auth as when checking user tokens and add the user to the service project, again we need to pick a password for this user so `Sekr3tPass` will do.. don’t forget to change it:
openstack user create swift --password Sekr3tPass --project service
openstack role add admin --project service --user swift

Now we will create the object-store (swift) service and add the endpoints for the service catelog:
openstack service create object-store --name swift --description "Swift Service"
openstack endpoint create swift public "http://localhost:8080/v1/AUTH_\$(tenant_id)s"
openstack endpoint create swift internal "http://localhost:8080/v1/AUTH_\$(tenant_id)s"

Note: We need to define the reseller_prefix we want to use in Swift. If you change it in Swift, make sure you update it here.

 

Now we can add roles that will match to roles in Swift, namely an operator (someone who will get a Swift account) and reseller_admins:
openstack role create SwiftOperator
openstack role create ResellerAdmin

Step 3 – Setup some keystone users to auth as.

TODO: create all the tempauth users here

 

Here, it would make sense to create the tempauth users devs are used to using, but I’ll just go create a user so you know how to do it. First create a project (tennent) for this example demo:
openstack project create --domain default --description "Demo Project" demo

Create a user:
openstack user create --domain default --password-prompt matt

We’ll also go create a basic user role:
openstack role create user

Now connect the 3 pieces together by adding user matt to the demo project with the user role:
openstack role add --project demo --user matt user

If you wanted user matt to be a swift operator (have an account) you’d:
openstack role add --project demo --user matt SwiftOperator

or even a reseller_admin:
openstack role add --project demo --user matt ResellerAdmin

If your in a virtual env, you can leave it now, because next we’re going to go back to your already setup swift to do the Swift -> Keystone part:
deactivate

Step 4 – Configure Swift

To get swift to talk to keystone we need to add 2 middlewares to the proxy pipeline. And in the case of a SAIO, remove the tempauth middleware. But before we do that we need to install the keystonemiddleware to get one of the 2 middlware’s, keystone’s authtoken:
sudo pip install keystonemiddleware

Now you want to replace your tempauth middleware in the proxy path pipeline with authtoken keystoneauth so it looks something like:
pipeline = catch_errors gatekeeper healthcheck proxy-logging cache bulk tempurl ratelimit crossdomain container_sync authtoken keystoneauth staticweb copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server

Then in the same ‘proxy-server.conf’ file you need to add the paste filter sections for both of these new middlewares:
[filter:authtoken]
paste.filter_factory = keystonemiddleware.auth_token:filter_factory
auth_host = localhost
auth_port = 35357
auth_protocol = http
auth_uri = http://localhost:5000/
admin_tenant_name = service
admin_user = swift
admin_password = Sekr3tPass
delay_auth_decision = True
# cache = swift.cache
# include_service_catalog = False

[filter:keystoneauth]
use = egg:swift#keystoneauth
# reseller_prefix = AUTH
operator_roles = admin, SwiftOperator
reseller_admin_role = ResellerAdmin

Note: You need to make sure if you change the reseller_prefix here, you change it in keystone. And notice this is where you map operator_roles and reseller_admin_role in swift to that in keystone. Here anyone in with the keystone role admin or SwiftOperator are swift operators and those with the ResellerAdmin role are reseller_admins.

 

And that’s it. Now you should be able to restart your swift proxy and it’ll go off and talk to keystone.

 

You can use your Python swiftclient now to go talk, and whats better swiftclient understands the OS_* variables, so you can just source your keystone.env and talk to your cluster (to be admin) or export some new envs for the user you’ve created. If you want to use curl you can. But _much_ easier to use swiftclient.

 

Tip: You can use: swift auth to get the auth_token if you want to then use curl.

 

If you want to authenticate via curl then for v3, use: https://docs.openstack.org/developer/keystone/devref/api_curl_examples.html

 

Or for v2, I use:
url="http://localhost:5000/v2.0/tokens"
auth='{"auth": {"tenantName": "demo", "passwordCredentials": {"username": "matt", "password": ""}}}'

 

curl -s -d "$auth" -H 'Content-type: application/json' $url |python -m json.tool

 

or

curl -s -d "$auth" -H 'Content-type: application/json' $url |python -c "import sys, json; print json.load(sys.stdin)['access']['token']['id']"

To just print out the token. Although a simple swift auth would do all this for you.

LSM mailing list archive: this time for sure!

Following various unresolved issues with existing mail archives for the Linux Security Modules mailing list, I’ve set up a new archive here.

It’s a mailman mirror of the vger list.

March 13, 2017

How to Create a Venn Diagram with Independent Intersections in PowerPoint

A Venn diagram can be a great way to explain a business concept. This is generally not difficult to create in modern presentation software. I often use Google Slides for its collaboration abilities.

Where it becomes difficult is when you want to add a unique colour/pattern to an intersection, where the circles overlap. Generally you will either get one circle overlapping another, or if you set some transparency then the intersection will become a blend of the colours of the circles.

I could not work out how to do this in Google Slides, so on this occasion I cheated and did it in Microsoft PowerPoint instead. I then imported the resulting slide into Slides.

This worked for me in PowerPoint for Mac 2016. The process is probably the same on Windows.

Firstly, create a SmartArt Venn Diagram

Insert > SmartArt > Relationship > Basic Venn

alt

Separate the Venn circles

SmartArt Design > Convert > Convert to Shapes

alt

Ungroup shapes

Shape Format > Group Objects > Ungroup

alt

Split out the intersections

Shape Format > Merge Shapes > Fragment

alt

From there, you can select the intersection as an independent shape. You can treat each piece separately. Try giving them different colours or even moving them apart.

alt

This can be a simple but impactful way to get your point across.

This Week in HASS – term 1, week 7

This week our youngest students are looking in depth at different types of celebrations; slightly older students are examining how people got around in the ‘Olden Days’; and our older primary students have some extra time to finish their activities from last week.

Foundation to Year 3

First car made in Qld, 1902

In the stand-alone Foundation (Prep) unit (F.1), students are discussing celebrations – which ones do we recognise in Australia, how these compare with celebrations overseas, and what were these celebrations like in days gone by. Our integrated Foundation (Prep) unit (F.5) and students in Years 1 (1.1), 2 (2.1) and 3 (3.1), are examining Transport in the Past – how did their grandparents get around? How did people get around 100 years ago? How did kids get to school? How did people do the shopping? Students even get to dream about how we might get around in the future…

Years 3 to 6

Making mud bricks

At OpenSTEM we recognise that good activities, which engage students and allow for real learning, take time. Nobody likes to get really excited about something and then be rushed through it and quickly moved on to something else. This part of the unit has lots of hands-on activities for Year 3 (3.5) students in an integrated class with Year 4, as well as Year 4 (4.1), 5 (5.1) and 6 (6.1) students. In recognition of that, two weeks are allowed for the students to really get into making Ice Ages and mud bricks, and working out how to survive the challenges of living in a Neolithic village – including how to trade, count and write. Having enough time allows for consolidation of learning, as well as allowing teachers to potentially split the class into different groups engaged in different activities, and then rotate the groups through the activities over a 2 week period.

March 12, 2017

Prophets/Genesis/Terraforming Mars, Seek Menu, and More

Obvious continuation from my previous other posts with regards to prophets/pre-cogs: http://dtbnguyen.blogspot.com/2017/03/prophetspre-cogsstargate-program-8.html http://dtbnguyen.blogspot.com/2017/02/life-in-india-prophetspre-cogsstargate_82.html http://dtbnguyen.blogspot.com/2017/02/life-in-iran-examining-prophetspre-cogs.html http://dtbnguyen.blogspot.com/2017/02/

March 10, 2017

Solarized xmobar, trayer and dmenu for xmonad

Like many other people, Ethan Schoonover's precision colours for machines and people, known as Solarized is my preferred colour pallet, especially as I spend most of my time in terminals.

I've "Solarized" most things in my work environment except my window manager, xmonad. Well to be more specific, the xmobar, trayer and dmenu applications that I use within xmonad.

The commits:

Modifications were made to xmonad.hs, .xmobarrc and trayer in .xsession that switches out the existing and default colours for those I've selected from the Solarized palette. I now have a fully Solarized window manager that fits in much better with the rest of my workspace.

Relevant code snippets:

xmonad.hs

                , logHook = dynamicLogWithPP $ xmobarPP
                    { ppOutput = hPutStrLn xmproc
                    , ppCurrent = xmobarColor "#859900" "" . wrap "[" "]"
                    , ppVisible = xmobarColor "#2aa198" "" . wrap "(" ")"
                    , ppLayout = xmobarColor "#2aa198" ""
                    , ppTitle = xmobarColor "#859900" "" . shorten 50
                    }
            , modMask = mod4Mask -- Rebind Mod to the Windows key
            --, borderWidth = 1
            } `additionalKeys`
                    -- Custom dmenu launcher
                    [ ((mod4Mask, xK_p ), spawn "exe=`dmenu_path | dmenu -fn \"Open Sans-10\" -p \"λ:\" -nb \"#073642\" -nf \"#93a1a1\" -sb \"#002b36\" -sf \"#859900\"` && eval \"exec $exe\"")

.xmobarrc

-- Appearance
  font = "xft:OpenSans:size=10:antialias=true"
, bgColor = "#073642"
, fgColor = "#93a1a1"

...

--Plugins
, commands =
    -- CPU Activity Monitor
    ...
                         , "--low"      , "#2aa198"
                         , "--normal"   , "#859900"
                         , "--high"     , "#dc322f"
    ...
    -- cpu core temperature monitor
    ...
                         , "--low"      , "#2aa198"
                         , "--normal"   , "#859900"
                         , "--high"     , "#dc322f"
    ...
    -- Memory Usage Monitor
    ...
                         , "--low"      , "#2aa198"
                         , "--normal"   , "#859900"
                         , "--high"     , "#dc322f"
    ...
    -- Battery Monitor
    ...
                         , "--low"      , "#dc322f"
                         , "--normal"   , "#859900"
                         , "--high"     , "#2aa198"

                         , "--" -- battery specific options
                                   -- discharging status
                                   , "-o"   , "<left>% (<timeleft>)"
                                   -- AC "on" status
                                   , "-O" , "<fc=#2aa198>Charging</fc>"
                                   -- charged status
                                   , "-i"   , "<fc=#859900>Charged</fc>"
    ...
    -- Weather Monitor
    ...
                         , "--low"      , "#2aa198"
                         , "--normal"   , "#859900"
                         , "--high"     , "#dc322f"
    ...
    -- Network Activity Monitor
    ...
                         , "--low"      , "#2aa198"
                         , "--normal"   , "#859900"
                         , "--high"     , "#dc322f"
    ...
    -- Time and Date Display
    , Run Date           "<fc=#268bd2>%a %b %_d %H:%M</fc>" "date" 10
    ...

.xsession

exec /usr/bin/trayer --edge top --align right --SetDockType true --SetPartialStrut true --expand true --width 10 --transparent true --alpha 0 --tint 0x073642 --height 22 &

That at least implements the colours I selected from Solarized and should be a good starting point for anyone else.

March 09, 2017

Hardening the LSM API

The Linux Security Modules (LSM) API provides security hooks for all security-relevant access control operations within the kernel. It’s a pluggable API, allowing different security models to be configured during compilation, and selected at boot time. LSM has provided enough flexibility to implement several major access control schemes, including SELinux, AppArmor, and Smack.

A downside of this architecture, however, is that the security hooks throughout the kernel (there are hundreds of them) increase the kernel’s attack surface. An attacker with a pointer overwrite vulnerability may be able to overwrite an LSM security hook and redirect execution to other code. This could be as simple as bypassing an access control decision via existing kernel code, or redirecting flow to an arbitrary payload such as a rootkit.

Minimizing the inherent security risk of security features, is, I believe, an essential goal.

Recently, as part of the Kernel Self Protection Project, support for marking kernel pages as read-only after init (ro_after_init) was merged, based on grsecurity/pax code. (You can read more about this in Kees Cook’s blog here). In cases where kernel pages are not modified after the kernel is initialized, hardware RO page protections are set on those pages at the end of the kernel initialization process. This is currently supported on several architectures (including x86 and ARM), with more architectures in progress.

It turns out that the LSM hook operations make an ideal candidate for ro_after_init marking, as these hooks are populated during kernel initialization and then do not change (except in one case, explained below). I’ve implemented support for ro_after_init hardening for LSM hooks in the security-next tree, aiming to merge it to Linus for v4.11.

Note that there is one existing case where hooks need to be updated, for runtime SELinux disabling via the ‘disable’ selinuxfs node. Normally, to disable SELinux, you would use selinux=0 at the kernel command line. The runtime disable feature was requested by Fedora folk to handle platforms where the kernel command line is problematic. I’m not sure if this is still the case anywhere. I strongly suggest migrating away from runtime disablement, as configuring support for it in the kernel (via CONFIG_SECURITY_SELINUX_DISABLE) will cause the ro_after_init protection for LSM to be disabled. Use selinux=0 instead, if you need to disable SELinux.

It should be noted, of course, that an attacker with enough control over the kernel could directly change hardware page protections. We are not trying to mitigate that threat here — rather, the goal is to harden the security hooks against being used to gain that level of control.

Quick Stats on zstandard (zstd) Performance

Was looking at using zstd for backup, and wanted to see the effect of different compression levels. I backed up my (built) bitcoin source, which is a decent representation of my home directory, but only weighs in 2.3GB. zstd -1 compressed it 71.3%, zstd -22 compressed it 78.6%, and here’s a graph showing runtime (on my laptop) and the resulting size:

zstandard compression (bitcoin source code, object files and binaries) times and sizes

For this corpus, sweet spots are 3 (the default), 6 (2.5x slower, 7% smaller), 14 (10x slower, 13% smaller) and 20 (46x slower, 22% smaller). Spreadsheet with results here.

March 08, 2017

Prophets/Pre-Cogs/Stargate Program 8, Github Download Script, and More

Obvious continuation from my previous other posts with regards to prophets/pre-cogs: http://dtbnguyen.blogspot.com/2017/02/life-in-india-prophetspre-cogsstargate_82.html http://dtbnguyen.blogspot.com/2017/02/life-in-iran-examining-prophetspre-cogs.html http://dtbnguyen.blogspot.com/2017/02/life-in-venezuela-examining-prophetspre.html http://dtbnguyen.blogspot.com/2017/01/

pudb debugging tips

As an OpenStack Swift dev I obviously write a lot of Python. Further Swift is cluster and so it has a bunch of moving pieces. So debugging is very important. Most the time I use pudb and then jump into the PyCharms debugger if get really stuck.

Pudb is curses based version of pdb, and I find it pretty awesome and you can use it while ssh’d somewhere. So I thought I’d write a tips that I use. Mainly so I don’t forget 🙂

The first and easiest way to run pudb is use pudb as the python runner.. i.e:

pudb <python script>

On first run, it’ll start with the preferences window up. If you want to change preferences you can just hit ‘<ctrl>+p’. However you don’t need to remember that, as hitting ‘?’ will give you a nice help screen.

I prefer to see line numbers, I like the dark vim theme and best part of all, I prefer my interactive python shell to be ipython.

While your debugging, like in pdb, there are some simple commands:

  • n – step over (“next”)
  • s – step into
  • c – continue
  • r/f – finish current function
  • t – run to cursor
  • o – show console/output screen
  • b – toggle breakpoint
  • m – open module
  • ! – Jump into interactive shell (most useful)
  • / – text search

There are obviously more then that, but they are what I mostly use. The open module is great if you need to set a breakpoint somewhere deeper in the code base, so you can open it, set a breakpoint and then happily press ‘c’ to continue until it hits. The ‘!’ is the most useful, it’ll jump you into an interactive python shell in the exact point the debugger is at. So you can jump around, check/change settings and poke in areas to see whats happening.

As with pdb you can also use code to insert a breakpoint so pudb will be triggered rather then having to start a script with pudb. I give an example of how in the nosetest section below.

nosetests + pudb

Sometimes the best way to use pudb is to debug unit tests, or even write a unit (or functaional or probe) test to get you into an area you want to test. You can use pudb to debug these too. And there are 2 ways to do it.

The first way is by installing the ‘nose-pudb’ pip package:

pip install nose-pudb

Now when you run nosetests you can add the –pudb option and it’ll break into pudb if there is an error, so you go poke around in ‘post-mortem’ mode. This is really useful, but doesn’t allow you to actually trace the tests as they run.

So the other way of using pudb in nosetests is actually insert some code in the test that will trigger as a breakpoint and start up pudb. To do so is exactly how you would with pdb, except substitute for pudb. So just add the following line of code to your test where you want to drop into pudb:

import pudb; pudb.set_trace()

And that’s it.. well mostly, because pudb is command line you need to tell nosetests to not capture stdout with the ‘-s’ flag:

nosetests -s test/unit/common/middleware/test_cname_lookup.py

testr + pudb

Not problem here, it uses the same approach as above. Where you programmatically set a trace, as you would for pdb. Just follow the  ‘Debugging (pdb) Tests’ section on this page (except substitute pdb for pudb)

 

Update – run_until_failure.sh

I’ve been trying to find some intermittent unit test failures recently. So I whipped up  a quick bash script that I run in a tmux session that really helps find and deal with them, I thought I’d add to this post as I then can add nose-pudb to make it pretty useful.

#!/bin/bash

n=0
while [ True ]
do 
  clear
  $@
  if [ $? -gt 0 ]
  then 
    echo 'ERROR'
    echo "number " $n
    break
  fi
  let "n=n+1"
  sleep 1
done

With this I can simply:
run_until_failure.sh tox -epy27

 

It’ll stop looping once the command passed returns something other then 0.

Once I have an error, I have then been focusing in on the area it happens (to speed up the search a bit), I can also use nose-pudb to drop me into post-mortem mode so I can poke around in ipython, for example, I’m currently running:

 

run_until_failure.sh nosetests --pudb test/unit/proxy/test_server.py

 

Then I can come back to the tmux session, if I’m dropped in a pudb interface, I can go poke around.

Oceanography and the Continents

Marie Tharp (30 July, 1920 – 23 August, 2006) was an oceanographer and cartographer who mapped the oceans of the world. She worked with Bruce Heezen, who collected data on a ship, mapping the ocean floor.

Tharp and Heezen

Tharp turned the data into detailed maps. At that time women were not allowed to work on research ships, as it was thought that they would bring bad luck! However, Tharp was a skilled cartographer, and as she made her maps of the floor of the oceans of the world, with their ridges and valleys, she realised that there were deep valleys which showed the boundaries of continental plates. She noticed that these valleys were also places with lots of earthquakes and she became convinced of the basics of plate tectonics and continental drift.

Between 1959 and 1963, Tharp was not mentioned in any of the scientific papers published by Heezen, and he dismissed her theories disparagingly as “girl talk”. As this video  from National Geographic shows, she stuck to her guns and was vindicated by the evidence, eventually managing to persuade Heezen, and the scientific community at large, of the validity of the theories. In 1977, Heezen and Tharp published a map of the entire ocean floor. Tharp obtained degrees in English, Music, Geology and Mathematics during the course of her life. In 2001, a few weeks before her 81st birthday, Marie Tharp was awarded the Lamont-Doherty Heritage Award at Columbia University, in the USA, as a pioneer of oceanography. She died of cancer in 2006.

The National Geographic video provides an excellent testimony to this woman pioneer in oceanography.

March 07, 2017

Multicore World 2017: A Review

Multicore World is a small conference held annually in New Zealand hosted by Open Parallel. What it lacks in numbers however it makes up in quality of the presenters. The 2017 conference included a typically impressive array of speakers dealing with some of the most difficult issues facing computational science, and included several important announcements in the fields of supercomputing, the Internet of Things, and manufacting issues.

read more

March 06, 2017

Adventures in Unemployment

Due to a recent corporate fire sale, implosion, what ever you'd like to call it, I found myself joining thousands of my former colleagues unemployed and looking for "new opportunities" (hire me, I'm dead set amazing).

As a parent who also has an ex-wife and children it is incumbent upon me to inform the Department of Human Services (DHS) of any changes to my income within strict time frames. So like a dutiful slave of the state, I called them to advise of my new $0 income status.

The following conversation actually happened:

[DHS] "So taking into account your new income of $0, you will need to pay $114 / month."

[McW] "With an income of $0, how would you expect me to pay that?"

[DHS] "Borrow money from family and friends."

[McW] "You know you just said that out loud, right?"

[DHS] "Yes sir."

[McW] "Okay, so let me clarify this. I have an income of $0, 3 dependent children living with me, one dependent adult and the DHS priority is not for me to provide food and shelter for them but to pay child support?"

[DHS] "That is correct."

[McW] "..and this is something you've not only said out loud but on a phone call that's being recorded for 'service quality and training purposes'."

[DHS] "That is the nature of the legislation and what we are trained to say."

[McW] "You do see the problem here, don't you?"

[DHS] "Yes sir, I do."

[McW] "Are there any other things you're trained to say that might help?"

[DHS] "You could apply for work benefits."

[McW] "Okay, let's think this one through. Let's say I did get the dole, which would be about $400 / fortnight, less than my fortnightly rent even before I commence buying food, would the DHS still want $114 from that?

[DHS] "Yes, child support would be taken from the benefits before they were paid to you."

[McW] [long pause] "Back to the $0 income and obvious incapacity to pay, when the inevitable non-payment occurs, what does the DHS do next?"

[DHS] "Despite your excellent payment history, the DHS would have to pursue avenues for collection."

[McW] "So I have a family of 6 to shelter and support and the DHS will still end up going collect to strip us of whatever they can? That's not particularly helpful to anyone, not those I'm directly supporting nor my children for whom the DHS is collecting child support."

[DHS] "That's correct sir, once a child support debt of $1,000 is accrued, DHS will pursue collection avenues. Is there anything else I can help you with?"

[McW] "Unless you can change the legislation, I think we're good here. Thank you."

Having been in the child support "game" for about 13 years, having seen female friends dudded by former male partners, have seen male friends rorted by former female partners, it's not as though I was unaware the system was truly broken and unfair to all parties in so many cases.

This conversation however, was truly breathtaking. I doubt Douglas Adams could have scripted this any better. :-)

The Week in HASS – term 1, week 6

HASS students have a global focus this week. The younger students are looking at calendars, celebrations and which countries classmates are connected to, around the world. Older students are starting to explore what happened at the end of the Ice Age and the beginnings of agriculture and trade. These students will also be applying the scientific method to practical examinations – creating their own mini Ice Ages in a bowl and making mud bricks.

OpenSTEM A0 world map: Country Outlines and Ice Age CoastlineFoundation to Year 3

Our standalone Foundation/Prep classes (F.1) are looking at calendars and celebrations this week, starting to explore the world beyond their own family and gain an identity relative to each other. Integrated Foundation/Prep (F.5) and Year 1 (1.1) classes; as well as Year 1 (1.1), 2 (2.1) and 3 (3.1) classes are examining our OpenSTEM blackline world map and putting coloured dots on all the countries that they and their families are connected to, either through relatives, or by having lived there themselves. It is through this sort of exercise that students can start to understand the concept of the “global family”.

Year 3 to Year 6

Making an Ice Age

Students in years 3 (3.5), 4 (4.1), 5 (5.1) and 6 (6.1) are consolidating their learning and expanding into subjects, such as Science and Economics and Business. The ever-popular Ice Ages and Mud Bricks activity links to core Science curricular strands and allows students to explore their learning in very tactile ways. Whilst undertaking the activity, students make a mini Ice Age in a bowl, attempting to predict what will happen to their clay landscape when it is flooded and frozen, and then comparing these predictions to their recorded observations, during empirical testing. Students also make their own mud bricks by hand, once again predicting how to make the bricks strongest and testing different construction techniques. We have even had classes test the strength of their mud brick walls under simulated flood conditions, working inside a tidy tray.

Making mud bricks

Students move on from studying the Ice Age, looking at what happened as the climate changed and global sea levels rose. The pressures that these changes brought to people’s lives is examined by looking at the origins of agriculture with domestic plants and animals. Students consider how people needed to wok together to survive. The cooperative Trade and Barter activity allows students to role play life in a Neolithic village. Faced with a range of challenges, such as floods and droughts, students discover how to prioritise their needs for food to survive the winter, against their wants. They also discover that trade, counting and writing all grew out of the needs for people to exchange items and help each other to survive. This activity covers all the basic concepts in the Economics and Business curriculum, whilst providing a context that is meaningful to the students and their own experiences. Replicating the way that people developed trade, counting and writing in the historic period, the students’ experiences during the Trade and Barter activity lay the foundations for a deeper understanding of the basic concepts of Economics and Business.

flooding the mud brick wall

March 05, 2017

Non self replicating reprap 3d printer

The reprap is designed to be able to "self replicate" to a degree. If a part on a reprap 3d printer breaks then a replacement part can be printed and attached. Parts can evolve as new ideas come along. Having parts crack or weaken on a 3d printer can be undesirable though.

A part on this printer was a mix of acrylic and PLA, both of which were cracked. Not quite what one would hope for as a foot of the y-axis. It is an interesting design with the two driving rods the same length as the alloy channel at the back of the printer.



A design I thought of called for 1/2 inch alloy in order to wrap the existing alloy extrusion with a 3mm cover. The dog bone on the slot is manually added in Fusion 360 so it is larger than needed. The whole thing being a learning exercise for me as to how to create 2.5D parts. The belt tensioning is on a 6mm subassembly that is mounted on the bracket in the right of the image below.


The bracket and subassembly are shown mounted below. Yes, using four M6 bolts to tension a belt is overkill. I would imagine you can stretch the belt to breaking point quite easily with these bolts. The two rods are locked into place using M3 tapped grub screws. The end brackets are bolted to the back extrusion using two M6 bolts.


The z-axis is now supported by a second 10mm alloy custom bracket. This combination makes it much, much harder to wobble the z-axis than the original design using plastic parts.




March 03, 2017

Service Innovation in New Zealand – the new digital transformation

Over the past fortnight I have had the pleasure and privilege of working closely with the Service Innovation team in the New Zealand Government to contribute to their next steps in achieving proper service integration. It was an incredible two weeks as part of an informal exchange between our agencies to share expertise and insights. I am very thankful for the opportunity to work with the team and to contribute in some small way to their visionary, ambitious and world leading agenda. I also recommend everyone watch closely the work of Service Innovation team, and contact them if you are interested in giving feedback on the model.

I spent a couple of weeks in New Zealand looking at their “Service Innovation” agenda, which is, I can confidently say, one of the most exciting things I’ve ever seen for genuine digital transformation. The Kiwis have in place already a strong and technically sound vision for service integration, a bunch of useful guidance (including one of the best gov produced API guides I’ve ever seen!) and a commitment to delivering integrated services as a key part of their agenda and programme along with brilliant skills and visionaries across government.

I believe New Zealand will be the one to watch over the coming year and, with a little luck, they could redefine the baseline for what everyone should be aspiring to. They could be the first government to properly demonstrate Gov as a Platform, not just better digital government, which is quite exciting! Systemic change and transformation generally happens once a generation if you are lucky, so do keep an eye on the Kiwis. They are set to  leave us all behind!

There is more internal documentation which I encourage the team to publish, like the Federated Services Model Reference Architecture and other gems.

In a couple of weeks, on the back of a raft of ongoing work, we analysed why it is with such great guidance available, why would siloed approaches still be happening? We found that the natural motivations of agencies would always drive an implementation that was designed to meet the specific agency needs rather than the system needs across government. That was unsurprising but the new insight was the service delivery teams themselves, who wanted to do the best possible implementations but with little time and resource, and high expectations, couldn’t take the time needed to find, read, interpret, translate into practice and verify implementation of the guidance. Which is quite fair! So we looked at models of reducing the barriers for those teams to do things better by providing reusable infrastructure and reference implementations, and either changing or tweaking the motivations of agencies themselves.

This is an ongoing piece of work, but fundamentally we looked at the idea that if we made the best technical path also the easiest path for service delivery teams to follow, then there would be a reasonable chance of a consumable systems approach to delivering these services. If support and skills was available with tools, code, dev environments, reference implementations, lab environments and other useful tools for designing and delivering government services faster, better and cheaper, then service delivery teams and agencies both would have a natural motivation to take that approach. Basically, we surmised that vision and guidance probably needed to be supplemented by implementation to make it real, moving from policy to application.

It is great to see other jurisdictions like New Zealand starting to experiment and implement the consumable mashable government model! I want to say a huge thank you to the New Zealand Government for sharing their ideas, but mostly for now picking up and being in such a great position to show everyone what Gov as a Platform and Gov as an API should look like. I wish you luck and hope to be a part of your success, even just in a small way!

Rock on.

March 02, 2017

Life in Iraq, Random Stuff, and More

- ~40M people. Main industries are petroleum, chemicals, textiles, leather, construction materials, food processing, fertilizer, metal fabrication/processing. Export goods are crude oil 84%, crude materials excluding fuels 8%, food and live animals 5%. Main export partners: China 22.6%, India 21.1%, South Korea 11.2%, United States 7.8%, Italy 6.7%, Greece 6% (2015). Main import partners: Turkey

February 28, 2017

Mildred Dresselhaus, the Queen of Carbon | NY Times

“Dr. Dresselhaus, who helped transform carbon into the superstar of modern materials science, was renowned for her efforts to promote the cause of women in science.”

Millie Dresselhaus (nee Spiewak) high school yearbook 19481948 A tribute at Hunter High School.

“Mildred (Millie) Dresselhaus, a professor emerita at the Massachusetts Institute of Technology whose research into the fundamental properties of carbon helped transform it into the superstar of modern materials science and the nanotechnology industry, died on Monday in Cambridge, Mass. She was 86.”

Read more.

February 27, 2017

LUV Main March 2017 Meeting: Multicore World / Patching with quilt

Mar 7 2017 18:30
Mar 7 2017 20:30
Mar 7 2017 18:30
Mar 7 2017 20:30
Location: 
Level 29, 570 Bourke St. Melbourne

PLEASE NOTE NEW LOCATION

Tuesday, March 7, 2017
6:30 PM to 8:30 PM
Level 29, 570 Bourke St. Melbourne

Speakers:

• Lev Lafayette, MultiCore World 2017 Wellington
• Russell Coker, Patching with quilt

570 Bourke St. Melbourne, between King and William streets

Late arrivals needing access to the building and the twenty-ninth floor please call 0490 627 326.
 

Before and/or after each meeting those who are interested are welcome to join other members for dinner. We are open to suggestions for a good place to eat near our venue.

LUV would like to acknowledge Dell for their help in obtaining the venue.

Linux Users of Victoria Inc., is an incorporated association, registration number A0040056C.

March 7, 2017 - 18:30

read more

LUV Beginners March Meeting: node.js workshop

Mar 18 2017 12:30
Mar 18 2017 16:30
Mar 18 2017 12:30
Mar 18 2017 16:30
Location: 
Infoxchange, 33 Elizabeth St. Richmond

node.js workshop

Node.js® is a JavaScript runtime built on Chrome's V8 JavaScript engine. Node.js uses an event-driven, non-blocking I/O model that makes it lightweight and efficient. Node.js' package ecosystem, npm, is the largest ecosystem of open source libraries in the world.

Members will be invited to install and learn about node.js with peer assistance.

The meeting will be held at Infoxchange, 33 Elizabeth St. Richmond 3121 (enter via the garage on Jonas St.) Late arrivals, please call (0421) 775 358 for access to the venue.

LUV would like to acknowledge Infoxchange for the venue.

Linux Users of Victoria Inc., is an incorporated association, registration number A0040056C.

March 18, 2017 - 12:30

read more

February 26, 2017

This Week in HASS – term 1, week 5

This week students are exploring a vast range of topics, across the year levels. From using a torch and a tennis ball to investigate how the Earth experiences Day and Night to case studies on natural disasters, celebrations and indigenous peoples, there is a broad range of topics to spark interest.

Foundation to Year 3

Our youngest students (Foundation/Prep – Unit F.1) are talking about where they, and other members of their family, were born. Once again, this activity gets them interacting with maps and thinking about how we represent locations, whilst reinforcing their sense of identity and how they relate to others. Students in Years 1 to 3 (Units 1.1, 2.1 and 3.1) are using a torch and a tennis ball to investigate how the Earth experiences Night and Day, Seasons, Equinoxes and Solstices. This activity ties in what we experience as weather, seasons and their related celebrations to the Physics of how it all works, allowing students to draw connections between what they experience and what they are learning, and providing essential context for the more abstract knowledge. Teachers can easily tailor this activity to the needs of each class and explore the concepts in as much detail as required.

Years 3 to 6

Charlotte St, Brisbane 1893 floodsCharlotte St, Brisbane 1893 floods

Students in Years 3 to 6 (Units 3.5, 4.1, 5.1 and 6.1) are looking at a range of different case studies pertinent to their year-level curriculum requirements, this week. Year 3 students are examining celebrations in Australia and around the world (the Celebrations Around the World resource has been updated this year, and contains some new material, please check that you have the latest copy, and re-download it if necessary) and Year 4 students examine areas of natural beauty in Australia. Year 5 students are looking at the effects of natural disasters, especially here in Australia. Case studies on floods, such as the Brisbane Floods of 1893, and bushfires, such as the infamous Black Friday fires in Victoria, are available for more in depth study by teachers and students wishing to explore the topic in more detail. Year 6 students are examining Indigenous groups of people from Australia and Asia. A range of case studies are available for this topic, from groups within Australia, holding Native Title, such as the Quandamooka People, to groups from the mountains of Southern China, such as the Yi people. The larger number of case studies available, which can be found in our store resource category Indigenous Peoples, allows for Year 6 students to pursue more individual lines of enquiry, suited to their developing abilities.

Life in Afghanistan, Random Stuff, and More

On Afghanistan: - the place reminds me of a lot of other Middle Eastern countries.... Despite this and their internal stability issues clear that they have their own 'identity'. The deeper you look the more you'll realise why the US/West is having such difficulties there Afghanistan, officially the Islamic Republic of Afghanistan, is a landlocked country located within South Asia and Central

Nyriad: An Agile Startup Done Right

I have recently spent a few days in the company of Nyriad, a New Zealand IT company specialisng in GPU software. I wish to make a point of a few observations of the company because they are an example of both a startup company that uses agile project management, two terms much maligned and subject to justified cynicism, and does it right. Because I have seen so many colleagues burned by companies and organisations which profess such values and do not do it right, I hope the following observations will be useful for future organisations.

read more

February 23, 2017

This Week in HASS – term 1, week 4

This week the Understanding Our World program for primary schools has younger students looking at time passing, both in their own lives and as marked by others, including the seasons recognised by different Aboriginal groups. Older students are looking at how Aboriginal people interacted with the Australian environment, as it changed at the end of the Ice Age, and how they learnt to manage the environment and codified that knowledge into their lore.

Foundation to Year 3

This week our standalone Foundation classes  (Unit F.1) are thinking about what they were like as babies. They are comparing photographs or drawings of themselves as babies, with how they are now. This is a great week to involve family members and carers into class discussions, if appropriate. Students in multi-age classes and Years 1 to 3 (Units F.5, 1.1, 2.1 and 3.1) , are examining how weather and seasons change throughout the year and comparing our system of seasons with those used by different groups of Aboriginal people in different parts of Australia. Students can compare these seasons to the weather where they live and think about how they would divide the year into seasons that work where they live. Students can also discuss changes in weather over time with older members of the community.

Years 3 to 6

Older students, having followed the ancestors of Aboriginal people all the way to Australia, are now examining how the climate changed in Australia after the Ice Age, and how this affected Aboriginal people. They learn how Aboriginal people adapted to their changing environment and learned to manage it in a sustainable way. This vitally important knowledge about how to live with, and manage, the Australian environment, was codified into Aboriginal lore and custom and handed down in stories and laws, from generation to generation. Students start to examine the idea of Country/Place, in this context.

February 22, 2017

Making a USB powered soldering iron that doesn't suck

Today's evil project was inspired by a suggestion after my talk on USB-C & USB-PD at this years's linux.conf.au Open Hardware miniconf.

Using a knock-off Hakko driver and handpiece I've created what may be the first USB powered soldering iron that doesn't suck (ok, it's not a great iron, but at least it has sufficient power to be usable).

Building this was actually trivial, I just wired the 20v output of one of my USB-C ThinkPad boards to a generic Hakko driver board, the loss of power from using 20v not 24v is noticeable, but for small work this would be fine (I solder in either the work lab or my home lab, where both have very nice soldering stations so I don't actually expect to ever use this).

If you were to turn this into a real product you could in fact do much better, by doing both power negotiation and temperature control in a single micro, the driver could instead be switched to a boost converter instead of just a FET, and by controlling the output voltage control the power draw, and simply disable the regulator to turn off the heater. By chance, the heater resistance of the Hakko 907 clone handpieces is such that combined with USB-PD power rules you'd always be boost converting, never needing to reduce voltage.

With such a driver you could run this from anything starting with a 5v USB-C phone charger or battery (15W for the nicer ones), 9v at up to 3A off some laptops (for ~25W), or all the way to 20V@5A for those who need an extremely high-power iron. 60W, which happens to be the standard power level of many good irons (such as the Hakko FX-888D) is also at 20v@3A a common limit for chargers (and also many cables, only fixed cables, or those specially marked with an ID chip can go all the way to 5A). As higher power USB-C batteries start becoming available for laptops this becomes a real option for on-the-go use.

Here's a photo of it running from a Chromebook Pixel charger:

February 21, 2017

$AUD 35k available in 2017 Grants Program

Linux Australia is delighted to announce the availability of $AUD 35,000
for open source, open data, open government, open education, open
hardware and open culture projects, as part of the organisation’s
commitment to free and open source systems and communities in the region.

This year, we have deliberately weighted some areas in which we strongly
welcome grant applications.

More information is available at: https://linux.org.au/projects/grants

Please do share this with colleagues who may find it of interest, and
feel free to contact the Linux Australia Council if you would like a
private discussion.

With kind regards,

Kathy Reid

President, Linux Australia

February 20, 2017

Life in India, Prophets/Pre-Cogs/Stargate Program 7, and More

On India: - your life is very much dependent on how you were born, how much you money you have, etc... In spite of being a capitalist, democracy it still bears aspects of being stuck with a caste/feudal/colonial system. Electrical power stability issues still. Ovens generally lacking. Pollution and traffic problems no matter where you live in India. They have the same number of hours in each

New LinkedIn Interface Delete Your Data? Here’s How to Bring it Back.

Over the past few years it has seemed like LinkedIn were positioning themselves to take over your professional address book. Through offering CRM-like features, users were able to see a summary of their recent communications with each connection as well as being able to add their own notes and categorise their connections with tags. It appeared to be a reasonable strategy for the company, and many users took the opportunity to store valuable business information straight onto their connections.

Then at the start of 2017 LinkedIn decided to progressively foist a new user experience upon its users, and features like these disappeared overnight in lieu of a more ‘modern’ interface. People who grew to depend on this integration were in for a rude shock — all of a sudden it was missing. Did LinkedIn delete the information? There was no prior warning given and I still haven’t seen any acknowledgement or explanation (leave alone an apology) from LinkedIn/Microsoft on the inconvenience/damage caused.

If anything, this reveals the risks in entrusting your career/business to a proprietary cloud service. Particularly with free/freemium (as in cost) services, the vendor is more likely to change things on a whim or move that functionality to a paid tier.

It’s another reason why I’ve long been an advocate for open standards and free and open source software.

Fortunately there’s a way to export all of your data from LinkedIn. This is what we’ll use to get back your tags and notes. These instructions are relevant for the new interface. Go to your account settings and in the first section (“Basics”) you should see an option called “Getting an archive of your data”.

altLinkedIn: Getting an archive of your data

Click on Request Archive and you’ll receive an e-mail when it’s available for download. Extract the resulting zip file and look for a file called Contacts.csv. You can open it in a text editor, or better yet a spreadsheet like LibreOffice Calc or Excel.

In my copy, my notes and tags were in columns D and E respectively. If you have many, it may be a lot of work to manually integrate them back into your address book. I’d love suggestions on how to automate this. Since I use Gmail, I’m currently looking into Google’s address book import/export format, which is CSV based.

As long as Microsoft/LinkedIn provide a full export feature, this is a good way to maintain ownership of your data. It’s good practice to take an export every now and then to give yourself some peace-of-mind and avoid vendor lock-in.

This article has also been published on LinkedIn.

February 19, 2017

HPC/Cloud Hybrids for Efficient Resource Allocation and Throughput

HPC systems running massively parallel jobs need a fairly static software operating environment running on bare metal hardware, a high speed interconnect to reach their full potential, and offer linear performance scaling for cleverly designed applications. Cloud computing, on the other hand, offers flexible virtual environments and can be used for pleasingly parallel workloads.

read more

CareerNexus: a new way to find work

The start-up that I have co-founded, CareerNexus, is looking for job seekers to take part in a product test and market experiment. If you, or someone you know, wants to know more and potentially take part, message me.

If we can help just a fraction of those people who have difficulty finding work through traditional means — people returning from parental leave, people looking for roles after being made redundant, mature workers, even some highly skilled professionals — we’ll be doing something great.

As an alternate means of finding work, it need not replace any mechanisms that you may already be engaged in. In other words, there is nothing for you to lose and hopefully much for you to gain.

Published in Engineers Without Borders Magazine

Engineers Without Borders asked me to write something for their Humanitarian Engineering magazine about One Laptop per Child. Here is what I wrote.

The school bell rings, and the children filter into the classroom. Each is holding an XO – their own personal learning device.

altStudents from Doomadgee often use their XOs for outdoors education. The sunlight-readable screen
combined with the built-in camera allow for hands-on exploration of their environment.

This is no ordinary classroom. As if by magic, the green and white XOs automatically see each other as soon as they are started up, allowing children to easily share information and collaborate on activities together. The kids converse on how they can achieve the tasks at hand. One girl is writing a story on her XO, and simultaneously on the same screen she can see the same story being changed by a boy across the room. Another group of children are competing in a game that involves maths questions.

altChildren in Kiwirrkurra, WA, collaborate on an activity with help from teachers.

Through the XO, the learning in this classroom has taken on a peer-to-peer character. By making learning more fun and engaging, children are better equipped to discover and pursue their interests. Through collaboration and connectivity, they can exchange knowledge with their peers and with the world. In the 21st century, textbooks should be digital and interactive. They should be up-to-date and locally relevant. They should be accessible and portable.

Of course, the teacher’s role remains vital, and her role has evolved into that of a facilitator in this knowledge network. She is better placed to provide more individual pathways for learning. Indeed the teacher is a learner as well, as the children quickly adapt to the new technology and learn skills that they can teach back.

altA teacher in Jigalong, WA, guides a workgroup of children in their class.

Helping to keep the classroom session smoothly humming along are children who have proven themselves to be proficient with assisting their classmates and fixing problems (including repairing hardware). These kids have taken part in training programmes that award them for their skills around the XO. In the process, they are learning important life skills around problem solving and teamwork.

altDozens of students in Doomadgee State School are proficient in fixing XO hardware.

This is all part of the One Education experience, an initiative from One Laptop per Child (OLPC) Australia. This educational programme provides a holistic educational scaffolding around the XO, the laptop developed by the One Laptop per Child Association that has its roots in the internationally-acclaimed MIT Media Lab in the USA.

The XO was born from a desire to empower each and every child in the world with their own personal learning device. Purpose-built for young children and using solid open source software, the XO provides an ideal platform for classroom learning. Designed for outdoors, with a rugged design and a high-resolution sunlight-readable screen, education is no longer confined to a classroom or even to the school grounds. Learning time needn’t stop with the school bell – many children are taking their XOs home. Also important is the affordability and full repairability of the devices, making it cost-effective versus non-durable and ephemeral items such as stationery, textbooks and other printed materials. There are over 3 million XOs in distribution, and in some countries (such as Uruguay) every child owns one.

altA One Education classroom in Kenya.

One Education’s mission is to provide educational opportunities to every child, no matter how remote or disadvantaged. The digital divide is a learning divide. This can be conquered through a combination of modern technology, training and support, provided in a manner that empowers local schools and communities. The story told above is already happening in many classrooms around the country and the world.

altA One Education classroom in northern Thailand.

With teacher training often being the Achilles’ heel of technology programmes in the field of education, One Education focuses only on teachers who have proven their interest and aptitude through the completion of a training course. Only then are they eligible to receive XOs (with an allocation of spare parts) into their classroom. Certified teachers are eligible for ongoing support from OLPC Australia, and can acquire more hardware and parts as required.

As a not-for-profit, OLPC Australia works with sponsors to heavily subsidise the costs of the One Education programme for low socio-economic status schools. In this manner, the already impressive total cost of ownership can be brought down even further.

High levels of teacher turnover are commonplace in remote Australian schools. By providing courses online, training can be scalable and cost-effective. Local teachers can even undergo further training to gain official trainer status themselves. Some schools have turned this into a business – sending their teacher-trainers out to train teachers in other schools.

altStudents in Geeveston in Tasmania celebrate their attainment of XO-champion status, recognising
their proficiency in using the XO and their helpfulness in the classroom.

With backing from the United Nations Development Programme, OLPC are tackling the Millennium Development Goals by focusing on Goal 2 (Achieve Universal Primary Education). The intertwined nature of the goals means that progress made towards this goal in turn assists the others. For example, education on health can lead to better hygiene and lower infant mortality. A better educated population is better empowered to help themselves, rather than being dependent on hand-outs. For people who cannot attend a classroom (perhaps because of remoteness, ethnicity or gender), the XO provides an alternative. OLPC’s focus on young children means that children are becoming engaged in their most formative years. The XO has been built with a minimal environmental footprint, and can be run off-grid using alternate power sources such as solar panels.

One Education is a young initiative, formed based on experiences learnt from technology deployments in Australia and other countries. Nevertheless, results in some schools have been staggering. Within one year of XOs arriving in Doomadgee State School in northern Queensland, the percentage of Year 3 pupils meeting national literacy standards leapt from 31% to 95%.

altA girl at Doomadgee State School very carefully removes the screen from an XO.

2013 will see a rapid expansion of the programme. With $11.7m in federal government funding, 50,000 XOs will be distributed as part of One Education. These schools will be receiving the new XO Duo (AKA XO-4 Touch), a new XO model developed jointly with the OLPC Association. This version adds a touch-screen user experience while maintaining the successful laptop form factor. The screen can swivel and fold backwards over the keyboard, converting the laptop into a tablet. This design was chosen in response to feedback from educators that a hardware keyboard is preferred to a touch-screen for entering large amounts of information. As before, the screen is fully sunlight-readable. Performance and battery life have improved significantly, and it is fully repairable as before.

As One Education expands, there are growing demands on OLPC Australia to improve the offering. Being a holistic project, there are plenty of ways in which we could use help, including in education, technology and logistics. We welcome you to join us in our quest to provide educational opportunities to the world’s children.

February 18, 2017

Choose your own adventure – keynote

This is a blog version of the keynote I gave at linux.conf.au 2017. Many thanks to everyone who gave such warm feedback, and I hope it helps spur people to think about systemic change and building the future. The speech can be watched at https://www.youtube.com/watch?v=J6IqGuxCKa8.

I genuinely believe we are at a tipping point right now. A very important tipping point where we have at our disposal all the philosophical and technical means to invent whatever world we want, but we’re at risk of reinventing the past with shiny new things. This talk is about trying to make active choices about how we want to live in future and what tools we keep or discard to get there. Passive choices are still a choice, they are choosing the status quo. We spend a lot of our time tinkering around the edges of life as it is, providing symptomatic relief for problems we find, but we need to take a broader systems based view and understand what systemic change we can make to properly address those problems.

We evolved over hundreds of thousands of years using a cooperative competitive social structure that helped us work together to flourish in every habitat, rapidly and increasingly evolve an learn, and establish culture, language, trade and travel. We were constantly building on what came before and we built our tools as we went.

In recent millennia we invented systems of complex differentiated and interdependent skills, leading to increasingly rapid advancements in how we live and organise ourselves physically, politically, economically and socially, especially as we started building huge cities. Lots of people meant a lot of time to specialise, and with more of our basic needs taken care of, we had more time for philosophy and dreaming.

Great progress created great surplus, creating great power, which we generally centralised in our great cities under rulers that weren’t always so great. Of course, great power also created great inequalities so sometimes we burned down those great cities, just to level the playing field. We often took a symptomatic relief approach to bad leaders by replacing them, without fundamentally changing the system.

But in recent centuries we developed the novel idea that all people have inalienable rights and can be individually powerful. This paved the way for a massive culture shift and distribution of power combined with heightened expectations of individuals in playing a role in their own destiny, leading us to the world as we know it today. Inalienable rights paved the way for people thinking differently about their place in the world, the control they had over their lives and how much control they were happy to cede to others. This makes us, individually, the most powerful we have ever beed, which changes the game moving forward.

You see, the internet was both a product and an amplifier of this philosophical transition, and of course it lies at the heart of our community. Technology has, in large part, only sped up the cooperative competitive models of adapting, evolving and flourishing we have always had. But the idea that anyone has a right to life and liberty started a decentralisation of power and introduced the need for legitimate governance based on the consent of citizens (thank you Locke).

Citizens have the powers of publishing, communications, monitoring, property, even enforcement. So in recent decades we have shifted fundamentally from kings in castles to nodes in a network, from scarcity to surplus or reuse models, from closed to open systems, and the rate of human progress only continues to grow towards an asymptoic climb we can’t even imagine.

To help capture this, I thought I’d make a handy change.log on human progress to date.

# Notable changes to homo sapiens – change.log
## [2.1.0] – 1990s CE “technology revolution & internet”
### Changed
– New comms protocol to distribute “rights”. Printing press patch unexpectedly useful for distributing resources. Moved from basic multi-core to clusters of independent processors with exponential growth in power distribution.

## [2.0.0] – 1789 CE “independence movements”
### Added
– Implemented new user permissions called “rights”, early prototype of multi-core processing with distributed power & comms.

## [1.2.0] – 1760 CE “industrial revolution”
### Changed
– Agricultural libraries replaced by industrial libraries, still single core but heaps faster.

## [1.1.1] – 1440 CE “gutenberg”
### Patched
– Printing press a minor patch for more efficient instructions distribution, wonder if it’d be more broadly useful?

## [1.1.0] – 2,000 BCE “cities era”
### Changed
– Switched rural for urban operating environment. Access to more resources but still on single core.

## [1.0.0] – 8,000 BCE “agricultural revolution”
### Added
– New agricultural libraries, likely will create surplus and population explosion. Heaps less resource intensive.

## [0.1.0] – 250,000 BCE “homo sapiens”
### Added
– Created fork from homo erectus, wasn’t confident in project direction though they may still submit contributions…

(For more information about human evolution, see https://www.bighistoryproject.com)

The point to this rapid and highly oversimplified historical introduction is threefold: 1) we are more powerful than ever before, 2) the rate of change is only increasing, and 3) we made all this up, and we can make it up again. It is important to recognise that we made all of this up. Intellectually we all understand this but it matters because we often assume things are how they are, and then limit ourselves to working within the constraints of the status quo. But what we invented, we can change, if we choose.

We can choose our own adventure, or we let others choose on our behalf. And if we unthinkingly implement the thinking, assumptions and outdated paradigms of the past, then we are choosing to reimplement the past.

Although we are more individually and collectively powerful than ever before, how often do you hear “but that’s just how we’ve always done it”, “but that’s not traditional”, or “change is too hard”. We are demonstrably and historically utter masters at change, but life has become so big, so fast, and so interrelated that change has become scary for many people, so you see them satisfied by either ignoring change or making iterative improvements to the status quo. But we can do better. We must do better.

I believe we are at a significant tipping point in history. The world and the very foundations our society were built on have changed, but we are still largely stuck in the past in how we think and plan for the future. If we don’t make some active decisions about how we live, think and act, then we will find ourselves subconsciously reinforcing the status quo at every turn and not in a position to genuinely create a better future for all.

So what could we do?

  • Solve poverty and hunger: distributed property through nanotechnology and 3D printing, universal education and income.
  • Work 2 days a week, automate the rest: work, see “Why the Future is Workless” by Tim Dunlop
  • Embrace and extend our selves: Transhumanism, para olympics, “He was more than a dolphin, but from another dolphin’s point of view he might have seemed like something less.” — William Gibson, from Johnny Mnemonic. Why are we so conservative about what it means to be human? About our picture of self? Why do we get caught up on what is “natural” when almost nothing we do is natural.
  • Overcome the tyranny of distance: rockets for international travel, interstellar travel, the opportunity to have new systems of organising ourselves
  • Global citizens: Build a mighty global nation where everyone can flourish and have their rights represented beyond the narrow geopolitical nature of states: peer to peer economy, international rights, transparent gov, digital democracy, overcome state boundaries,
  • ?? What else ?? I’m just scratching the surface!

So how can we build a better world? Luckily, the human species has geeks. Geeks, all of us, are special because we are the pioneers of the modern age and we get to build the operating system for all our fellow humans. So it is our job to ensure what we do makes the world a better place.

rOml is going to talk more about future options for open source in the Friday keynote, but I want to explore how we can individually and collectively build for the future, not for the past.

I would suggest, given our role as creators, it is incumbent on us to both ensure we build a great future world that supports all the freedoms we believe in. It means we need to be individually aware of our unconscious bias, what beliefs and assumptions we hold, who benefits from our work, whether diversity is reflected in our life and work, what impact we have on society, what we care about and the future we wish to see.

Collectively we need to be more aware of whether we are contributing to future or past models, whether belief systems are helping or hindering progress, how we treat others and what from the past we want to keep versus what we want to get rid of.

Right now we have a lot going on. On the one hand, we have a lot of opportunities to improve things and the tools and knowledge at our disposal to do so. On the other hand we have locked up so much of our knowledge and tools, traditional institutions are struggling to maintain their authority and control, citizens are understandably frustrated and increasingly taking matters into their own hands, we have greater inequality than ever before, an obsession with work at the cost of living, and we are expected to sacrifice our humanity at the alter of economics

Questions to ask yourself:

Who are/aren’t you building for?
What is the default position in society?
What does being human mean to you?
What do we value in society?
What assumptions and unconscious bias do you have?
How are you helping non-geeks help themselves?
What future do you want to see?

What should be the rights, responsibilities and roles of
citizens, governments, companies, academia?

Finally,we must also help our fellow humans shift from being consumers to creators. We are all only as free as the tools we use, and though geeks will always be able to route around damage, be that technical or social, many of our fellow humans do not have the same freedoms we do.

Fundamental paradigm shifts we need to consider in building the future.

Scarcity → Surplus
Closed → Open
Centralised → Distributed
Analogue → Digital
Belief → Rationalism
Win/lose → Cooperative competitive
Nationalism → Transnationalism
Normative humans → Formative humans

Open source is the best possible modern expression of cooperative competitiveness that also integrates our philosophical shift towards human rights and powerful citizens, so I know it will continue to thrive and win when pitted against closed models, broadly speaking.

But in inventing the future, we need to be so very careful that we don’t simply rebuild the past with new shiny tools. We need to keep one eye always on the future we want to build, on how what we are doing contributes to that future, and to ensuring we have enough self awareness and commitment to ensuring we don’t accidentally embed in our efforts the outdated and oftentimes repressive habits of the past.

To paraphrase Gandhi, build the change you want to see. And build it today.

Thank you, and I hope you will join me in forging a better future.

Life in Sudan (technically North and South Sudan), Life in Somalia, and More

- Kingdom of Kush was one of first recorded instances of Sudan in history. Strong influence of ancient Egyptian empire and their Gods. Strong religious influence with Christianity and then Islam thereafter. Has struggled with internal conflict for a long time  sudan history https://www.youtube.com/results?search_query=sudan+history https://en.wikipedia.org/wiki/Foreign_relations_of_Sudan -

February 16, 2017

New Research on Our Little Cousins to the North!

Homo floresiensis

Last year, several research papers were published on the ongoing excavations and analysis of material from the island of Flores in Indonesia, where evidence of very small stature hominins was found in the cave of Liang Bua, in 2003. The initial dates dated these little people to between 50,000 and about 14,000 years ago, which would have meant that they lived side-by-side with anatomically modern humans in Indonesia, in the late Ice Age. The hominins, dubbed Homo floresiensis, after the island on which they were found, stood about 1m tall – smaller than any group of modern humans known. Their tiny size included a tiny brain – more in the range of 4 million year old Australopithecus than anything else. However, critical areas of higher order thinking in their brains were on par with modern humans.

Baffled by the seeming wealth of contradictions, these little people raised, researchers returned to the island, and the cave of Liang Bua, determined to check all of their findings in even more detail. Last year, they reported that they had in fact made some mistakes, the first time around. Very, very subtle changes in the sediments of the deposits, revealed that the Homo floresiensis bones belonged to some remnant older deposits, which had been eroded away in other parts of the cave, and replaced by much younger layers. Despite the samples for dating having been taken from close to the hominin bones, as luck would have it, they were all in the younger deposits! New dates, run on the actual sediments containing the bones, gave ages of between 190,000 to 60,000 years. Dates from close to the stone tools found with the hominins gave dates down to 50,000 years ago, but no later.

Liang Bua. Image by Rosino

The researchers – demonstrating a high level of ethics and absolutely correct scientific procedure, published the amended stratigraphy and dates, showing how the errors had occurred. At another site, Mata Menge, they had also found some ancestral hominins – very similar in body type to the ones from Liang Bua, dated to 700,000 years ago. Palaeoanthropologists were able to find similarities linking these hominins to the early Homo erectus found on Java and dated to about 1.2 million years ago, leading researchers to suggest that Homo floresiensis was a parallel evolution to modern humans, out of early Homo erectus in Indonesia, making them a fairly distant cousin on the grand family tree.

Careful examination of the deposits has now also called in to question whether Homo floresiensis could control fire. We know that they made stone tools – of a type pretty much unchanged over more than 600,000 years, and they used these tools to help them hunt Stegodon – an Ice Age dwarf elephant, which was as small as 1.5m at the shoulder. However, researchers now think that evidence of controlled fire is only in layers associated with modern humans. It is this cross-over between Homo floresiensis and modern humans, arriving about 60,000 – 50,000 years ago, that is a focus of current research – including that of teams working there now. At the moment, it looks as if Homo floresiensis disappears at about the same time that modern humans arrive, which sadly, is a not totally unlikely pattern.

Stegodon. Image by I, Vjdchauhan.

What does this have to do with Australia? Well, it’s always interesting to get information about our immediate neighbours and their history (and prehistory). But beyond that – we know that the ancestors of Aboriginal people (modern humans) were in Australia by about 60,000 – 50,000 years ago, so understanding how they arrived is part of understanding our own story. For more case studies on interesting topics in archaeology and palaeontology see our Archaeology Textbook resources for Year 11 students.

New D&D Cantrip

Name: Alternative Fact
Level: 0
School: EN
Time: 1 action
Range: global, contagious
Components: V, S, M (one racial, cultural or religious minority to blame)
Duration: Permanent (irrevocable)
Classes: Cleric, (Grand) Wizard, Con-man Politician

The caster can tell any lie, no matter how absurd or outrageous (in fact, the more outrageous the better), and anyone hearing it (or hearing about it later) with an INT of 10 or less will believe it instantly, with no saving throw. They will defend their new belief to the death – theirs or yours.

This belief can not be disbelieved, nor can it be defeated by any form of education, logic, evidence, or reason. It is completely incurable. Dispel Magic does not work against it, and Remove Curse is also ineffectual.

New D&D Cantrip is a post from: Errata

Two Weeks’ Notice

Last week, a rather heavy document envelope showed up in the mail.

Inside I found a heavy buff-coloured envelope, along with my passport — now containing a sticker featuring an impressive collection of words, numbers, and imagery of landmarks from the United States of America. I’m reliably informed that sticker is the valid US visa that I’ve spent the last few months applying for.

Having that visa issued has unblocked a fairly important step in my path to moving in with Josh (as well as eventually getting married, but that’s another story). I’m very very excited about making the move, though very sad to be leaving the city I’ve grown up in and come to love, for the time being.

Unrelatedly, I happened to have a trip planned to Montréal to attend ConFoo in March. Since I’ll already be in the area, I’m using that trip as my opportunity to move.

My last day in Hobart will be Thursday 2 March. Following that, I’ll be spending a day in Abu Dhabi (yes, there is a good reason for this), followed by a week in Montréal for ConFoo.

After that, I’ll be moving in with Josh in Petaluma, California on Saturday 11 March.

But until then, I definitely want to enjoy what remaining time I have in Hobart, and catch up with many many people.

Over the next two weeks I’ll be:

  • Attending, and presenting a talk at WD42 — my talk will be one of my pieces for ConFoo, and is entirely new material. Get along!
  • Having a farewell do, *probably* on Tuesday 28 February (but that’s not confirmed yet). I’ll post details about where and when that’ll be in the near future (once I’ve made plans)
  • Madly packing and making sure that that I use up as close to 100% of my luggage allowance as possible

If you want to find some time to catch up over the next couple of weeks, before I disappear for quite some time, do let me know.

February 15, 2017

FreeDV 700C

Over the past month the FreeDV 700C mode has been developed, integrated into the FreeDV GUI program version 1.2, and tested. Windows versions (64 and 32 bit) of this program can be downloaded from freedv.org. Thanks Richard Shaw for all your hard work on the release and installers.

FreeDV 700C uses the Codec 2 700C vocoder with the COHPSK modem. Some early results:

  • The US test team report 700C contacts over 2500km at SNRs down to -2dB, in conditions where SSB cannot be heard.
  • My own experience: the 700C speech quality is not quite as good as FreeDV 1600, but usable for conversation. That’s OK – it’s very early days for the 700C codec, and hey, it’s half the bit rate of 1600. I’m actually quite excited that 700C can be used conversationally at this early stage! I experienced a low SNR channel where FreeDV 700C didn’t work but SSB did, however 700C certainly works at much lower SNRs than 1600.
  • Some testers in Europe report 700C falling over at relatively high SNRs (e.g. 8dB). I also experienced this on a 1500km contact. Suspect this is a bug or corner case we can fix, especially in light of the US teams results.

Tony, K2MO, has put together this fine video demonstrating the various FreeDV modes over a simulated HF channel:

It’s early days for 700C, and there are mixed reports. However it’s looking promising. My next steps are to further explore the real world operation of FreeDV 700C, and work on improving the low SNR performance further.

Life in Libya, Going off the Grid, and More

On Libya: - Berber tribes are indigenous people. For most of its history, Libya has been subjected to varying degrees of foreign control, from Europe, Asia, and Africa. The modern history of independent Libya began in 1951. The history of Libya comprises six distinct periods: Ancient Libya, the Roman era, the Islamic era, Ottoman rule, Italian rule, and the Modern era. Very small population of

Modems for HF Digital Voice Part 2

In the previous post I argued that pushing bits through a HF channel involves much wailing and gnashing of teeth. Now we shall apply numbers and graphs to the problem, which is – in a nutshell – Engineering.

QPSK Modem Simulation

I have worked up a GNU Octave modem simulation called hf_modem_curves.m. This operates at 1 sample/symbol, i.e. the sample rate is the symbol rate. So we takes some random bits, map them to QPSK symbols, add noise, then turn the noisy symbols back into bits and count errors:

The simulation ignores a few real world details like timing and phase synchronisation, so is a best case model. That’s OK for now. QPSK uses symbols that each carry 2 bits of information, here is the symbol set or “constellation”:

Four different points, each representing a different 2 bit combination. For example the bits ’00’ would be the cross at 45 degrees, ’10’ at 135 degrees etc. The plot above shows all possible symbols, but we just send one at a time. However it’s useful to plot all of the received symbols like this, as an indication of received signal quality. If the channel is playing nice, we receive something like this:

Each cross is now a fuzzy dot, as noise has been added by the channel. No bit errors yet – a bit error happens when we get enough noise to move received symbols into another quadrant. This sort of channel is called Additive White Gaussian Noise (AWGN). Line of site UHF radio is a good example of a real world AWGN channel – all you have to worry about is additive noise.

With a fading or multipath channel like HF we end up with something like:

In a fading channel the received symbol amplitudes bounce up and down as the channel fades in and out. Sometimes the symbols dip down into the noise and we get lots of bit errors. Sometimes the signal is reinforced, and the symbol amplitude gets bigger.

The simulation used for the multipath or HF channel uses a two path model, with additive noise as per the AWGN simulation:

Graphs and Modem Performance

Turns out there are some surprisingly good models to help us work out the expected Bit Error Rate (BER) for a modem. By “model” I mean people have worked out the maths to describe the Bit Error Rate (BER) for a QPSK Modem. This graph shows us how to work out the BER for QPSK (and BPSK):

So the red line shows us the BER given Eb/No (E-B on N-naught), which is a normalised form of Signal to Noise Ratio (SNR). Think about Eb/No as a modem running at 1 bit per second, with the noise power measured in 1 Hz of bandwidth. It’s a useful scale for comparing modems and modulation schemes.

Looking at the black lines, we can see that for an Eb/No or 4dB, we can expect a BER of 1E-2 or 0.01 or 1% of our bits will be received in error over an AWGN channel. This curve is for QPSK or BPSK, different curves would be used for other modems like FSK.

Given Eb/No you can work out the SNR if you know the bit rate and noise bandwidth:

    SNR = S/N = EbRb/NoB

or in dB:

    SNR(dB) = Eb/No(dB) + 10log10(Rb/B)

For example at Rb = 1600 bit/s and a noise bandwidth B = 3000 Hz:

    SNR(dB) = 4 + 10log10(1600/3000) = 1.27 dB

OK, so that was for ideal QPSK. Lets add a few more curves to our graph:

We have added the experimental results for our QPSK simulation (green), and for Differential QPSK (DQPSK – blue). Our QPSK modem simulation (green) is right on top of the theoretical QPSK curve (red) – this is good and shows our simulation is working really well.

DQPSK was discussed in Part 1. Phase differences are sent, which helps with phase errors in the channels but costs us extra bit errors. This is evident on the curves – at the 1E-2 BER line, DQPSK requires 7dB Eb/No, 3dB more (double the power) of QPSK.

Now lets look at modem performance for HF (multipath) channels, on this rather busy graph (click for larger version):

Wow, HF sucks. Looking at the theoretical HF QPSK performance (straight red line) to achieve a BER of 1E-2, we need 14dB of Eb/No. That’s 10dB worse than QPSK on the AWGN channel. With DQPSK, we need about 16dB.

For HF, a lot of extra power is required to make a small difference in BER.

Some of the kinks in the HF curves (e.g. green QPSK HF simulated just under red QPSK HF theory) are due to not enough simulation points – it’s not actually possible to do better than theory!

Estimated Performance of FreeDV Modes

Now we have the tools to estimate the performance of FreeDV modes. FreeDV 1600 uses Codec 2 at 1300 bit/s, plus a little FEC at 300 bit/s to give a total of 1600 bit/s. With the FEC, lets say we can get reasonable voice quality at 4% BER. FreeDV 1600 uses a DQPSK modem.

On an AWGN channel, that’s an Eb/No of 4.4dB for DQPSK, and a SNR of:

    SNR(dB) = 4.4 + 10log10(1600/3000) = 1.7 dB

On a multipath channel, that’s an Eb/No of 11dB for DQPSK, and a SNR of:

    SNR(dB) = 11 + 10log10(1600/3000) = 8.3 dB

As discussed in Part 1, FreeDV 700C uses diversity and coherent QPSK, and has a multipath (HF) performance curve plotted in cyan above, and close to ideal QPSK on AWGN channels. The payload data rate is 700 bit/s, however we have an overhead of two pilot symbols for every 4 data symbols. This means we effectively need a bit rate of Rb = 700*(4+2)/4 = 1050 bit/s to pump 700 bits/s through the channel. It doesn’t have any FEC (yet, anyway), so we need a BER of a little lower than FreeDV 1600, about 2%. Running the numbers:

On an AWGN channel, for 2% BER we need an Eb/No of 3dB for QPSK, and a SNR of:

    SNR(dB) = 3 + 10log10(1050/3000) = -1.5 dB

On a multipath channel, diversity (cyan line) helps a lot, that’s an Eb/No of 8dB, and a SNR of:

    SNR(dB) = 8 + 10log10(1050/3000) = 3.4 dB

The diversity model in the simulation uses two carriers. The amplitudes of each carrier after passing through the multipath model are plotted below:

Often when one carrier is faded, the other is not faded, so when we recombine them at the receiver we get an average that is closer to AWGN performance. However diversity is not perfect, occasionally both carriers are wiped out at the same time by a fade.

So we can see FreeDV 700C is about 4 dB in front of FreeDV 1600, which matches the best reports from early adopters. I’ve had reports of FreeDV 700C operating at as low as -2dB , which is presumably on channels that don’t have heavy fading and are more like AWGN. Also some reports of 700C falling over at high SNRs (around like 8dB)! However that is probably a bug, e.g. a sync issue or something else we can track down in time.

Real world channels can vary. The multipath model above doesn’t take into account fast or slow fading, it just calculates the average bit errors rate. In practice, slow fading is hard to handle in digital voice applications, as the whole channel might be wiped out for a few seconds.

Now that we have a reasonable 700 bit/s codec – we can also consider other schemes, such as a more powerful FEC code rather than diversity. Like diversity, FEC codes provide “coding gain”, moving our operating point to the left. Really good codes operate at 10% BER, right over on the Eb/No = 2dB region of the curve. No free lunch of course – such codes may require long latency (seconds) or be expensive to decode.

Next Steps

I’d like to “instrument” FreeDV 700C and work with the 700C early adopters to find out how well it’s working, why and how it falls over, and work through any obvious bugs. Then start experimenting with ways to make it operate at lower SNRs, such as more powerful FEC codes or even non-redundant techniques like Trellis decoding.

Now we have shown Codec 700C has sufficient quality for conversations over the air, I’m planning another iteration of the Codec 2 700C vocoder design to see if we can improve speech quality.

Links

Modems for HF Digital Voice Part 1.

More Eb/No to SNR worked examples.

Similar modem calculations were used to develop a 100 kbit/s telemetry system to send HD images from High Altitude Balloons.

February 14, 2017

j-core + Numato Spartan 6 board + Fedora 25

A couple of changes to http://j-core.org/#download_bitstream made it easy for me to get going:

  • In order to make ModemManager not try to think it’s a “modem”, create /etc/udev/rules.d/52-numato.rules with the following content:
    # Make ModemManager ignore Numato FPGA board
    ATTRS{idVendor}=="2a19", ATTRS{idProduct}=="1002", ENV{ID_MM_DEVICE_IGNORE}="1"
  • You will need to install python3-pyserial and minicom
  • The minicom command line i used was:
    sudo stty -F /dev/ttyACM0 -crtscts && minicom -b 115200 -D /dev/ttyACM0

and along with the instructions on j-core.org, I got it to load a known good build.

This Week in HASS – term 1, week 3

This is a global week in HASS for primary students. Our youngest students are marking countries around the world where they have family members, slightly older students are examining the Mayan calendar, while older students get nearer to Australia, examining how people reached Australia and encountered its unique wildlife.

Foundation to Year 3

Mayan date

Foundation students doing the Me and My Global Family unit (F.1) are working with the world map this week, marking countries where they have family members with coloured sticky dots. Those doing the My Global Family unit (F.6), and students in Years 1 to 3 (Units 1.1; 2.1 and 3.1), are examining the Mayan calendar this week. The Mayan calendar is a good example of an alternative type of calendar, because it is made up of different parts, some of which do not track the seasons, and is cyclical, based on nested circles. The students learn about the 2 main calendars used by the Mayans – a secular and a celebratory sacred calendar, as well as how the Mayans divided time into circles running at different scales – from the day to the millennium and beyond. And no, in case anyone is still wondering – they did not predict the end of the world in 2012, merely the end of one particular long-range cycle, and hence, the beginning of a new one…

Years 3 to 6

Lake Mungo, where people lived at least 40,000 years ago.

Students doing the Exploring Climates unit (3.6), and those in Years 4 to 6 (Units 4.1, 5.1 and 6.1), are examining how people reached Australia during the Ice Age, and what Australia was like when they arrived. People had to cross at least 90 km of open sea to reach Australia, even during the height of the Ice Age, and this sea gap led to the relative isolation of animals in Australia from others in Asia. This phenomenon was first recorded by Alfred Wallace, who drew a line on a map marking the change in fauna. This line became known as the Wallace line, as a result. Students will also examine the archaeological evidence, and sites of the first people in Australia, ancestors of Aboriginal people. The range of sites across Australia, with increasingly early dates, amply demonstrate the depth of antiquity of Aboriginal knowledge and experience in Australia.

February 13, 2017

Life in Syria, Why the JSF isn't Worth It, and More

Given what has happened wanted to see what has been happening inside Syria: https://en.wikipedia.org/wiki/Syria http://www.aljazeera.com/topics/country/syria.html http://www.aljazeera.com/programmes/specialseries/2017/02/boy-started-syrian-war-170208093451538.html - complicated colonial history with both British and French. Has had limited conflict with some of it's neighbours including

High Power Lustre

(Most of the hard work here was done by fellow blogger Rashmica - I just verified her instructions and wrote up this post.)

Lustre is a high-performance clustered file system. Traditionally the Lustre client and server have run on x86, but both the server and client will also work on Power. Here's how to get them running.

Server

Lustre normally requires a patched 'enterprise' kernel - normally an old RHEL, CentOS or SUSE kernel. We tested with a CentOS 7.3 kernel. We tried to follow the Intel instructions for building the kernel as much as possible - any deviations we had to make are listed below.

Setup quirks

We are told to edit ~/kernel/rpmbuild/SPEC/kernel.spec. This doesn't exist because the directory is SPECS not SPEC: you need to edit ~/kernel/rpmbuild/SPECS/kernel.spec.

I also found there was an extra quote mark in the supplied patch script after -lustre.patch. I removed that and ran this instead:

for patch in $(<"3.10-rhel7.series"); do \
      patch_file="$HOME/lustre-release/lustre/kernel_patches/patches/${patch}" \
      cat "${patch_file}" >> $HOME/lustre-kernel-x86_64-lustre.patch \
done

The fact that there is 'x86_64' in the patch name doesn't matter as you're about to copy it under a different name to a place where it will be included by the spec file.

Building for ppc64le

Building for ppc64le was reasonably straight-forward. I had one small issue:

[build@dja-centos-guest rpmbuild]$ rpmbuild -bp --target=`uname -m` ./SPECS/kernel.spec
Building target platforms: ppc64le
Building for target ppc64le
error: Failed build dependencies:
       net-tools is needed by kernel-3.10.0-327.36.3.el7.ppc64le

Fixing this was as simple as a yum install net-tools.

This was sufficient to build the kernel RPMs. I installed them and booted to my patched kernel - so far so good!

Building the client packages: CentOS

I then tried to build and install the RPMs from lustre-release. This repository provides the sources required to build the client and utility binaries.

./configure and make succeeded, but when I went to install the packages with rpm, I found I was missing some dependencies:

error: Failed dependencies:
        ldiskfsprogs >= 1.42.7.wc1 is needed by kmod-lustre-osd-ldiskfs-2.9.52_60_g1d2fbad_dirty-1.el7.centos.ppc64le
    sg3_utils is needed by lustre-iokit-2.9.52_60_g1d2fbad_dirty-1.el7.centos.ppc64le
        attr is needed by lustre-tests-2.9.52_60_g1d2fbad_dirty-1.el7.centos.ppc64le
        lsof is needed by lustre-tests-2.9.52_60_g1d2fbad_dirty-1.el7.centos.ppc64le

I was able to install sg3_utils, attr and lsof, but I was still missing ldiskfsprogs.

It seems we need the lustre-patched version of e2fsprogs - I found a mailing list post to that effect.

So, following the instructions on the walkthrough, I grabbed the SRPM and installed the dependencies: yum install -y texinfo libblkid-devel libuuid-devel

I then tried rpmbuild -ba SPECS/e2fsprogs-RHEL-7.spec. This built but failed tests. Some failed because I ran out of disk space - they were using 10s of gigabytes. I found that there were some comments in the spec file about this with suggested tests to disable, so I did that. Even with that fix, I was still failing two tests:

  • f_pgsize_gt_blksize: Intel added this to their fork, and no equivalent exists in the master e2fsprogs branches. This relates to Intel specific assumptions about page sizes which don't hold on Power.
  • f_eofblocks: This may need fixing for large page sizes, see this bug.

I disabled the tests by adding the following two lines to the spec file, just before make %{?_smp_mflags} check.

rm -rf tests/f_pgsize_gt_blksize
rm -rf tests/f_eofblocks

With those tests disabled I was able to build the packages successfully. I installed them with yum localinstall *1.42.13.wc5* (I needed that rather weird pattern to pick up important RPMs that didn't fit the e2fs* pattern - things like libcom_err and libss)

Following that I went back to the lustre-release build products and was able to successfully run yum localinstall *ppc64le.rpm!

Testing the server

After disabling SELinux and rebooting, I ran the test script:

sudo /usr/lib64/lustre/tests/llmount.sh

This spat out one scary warning:

mount.lustre FATAL: unhandled/unloaded fs type 0 'ext3'

The test did seem to succeed overall, and it would seem that is a known problem, so I pressed on undeterred.

I then attached a couple of virtual harddrives for the metadata and object store volumes, and having set them up, proceeded to try to mount my freshly minted lustre volume from some clients.

Testing with a ppc64le client

My first step was to test whether another ppc64le machine would work as a client.

I tried with an existing Ubuntu 16.04 VM that I use for much of my day to day development.

A quick google suggested that I could grab the lustre-release repository and run make debs to get Debian packages for my system.

I needed the following dependencies:

sudo apt install module-assistant debhelper dpatch libsnmp-dev quilt

With those the packages built successfully, and could be easily installed:

dpkg -i lustre-client-modules-4.4.0-57-generic_2.9.52-60-g1d2fbad-dirty-1_ppc64el.deblustre-utils_2.9.52-60-g1d2fbad-dirty-1_ppc64el.deb

I tried to connect to the server:

sudo mount -t lustre $SERVER_IP@tcp:/lustre /lustre/

Initially I wasn't able to connect to the server at all. I remembered that (unlike Ubuntu), CentOS comes with quite an aggressive firewall by default. I ran the following on the server:

systemctl stop firewalld

And voila! I was able to connect, mount the lustre volume, and successfully read and write to it. This is very much an over-the-top hack - I should have poked holes in the firewall to allow just the ports lustre needed. This is left as an exercise for the reader.

Testing with an x86_64 client

I then tried to run make debs on my Ubuntu 16.10 x86_64 laptop.

This did not go well - I got the following error:

liblustreapi.c: In function ‘llapi_get_poollist’:
liblustreapi.c:1201:3: error: ‘readdir_r’ is deprecated [-Werror=deprecated-declarations]

This looks like one of the new errors introduced in recent GCC versions, and is a known bug. To work around it, I found the following stanza in a lustre/autoconf/lustre-core.m4, and removed the -Werror:

AS_IF([test $target_cpu == "i686" -o $target_cpu == "x86_64"],
        [CFLAGS="$CFLAGS -Wall -Werror"])

Even this wasn't enough: I got the following errors:

/home/dja/dev/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lustre/llite/dcache.c:387:22: error: initialization from incompatible pointer type [-Werror=incompatible-pointer-types]
         .d_compare = ll_dcompare,
                  ^~~~~~~~~~~
/home/dja/dev/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lustre/llite/dcache.c:387:22: note: (near initialization for ‘ll_d_ops.d_compare’)

I figured this was probably because Ubuntu 16.10 has a 4.8 kernel, and Ubuntu 16.04 has a 4.4 kernel. Work on supporting 4.8 is ongoing.

Sure enough, when I fired up a 16.04 x86_64 VM with a 4.4 kernel, I was able to build and install fine.

Connecting didn't work first time - the guest failed to mount, but I did get the following helpful error on the server:

LNetError: 2595:0:(acceptor.c:406:lnet_acceptor()) Refusing connection from 10.61.2.227: insecure port 1024

Refusing insecure port 1024 made me thing that perhaps the NATing that qemu was performing for me was interfering - perhaps the server expected to get a connection where the source port was privileged, and qemu wouldn't be able to do that with NAT.

Sure enough, switching NAT to bridging was enough to get the x86 VM to talk to the ppc64le server. I verified that ls, reading and writing all succeeded.

Next steps

The obvious next steps are following up the disabled tests in e2fsprogs, and doing a lot of internal performance and functionality testing.

Happily, it looks like Lustre might be in the mainline kernel before too long - parts have already started to go in to staging. This will make our lives a lot easier: for example, the breakage between 4.4 and 4.8 would probably have already been picked up and fixed if it was the main kernel tree rather than an out-of-tree patch set.

In the long run, we'd like to make Lustre on Power just as easy as Lustre on x86. (And, of course, more performant!) We'll keep you up to date!

(Thanks to fellow bloggers Daniel Black and Andrew Donnellan for useful feedback on this post.)

February 12, 2017

Life in Egypt, Life in Saudi Arabia, and More

On Egypt: - the thing that we mostly know Egypt for is the ancient Egyptian empire. Once upon a time the Middle East was effectively the global centre of knowledge, culture, etc... The pyramids were so spectacular for their age that their have been rumors throughout time of aliens being in contact with them. Would actually make a lot of stories in the Holy Scriptures a lot more sense as well?

Printer bracket fix

Similar to many 3d printer designs, many of the parts on this 3d printer are plastic. Where the Z-Axis meets the Y-Axis is held in place by two top brackets (near the gear on the stepper is a bolt to the z alloy extrusion) and the bottom bracket. One flaw here is that there are no bolts to the z-axis on the bottom bracket. It was also cracked in two places so the structural support was low and the x-axis would droop over time. Not so handy.


The plastic is about 12mm thick and smells like a 2.5D job done by a 3d printer 'just because'.  So a quick tinker in Fusion 360 and the 1/2 inch thick flatland part was born. After removing the hold down tabs and flapping the remains away 3 M6 bolt holds were hand drilled. Notice the subtle shift on the inside of the part where the extrusion and stepper motor differ in size.


It was quicker to just do that rather than try to remount and register on the cnc and it might not have even worked with the limited z range of the machine.


The below image only has two of the three bolts in place. With the addition of the new bolt heading into the z axis the rigidity of the machine went right up. The shaft that the z axis is mounted onto goes into the 12mm empty hole in the part.


This does open up the mental thoughts of how many other parts would be better served by not being made out of plastic.


February 11, 2017

Multicore World 2017

The 6th Multicore World will be held on Monday 20th to Wednesday 22nd of February 2017 at Shed 6 on the Wellington (NZ) waterfront. Nicolás Erdödy (Open Parallel) has once again done an amazing job at finding some the significant speakers in the world in parallel programming and multicore systems to attend. Although a short - and not an enormous conference - the technical quality is always extremely high, dealing with some of the most fundamental problems and recent experiences in these fields.

read more

Librarians take up arms against fake news | Seattle Times

Librarians are stepping into the breach to help students become smarter evaluators of the information that floods into their lives. That’s increasingly necessary in an era in which fake news is a constant.

Spotting fake news - librarian Janelle Hagen - Lakeside School SeattleSpotting fake news – by librarian Janelle Hagen – Lakeside School Seattle

Read more: http://www.seattletimes.com/seattle-news/librarians-take-up-arms-against-fake-news

5 Tech Non Profs you should support right now!

icons for the 5 tech non profs whose actual icons appear below anyway.

Join 'em, support 'em, donate, promote... whatever. They all do good work. Really good work. And we should all support them as much as we can. Help me, help them, by following them, amplifying their voices, donating or even better?Joining them! And if all you've got is gratitude for the work they do, then drop 'em a line and just say a simple thank you :)

 

Software Freedom Conservancy Logo

Software Freedom Conservancy

Follow: @conservancy

Donate: sfconservancy.org/donate

Join: sfconservancy.org/supporter

 

Open Source Initiative

Follow: @OpenSourceOrg

Donate: opensource.org/donate

Join: opensource.org/join

 

 

Drupal Association

Follow: @drupalassoc

Donate: assoc.drupal.org/donate

Join: www.drupal.org/association/individual-membership

 

 

Internet Archive

Follow: @internetarchive

Donate: archive.org/donate

Join: as above, just choose monthly sustaining member

 

 

Wikimedia Foundation

Follow: @Wikimedia

Donate: donate.wikimedia.org

Join: wikimediafoundation.org/wiki/Volunteer_opportunities

February 10, 2017

This Week in HASS: term 1 week 2

OpenSTEM A0 world map: Country Outlines and Ice Age CoastlineFoundation to Year 3

Our standalone Foundation (Prep/Kindy etc) students are introduced to the World Map this week, as they start putting stickers on it, showing where in the world they and their families come from – the origin of the title of this unit (Me and My Global Family). This helps students to feel connected with each other and to start to understand both the notion of the ‘global family’, as well as the idea that places can be represented by pictures (maps). Of course, we don’t expect most 5 year olds to understand the world map, but the sooner they start working with it, the deeper the familiarity and understanding later on.

Year 1-3 Building Stonehenge Activity - OpenSTEM History/Geography program for Primary SchoolsStudents building Stonehenge with blocks

All the other younger students are learning about movements of celestial bodies (the Earth and Moon, as they go around the Sun and each other) and that people have measured time in the past with reference to both the Sun and the Moon – Solar and Lunar calendars. To make these ideas more concrete, students study ancient calendars, such as Stonehenge, Newgrange and Abu Simbel, and take part in an activity building a model of Stonehenge from boxes or blocks.

Years 3 to 6

Demon Duck of Doom

Our older primary students are going back into the Ice Age (and who wouldn’t want to, in this weather!), as they explore the routes of modern humans leaving Africa, as part of understanding how people reached Australia. Aboriginal people arrived in Australia as part of the waves of modern humans spreading across the world. However, the Australia they encountered was very different from today. It was cold, dry and very dusty, inhabited by giant Ice Age animals (the Demon Duck of Doom is always a hot favourite with the students!) and overall, a pretty dangerous place. We challenge students to imagine life in those times, and thereby start to understand the basis for some of the Dreamtime stories, as well as the long and intricate relationship between Aboriginal people and the Australian environment.

This Week in HASS: term 1 week 1

We thought it would be fun to track what’s happening in schools using our primary HASS program, on a weekly basis. Now we know that some of you are doing different units and some will start in different weeks, depending on what state you’re in, what term dates you have etc, but we will run these posts based off those schools which are implementing the units in numerical order and starting in the week beginning 30 January, 2017.

Week 1 is an introductory week for all units, and usually sets some foundations for the rest of the unit.

Foundation to Year 3

Our youngest students are still finding their feet in the new big world of school! We have 2 units for Term 1, depending on whether the class is standalone, or integrating with some Year 1 students. This week standalone classes will be starting a discussion about their families – geared towards making our newest students feel welcome and comfortable at school.

Those integrating with Year 1 or possibly Year 2, as well, will start working with their teachers on a Class Calendar, marking terms and holidays, as well as celebrations such as birthdays and public holidays. This helps younger students start to map out the coming year, as well as provide a platform for discussions about how they spent the holidays. Year 2 and 3 students may choose to focus more on discussing which season we are in now, and what the weather’s like at the moment (I’m sure most of you are in agreement that it’s too hot!). Students can track the weather on the calendar as well.

Years 3 to 6

Some Year 3 students may be in classes integrating with Year 4 students, rather than Year 2. Standalone Year 3 classes have a choice of doing either unit. These older students will be undertaking the Timeline Activity and getting a physical sense of history and spans of time. Students love an excuse to get outdoors, even when it’s hot, and this activity gives them a preview of material they will be covering later in the year, as well as giving them a hands-on understanding of how time has passed and how where we are compares to past events. This activity can even reinforce the concept of a number line from Maths, in a very kinaesthetic way.

February 09, 2017

Modems for HF Digital Voice Part 1

The newly released FreeDV 700C mode uses the Coherent PSK (COHPSK) modem which I developed in 2015. This post describes the challenges of building HF modems for DV, and how the COHPSK modem evolved from the FDMDV modem used for FreeDV 1600.

HF channels are tough. You need a lot of SNR to push bits through them. There are several problems to contend with:

When the transmit signal is reflected off the ionosphere, two or more copies arrive at the receiver antenna a few ms apart. These echoes confuse the demodulator, just like a room with bad echo can confuse a listener.

Here is a plot of a BPSK baseband signal (top). Lets say we receive two copies of this signal, from two paths. The first is identical to what we sent (top), but the second is delayed a few samples and half the amplitude (middle). When you add them together at the receiver input (bottom), it’s a mess:

The multiple paths combining effectively form a comb filter, notching out chunks of the modem signal. Loosing chunks of the modem spectrum is bad. Here is the magnitude and phase frequency response of a channel with the two paths used for the time domain example above:

Note that comb filtering also means the phase of the channel is all over the place. As we are using Phase Shift Keying (PSK) to carry our precious bits, strange phase shifts are more bad news.

All of these impairments are time varying, so the echoes/notches, and phase shifts drift as the ionosphere wiggles about. As well as the multipath, it must deal with noise and operate at SNRs of around 0dB, and frequency offsets between the transmitter and receiver of say +/- 100 Hz.

If commodity sound cards are used for the ADC and DAC, the modem must also handle large sample clock offsets of +/-1000 ppm. For example the transmitter DAC sample clock might be 7996 Hz and the receiver ADC 8004 Hz, instead of the nominal 8000 Hz.

As the application is Push to Talk (PTT) Digital Voice, the modem must sync up quickly, in the order of 100ms, even with all the challenges above thrown at it. Processing delay should be around 100ms too. We can’t wait seconds for it to train like a data modem, or put up with several seconds of delay in the receive speech due to processing.

Using standard SSB radio sets we are limited to around 2000 Hz of RF bandwidth. This bandwidth puts a limit on the bit rate we can get through the channel. The amplitude and phase distortion caused by typical SSB radio crystal filters is another challenge.

Designing a modem for HF Digital Voice is not easy!

FDMDV Modem

In 2012, the FDMDV modem was developed as our first attempt at a modem for HF digital voice. This is more or less a direct copy of the FDMDV waveform which was developed by Francesco Lanza, HB9TLK and Peter Martinez G3PLX. The modem software was written in GNU Octave and C, carefully tested and tuned, and most importantly – is open source software.

This modem uses many parallel carriers or tones. We are using Differential QPSK, so every symbol contains 2 bits encoded as one of 4 phases.

Lets say we want to send 1600 bits/s over the channel. We could do this with a single QPSK carrier at Rs = 800 symbols a second. Eight hundred symbols/s times two bit/symbol for QPSK is 1600 bit/s. The symbol period Ts = 1/Rs = 1/800 = 1.25ms. Alternatively, we could use 16 carriers running at 50 symbols/s (symbol period Ts = 20ms). If the multipath channel has echoes 1ms apart it will make a big mess of the single carrier system but the parallel tone system will do much better, as 1ms of delay spread won’t upset a 20ms symbol much:

We handle the time-varying phase of the channel using Differential PSK (DPSK). We actually send and receive phase differences. Now the phase of the channel changes over time, but can be considered roughly constant over the duration of a few symbols. So when we take a difference between two successive symbols the unknown phase of the channel is removed.

Here is an example of DPSK for the BPSK case. The first figure shows the BPSK signal top, and the corresponding DBPSK signal (bottom). When the BPSK signal changes, we get a +1 DBPSK value, when it is the same, we get a -1 DBPSK value.

The next figure shows the received DBPSK signal (top). The phase shift of the channel is a constant 180 degrees, so the signal has been inverted. In the bottom subplot the recovered BPSK signal after differential decoding is shown. Despite the 180 degree phase shift of the channel it’s the same as the original Tx BPSK signal in the first plot above.

This is a trivial example, in practice the phase shift of the channel will vary slowly over time, and won’t be a nice neat number like 180 degrees.

DPSK is a neat trick, but has an impact on the modem Bit Error Rate (BER) – if you get one symbol wrong, the next one tends to be corrupted as well. It’s a two for one deal on bit errors, which means crappier performance for a given SNR than regular (coherent) PSK.

To combat frequency selective fading we use a little Forward Error Correction (FEC) on the FreeDV 1600 waveform. So if one carrier gets notched out, we can use bits in the other carriers to recover the missing bits. Unfortunately we don’t have the bandwidth available to protect all bits, and the PTT delay requirement means we have to use a short FEC code. Short FEC codes don’t work as well as long ones.

COHPSK Modem

Over the next few years I spent some time thinking about different modem designs and trying a bunch of different ideas, most of which failed. Research and disappointment. You just have to learn from your mistakes, talk to smart people, and keep trying. Then, towards the end of 2014, a few ideas started to come together, and the COHPSK modem was running in real time in mid 2015.

The major innovations of the COHPSK modem are:

  1. The use of diversity to help combat frequency selective fading. The baseline modem has 7 carriers. A copy of these are made, and sent at a higher frequency to make 14 tones in total. Turns out the HF channel giveth and taketh away. When one tone is notched out another is enhanced (an anti-fade). So we send each carrier twice and add them back together at the demodulator, averaging out the effect of frequency selective fades:
  2. To use diversity we need enough bandwidth to fit a copy of the baseline modem carriers. This implies the need for a vocoder bit rate of much less than 1600 bit/s – hence several iterations at a 700 bits/s speech codec – a completely different skill set – and another 18 months of my life to develop Codec 2 700C.
  3. Coherent QPSK detection is used instead of differential detection, which halves the number of bit errors compared to differential detection. This requires us to estimate the phase of the channel on the fly. Two known symbols are sent followed by 4 data symbols. These known, or Pilot symbols, allow us to measure and correct for the current phase of each carrier. As the pilot symbols are sent regularly, we can quickly acquire – then track – the phase of the channel as it evolves.

Here is a figure that shows how the pilot and data symbols are distributed across one frame of the COHPSK modem. More information of the frame design is available in the cohpsk frame design spreadsheet, including performance calculations which I’ll explain in the next blog post in this series.

Coming Next

In the next post I’ll show how reading a few graphs and adding a few dBs together can help us estimate the performance of the FDMDV and COHPSK modems on HF channels.

Links

Modems for HF Digital Voice Part 2

cohpsk_plots.m Octave script used to generate plots for this post.

FDMDV Modem Page

Some earlier musings on FreeDV 1600 and why SSB works so well:

FreeDV Robustness Part 1

FreeDV Robustness Part 2

FreeDV Robustness Part 3

Life in Iran, Examining Prophets/Pre-Cogs 6/Hyperspace Travel, and More

Wanted to take a look inside Iran given how much trouble it seems to in: https://en.wikipedia.org/wiki/Iran https://en.wikipedia.org/wiki/Supreme_Leader_of_Iran http://president.ir/en/ http://smartraveller.gov.au/Countries/middle-east/Pages/iran.aspx https://www.theguardian.com/world/iran http://www.aljazeera.com/topics/country/iran.html https://www.lonelyplanet.com/iran - ancient

Adding a Docker Runner to GitLab

In my particular scenario, I need to run both docker and docker-compose to test and build our changes. The first step to achieving this is to add an appropriate GitLab runner.

We especially need to run a privileged runner to make this happen.

Assuming that GitLab Runner has already been successfully installed, head to Admin -> Runner in the webUI of your GitLab instance and note your Registration token.

From a suitable account on your GitLab instance register a shared runner:

% sudo /usr/bin/gitlab-ci-multi-runner register --docker-privileged \
    --url https://gitlab.my.domain/ci \
    --registration-token REGISTRATION_TOKEN \
    --executor docker \
    --description "My Docker Runner" \
    --docker-image "docker:latest" \

Your shared runner should now be ready to run.

This applies to self-hosting a GitLab instance. If you are using the gitlab.com hosted services, a suitable runner is already supplied.

There are many types of executors for runners, suiting a variety of scenarios. This example's scenario is that both GitLab and the desired runner are on the same instance.

February 08, 2017

Manage Intel Turbo Boost with systemd

If you have a little laptop with an Intel CPU that supports turbo boost, you might find that it’s getting a little hot when you’re using it on your lap.

For example, taking a look at my CPU:
lscpu |egrep "Model name|MHz"

We can see that it’s a 2.7GHz CPU with turbo boost taking it up to 3.5GHz.

Model name: Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz
CPU MHz: 524.633
CPU max MHz: 3500.0000
CPU min MHz: 400.0000

Here’s a way that you can enable and disable turbo boost with a systemd service, which lets you hook it into other services or disable it on boot.

By default, turbo boost is on, so starting our service will disable it.

Create the service.
cat << EOF | sudo tee \
/etc/systemd/system/disable-turbo-boost.service
[Unit]
Description=Disable Turbo Boost on Intel CPU
 
[Service]
ExecStart=/bin/sh -c "/usr/bin/echo 1 > \
/sys/devices/system/cpu/intel_pstate/no_turbo"
ExecStop=/bin/sh -c "/usr/bin/echo 0 > \
/sys/devices/system/cpu/intel_pstate/no_turbo"
RemainAfterExit=yes
 
[Install]
WantedBy=sysinit.target
EOF

Reload systemd manager configuration.
sudo systemctl daemon-reload

Test it by running something CPU intensive and watching the current running MHz.

cat /dev/urandom > /dev/null &
lscpu |grep "CPU MHz"

CPU MHz: 3499.859

Now disable turbo boost and check the CPU speed again.
sudo systemctl start disable-turbo-boost
lscpu |grep "CPU MHz"

CPU MHz: 2699.987

Don’t forget to kill the CPU intensive process 🙂

kill %1

If you want to disable turbo boost on boot by default, just enable the service.

sudo systemctl enable disable-turbo-boost

February 07, 2017

Fixing webcam flicker in Linux with udev

I recently got a new Dell XPS 13 (9360) laptop for work and it’s running Fedora pretty much perfectly.

However, when I load up Cheese (or some other webcam program) the video from the webcam flickers. Given that I live in Australia, I had to change the powerline frequency from 60Hz to 50Hz to fix it.

sudo dnf install v4l2-ctl
v4l2-ctl --set-ctrl power_line_frequency=1

I wanted this to be permanent each time I turned my machine on, so I created a udev rule to handle that.

cat << EOF | sudo tee /etc/udev/rules.d/50-dell-webcam.rules
SUBSYSTEM=="video4linux", \
SUBSYSTEMS=="usb", \
ATTRS{idVendor}=="0c45", \
ATTRS{idProduct}=="670c", \
PROGRAM="/usr/bin/v4l2-ctl --set-ctrl \
power_line_frequency=1 --device /dev/%k", \
SYMLINK+="dell-webcam"
EOF

It’s easy to test. Just turn flicker back on, reload the rules and watch the flicker in Cheese automatically disappear 🙂

v4l2-ctl --set-ctrl power_line_frequency=0
sudo udevadm control --reload-rules && sudo udevadm trigger

Of course I also tested with a reboot.

It’s easy to do with any webcam, just take a look on the USB bus for the vendor and product IDs. For example, here’s a Logitech C930e (which is probably the nicest webcam I’ve ever used, and also works perfectly under Fedora).

Bus 001 Device 022: ID 046d:0843 Logitech, Inc. Webcam C930e

So you would replace the following in your udev rule:

  • ATTRS{idVendor}==“046d”
  • ATTRS{idProduct}==“0843”
  • SYMLINK+=“c930e”

Note that SYMLINK is not necessary, it just creates an extra /dev entry, such as /dev/c930e, which is useful if you have multiple webcams.

OpenStack and the OpenStack Barcelona Summit

Presentation to Linux Users of Victoria, 7th February, 2017

An overview of cloud computing platforms in general, and OpenStack in particular, is provided introduces this presentation. Cloud computing is one of the most significant changes to IT infrastructure and employment in the past decade, with major corporate services (Amazon, Microsoft) gaining particular significance in the late 2000s. In mid-2010, Rackspace Hosting and NASA jointly launched an open-source cloud-software initiative known as OpenStack, with initial code coming from NASA's Nebula project and Rackspace's Cloud Files project, and soon gained prominence as the largest open-source cloud platform. Although a cross-platform service, it was quickly available on various Linux distributions including Debian, Ubuntu, SuSE (2011), and Red Hat (2012).

OpenStack is governed by the OpenStack Foundation, a non-profit corporate entity established in September 2012. Correlating with the release cycle of the product, OpenStack Summits are held every six months for developers, users and managers. The most recent Summit was held in Barcelona in late October 2016, with over 5000 attendees, almost 1000 organisations and companies, and 500 sessions, spread out over three days, plus one day of "Upstream University" prior to the main schedule, plus one day after the main schedule for contributor working parties. The presentation will cover the major announcements of the conference as well as a brief overview of the major streams, as well the direction of OpenStack as the November Sydney Summit approaches.

read more

February 06, 2017

Life in Venezuela, Examining Prophets/Pre-Cogs 5, and More

Wanted to see what life was like in Venezuela given their recent problems: - complicated colonial history with conflict between Spaniards and local indigenous people (led by Native caciques, such as Guaicaipuro and Tamanaco). One of first to declare independence in Latin America. History of military strongmen and corruption? Political and economic instability over many years... venezuela

SE Linux in Debian/Stretch

Debian/Stretch has been frozen. Before the freeze I got almost all the bugs in policy fixed, both bugs reported in the Debian BTS and bugs that I know about. This is going to be one of the best Debian releases for SE Linux ever.

Systemd with SE Linux is working nicely. The support isn’t as good as I would like, there is still work to be done for systemd-nspawn. But it’s close enough that anyone who needs to use it can use audit2allow to generate the extra rules needed. Systemd-nspawn is not used by default and it’s not something that a new Linux user is going to use, I think that expert users who are capable of using such features are capable of doing the extra work to get them going.

In terms of systemd-nspawn and some other rough edges, the issue is the difference between writing policy for a single system vs writing policy that works for everyone. If you write policy for your own system you can allow access for a corner case without a lot of effort. But if I wrote policy to allow access for every corner case then they might add up to a combination that can be exploited. I don’t recommend blindly adding the output of audit2allow to your local policy (be particularly wary of access to shadow_t and write access to etc_t, lib_t, etc). But OTOH if you have a system that’s running in enforcing mode that happens to have one daemon with more access than is ideal then all the other daemons will still be restricted.

As for previous releases I plan to keep releasing updates to policy packages in my own apt repository. I’m also considering releasing policy source to updates that can be applied on existing Stretch systems. So if you want to run the official Debian packages but need updates that came after Stretch then you can get them. Suggestions on how to distribute such policy source are welcome.

Please enjoy SE Linux on Stretch. It’s too late for most bug reports regarding Stretch as most of them won’t be sufficiently important to justify a Stretch update. The vast majority of SE Linux policy bugs are issues of denying wanted access not permitting unwanted access (so not a security issue) and can be easily fixed by local configuration, so it’s really difficult to make a case for an update to Stable. But feel free to send bug reports for Buster (Stretch+1).

February 05, 2017

IPv6 and OpenVPN on Linode Debian/Ubuntu VPS

Here is how I managed to extend my OpenVPN setup on my Linode VPS to include IPv6 traffic. This ensures that clients can route all of their traffic through the VPN and avoid leaking IPv6 traffic, for example. It also enables clients on IPv4-only networks to receive a routable IPv6 address and connect to IPv6-only servers (i.e. running your own IPv6 broker).

Request an additional IPv6 block

The first thing you need to do is get a new IPv6 address block (or "pool" as Linode calls it) from which you can allocate a single address to each VPN client that connects to the server.

If you are using a Linode VPS, there are instructions on how to request a new IPv6 pool. Note that you need to get an address block between /64 and /112. A /116 like Linode offers won't work in OpenVPN. Thankfully, Linode is happy to allocate you an extra /64 for free.

Setup the new IPv6 address

If your server only has an single IPv4 address and a single IPv6 address, then a simple DHCP-backed network configuration will work fine. To add the second IPv6 block on the other hand, I had to change my network configuration (/etc/network/interfaces) to this:

auto lo
iface lo inet loopback

allow-hotplug eth0
iface eth0 inet dhcp
    pre-up iptables-restore /etc/network/iptables.up.rules

iface eth0 inet6 static
    address 2600:3c01::xxxx:xxxx:xxxx:939f/64
    gateway fe80::1
    pre-up ip6tables-restore /etc/network/ip6tables.up.rules

iface tun0 inet6 static
    address 2600:3c01:xxxx:xxxx::/64
    pre-up ip6tables-restore /etc/network/ip6tables.up.rules

where 2600:3c01::xxxx:xxxx:xxxx:939f/64 (bound to eth0) is your main IPv6 address and 2600:3c01:xxxx:xxxx::/64 (bound to tun0) is the new block you requested.

Once you've setup the new IPv6 block, test it from another IPv6-enabled host using:

ping6 2600:3c01:xxxx:xxxx::1

OpenVPN configuration

The only thing I had to change in my OpenVPN configuration (/etc/openvpn/server.conf) was to change:

proto udp

to:

proto udp6

in order to make the VPN server available over both IPv4 and IPv6, and to add the following lines:

server-ipv6 2600:3c01:xxxx:xxxx::/64
push "route-ipv6 2000::/3"

to bind to the right V6 address and to tell clients to tunnel all V6 Internet traffic through the VPN.

In addition to updating the OpenVPN config, you will need to add the following line to /etc/sysctl.d/openvpn.conf:

net.ipv6.conf.all.forwarding=1

and the following to your firewall (e.g. /etc/network/ip6tables.up.rules):

# openvpn
-A INPUT -p udp --dport 1194 -j ACCEPT
-A FORWARD -m state --state NEW -i tun0 -o eth0 -s 2600:3c01:xxxx:xxxx::/64 -j ACCEPT
-A FORWARD -m state --state NEW -i eth0 -o tun0 -d 2600:3c01:xxxx:xxxx::/64 -j ACCEPT
-A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

in order to ensure that IPv6 packets are forwarded from the eth0 network interface to tun0 on the VPN server.

With all of this done, apply the settings by running:

sysctl -p /etc/sysctl.d/openvpn.conf
ip6tables-apply
systemctl restart openvpn.service

Testing the connection

Now connect to the VPN using your desktop client and check that the default IPv6 route is set correctly using ip -6 route.

Then you can ping the server's new IP address:

ping6 2600:3c01:xxxx:xxxx::1

and from the server, you can ping the client's IP (which you can see in the network settings):

ping6 2600:3c01:xxxx:xxxx::1002

Once both ends of the tunnel can talk to each other, you can try pinging an IPv6-only server from your client:

ping6 ipv6.google.com

and then pinging your client from an IPv6-enabled host somewhere:

ping6 2600:3c01:xxxx:xxxx::1002

If that works, other online tests should also work.

LUV Main February 2017 Meeting: OpenStack Summit/Data Structures and Algorithms

Feb 7 2017 18:30
Feb 7 2017 18:30
Location: 
6th Floor, Trinity College (EPA Victoria building), 200 Victoria St., Carlton

Tuesday, February 7, 2017
6:30 PM to 8:30 PM
6th Floor, Trinity College (EPA Victoria building)
200 Victoria St., Carlton

Speakers:

• Lev Lafayette, OpenStack and the OpenStack Barcelona Summit
• Jacinta Richardson, Data Structures and Algorithms in the 21st Century

200 Victoria St. Carlton VIC 3053 (the EPA building)

Late arrivals needing access to the building and the sixth floor please call 0490 049 589.
 

Before and/or after each meeting those who are interested are welcome to join other members for dinner. We are open to suggestions for a good place to eat near our venue. Maria's on Peel Street in North Melbourne is currently the most popular place to eat after meetings.

LUV would like to acknowledge Red Hat for their help in obtaining the venue.

Linux Users of Victoria Inc., is an incorporated association, registration number A0040056C.

February 7, 2017 - 18:30

read more

Astrophotography with Mac OS X

It's been a good three years now since I swapped my HP laptop for a Macbook Pro. In the mean time, I've started doing a bit more astrophotography and of course the change of operating system has affected the tools I use to obtain and process photos.

Amateur astronomers have traditionally mostly used Windows, so there are a lot of Windows tools, both freeware and payware, to help. I used to run the freeware ones in Wine on Ubuntu with varying levels of success.

When I first got the Mac, I had a lot of trouble getting Wine to run reliably and eventually ended up doing my alignment and processing manually in The Gimp. However, that's time consuming and rather fiddly and limited to stacking static exposures.

However, I've recently started finding quite a bit of Mac OS based astrophotography software. I don't know if that means it's all fairly new or whether my Google skills failed me over the past years :-)

Software

I thought I'd document what I use, in the hope that I can save others who want to use their Macs some searching.

Some are Windows software, but run OK on Mac OS X. You can turn them into normal double click applications using a utility called WineSkin Winery.

Obtaining data from video camera:

Format-converting video data:

Processing video data:

  • AutoStakkert! (Windows + Wine, free for non-commercial use, donationware)

Obtaining data from DSLR:

Processing and stacking DSLR files and post-processing video stacks:

Post-processing:

Telescope guiding:

  • AstroGuider (Mac OS X, payware, free trial)
  • PHD2 (Mac OS X, free, open source)

Hardware

M42 - Orion NebulaA few weeks ago I bought a ZWO ASI120MC-S astro camera, as that was on sale and listed by Nebulosity as supported by OSX. Until then I'd messed around with a hacked up Logitech webcam, which seemed to only be supported by the Photo Booth app.

I've not done any guiding yet (I need a way to mount the guide scope on the main scope - d'oh) but the camera works well with Nebulosity 4 and oaCapture. I'm looking forward to being able to grab Jupiter with it in a month or so and Saturn and Mars later this year.

The image to the right is a stack of 24x5 second unguided exposures of the trapezium in M42. Not too bad for a quick test on a half-moon night.

Settings

I've been fiddling with Nebulosity  abit, to try and get it to stack the RAW images from my Nikon D750 as colour. I found a conversion matrix that was supposed to be decent, but as it turns out that made all images far too blue.

The current matrix I use is listed below. If you find a better one, please let me know.

  R G B
R 0.50 0.00 1.00
G 0.00 1.00 0.00
B 1.00 0.00 0.50

February 04, 2017

Career Opportunities

Had a friendly meeting a few days ago with a young person debating their future career path. They had a very good IT-orientated resume (give this person a job, seriously) but were debating whether they should go down the path of a Business Analyst. It was fairly clear that they lived and breathed IT, whereas the BA choice was one of some indifference. In reverse, there was a situation when VPAC had a year of summer school graduates where it became quickly obvious that none of them had any passion for IT.

read more

CMA Equalisation of FSK

We’ve just released a new experimental mode for Digital Voice called FreeDV 800XA. This uses the Codec 700C mode, 100 bit/s for synchronisation, and a 4FSK modem, actually the same modem that has been so successful for images from High Altitude Balloons.

FSK has the advantage of being a constant amplitude waveform, so efficient class C amplifiers can be used. However as it currently stands, 800XA has no real protection for the multipath common on HF channels, for example symbols that have an echo delayed by a few ms.

So I decided to start looking at equalisers. Some Googling suggested the Constant Modulus Algorithm (CMA) Equaliser might be a suitable choice for FSK, and turned up some sample code on DSP stack exchange.

I had a bit of trouble getting the algorithm to work for bandpass FSK signals, so posted this question on CMA equalisation for FSK. I received some kind help, and eventually made the equaliser work on a simulated HF channel. Here is the Octave simulation cma.m

How it works

The equaliser attempts to correct for the channel using the received signal, which is corrupted by noise.

There is a “gotcha” in using a FIR filter to equalise a channel response. Consider a channel H(z) with a simple 3 sample impulse response h(n). Now we could equalise this with the exact inverse 1/H(z). Here is a plot of our example channel frequency response and the ideal equaliser which is exactly the inverse:

Now here is a plot of the impulse responses of the channel h(n), and equaliser h'(n):

The ideal equaliser response h'(n) is much longer than the 3 samples of the channel impulse response h(n). The CMA algorithm requires our equaliser to be a FIR filter. Counter-intuitively, we need to use an FIR equaliser with a number of taps significantly larger than the expected channel impulse response we are trying to equalise.

One explanation for this – the channel response can be considered to be a Finite Impulse response (FIR) filter H(z). The exact inverse 1/H(z), when expressed in the time domain, is an Infinite Impulse Response (IIR) filter, which have, you know, an infinitely long impulse response!

Simulation

The figures below show the CMA equaliser doing it’s thing in a multipath channel with AWGN noise. In Figure 1 the error is reduced over time, and the lower plot shows the combined channel-equaliser impulse response. If the equaliser was perfect the combined channel-equaliser response would be 1.

Figure 2 below shows the CMA going to work on a FSK signal. The top subplot is the transmitted FSK signal, you can see the two different frequencies in the waveform. The middle plot shows the received signal, after it has been messed up by the multipath channel. It’s clear that the tone amplitudes are different. Looking carefully at the point where the tones transition (e.g. around sample 25 and 65) there is intersymbol interference due to multipath echoes, messing up the start of each FSK symbol.

However in the bottom subplot the equaliser has worked it’s magic and the waveform is looking quite nice. The tone levels are nearly equal and much of the ISI removed. Yayyyyyy.

Figure 4 shows the magnitude frequency response at several stages in the simulation. The top subplot is the channel response. It’s a comb filter, typical of multipath channels. The middle subplot is the equaliser response. Ideally, this should be the exact inverse of the channel. It’s pretty close at the low end but seems to lose it’s way at very low and high frequencies. The lower plot is the combined response, which is close to 0dB at the low frequencies. Cool.

Figure 4 is the transmit spectrum of the modem signal (top), and the spectrum after the channel has mangled it (lower). Note one tone is now lower than the other. Also note that the modem signal only has energy in the low-mid range of the spectrum. This might explain why the equaliser does a good job in that region of the spectrum – it’s where we have energy to drive the adaption.

Problems for HF Digital Voice

Unfortunately the CMA equaliser only works well at high SNRs, and takes seconds to converge. I am interested in low SNR (around 0dB in a 3000 Hz noise bandwidth) and it’s Push To Talk (PTT) radio so we a need fast initial training, around 100ms. Then it must follow the time varying HF channel, continually retraining on the fly.

For further work I really should measure BER versus Eb/No for a variety of SNRs and convergence times, and measure what BER improvement we are buying with equalisation. BER is King, much easier that squinting at time domain waveforms.

If the CMA cost function was used with known information (like pilot symbols or the Unique Word we have in 800XA) it might be able to work faster. This would involve deconvolution on the fly, rather than using iterative or adaptive techniques.

Links

Jonathan Olds (Jonti) has been Experimening with digital SCA signals. This includes an OQPSK modem with a C implementation of a CMA equaliser.

February 03, 2017

Trump Background, Random Stuff, and More

Given his recent inauguration, I thought it would be interesting to take a look at the background of the new US president, Donald Trump: https://www.bloomberg.com/politics/articles/2017-01-21/merkel-said-to-scour-trump-archive-for-clues-on-how-to-read-him https://www.rt.com/viral/374666-twitter-gifts-trump-followers/?utm_source=rss&utm_medium=rss&utm_campaign=RSS - well known background,

Nova vendordata deployment, an excessively detailed guide

Nova presents configuration information to instances it starts via a mechanism called metadata. This metadata is made available via either a configdrive, or the metadata service. These mechanisms are widely used via helpers such as cloud-init to specify things like the root password the instance should use. There are three separate groups of people who need to be able to specify metadata for an instance.

User provided data

The user who booted the instance can pass metadata to the instance in several ways. For authentication keypairs, the keypairs functionality of the Nova APIs can be used to upload a key and then specify that key during the Nova boot API request. For less structured data, a small opaque blob of data may be passed via the user-data feature of the Nova API. Examples of such unstructured data would be the puppet role that the instance should use, or the HTTP address of a server to fetch post-boot configuration information from.

Nova provided data

Nova itself needs to pass information to the instance via its internal implementation of the metadata system. Such information includes the network configuration for the instance, as well as the requested hostname for the instance. This happens by default and requires no configuration by the user or deployer.

Deployer provided data

There is however a third type of data. It is possible that the deployer of OpenStack needs to pass data to an instance. It is also possible that this data is not known to the user starting the instance. An example might be a cryptographic token to be used to register the instance with Active Directory post boot -- the user starting the instance should not have access to Active Directory to create this token, but the Nova deployment might have permissions to generate the token on the user's behalf.

Nova supports a mechanism to add "vendordata" to the metadata handed to instances. This is done by loading named modules, which must appear in the nova source code. We provide two such modules:

  • StaticJSON: a module which can include the contents of a static JSON file loaded from disk. This can be used for things which don't change between instances, such as the location of the corporate puppet server.
  • DynamicJSON: a module which will make a request to an external REST service to determine what metadata to add to an instance. This is how we recommend you generate things like Active Directory tokens which change per instance.


Tell me more about DynamicJSON

Having said all that, this post is about how to configure the DynamicJSON plugin, as I think its the most interesting bit here.

To use DynamicJSON, you configure it like this:

  • Add "DynamicJSON" to the vendordata_providers configuration option. This can also include "StaticJSON" if you'd like.
  • Specify the REST services to be contacted to generate metadata in the vendordata_dynamic_targets configuration option. There can be more than one of these, but note that they will be queried once per metadata request from the instance, which can mean a fair bit of traffic depending on your configuration and the configuration of the instance.


The format for an entry in vendordata_dynamic_targets is like this:

<name>@<url>


Where name is a short string not including the '@' character, and where the URL can include a port number if so required. An example would be:

testing@http://127.0.0.1:125


Metadata fetched from this target will appear in the metadata service at a new file called vendordata2.json, with a path (either in the metadata service URL or in the configdrive) like this:

openstack/2016-10-06/vendor_data2.json


For each dynamic target, there will be an entry in the JSON file named after that target. For example::

        {
            "testing": {
                "value1": 1,
                "value2": 2,
                "value3": "three"
            }
        }


Do not specify the same name more than once. If you do, we will ignore subsequent uses of a previously used name.

The following data is passed to your REST service as a JSON encoded POST:

  • project-id: the UUID of the project that owns the instance
  • instance-id: the UUID of the instance
  • image-id: the UUID of the image used to boot this instance
  • user-data: as specified by the user at boot time
  • hostname: the hostname of the instance
  • metadata: as specified by the user at boot time


Deployment considerations

Nova provides authentication to external metadata services in order to provide some level of certainty that the request came from nova. This is done by providing a service token with the request -- you can then just deploy your metadata service with the keystone authentication WSGI middleware. This is configured using the keystone authentication parameters in the vendordata_dynamic_auth configuration group.

This behavior is optional however, if you do not configure a service user nova will not authenticate with the external metadata service.

Deploying the same vendordata service

There is a sample vendordata service that is meant to model what a deployer would use for their custom metadata at http://github.com/mikalstill/vendordata. Deploying that service is relatively simple:

$ git clone http://github.com/mikalstill/vendordata
$ cd vendordata
$ apt-get install virtualenvwrapper
$ . /etc/bash_completion.d/virtualenvwrapper (only needed if virtualenvwrapper wasn't already installed)
$ mkvirtualenv vendordata
$ pip install -r requirements.txt


We need to configure the keystone WSGI middleware to authenticate against the right keystone service. There is a sample configuration file in git, but its configured to work with an openstack-ansible all in one install that I setup up for my private testing, which probably isn't what you're using:

[keystone_authtoken]
insecure = False
auth_plugin = password
auth_url = http://172.29.236.100:35357
auth_uri = http://172.29.236.100:5000
project_domain_id = default
user_domain_id = default
project_name = service
username = nova
password = 5dff06ac0c43685de108cc799300ba36dfaf29e4
region_name = RegionOne


Per the README file in the vendordata sample repository, you can test the vendordata server in a stand alone manner by generating a token manually from keystone:

$ curl -d @credentials.json -H "Content-Type: application/json" http://172.29.236.100:5000/v2.0/tokens > token.json
$ token=`cat token.json | python -c "import sys, json; print json.loads(sys.stdin.read())['access']['token']['id'];"`


We then include that token in a test request to the vendordata service:

curl -H "X-Auth-Token: $token" http://127.0.0.1:8888/


Configuring nova to use the external metadata service

Now we're ready to wire up the sample metadata service with nova. You do that by adding something like this to the nova.conf configuration file:

[api]
vendordata_providers=DynamicJSON
vendordata_dynamic_targets=testing@http://metadatathingie.example.com:8888


Where metadatathingie.example.com is the IP address or hostname of the server running the external metadata service. Now if we boot an instance like this:

nova boot --image 2f6e96ca-9f58-4832-9136-21ed6c1e3b1f --flavor tempest1 --nic net-name=public --config-drive true foo


We end up with a config drive which contains the information or external metadata service returned (in the example case, handy Carrie Fischer quotes):

# cat openstack/latest/vendor_data2.json | python -m json.tool
{
    "testing": {
        "carrie_says": "I really love the internet. They say chat-rooms are the trailer park of the internet but I find it amazing."
    }
}


Tags for this post: openstack nova metadata vendordata configdrive cloud-init
Related posts: One week of Nova Kilo specifications; Specs for Kilo; Juno nova mid-cycle meetup summary: nova-network to Neutron migration; Juno Nova PTL Candidacy; Juno nova mid-cycle meetup summary: scheduler; Juno nova mid-cycle meetup summary: ironic

Comment

February 01, 2017

LUV Beginners February Meeting: Static websites with Jekyll, Hugo and Forestry

Feb 25 2017 12:30
Feb 25 2017 16:30
Feb 25 2017 12:30
Feb 25 2017 16:30
Location: 
Infoxchange, 33 Elizabeth St. Richmond

PLEASE NOTE CHANGE OF DATE THIS MONTH ONLY

Static websites with Jekyll, Hugo and Forestry

Andrew Pam will demonstrate a new way to make websites complete with content management that doesn't require software running on a web server.  This technique enhances both performance and security.  More information at:

 

The meeting will be held at Infoxchange, 33 Elizabeth St. Richmond 3121 (enter via the garage on Jonas St.) Late arrivals, please call (0421) 775 358 for access to the venue.

LUV would like to acknowledge Infoxchange for the venue.

Linux Users of Victoria Inc., is an incorporated association, registration number A0040056C.

February 25, 2017 - 12:30

read more

LUV Main February 2017 Meeting: OpenStack Barcelona Summit / Data Structures and Algorithms

Feb 7 2017 18:30
Feb 7 2017 20:30
Feb 7 2017 18:30
Feb 7 2017 20:30
Location: 
6th Floor, 200 Victoria St. Carlton VIC 3053

Speakers:

• Lev Lafayette, OpenStack and the OpenStack Barcelona Summit
• Jacinta Richardson, Data Structures and Algorithms in the 21st Century

200 Victoria St. Carlton VIC 3053 (the EPA building)

Late arrivals needing access to the building and the sixth floor please call 0490 049 589.

Before and/or after each meeting those who are interested are welcome to join other members for dinner. We are open to suggestions for a good place to eat near our venue. Maria's on Peel Street in North Melbourne is currently the most popular place to eat after meetings.

LUV would like to acknowledge Red Hat for their help in obtaining the venue.

Linux Users of Victoria Inc. is an incorporated association, registration number A0040056C.

February 7, 2017 - 18:30

read more

January 31, 2017

Giving serial devices meaningful names

This is a hack I've been using for ages, but I thought it deserved a write up.

I have USB serial devices. Lots of them. I use them for home automation things, as well as for talking to devices such as the console ports on switches and so forth. For the permanently installed serial devices one of the challenges is having them show up in predictable places so that the scripts which know how to drive each device are talking in the right place.

For the trivial case, this is pretty easy with udev:

$  cat /etc/udev/rules.d/60-local.rules 
KERNEL=="ttyUSB*", \
    ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6001", \
    ATTRS{serial}=="A8003Ye7", \
    SYMLINK+="radish"


This says for any USB serial device that is discovered (either inserted post boot, or at boot), if the USB vendor and product ID match the relevant values, to symlink the device to "/dev/radish".

You find out the vendor and product ID from lsusb like this:

$ lsusb
Bus 003 Device 003: ID 0624:0201 Avocent Corp. 
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 007 Device 002: ID 0665:5161 Cypress Semiconductor USB to Serial
Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 002: ID 0403:6001 Future Technology Devices International, Ltd FT232 Serial (UART) IC
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 009 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 008 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub


You can play with inserting and removing the device to determine which of these entries is the device you care about.

So that's great, until you have more than one device with the same USB serial vendor and product id. Then things are a bit more... difficult.

It turns out that you can have udev execute a command on device insert to help you determine what symlink to create. So for example, I have this entry in the rules on one of my machines:

KERNEL=="ttyUSB*", \
    ATTRS{idVendor}=="067b", ATTRS{idProduct}=="2303", \
    PROGRAM="/usr/bin/usbtest /dev/%k", \
    SYMLINK+="%c"


This results in /usr/bin/usbtest being run with the path of the device file on its command line for every device detection (of a matching device). The stdout of that program is then used as the name of a symlink in /dev.

So, that script attempts to talk to the device and determine what it is -- in my case either a currentcost or a solar panel inverter.

Tags for this post: linux udev serial usb usbserial
Related posts: SMART and USB storage; Video4Linux, ov511, and RGB24 palettes; ov511 hackery; Ubuntu, Dapper Drake, and that difficult Dell e310; Roomba serial cables; Via M10000, video, and a Belkin wireless USB thing

Comment

NAMD on NVLink

NAMD is a molecular dynamics program that can use GPU acceleration to speed up its calculations. Recent OpenPOWER machines like the IBM Power Systems S822LC for High Performance Computing (Minsky) come with a new interconnect for GPUs called NVLink, which offers extremely high bandwidth to a number of very powerful Nvidia Pascal P100 GPUs. So they're ideal machines for this sort of workload.

Here's how to set up NAMD 2.12 on your Minsky, and how to debug some common issues. We've targeted this script for CentOS, but we've successfully compiled NAMD on Ubuntu as well.

Prerequisites

GPU Drivers and CUDA

Firstly, you'll need CUDA and the NVidia drivers.

You can install CUDA by following the instructions on NVidia's CUDA Downloads page.

yum install epel-release
yum install dkms
# download the rpm from the NVidia website
rpm -i cuda-repo-rhel7-8-0-local-ga2-8.0.54-1.ppc64le.rpm
yum clean expire-cache
yum install cuda
# this will take a while...

Then, we set up a profile file to automatically load CUDA into our path:

cat >  /etc/profile.d/cuda_path.sh <<EOF
# From http://developer.download.nvidia.com/compute/cuda/8.0/secure/prod/docs/sidebar/CUDA_Quick_Start_Guide.pdf - 4.4.2.1
export PATH=/usr/local/cuda-8.0/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
EOF

Now, open a new terminal session and check to see if it works:

cuda-install-samples-8.0.sh ~
cd ~/NVIDIA_CUDA-8.0_Samples/1_Utilities/bandwidthTest
make && ./bandwidthTest

If you see a figure of ~32GB/s, that means NVLink is working as expected. A figure of ~7-8GB indicates that only PCI is working, and more debugging is required.

Compilers

You need a c++ compiler:

yum install gcc-c++

Building NAMD

Once CUDA and the compilers are installed, building NAMD is reasonably straightforward. The one hitch is that because we're using CUDA 8.0, and the NAMD build scripts assume CUDA 7.5, we need to supply an updated Linux-POWER.cuda file. (We also enable code generation for the Pascal in this file.)

We've documented the entire process as a script which you can download. We'd recommend executing the commands one by one, but if you're brave you can run the script directly.

The script will fetch NAMD 2.12 and build it for you, but won't install it. It will look for the CUDA override file in the directory you are running the script from, and will automatically move it into the correct place so it is picked up by the build system..

The script compiles for a single multicore machine setup, rather than for a cluster. However, it should be a good start for an Ethernet or Infiniband setup.

If you're doing things by hand, you may see some errors during the compilation of charm - as long as you get charm++ built successfully. at the end, you should be OK.

Testing NAMD

We have been testing NAMD using the STMV files available from the NAMD website:

cd NAMD_2.12_Source/Linux-POWER-g++
wget http://www.ks.uiuc.edu/Research/namd/utilities/stmv.tar.gz
tar -xf stmv.tar.gz
sudo ./charmrun +p80 ./namd2 +pemap 0-159:2 +idlepoll +commthread stmv/stmv.namd

This binds a namd worker thread to every second hardware thread. This is because hardware threads share resources, so using every hardware thread costs overhead and doesn't give us access to any more physical resources.

You should see messages about finding and using GPUs:

Pe 0 physical rank 0 binding to CUDA device 0 on <hostname>: 'Graphics Device'  Mem: 4042MB  Rev: 6.0

This should be significantly faster than on non-NVLink machines - we saw a gain of about 2x in speed going from a machine with Nvidia K80s to a Minsky. If things aren't faster for you, let us know!

Downloads

Other notes

Namd requires some libraries, some of which they supply as binary downloads on their website. Make sure you get the ppc64le versions, not the ppc64 versions, otherwise you'll get errors like:

/bin/ld: failed to merge target specific data of file .rootdir/tcl/lib/libtcl8.5.a(regfree.o)
/bin/ld: .rootdir/tcl/lib/libtcl8.5.a(regerror.o): compiled for a big endian system and target is little endian
/bin/ld: failed to merge target specific data of file .rootdir/tcl/lib/libtcl8.5.a(regerror.o)
/bin/ld: .rootdir/tcl/lib/libtcl8.5.a(tclAlloc.o): compiled for a big endian system and target is little endian

The script we supply should get these right automatically.

linux.conf.au 2017 review

I recently attended LCA 2017, where I gave a talk at the Linux Kernel miniconf (run by fellow sthbrx blogger Andrew Donnellan!) and a talk at the main conference.

I received some really interesting feedback so I've taken the opportunity to write some of it down to complement the talk videos and slides that are online. (And to remind me to follow up on it!)

Miniconf talk: Sparse Warnings

My kernel miniconf talk was on sparse warnings (pdf slides, 23m video).

The abstract read (in part):

sparse is a semantic parser for C, and is one of the static analysis tools available to kernel devs.

Sparse is a powerful tool with good integration into the kernel build system. However, we suffer from warning overload - there are too many sparse warnings to spot the serious issues amongst the trivial. This makes it difficult to use, both for developers and maintainers.

Happily, I received some feedback that suggests it's not all doom and gloom like I had thought!

  • Dave Chinner told me that the xfs team uses sparse regularly to make sure that the file system is endian-safe. This is good news - we really would like that to be endian-safe!

  • Paul McKenney let me know that the 0day bot does do some sparse checking - it would just seem that it's not done on PowerPC.

Main talk: 400,000 Ephemeral Containers

My main talk was entitled "400,000 Ephemeral Containers: testing entire ecosystems with Docker". You can read the abstract for full details, but it boils down to:

What if you want to test how all the packages in a given ecosystem work in a given situation?

My main example was testing how many of the Ruby packages successfully install on Power, but I also talk about other languages and other cool tests you could run.

The 44m video is online. I haven't put the slides up yet but they should be available on GitHub soonish.

Unlike with the kernel talk, I didn't catch the names of most of the people with feedback.

Docker memory issues

One of the questions I received during the talk was about running into memory issues in Docker. I attempted to answer that during the Q&A. The person who asked the question then had a chat with me afterwards, and it turns out I had completely misunderstood the question. I thought it was about memory usage of running containers in parallel. It was actually about memory usage in the docker daemon when running lots of containers in serial. Apparently the docker daemon doesn't free memory during the life of the process, and the question was whether or not I had observed that during my runs.

I didn't have a good answer for this at the time other than "it worked for me", so I have gone back and looked at the docker daemon memory usage.

After a full Ruby run, the daemon is using about 13.9G of virtual memory, and 1.975G of resident memory. If I restart it, the memory usage drops to 1.6G of virtual and 43M of resident memory. So it would appear that the person asking the question was right, and I'm just not seeing it have an effect.

Other interesting feedback

  • Someone was quite interested in testing on Sparc, once they got their Go runtime nailed down.

  • A Rackspacer was quite interested in Python testing for OpenStack - this has some intricacies around Py2/Py3, but we had an interesting discussion around just testing to see if packages that claim Py3 support provide Py3 support.

  • A large jobs site mentioned using this technique to help them migrate their dependencies between versions of Go.

  • I was 'gently encouraged' to try to do better with how long the process takes to run - if for no other reason than to avoid burning more coal. This is a fair point. I did not explain very well what I meant with diminishing returns in the talk: there's lots you could do to make the process faster, it's just comes at the cost of the simplicity that I really wanted when I first started the project. I am working (on and off) on better ways to deal with this by considering the dependency graph.

January 30, 2017

Linux BASH CLI RSS Reader, Explaining Prophets 4, and More

- built my own RSS feed reader yesterday. It actually took a lot less time then going out to search for one that suited my needs. It's based on someone else's code (credit given in code but since that code was so buggy that it wouldn't work) I guess it's mine now? https://sites.google.com/site/dtbnguyen/rssread-1.11.tar.gz https://sites.google.com/site/dtbnguyen/ - code to extract from

My Personal Travel Ban

I plan to avoid any and all travel to the USA for the foreseeable future due to the complete mess unfolding there with Trump’s executive orders banning immigration from some Muslim-majority countries, related protests, illegal detainment, etc. etc. (the list goes on, and I expect it to get longer).

It’s not that I’m from one of the blacklist countries, and I’m not a Muslim. I’m even white. But I no longer consider travel to the USA safe (especially bearing in mind my ridiculous beard and long hair), and even if I did, I’d want to stand in solidarity with the people who are currently being screwed. The notion of banning entire groups of people based on a single shared trait (in this case, probable adherence to a particular religion) is abhorrent; it demonizes our fellow humans, divides us and builds walls – whether metaphorical or physical – between our various communities. The fact that this immigration ban will impact refugees and asylum seekers just makes matters worse. I am deeply ashamed by Australia’s record on that front too, and concerned that our government will not do much better.

So I won’t be putting in any talks for Cephalocon - which is a damn shame, as I’m working on Ceph – or for any other US-based tech conference unless and until the situation over there changes.

I realise this post may not make much difference in the grander scheme of things, but one more voice is one more voice.

A pythonic example of recording metrics about ephemeral scripts with prometheus

In my previous post we talked about how to record information from short lived scripts (I call them ephemeral scripts by the way) with prometheus. The example there was a script which checked the SMART status of each of the disks in a machine and reported that via pushgateway. I now want to work through a slightly more complicated example.

I think you hit the limits of reporting simple values in shell scripts via curl requests fairly quickly. For example with the SMART monitoring script, SMART is capable of returning a whole heap of metrics about the performance of a disk, but we boiled that down to a single "health" value. This is largely because writing a parser for all the other values that smartctl returns would be inefficient and fragile in shell. So for this post, we're going to work through an example of how to report a variety of values from a python script. Those values could be the parsed output of smartctl, but to mix things up a bit, I'm going to use a different script I wrote recently.

This new script uses the Weather Underground API to lookup weather stations near my house, and then generate graphics of the weather forecast. These graphics are displayed on the various Cisco SIP phones I already had around the house. The forecasts look like this:



The script to generate these weather forecasts is relatively simple python, and you can see the source code on github.

My cunning plan here is to use prometheus' time series database and alert capabilities to drive home automation around my house. The first step for that is to start gathering some simple facts about the home environment so that we can do trending and decision making on them. The code to do this isn't all that complicated. First off, we need to add the python prometheus client to our python environment, which is hopefully a venv:

pip install prometheus_client
pip install six


That second dependency isn't a strict requirement for prometheus, but the script I'm working on needs it (because it needs to work out what's a text value, and python 3 is bonkers).

Next we import the prometheus client in our code and setup the counter registry. At the same time I record when the script was run:

from prometheus_client import CollectorRegistry, Gauge, push_to_gateway

registry = CollectorRegistry()
Gauge('job_last_success_unixtime', 'Last time the weather job ran',
      registry=registry).set_to_current_time()


And then we just add gauges for any values we want to add to the pushgateway

Gauge('_'.join(field), '', registry=registry).set(value)


Finally, the values don't exist in the pushgateway until we actually push them there, which we do like this:

push_to_gateway('localhost:9091', job='weather', registry=registry)


You can see the entire patch I wrote to add prometheus support on github if you're interested in an example with more context.

Now we can have pretty graphs of temperature and stuff!

Tags for this post: prometheus monitoring python pushgateway
Related posts: Recording performance information from short lived processes with prometheus; Basic prometheus setup; Implementing SCP with paramiko; Mona Lisa Overdrive; Packet capture in python; mbot: new hotness in Google Talk bots

Comment

Creating a home music server using mpd

I recently setup a music server on my home server using the Music Player Daemon, a cross-platform free software project which has been around for a long time.

Basic setup

Start by installing the server and the client package:

apt install mpd mpc

then open /etc/mpd.conf and set these:

music_directory    "/path/to/music/"
bind_to_address    "192.168.1.2"
bind_to_address    "/run/mpd/socket"
zeroconf_enabled   "yes"
password           "Password1"

before replacing the alsa output:

audio_output {
   type    "alsa"
   name    "My ALSA Device"
}

with a pulseaudio one:

audio_output {
   type    "pulse"
   name    "Pulseaudio Output"
}

In order for the automatic detection (zeroconf) of your music server to work, you need to prevent systemd from creating the network socket:

systemctl stop mpd.service
systemctl stop mpd.socket
systemctl disable mpd.socket

otherwise you'll see this in /var/log/mpd/mpd.log:

zeroconf: No global port, disabling zeroconf

Once all of that is in place, start the mpd daemon:

systemctl start mpd.service

and create an index of your music files:

MPD_HOST=Password1@/run/mpd/socket mpc update

while watching the logs to notice any files that the mpd user doesn't have access to:

tail -f /var/log/mpd/mpd.log

Enhancements

I also added the following in /etc/logcheck/ignore.server.d/local-mpd to silence unnecessary log messages in logcheck emails:

^\w{3} [ :0-9]{11} [._[:alnum:]-]+ systemd\[1\]: Started Music Player Daemon.$
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ systemd\[1\]: Stopped Music Player Daemon.$
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ systemd\[1\]: Stopping Music Player Daemon...$

and created a cronjob in /etc/cron.d/mpd-francois to update the database daily and stop the music automatically in the evening:

# Refresh DB once a day
5 1 * * *  mpd  MPD_HOST=Password1@/run/mpd/socket /usr/bin/mpc --quiet update
# Think of the neighbours
0 22 * * 0-4  mpd  MPD_HOST=Password1@/run/mpd/socket /usr/bin/mpc --quiet stop
0 23 * * 5-6  mpd  MPD_HOST=Password1@/run/mpd/socket /usr/bin/mpc --quiet stop

Clients

To let anybody on the local network connect, I opened port 6600 on the firewall (/etc/network/iptables.up.rules since I'm using Debian's iptables-apply):

-A INPUT -s 192.168.1.0/24 -p tcp --dport 6600 -j ACCEPT

Then I looked at the long list of clients on the mpd wiki.

Desktop

The official website suggests two clients which are available in Debian and Ubuntu:

Both of them work well, but haven't had a release since 2011, even though there is some activity in 2013 and 2015 in their respective source control repositories.

Ario has a simpler user interface but gmpc has cover art download working out of the box, which is why I might stick with it.

In both cases, it is possible to configure a polipo proxy so that any external resources are fetched via Tor.

Android

On Android, I got these two to work:

I picked M.A.L.P. since it includes a nice widget for the homescreen.

iOS

On iOS, these are the most promising clients I found:

since MPoD and MPaD don't appear to be available on the AppStore anymore.

Extracting Early Boot Messages in QEMU

Be me, you're a kernel hacker, you make some changes to your kernel, you boot test it in QEMU, and it fails to boot. Even worse is the fact that it just hangs without any failure message, no stack trace, no nothing. "Now what?" you think to yourself.

You probably do the first thing you learnt in debugging101 and add abundant print statements all over the place to try and make some sense of what's happening and where it is that you're actually crashing. So you do this, you recompile your kernel, boot it in QEMU and lo and behold, nothing... What happened? You added all these shiny new print statements, where did the output go? The kernel still failed to boot (obviously), but where you were hoping to get some clue to go on you were again left with an empty screen. "Maybe I didn't print early enough" or "maybe I got the code paths wrong" you think, "maybe I just need more prints" even. So lets delve a bit deeper, why didn't you see those prints, where did they go, and how can you get at them?

__log_buf

So what happens when you call printk()? Well what normally happens is, depending on the log level you set, the output is sent to the console or logged so you can see it in dmesg. But what happens if we haven't registered a console yet? Well then we can't print the message can we, so its logged in a buffer, kernel log buffer to be exact helpfully named __log_buf.

Console Registration

So how come I eventually see print statements on my screen? Well at some point during the boot process a console is registered with the printk system, and any buffered output can now be displayed. On ppc it happens that this occurs in register_early_udbg_console() called in setup_arch() from start_kernel(), which is the generic kernel entry point. From this point forward when you print something it will be displayed on the console, but what if you crash before this? What are you supposed to do then?

Extracting Early Boot Messages in QEMU

And now the moment you've all been waiting for, how do I extract those early boot messages in QEMU if my kernel crashes before the console is registered? Well it's quite simple really, QEMU is nice enough to allow us to dump guest memory, and we know the log buffer is in there some where, so we just need to dump the correct part of memory which corresponds to the log buffer.

Locating __log_buf

Before we can dump the log buffer we need to know where it is. Luckily for us this is fairly simple, we just need to dump all the kernel symbols and look for the right one.

> nm vmlinux > tmp; grep __log_buf tmp;
c000000000f5e3dc b __log_buf

We use the nm tool to list all the kernel symbols and output this into some temporary file, we can then grep this for the log buffer (which we know to be named __log_buf), and presto we are told that it's at kernel address 0xf5e3dc.

Dumping Guest Memory

It's then simply a case of dumping guest memory from the QEMU console. So first we press ^a+c to get us to the QEMU console, then we can use the aptly named dump-guest-memory.

> help dump-guest-memory
dump-guest-memory [-p] [-d] [-z|-l|-s] filename [begin length] -- dump guest memory into file 'filename'.
            -p: do paging to get guest's memory mapping.
            -d: return immediately (do not wait for completion).
            -z: dump in kdump-compressed format, with zlib compression.
            -l: dump in kdump-compressed format, with lzo compression.
            -s: dump in kdump-compressed format, with snappy compression.
            begin: the starting physical address.
            length: the memory size, in bytes.

We just give it a filename for where we want our output to go, we know the starting address, we just don't know the length. We could choose some arbitrary length, but inspection of the kernel code shows us that:

#define __LOG_BUF_LEN (1 << CONFIG_LOG_BUF_SHIFT)
static char __log_buf[__LOG_BUF_LEN] __aligned(LOG_ALIGN);

Looking at the pseries_defconfig file shows us that the LOG_BUF_SHIFT is set to 18, and thus we know that the buffer is 2^18 bytes or 256kb. So now we run:

> dump-guest-memory tmp 0xf5e3dc 262144

And we now get our log buffer in the file tmp. This can simply be viewed with:

> hexdump -C tmp

This gives a readable, if poorly formatted output. I'm sure you can find something better but I'll leave that as an exercise for the reader.

Conclusion

So if like me your kernel hangs somewhere early in the boot process and you're left without your console output you are now fully equipped to extract the log buffer in QEMU and hopefully therein lies the answer to why you failed to boot.

Git hook to help with OpenStack development

I wrote a small Git hook which may be useful in helping OpenStack devs run tests (and any script they like) before a commit is made (see Superuser magazine article).

This way we can save everyone time in the review process by fixing simple issues before they break in the check-pipeline.

Installation is easy (see the GitHub page) and all prompts default to no, so that the dev can easily just hit Enter to skip and continue (but still be reminded).