Planet Linux Australia
Celebrating Australians & Kiwis in the Linux and Free/Open-Source community...

January 26, 2015

First jog, and a walk to Los Altos

Today was a busy day, not only did I foolishly go for a jog 5 minutes after sunrise...

Interactive map for this route. ...but then I went for a walk with James in the afternoon as well.

Interactive map for this route. Let's just say my fitbit is very impressed with me.

Tags for this post: blog walk california running

Related posts: Walking to work; Did I mention it's hot here?; Summing up Santa Monica; Noisy neighbours at Central Park in Mountain View; So, how am I getting to the US?; VTA station for the Santa Clara Convention Center



Notwork, due to Australia day. Spending an inordinate amount of time trying to find some aircon so I don’t sweat all day long. I did get to pre-poll vote in the morning, so not all aircon hunting time was wasted.

My headphones have died in one ear, time for another set of consumables. The wirleless in the library is hopeless. This combination is making me very unproductive at both tasks I set myself for today.

Filed under: Uncategorized

January 25, 2015

Twitter posts: 2015-01-19 to 2015-01-25


Finished the Learning to Learn MOOC course. I missed a few of the deadlines due to lca2015, so I’m not not bothering to do the written parts, which does make me feel rather like I haven’t finished the course. It’ll be interesting to see if I can apply the techniques going forward. I’m pretty sure I’ll chase up their book at some point as well.

Filed under: diary

January 24, 2015

Craige McWhirter: A Little Vim Hack For Go

After LCA2015 I've starting playing with Go (I blame Sven Dowideit). If you already use VIM-YouCompleteMe) then you should be right for most things Go. However I tinker in a few languages and you'll never guess that they have different rules around style and formatting of code.

Go is one out for me requiring settings unique to Go among the languages I tinker in. I made the below changes to my ~/.vimrc to suit Go:

function! GoSettings()
    set tabstop=7
    set shiftwidth=7
    set noexpandtab
autocmd BufNewFile,BufFilePre,BufRead *.go :call GoSettings()

Now when I edit a file with the .go extension, my Vim session will be formatting the file correctly from the start.

You can also configure Vim to run gofmt but I preferred this approach.


Caught up with a friend in the morning.

Booked the local bowling place for my birthday celebration.

Caught up on the ‘learning to learn’ mooc. I’ve missed the deadline on the quiz and the written material, but I’m continuing through with everything else. I should be able to get through the final week of content tomorrow.

Watching more LCA2015 videos.

Filed under: diary



A very wet day, I was drenched only about fifty metres from home.

Watching and noting on more lca2015 videos.

Filed under: diary



Finished typing up my hand written LCA2015 notes into the humbug wiki. I’ve now started the videos of the talks that I’ve missed.

Went and saw the move “Birdman” and I felt like I was back at BIFF, what a lovely little film, that covers a range of themes, without being complicated.

Home at midnight though, which made for a short night’s sleep.

Filed under: diary



Caught up with a close friend this evening. I may well lose her in the coming weeks, so I’m trying to organise a nice night out for her in a few weeks, under the guise of my birthday.

Filed under: diary

January 23, 2015

Indoor Skydiving

Dear Diary,

Today I went skydiving, whilst barely leaving the ground. It was a bunch of fun.

<3, Gary.

[life] Day 359: I guess I'm not a stay at home Dad any more

It's just occurred to me that this will be my last "stay at home Dad" post, because this morning was the last time I'll have Zoe until after she's started at school.

I woke to quite heavy rainfall. Zoe slept late, in think due to a combination of the cooler temperature, lack of direct sunlight, the rain, and the busy previous day and slightly later bedtime. I used the extra time to get ready for the day.

After a quick breakfast, we got straight in the car to get to Sarah's place, because I had to be back home again by 9:15am for a video interview with Google. It was tight, with the wet road conditions, but fortunately my interviewer was a few minutes later than me, so everything worked out okay.

Since then, I've been wandering around in a bit of a daze. I collected the last unit of my real estate licence course from my PO box. All I need now is the actual bit of paper saying I've achieved all of the requisite units of study, and I can go and apply for my licence. Oh, a business name will help too. Coming up with a name/domain name combination is proving phenomenally difficult.

I guess this lull in proceedings before a whole bunch of new stuff starts is a good time to stop and reflect.

I've had a fantastic year. This would have to be the sustained best year of my adult life. On a personal front, I've got to take a break from work, and go off and explore a bunch of other things. I did a barista course. I did a top rope climbing and abseiling course. I've finished my real estate licence course. I've had the opportunity to explore a few different startup ideas, and meet a bunch of different people.

My running has fallen off a bit, but I'm trying to fix that up again now. Maybe this year will be the year I do the Gold Coast Half Marathon. In fact I think I will actively work to that goal this year, since I shouldn't have any date conflicts like I ended up having last year.

On the fathering front, I had an irreplaceably wonderful year with Zoe. I have cemented my relationship with her, and I'm very confident in my capabilities as a single parent. I'm grateful that Zoe got to have the benefits of a proper Kindergarten program. It will give her a strong foundation entering Prep this year.

I'm also glad I managed to keep up the blogging for the whole year. It'll be nice to have this record to look back on.

As for 2015 for me, it's still a bit of a blank slate. I'm open to returning to Google, if that works out best for everyone. There's also a local job that I've found that I believe I'll be a good fit for, which I'm interviewing for next week. I could, if I was feeling really game, just freelance, and do a whole bunch of different things, but I think after a year of living off my savings, I'd prefer the security of a stable income for a while.

Right now, I think I'll try and clean up my desk, so I can start whatever comes next with a tidy desk.

January 22, 2015

Harcourt and Rogers Trigs

I needed to visit someone in deepest darkest North Canberra yesterday, and there was an hour to kill between that meeting and the local Linux User's Group meeting. It seemed silly to have driven all that way and to not see a couple of trigs, so I visited these two. Both these trigs were easy to get to and urban. Frankly a little boring.

Harcourt trig is in what I will call a cow paddock -- it doesn't have a lot of trees happening and feels a bit like left over land. Access to the nature reserve wasn't very obvious to me from the suburban streets, but the KML file below might help others to work it out. It wasn't too bad once I'd navigated the maze of streets and weird paved areas.


Interactive map for this route.

Rogers was similar, except access was more obvious because it is in an older suburb. This is a nicer reserve than Harcourt's, with a nice peak and some walking opportunities around the base of the hill. I think I'll probably end up coming back to this one as my wife is nostalgic about growing up backing on to this reserve.


Interactive map for this route.

Tags for this post: blog pictures 20150122-harcourt_and_rogers photo canberra gungahlin belconnen bushwalk trig_point urban_trig

Related posts: A quick walk to Tuggeranong Trig; A walk around Mount Stranger; Taylor Trig; Urambi Trig; Walk up Tuggeranong Hill; Wanniassa Trig


[life] Day 358: Doctor, class lists, shopping, swimming, haircuts, dinner

Yesterday was a rather jam packed day. I'm not surprised Zoe's sleeping in, although the rain is probably helping.

We got going in the morning, and first off was the weekly wart freezing at the doctor. We're going to have to take a different approach once school starts next week.

After that, I checked my PO box, and got back one more unit of my real estate licence course (marked competent). So I'm just waiting for one I had to resubmit then I've officially completed everything. Hopefully that will come through early next week.

After that, we popped over to Chloe's place so we could all walk around to the school together to check the class lists, which were published.

Zoe did really well. She got the teacher that I'd wanted her to get, and Chloe is in her class, along with a boy, Flynn, she got to know through the Poppet program they ran last year. I met the mother of another little girl, Milla, who will be in Zoe's class too, and introduced the girls to each other. I'm very excited to see how Zoe's going to go at school next week. I think she's going to do really well.

After that, we went out to Carindale to do some birthday present related shopping for Eva and Layla's upcoming birthday, and some other general shopping. We also bumped into Mackensie's dad.

We ended up at Carindale for quite a while, so by the time we were done, we had enough time to race home, make a batch of hummus for dinner, and race out again to swim class.

Straight after swim class, we had haircuts, so we had to hurry over to the hairdresser, and then head back to Chloe's place for dinner.

Zoe had a great time playing with Chloe and her older sister, and I got to put my feet up for a bit and hang out with Kelley and Mark. It was a nice evening.

We got home to a welcome letter in the mail from Zoe's new teacher, which was nice. We made that the bedtime reading, as it was getting late.

Mount Burnett Observatory Open Day and Third Birthday – Saturday 24th January 2015

As some of you know I’m involved with the Mount Burnett Observatory, a community run astronomical observatory in the Dandenong Ranges of Victoria near Emerald to the south-east of Melbourne. Originally built by Monash University in the early 1970’s it’s 3 years since a small group of people formed a community association, took over the site and starting resurrecting it as an observatory by and for the people. It’s now three years on and by the end of last year we were the second largest astronomical association in Victoria!


This Saturday (24th January) is our third birthday celebration so we’re having an open day running from 1pm through to 6pm with tours, activities, a solar telescope and components from the Murchison Widefield Array (MWA), one of the precursor telescopes to the massive Square Kilometer Array telescope project!

At 6pm we have a barbecue and then at 7pm there will be a talk by Perry Vlahos on what there will be to see in the sky over the coming month. After that we’ll be socialising and, if the weather behaves itself, viewing the stars through the many observatory telescopes.


This item originally posted here:

Mount Burnett Observatory Open Day and Third Birthday – Saturday 24th January 2015

Thank you all for being awesome!

Just over a year ago in Perth, we invited you to a party at our place, and we feel as if that is exactly what #lca2015 has been.

You have Been Awesome guests, and it's been a great party.

We had fantastic feedback from our event venues. MOTAT told us that their volunteer enthusiasts who were staffing the exhibits really enjoyed the intelligent conversations that they had with you. The staff at Sweat Shop said that you were all extremely well behaved which was why they were happy to keep the bar open for as long as you wanted.

We couldn't have asked for more from our guests.

We had a great time, and now it's time to relax for a little while. The videos of the presentations and keynotes are now online, and we're uploading the slides as they come in.

Our Thanks To

  • Linux Australia for trusting us with this amazing event
  • Our Speakers, Miniconf Organisers and Keynote / Plenary presenters
  • Linus, Bdale, Andrew and Rusty for the Q&A Session on Friday
  • The delegates for joining us in Auckland for #lca2015
  • The Sponsors for their contributions to
  • AV, Video and Network Team
  • Rego Desk
  • Partners Program Team
  • Room Runners
  • Our Drivers
  • Graphics and design team for our website, logos and associated swag
  • The ghosts of conferences past
  • ...and the team who have spent so much of the last year putting this event together

Some Numbers

As part of our closing session we provided some numbers regarding LCA 2015.
  • 0 - Unclaimed badges or bags at rego desk
  • 1 - Virtual session
  • 3 - Official social events for our attendees
  • 8 - Years since Cherie and Steven first attended #lca2007 in Sydney
  • 9 - Participants in our Keynote, Plenary and Q&A Sessions
  • 13 - Miniconfs
  • 89 - Main Program talks / tutorials
  • 470 - Days from being notified that Auckland would host #lca2015 until our closing ceremony
  • 650 - Average number of distinct devices on the conference delegate WiFi network
  • 715 - Days from the #lca2015 BoF held at #lca2013 in Canberra until the close of #lca2015
  • 3113 - Coffees served up by Tuihana Cafe
  • 3GB/s - Typical daily peak data utilisation on the conference WiFi network

To the future

The whole #lca2015 Auckland team wish the best of luck to the #lca2016 Geelong team...

Haere rā

Cherie and Steven Ellis

January 21, 2015 – so much all the things!

Well, I’ve said it before and not followed through, but I am intending to blog about various stuff from last weeks LCA over the next month or two.

One things about LCA of course is how much you learn. Especially when you stand up in front of a room to share something and discover errors in your own understanding! In my own case, I had a talk at the Open Hardware miniconf about some security things related to embedded devices. Literally an hour before I had a ping on twitter alterting me to a factual error in my blog, which was also loudly proclaimed in the talk I was about to deliver. Luckily it was only one slide, and the misunderstanding did not impact the rest of the talk (or for that matter, most of the offending blog article.) So I have updated the original blog article with a correction.

January 20, 2015


Slowly getting back into the swing of things, walked into work at stupid o’clock in the morning. Spending the afternoon at The Edge. Catching up on a few days of this diary.

Gearing up for the LCA debrief at Humbug, trying to not do it so off-the-cuff this year, organising the notes online in our wiki.

Filed under: diary


Waking up in a stupidly muggy Brisbane. Realised how happy I should be at having avoided this weather for a week.

Bussed to work, had to take have a shower when I got into work!

Filed under: diary


Windy Wellington!

Breaky and fancy coffee at..the coffee hangar I think? Then trooped back for the free Air New Zealand exhibit at Te Papa.

Headed to the airport quite early as I was completely drained by this point. The temperature and humidity levels back in Brisbane were quite confronting, I got very little sleep this night.

Filed under: diary


New Zealand!

Partook in Geeks On A Train today, from Auckland down to Wellington, quite happy with how it all went. Just about all of the photos in my LCA2015 album are taken on the train.

It was nice disconnecting from the conference and technology for a bit. And I have a feeling that the trip will be quite an important one to remember in the future.

Filed under: diary

Another lunch time walk

My arm still hurts, so no gym again. Instead, another lunch time walk although this one was shorter. The skies were dramatic, but no rain unfortunately. I found GC1DEFB during this walk.


Interactive map for this route.

Tags for this post: blog pictures 20150120-geocaching photo canberra tuggeranong bushwalk geocaching

Related posts: Lunchtime geocaching; A walk around Mount Stranger; Taylor Trig; Urambi Trig; Walk up Tuggeranong Hill; A quick walk to William Farrer's grave




The morning keynote really did feel like a kick in the guts to all the work that we’ve been doing, and is a horrible tail end to a conference that started with the wonderful community leadership summit. I later quipped that keynotes should be at the end of the day in case the only rational response was drinking.

Fortunately there was a light hearted Paul McKenny talk later in the day that lifted my spirits.

And on an even better note, the main organiser for Geelong is not going to put up with such shit from Linus.

Filed under: diary

‘Sup With The Tablet?

As I mentioned on Twitter last week, I’m very happy SUSE was able to support 2015 with a keynote giveaway on Wednesday morning and sponsorship of the post-conference Beer O’Clock at Catalyst:

For those who were in attendance, I thought a little explanation of the keynote gift (a Samsung Galaxy Tab 4 8″) might be in order, especially given the winner came up to me during the post-conference drinks and asked “what’s up with the tablet?”

To put this in perspective, I’m in engineering at SUSE (I’ve spent a lot of time working on high availabilitydistributed storage and cloud software), and while it’s fair to say I represent the company in some sense simply by existing, I do not (and cannot) actually speak on behalf of my employer. Nevertheless, it fell to me to purchase a gift for us to provide to one lucky delegate sensible enough to arrive on time for Wednesday’s keynote.

I like to think we have a distinct engineering culture at SUSE. In particular, we run a hackweek once or twice a year where everyone has a full week to work on something entirely of their own choosing, provided it’s related to Free and Open Source Software. In that spirit (and given that we don’t make hardware ourselves) I thought it would be nice to be able to donate an Android tablet which the winner would either be able to hack on directly, or would be able to use in the course of hacking something else. So I’m not aware of any particular relationship between my employer and that tablet, but as it says on the back of the hackweek t-shirt I was wearing at the time:

Some things have to be done just because they are possible.

Not because they make sense.


January 19, 2015

Lunchtime geocaching

Woke up this morning with a sore left arm, which ruled out going to the gym. Instead, I decided to go for a geocaching walk at lunch time. I found these caches: GC235FM; GC56N78; GC5B9WT; GC5F6G3; and GC5F0PE. A nice walk.


Interactive map for this route.

Tags for this post: blog pictures 20150119-geocaching photo canberra tuggeranong bushwalk geocaching

Related posts: Another lunch time walk; A walk around Mount Stranger; Taylor Trig; Urambi Trig; Walk up Tuggeranong Hill; A quick walk to William Farrer's grave


Fedora: easy recovery from corrupt root partition

When you boot Fedora with a corruption which is not automatically repaired when systemd runs fsck -a then you are asked on the console if to enter single user mode, or if to continue. If you choose to enter single user mode then you'll find that you can't run fsck /dev/md0 as the root filesystem is mounted.

Dracut has a debugging mode with named breakpoints: it will boot up to the break-point, and then dracut will drop the console into a shell.

This is useful for solving a corrupted root filesystem, we can boot up to just before the disk is mounted, breakpoint into the Dracut shell, and then run fsck on the yet-to-be-mounted root filesystem. To do this temporarily add the Dracut breakpoint parameter


to the Linux kernel.

In Fedora you do can temporarily modify the Linux kernel parameters by pressing e at the Grub bootloader prompt, arrow-ing down to the "linux" command, adding the parameter to the end of that line, and pressing F10 to run the Grub command list you see on the screen.

Dracut will load the logical volumes, assemble any RAID, and then present a shell on the console. Say fsck /dev/md0 (or whereever /etc/fstab says your / filesytem lives) and then reboot. This is a world easier than booting from a CD or USB and working out which partitions are on what logical volumes, and which logical volumes are in which RAID devices.

Breakpoints are a very fine feature of Dracut and, as this blog posting shows, very useful for solving problems which appear during the early stages of booting the machine.

The Growth of Modern C++ Support


Completing what I started here, I’ve charted the numbers from Christophe’s data for C++11, C++11 Concurrency, C++14 and C++17.

The data is taken entirely from the linked pdf with one exception: N3664 is a clarification that permits optimization, not a requirement for compliance. Compilers that do not perform this optimization are no less compliant with C++14. I’ve recomputed the percentages for all compiler versions to take this into account.

In addition to the references from the previous post, the approval date of C++14 was taken from

January 18, 2015

[life] Day 352: Camping again, beach time and visitors

We were camping underneath a wattle tree, and this wattle tree seemed very popular with the rainbow lorikeets, so we were up at 5:30am, literally with the birds. Zoe was still very excited about camping.

We had a shower first, and then I cooked some bacon and eggs for breakfast before heading down to the beach. Eva and Layla were coming to visit us for the day, so I thought we could just meet them on the beach before coming back to the camp site for lunch.

Zoe had a great time playing around in the waves, and then we did some sand play, making sand castles. I showed her how she could dig down to the sea water in the sand above the wave, and we found heaps of small bivalves. Zoe thought they were pretty cool.

Eva and Layla were running a bit late, so we finished up at the beach and met them at the caravan park, where we cooked some hotdogs for lunch, before having a swim in the pool and then heading back to the beach. It was a really nice afternoon.

After they left, we took it easy for a while before going out for fish and chips for dinner and then calling it a night. Zoe was fast asleep by 7pm, despite the light outside the tent.

Twitter posts: 2015-01-12 to 2015-01-18

SaltStack Ubuntu Hostname

SaltStack currently doesn’t set the hostname correctly on Debian/Ubuntu. For example, this won’t work:

      - enabled: True
      - hostname:

Here’s a little shell script I wrote, to get around this problem:

% cat 

hostname $hn
echo $hn > /etc/hostname
sed -i "1s/.*/ localhost $hn/" /etc/hosts

Then apply it using cmd.script, for example:

    - source: salt://soe/
    - args:
    - unless: grep -q "" /etc/hosts

[life] Day 351: Camping set up

Today was the big day. Pack up and drive to Bribie Island for two nights camping. This was the first time I've attempted camping since moving back to Australia (and being on my own). I like camping, and Zoe absolutely loves it, but I've found the idea of trying to do it all on my own a bit daunting, and it's taken me this long to get around to tackling it.

We managed to get the car packed up and be on the road by about 9:30am. Zoe was really helpful, and was able to help carry some of the lighter, less bulky stuff down to the car for me, that that was an unexpected bonus. Everything fit reasonably well with half of the back seats folded down.

We made good time getting there, and arrived at the caravan park by about 10:45am, but check in wasn't until 1:30pm, so we pottered around, and ended up back on the calm side of the island and had some lunch in the park there. After lunch, we caught up with Zoe's Great Aunty Pam for an ice cream before returning to the caravan park to check in.

We'd just started pegging down the tent when it became apparent that the very large caravan that was trying to maneuver into the site next to ours wasn't going to fit, so I offered to do a swap with them before I got any further invested in our site, so we had to pull up the tent and start over on the site next door.

This was the first time I'd put up this tent, so there was all the attendant trial and error of putting it up. I've learned all the lessons now, so I'll be better next time. Zoe was again super helpful, and we were able to put the tent up together successfully. It was ridiculously hot, and very sweaty work.

After we got the tent up, we went for a swim in the pool to cool off, before I started on dinner. The first night's dinner was just some spaghetti bolognaise. I'd pre-made the bolognaise at home and frozen it, so I just had to reheat it on the stove. The caravan park had a pretty decent camp kitchen, but I was trying to see how much I could be self-sufficient, so I did all the cooking on the gas stove I'd brought with me.

I got Zoe to bed a little bit later than normal, but she slept pretty well, despite the heat. It took me hours to get to sleep, despite being exhausted, because I found the tent uncomfortably hot. I was really happy with how the set up day had gone though, and Zoe had an absolute ball. It was totally worth all the effort.

[life] Day 350: Doctor, laying low, camping preparation

We didn't have a lot planned for the day, and given that we were embarking on a camping trip the following day, I decided to keep it that way. As it was, we still managed to have a pretty busy day.

I started the day off with a run, and managed to do 10 kilometres for the first time in ages. It was a dreadful time, but I was going for progress over perfection.

I didn't realise I had a chiropractic adjustment, so I had to go straight to the chiropractor after my run and sweat all over everything. I felt so bad, but I haven't had an adjustment since before Christmas, so it was great.

Then Sarah dropped Zoe off, and I finally got to have a shower and some breakfast. After that, we headed over to the doctor for the obligatory weekly wart freezing appointment. I think it's been the production it has been because the doctor hasn't been able to give it a really decent hit with the liquid nitrogen, but it's definitely shrinking. She was super brave and even let the doctor give it a bit of a scrape with a scalpel to take off some of the top layers of dead skin. I'm grateful that we have the relationship that we do, because she was a bit scared, but she trusted me anyway, and it all worked out fine.

On the way home, we picked up some mail from the post office. I have to resubmit one unit of my real estate licence course, because I made a mistake, but I passed the other one. I think I'm waiting for one more unit to come back.

After that, we just hung out at home until after lunch, and then went to Woolworths to do some grocery shopping. We ran into Lachlan there, and Zoe and Lachlan had a great time hanging out while we did the grocery shopping.

Instead of getting ready for camping, I decided to have a crack at baking one of the things I want to put in Zoe's school lunchbox, some Hidden Veggie Lunchbox Scrolls. They turned out pretty good, like something you'd get from Baker's Delight. The challenge now is to make space for them in the freezer.

Taylor Trig

At the top of Mount Taylor lies the first trig point defeated by a group walk I've been on. Steve * 3, Erin, Michael *2, Andrew, Cadell, Maddie, Mel, Neill and Jenny all made it to the top of this one, so I'm super proud of us as a group. A nice walk and Mount Taylor clearly has potential for other walks as well, so I am sure I'll return here again.


Interactive map for this route.

Tags for this post: blog pictures 20150118-mount_taylor photo canberra tuggeranong bushwalk trig_point

Related posts: A walk around Mount Stranger; Urambi Trig; Walk up Tuggeranong Hill; A quick walk to Tuggeranong Trig; Wanniassa Trig; Another lunch time walk


January 17, 2015

Creating a DMZ in OpenWRT

In computing, a DMZ (demilitarized zone) is a method for separating untrusted traffic from a trusted network. One of the most common implementations of this would be for supporting a publicly accessible server (such as web) on a local internet connection. The server sits in the DMZ and can be accessed from the Internet, but it cannot access the trusted network.

OpenWRT probably needs no introduction, the brilliant open source and community driven Linux based embedded router stack. I run it on my Netgear WNDR3800.

Netgear WNDR3800 running OpenWRT

Netgear WNDR3800 running OpenWRT

I have an ODRIOD-U3 (little ARM box) running Fedora, which runs a web server. This is what I want to make publicly available in my DMZ.

So, how to create a DMZ in OpenWRT? Some commercial routers have a single button “make a DMZ” and everything is handled behind the scenes for you. Not so with OpenWRT; it’s powerful, transparent, and only does what you tell it to, so we have to create it manually.

Physical devices

My router has a bunch of physical interfaces:

  • eth0 (switch)
  • eth1 (ethernet)
  • wlan0 (wireless card)
  • wlan1 (5GHz wireless card)

The eth1 device maps to the physical WAN port on the back of the router. It’s important to note that the physical interfaces may differ from router to router, depending on the chipsets.

The Switch

The switch (eth0) includes a number of ports, including the four physical ones on the back of the router, a fifth one that’s not used, as well as one that connects to the CPU.

The switch supports VLANs (virtual LANs), and by default OpenWRT puts all of those ports into VLAN 1. This means that physical connections in those four ports at the back are on the same virtual switch and are able to communicate with each other. You can imagine that if I changed the VLAN of one of those ports to VLAN 10, that the device plugged into that port would no-longer be able to communicate with other devices on the switch. This is the basis for our DMZ.

That VLAN 1 actually creates a new interface on the router:

  • eth0.1 (VLAN 1)

The configuration of the switch (including the mapping of ports to VLANs) is available under the switch menu, Network -> Switch.

Note: The port numbers on the switch in OpenWRT do not necessarily map in the right direction to the back of the router. In my case, port 0 on the switch is port 4 on the back of the router.

Creating a new VLAN

The first thing we want to do is create VLAN 10 and then assign one of the ports to that VLAN, removing it from VLAN 1.

  • Browse to Network -> Switch
  • Click Add to make a new VLAN entry
  • Set this new entry’s VLAN ID to 10
  • In the VLAN 1 row, change Port 0 to off
  • In the VLAN 10 row, change Port 0 to untagged
  • In the VLAN 10 row, change CPU port to tagged
Create VLAN

Create VLAN

Setting VLAN to untagged tells the switch to add the appropriate VLAN tag to each ethernet frame as the traffic exits that port. The setting tagged means that the switch should expect that traffic leaving the port has already been tagged, perhaps by the operating system running on the device which is attached to the port.

Port 0 (port 4 on the back of the router) is now in VLAN 10, while the remaining three ports are in VLAN 1 and so it is now isolated from the others. The CPU is also in VLAN 10, else we would not be able to pass any traffic to port 0.

That new VLAN 10 creates a new interface on the router:

  • eth0.10 (VLAN 10)


In OpenWRT you create virtual network interfaces which map to physical devices on the router. These are available under the Network -> Interfaces menu.

For example, my router has:

  • LAN (for my internal local area network)
  • WAN (for the external Internet connection)

One or more physical devices are attached to these zones, for example in my case:

  • LAN (bridges VLAN 1 eth0.1, wlan1 and wlan0 together)
  • WAN (eth1)

The LAN bridge creates a new interface on the router:

  • br-lan (bridged LAN)

Creating a new interface

Once we have created our new VLAN, we want to create a new a interface for the DMZ. In the same way that the VLAN 1 device, eth0.1, is attached to the LAN interface, we will attach VLAN 10 device, eth0.10, to our new DMZ interface.

  • Browse to Network -> Interfaces
  • Click Add New Interface to make a new DMZ zone
  • Set the name of the new interface to DMZ
  • Leave the protocol of the new interface to static
  • Ensure bridge over multiple interfaces remains unchecked
  • For the interface, select only VLAN Interface: “eth0.10″
  • Click Submit
Create Interface

Create Interface

You should be presented with a new configuration screen for this interface.

  • Set IPv4 address to something in a new range different to LAN, e.g. if your LAN is then set DMZ to
  • Leave the rest of the settings blank, you do not need to set routes, or IPv6 if you don’t want to
Interface Configuration

Interface Configuration

  • Click on the Advanced Settings tab
  • Ensure Bring up on boot is ticked
  • If you don’t want IPv6, untick Use builtin IPv6-management
Interface Configuration - Advanced

Interface Configuration – Advanced

  • Click on the Physical Settings tab, should already be set to eth0.10
Interface Configuration - Physical

Interface Configuration – Physical

  • Click on the Firewall Settings tab
  • Under Create / Assign firewall-zone select unspecified -or- create and type dmz
  • Click Save and Apply
Interface Config - Firewall

Interface Config – Firewall

  • If you want to run DHCP on your DMZ, then under DHCP Server click Setup DHCP Server button, leave default settings
Interface Config - DHCP

Interface Config – DHCP

We now have a new interface or zone called for the DMZ that’s set to use out DMZ VLAN. It has a new firewall policy assigned to it, dmz, which we now need to configure.


Now we need to configure the firewall to do a few things:

  • Allow the DMZ to talk to the WAN zone, so that devices can access the Internet
  • Allow the LAN zone to talk to the DMZ, but not the other way around
  • Add some traffic rules opening ports 53 and 67, so that devices from the DMZ can access DNS and DHCP services on the router’s DMZ IP address
  • Finally, forward the HTTP port (80) from external internet WAN interface onto a device in the DMZ

Let’s do zone settings first.

  • Browse to Network -> Firewall
  • Under the Zones section on General Settings page, edit the dmz zone
  • Leave the name set to dmz
  • Set input to reject, so that we drop all incoming packets by default
  • Leave output as accept, although you could set this to reject by default but you’ll require specific outgoing rules as required (like for Yum updates)
  • Leave Masquerading and MSS clamping disabled
  • Under Covered networks ensure that only dmz is selected


  • Under the section Inter-Zone Forwarding, ensure Allow forward to destination zones is set only to WAN
  • ensure Allow forward from source zones is set only to LAN
Zone Forwarding

Zone Forwarding

  • Click Advanced Settings tab
  • If you don’t want IPv6, you can set Restrict to address family to IPv4 only
  • Tick Enable logging on this zone, so that we can see what’s happening
Firewall Configuration - Advanced

Firewall Configuration – Advanced

Now let’s do port forwards.

  • Click on the Port Forwards tab
  • Under New port forward section, give a name, such as dmz-http
  • Set Protocol to TCP
  • Set External zone to WAN
  • Set External port to 80
  • Set Internal zone to DMZ
  • Set Internal IP address to your DMZ server, e.g.
  • Set Internal port to 80
  • Click Add when you’re happy
  • Repeat for HTTPS port 443 if you want to run a secure server
Port Forwarding

Port Forwarding

Finally, let’s finish with traffic rules.

  • Click on the Traffic Rules tab
  • Under Open ports on router, set a name like dhcp-dns
  • Under Protocol, select UDP
  • Under Port, set 53
  • Click Add
  • Find your new rule in the list and click edit
  • Set Destination address to your router’s DMZ IP address
  • Repeat for DHCP port 67 UDP if you want to use router’s DHCP server, but don’t set the destination address as DHCP is broadcast
Firewall Traffic - DHCP & DNS

Firewall Traffic – DHCP & DNS

If you want to be able to ping the router from the DMZ clients, do this.

  • Set a name like ping-dmz
  • Set protocol to Other
  • Click Add
  • In the new configuration page, set Protocol to ICMP
  • Set Match ICMP type to echo reply
  • Set Source zone to dmz
  • Leave Destination zone to Device (input)
  • Set Destination address to your router’s DMZ IP address
  • Click Save
Firewall Traffic - Ping

Firewall Traffic – Ping

Checking the logs

Remember we told the router to log the DMZ? Well now we can monitor the firewall rules by browsing to Status -> Kernel Log. Here you should be able to see any rejects that are happening, which is useful to work out why something isn’t happening as you expect on the DMZ.

For example, disable the dmz-ping rule and then try to ping the router from your DMZ server. Refresh the Kernel Log and you should see entries appear.


Plug in a device, see if it gets an IP address. Try to ping (Google DNS server), then try to ping

Set up a web server on your DMZ box, or use netcat to listen on port 80. Get your external IP address from the router, or Google “my ip”. Now get a friend to browse to your IP and see if you see your web server.


January 16, 2015

The Growth of C++11 Support

Update: This chart has been updated and I’ve added charts for C++11 Concurrency, C++14, and C++17 here.


A few days ago, Christophe Riccio tweeted a link to a pdf that shows the level of support for “Modern C++” standards in four C++ compilers: Visual C++, GCC, Clang, and ICC.

One of the things I wanted to see was not just how support had advanced between versions of each compiler, but how compilers had changed relative to one another over time. I extracted the numbers for C++11 from Christophe’s document, found the release dates for each compiler, and created a chart that puts it all together.

It’s interesting to see how far behind Clang starts in comparison to the others, and that it ends up in a close dance with GCC on the way to full C++11 support. It also highlights how disappointing VC++ has been in terms of language feature advancement — particularly when VS2010 was ahead of Clang and ICC for C++11 features.

Creating the chart also served as an opportunity to play around with data visualization using Bokeh. As such, you can click on the chart above and you’ll see a version that you can zoom, pan, and resize (which is only a small part of what Bokeh offers). I intend to write about my experiences with Bokeh at a later date.


Release dates for each compiler were taken from the following pages:

The date used to mark the approval of the C++11 standard is taken from

Building Reaktor Synthesisers, Download Scripts, and Re-Spin Revenue

After a bit of fiddling I've figured out how to build non-trivial Reaktor software synthesisers. By the looks of things, you can do quite a lot but there seems to be some gaps in the software which makes building a full blown synthesiser ready for sale to the public (which they will want to buy) a non-option for the moment (unless there is some information that I'm missing which is likely the case)...

For the lazy among you the easiest Reaktor Synthesisers that can be built can be made as so. Right click in the workspace, Instrument -> Synthesizers -> Option and then hook up to correct/relevant Voice Combiner. My designs obviously start from scratch though, as I'd like to be able to design some both for educational purposes, for resale, and if that's not possible simply to give away.

You can download my updated experiments from here:

I've been looking to build some Android software applications for a while now (curious to know whether this is a viable long term option). It's interesting how many people actually Open Source their software on the various web stores.

I recently wanted to download al the applications/archives from a particular website, so I looked at various website download programs (HTTrack, Teleport Pro, wget, curl, etc...). In spite of the filters/wildcards that were available they were too slow to be realistic.

####Start Quote####

Use wildcards to exclude or include URLs or links. You can put several scan strings on the same line. Use spaces as separators. Example: +*.zip -www.*.com -www.*.edu/cgi-bin/*.cgi

+*.png +*.gif +*.jpg +*.css +*.js*

+*.zip +*.exe +*.msi +*.tar.gz +*.tar +*.rar

+*.css +*.js*

####End Quote#####

What did I do? I built something because I noticed patterns in the way files were encoded.

####Start Quote####

Range for Instrument VSTs

was the same as

which converted to

which could then be parsed for automated download.

Range for Effects VSTs

was the same as

which converted to

which could then be parsed for automated download.

Range for Midi VSTs

####End Quote#####

You can download my script from here:

As I've stated previously I've been thinking of re-spinning some versions of Linux for fun and possibly profit. The irony is that it's actually much easier to go down than it is go up. Namely, the smaller distributions such as DamnSmall don't really lend themselves to customisation going up because there are too many dependencies that need to be remedied prior to being able to come up with something workable. This has led me to work on scripts to achieve the exact opposite on smaller (but large such as Knoppix) DVD/CD based live distributions. They work based on class of program based on yum or apt package information. It'll be interesting to see what we can do.

Several of the ways in which I was thinking about making revenue was:
  • distributing/re-sale on chosen media such as USB, CD, DVD, etc...
  • creating custom versions for who ever wants them. After all, if I'm currently building the code to allow for this why not? (You need to send a portion of payment now and rest on delivery.) Working perferably only on smaller distributions at this point unless the project is really interesting.
  • donations
  • figuring out what the public wants and then attempting to build that for them
  • figuring out what the best possible distribution is and attempting to build that for the public
  • support via of these distributions
Interesting stuff in general I came across during the week. 2015 – Day 5 – Session 3

NoOps with Ansible and Puppet – Monty Taylor

  • NoOps
    • didn’t know it was a contentious term
    • “devs can code and let a service deploy, manage and scale their code”
    • I want to change the system by landing commits. don’t want to “do ops”
    • if I have to use my root access it is a bug
  • Cloud Native
    • Ephemeral Compute
    • Data services
    • Design your applications to be resilient via scale out
    • Cloud scale out, forget HA for one system, forget long-lived system, shared-nothing for everything. Cloud provides the hard scale-out/HA/9s stuff
    • Great for new applications
  • OpenStack Infra
    • Tooling, automation, and CI for the openstack project
    • 2000 devs
    • every commit is fully tested.
    • each test runs on a single use cloud slave
    • 1.7 million test jobs in the last 6 months. 18 TB of log data
    • all runs in HP and rackspace public clouds
  • Create Servers manually at 1st
  • Step 1 – Puppet
    • extra hipster because it is in ruby
    • If you like ruby it is awesome. If don’t is it less-awesome
    • collaboration from non-root users
    • code review
    • problem that it blows up when you try and install the same thing in two different places
    • 3 ways to run. masterless puppet apply. master + puppet agent daemon . master + puppet agent non-daemons
  • Secret stuff that you don’t want into you puppet git repo
    • hiera
  • Step 2 – Ansible for orchestration
    • Control the puppet agent so it runs it nicely and in schedule and on correct hosts first
    • Open source system management tool
    • Sequence of steps not description of state like puppet
    • ad-hoc operation. run random commands
    • easy to slowly grow over time till it takes over puppet
    • yaml syntax of config files
  • Step 3 – Ansible for cloud management
  • Ansible config currently mixed in with puppet under –


Conference Closing

  • Steve Walsh wins Rusty Wrench award
  • Preview of 2016 in Geelong
    • Much flatter than Auckland
    • Deakin University – Waterfront Campus
    • Waurn Ponds student accomadation 15 minutes with shuttles
    • Feb 8th – 12th 2016
    • CFP 1st of June 2015
    • Theme “life is better with linux”
    • 4 keynotes confirmed or in final stages of discussion, 2 female, 2 male
    • NFS keytags
  • Announcement for 2017 will be in Hobart


Another Nova spec update

I started chasing down the list of spec freeze exceptions that had been requested, and that resulted in the list of specs for Kilo being updated. That updated list is below, but I'll do a separate post with the exception requests highlighted soon as well.


  • Add more detailed network information to the metadata server: review 85673 (approved).
  • Add separated policy rule for each v2.1 api: review 127863 (requested a spec exception).
  • Add user limits to the limits API (as well as project limits): review 127094.
  • Allow all printable characters in resource names: review 126696 (approved).
  • Consolidate all console access APIs into one: review 141065 (approved).
  • Expose the lock status of an instance as a queryable item: review 127139 (abandoned); review 85928 (approved).
  • Extend api to allow specifying vnic_type: review 138808 (requested a spec exception).
  • Implement instance tagging: review 127281 (fast tracked, approved).
  • Implement the v2.1 API: review 126452 (fast tracked, approved).
  • Improve the return codes for the instance lock APIs: review 135506.
  • Microversion support: review 127127 (approved).
  • Move policy validation to just the API layer: review 127160 (approved).
  • Nova Server Count API Extension: review 134279 (fast tracked).
  • Provide a policy statement on the goals of our API policies: review 128560 (abandoned).
  • Sorting enhancements: review 131868 (fast tracked, approved, implemented).
  • Support JSON-Home for API extension discovery: review 130715 (requested a spec exception).
  • Support X509 keypairs: review 105034 (approved).


  • Expand support for volume filtering in the EC2 API: review 104450.
  • Implement tags for volumes and snapshots with the EC2 API: review 126553 (fast tracked, approved).


  • Actively hunt for orphan instances and remove them: review 137996 (abandoned); review 138627.
  • Add totalSecurityGroupRulesUsed to the quota limits: review 145689.
  • Check that a service isn't running before deleting it: review 131633.
  • Enable the nova metadata cache to be a shared resource to improve the hit rate: review 126705 (abandoned).
  • Implement a daemon version of rootwrap: review 105404 (requested a spec exception).
  • Log request id mappings: review 132819 (fast tracked).
  • Monitor the health of hypervisor hosts: review 137768.
  • Remove the assumption that there is a single endpoint for services that nova talks to: review 132623.

Block Storage

  • Allow direct access to LVM volumes if supported by Cinder: review 127318.
  • Cache data from volumes on local disk: review 138292 (abandoned); review 138619.
  • Enhance iSCSI volume multipath support: review 134299 (requested a spec exception).
  • Failover to alternative iSCSI portals on login failure: review 137468 (requested a spec exception).
  • Give additional info in BDM when source type is "blank": review 140133.
  • Implement support for a DRBD driver for Cinder block device access: review 134153 (requested a spec exception).
  • Poll volume status: review 142828 (abandoned).
  • Refactor ISCSIDriver to support other iSCSI transports besides TCP: review 130721 (approved).
  • StorPool volume attachment support: review 115716 (approved, requested a spec exception).
  • Support Cinder Volume Multi-attach: review 139580 (approved).
  • Support iSCSI live migration for different iSCSI target: review 132323 (approved).


Containers Service


  • Develop and implement a profiler for SQL requests: review 142078 (abandoned).
  • Enforce instance uuid uniqueness in the SQL database: review 128097 (fast tracked, approved, implemented).
  • Nova db purge utility: review 132656.
  • Online schema change options: review 102545 (approved).
  • Support DB2 as a SQL database: review 141097 (fast tracked, approved).
  • Validate database migrations and model': review 134984 (approved).

Hypervisor: Docker

Hypervisor: FreeBSD

  • Implement support for FreeBSD networking in nova-network: review 127827.

Hypervisor: Hyper-V

  • Allow volumes to be stored on SMB shares instead of just iSCSI: review 102190 (approved, implemented).
  • Instance hot resize: review 141219.

Hypervisor: Ironic

Hypervisor: VMWare

  • Add ephemeral disk support to the VMware driver: review 126527 (fast tracked, approved).
  • Add support for the HTML5 console: review 127283 (requested a spec exception).
  • Allow Nova to access a VMWare image store over NFS: review 126866.
  • Enable administrators and tenants to take advantage of backend storage policies: review 126547 (fast tracked, approved).
  • Enable the mapping of raw cinder devices to instances: review 128697.
  • Implement vSAN support: review 128600 (fast tracked, approved).
  • Support multiple disks inside a single OVA file: review 128691.
  • Support the OVA image format: review 127054 (fast tracked, approved).

Hypervisor: libvirt

Instance features


  • A lock-free quota implementation: review 135296 (approved).
  • Automate the documentation of the virtual machine state transition graph: review 94835.
  • Fake Libvirt driver for simulating HW testing: review 139927 (abandoned).
  • Flatten Aggregate Metadata in the DB: review 134573 (abandoned).
  • Flatten Instance Metadata in the DB: review 134945 (abandoned).
  • Implement a new code coverage API extension: review 130855.
  • Move flavor data out of the system_metadata table in the SQL database: review 126620 (approved).
  • Move to polling for cinder operations: review 135367.
  • PCI test cases for third party CI: review 141270.
  • Transition Nova to using the Glance v2 API: review 84887 (abandoned).
  • Transition to using glanceclient instead of our own home grown wrapper: review 133485 (approved).


  • Enable lazy translations of strings: review 126717 (fast tracked, approved).


  • Add a new linuxbridge VIF type, macvtap: review 117465 (abandoned).
  • Add a plugin mechanism for VIF drivers: review 136827 (abandoned).
  • Add support for InfiniBand SR-IOV VIF Driver: review 131729 (requested a spec exception).
  • Neutron DNS Using Nova Hostname: review 90150 (abandoned).
  • New VIF type to allow routing VM data instead of bridging it: review 130732 (approved, requested a spec exception).
  • Nova Plugin for OpenContrail: review 126446 (approved).
  • Refactor of the Neutron network adapter to be more maintainable: review 131413.
  • Use the Nova hostname in Neutron DNS: review 137669.
  • Wrap the Python NeutronClient: review 141108.


  • Dynamically alter the interval nova polls components at based on load and expected time for an operation to complete: review 122705.


  • A nested quota driver API: review 129420.
  • Add a filter to take into account hypervisor type and version when scheduling: review 137714.
  • Add an IOPS weigher: review 127123 (approved, implemented); review 132614.
  • Add instance count on the hypervisor as a weight: review 127871 (abandoned).
  • Add soft affinity support for server group: review 140017 (approved).
  • Allow extra spec to match all values in a list by adding the ALL-IN operator: review 138698 (fast tracked, approved).
  • Allow limiting the flavors that can be scheduled on certain host aggregates: review 122530 (abandoned).
  • Allow the remove of servers from server groups: review 136487.
  • Cache aggregate metadata: review 141846.
  • Convert get_available_resources to use an object instead of dict: review 133728 (abandoned).
  • Convert the resource tracker to objects: review 128964 (fast tracked, approved).
  • Create an object model to represent a request to boot an instance: review 127610 (approved).
  • Decouple services and compute nodes in the SQL database: review 126895 (approved).
  • Distribute PCI Requests Across Multiple Devices: review 142094.
  • Enable adding new scheduler hints to already booted instances: review 134746.
  • Fix the race conditions when migration with server-group: review 135527 (abandoned).
  • Implement resource objects in the resource tracker: review 127609 (approved, requested a spec exception).
  • Improve the ComputeCapabilities filter: review 133534 (requested a spec exception).
  • Isolate Scheduler DB for Filters: review 138444 (requested a spec exception).
  • Isolate the scheduler's use of the Nova SQL database: review 89893 (approved).
  • Let schedulers reuse filter and weigher objects: review 134506 (abandoned).
  • Move select_destinations() to using a request object: review 127612 (approved).
  • Persist scheduler hints: review 88983.
  • Refactor allocate_for_instance: review 141129.
  • Stop direct lookup for host aggregates in the Nova database: review 132065 (abandoned).
  • Stop direct lookup for instance groups in the Nova database: review 131553 (abandoned).
  • Support scheduling based on more image properties: review 138937.
  • Trusted computing support: review 133106.



  • Make key manager interface interoperable with Barbican: review 140144 (fast tracked, approved).
  • Provide a reference implementation for console proxies that uses TLS: review 126958 (fast tracked, approved).
  • Strongly validate the tenant and user for quota consuming requests with keystone: review 92507 (approved).

Service Groups

Comment 2015 – Day 5 – Session 2

When Everything Falls Apart: Stories of Version Control System Scaling – Ben Kero

  • Sysadmin at Mozilla looking after VCS
  • Primarily covering mercurial
  • Background
    • Primarily mercurial
    • 3445 repos (1223 unique)
    • 32 million commits
    • 2TB+ transfer per day
    • 1000+ clones per day
    • Biggest customer = ourselves
    • tested platforms > 12
  • Also use  git (a lot) and a bit of:  subversion, CVS, Bazaar, RCS
  • 2 * ssh servers, 10 machines mirror http traffic behind load balancer
  • 1st story – know what you are hosting
    • Big git repo 1.7G somebody asked to move off github
    • Turned out to be mozilla git mirror, so important to move
    • plenty of spare resources
    • But high load straight away
    • turned out to be mercurial->git converter, huge load
    • Ran garbage collection – took several hours
    • tweaked some other settings
  • 2nd story
    • 2003 . “Try” CI system
    • Simple CI system (before the term existed or they were common)
    • flicks off to build server, sends status back to dev
    • mercurial had history being immutable up until v2.1 and mozilla was stuck on old version
    • ended up with 29,000 brashes in repo
    • Around 10,000 heads some operations just start to fail
    • Wait times for pushes over 45 minutes. Manual fixes for this
    • process was “hg serve” only just freezein gup, not any debug info
    • had to attached debugging. trying to update the cache.
    • cache got nuked by cached push, long process to rebuild it.
    • mercurial bug 4255 in process of being looked at, no fix yet
  • The new system
    • More web-scalable to replace old the system
    • Closer to the pull-request model
    • multi-homing
    • leverage mercurial bundles
    • stores bundles in scalable object store
    • hopefully minimal retooling from other groups (lots of weird systems supported)
  • Planet release engineering @ mozilla

SL[AUO]B: Kernel memory allocator design and philosophy – Christopher Lameter

  • NOTE: I don’t do kernel stuff so much of this is over my head.
  • Role of the allocator
    • page allocator only works in full page size (4k) and is fairly slow
    • slab allocator for smaller allocation
    • SLAB is one of the “slab allocators”
  • kmeme_cache , numa aware, etc
  • History
    • SLOB: K&R 1991-1999 . compact
    • SLAB: Solaris 199-2008 . cache friendly, benchmark friendly
    • SLUB: 2008-today , simple and instruction costs count, better debugging, defrag, execution time friendly
  • 2013 – work to split out common code for allocators
  • SOLB
    • manages list of free objects with the space of free objects
    • have to traverse list to find object of sufficient size
    • rapid fragmentation of memory
  • SLAB
    • queues per cpu and per node to track cache hotness
    • queues for each remote node
    • complete data structures
    • cold object expiration every 2 seconds on each CPU
    • large systems with LOTS of CPUs have huge amount of memory trapped, spending lots of time cleaning cache
  • SLUB
    • A lot less queuing
    • Pages associated with per-cpu. increased locality
    • page based policies and interleave
    • de-fragmentation on multiple levels
    • current default in the kernel
  • slabinfo tool for SLUB. tune, modify, query, control objects and settings
  • can be asked to go into debug mode even when debugging not enabled with rest of the kernel
  • Comparing
    • SLUB faster (SLAB good for benchmarks)
    • SLOB slow
    • SLOB less memory overhead for small/simple systems (only, doesn’t handle lots of reallocations that fragment)
  • Roadmap
    • More common framework
    • Various other speedups and features


Thank you to Linus Torvalds for this mornings Q&A

The #lca2015 team want to thank Linus, Bdale, Rusty and Andrew for the Q&A session which opened the conference this morning.

Linus Torvalds with Steven and Cherie Linus Q&A Audience

January 15, 2015

Craige McWhirter: Configuring CoreOS Toolbox to Use Debian

The toolbox command in CoreOS uses Fedora by default. If you'd rather it used Debian by default, you can add the following lines to .toolboxrc:


When you next run toolbox, you should see it pull down the requested image.

$ toolbox
Pulling repository debian
835c4d274060: Download complete
511136ea3c5a: Download complete
16386e29a1f4: Download complete
Status: Downloaded newer image for debian:jessie
Spawning container core-debian-jessie on /var/lib/toolbox/core-debian-jessie.
Press ^] three times within 1s to kill container.

It's that simple. 2015 – Day 5 – Session 1

How to get one of those Open Source jobs – Mark Atwood

  • Warns talk might still have some US-centric stuff still in it
  • “Open Source Job” – most important word is “Job”
    • The Open Source bit means you are a bit more transferable than a closed-source programmer
    • Don’t have to move to major tech city
  • Communication skills
    • Have to learn to Write clearly in English
    • Heave to learn how to speak, including in meetings and give some talks
    • Reachable – Have a public email address
    • Don’t be a jerk, reputation very important
  • Technical skills
    • Learn how to program
    • Start with python and javascript
    • Learn other languages eg scale, erlang, clojure, c, C++
    • How to use debugger and IDE
    • Learn to use git well
    • Learn how to code test (especially to work with CI testers like jenkins)
    • Idea: Do lots of simple practise problems in programming using specific technique or language
  • Relationships & Peers
    • Work with people remote and nearby
    • stackoverflow
    • Don’t be a jerk
  • Work
    • Have to “do the work” then “get the job”
    • Start by fixing bugs on a project
    • Your skills will improve and others will see you have those skills
  • Collaborate
    • Many projects use IRC
    • Most projects have bug tracker
    • Learn how to use the non-basic stuff in git
    • Peer programming
  • Reputation
    • Portfolio vs resume
    • github account is your portfolio
    • Need to be on social media, at least a little bit, most be reachable
  • Getting the Job
    • If you have a good enough a rep the jobs will seek you out
    • Keywords on github and linkedin will attract recruiters
    • People will suggest you that apply
    • Conferences like
    • Remember to counter-offer the offer letter
    • Once you are working for them, work out what is job related an the company might have a claim on. make sure you list in your agreement any projects you are already working on
  • Health
    • Don’t work longer than 40h a week regularly
    • 60h weeks can only be sustained for a couple of weeks
    • Just eat junk-food
    • Don’t work for jerks
  • Money
    • Startups – bad for your health. Do not kill yourself for a nickle, have real equity
  • Keep Learning
  • 3 books to read
    • Oh the palces you will go – Dr Seuss
    • Getting things Done – David Allen
    • How to fail at almost everything and still win big – Scott Adams


Pettycoin: Towards 1.0 – Rusty Russell

  • Problem it bitcoining mining is expensive, places lower limit on transaction fees
  • Took 6 months of to mostly work on pettycoin
  • Petty coin
    • Simple
    • gateway to bitcoin
    • small amounts
    • partial knowledge, don’t need to know everything
    • fast block times
  • Altcoins – bitcoin like things that are not bitcoin
    • 2 million posts to altcoin announce forum
    • lots of noise to talk to people
  • review
    • Paper released saying how it should have been done
    • hash functions
    • bitcoin blocks
    • Bitcoin transactions
  • Sidechain
    • alternative chains that use real bitcoins
    • Lots of wasted work? – bitcoin miners can mine other chains at the same time
    • too fast to keep notes
    • Compact CVP Proofs (reduce length of block header to go all the way back )


Gender diversity in speakers

My first was 2003 and it was absolutely fantastic and I’ve been to every one since. Since I like this radical idea of equality and the LCA2015 organizers said there were 20% female speakers this year, I thought I’d look through the history.

So, since there isn’t M or F on the conference program, I have to guess. This probably means I get things wrong and have a bias. But, heck, I’ll have a go and this is my best guess (and mostly excludes miniconfs as I don’t have programmes for them)

  • 2003: 34 speakers: 5.8% women.
  • 2004: 46 speakers: 4.3% women.
  • 2005: 44 speakers: 4.5% women
  • 2006: 66 speakers: 0% women (somebody please correct me, there’s some non gender specific names without gender pronouns in bios)
  • 2007: 173 speakers: 12.1% women (and an order of magnitude more than previously). Includes miniconfs

    (didn’t have just a list of speakers, so this is numbers of talks and talks given by… plus some talks had multiple presenters)
  • 2008: 72 speakers: 16.6% women
  • 2009: 177 speakers (includes miniconfs): 12.4% women
  • 2010: 207 speakers (includes miniconfs): 14.5% women
  • 2011: 194 speakers (includes miniconfs): 14.4% women
  • 2012: (for some reason site isn’t responding…)
  • 2013: 188 speakers (includes most miniconfs), 14.4% women
  • 2014: 162 speakers (some miniconfs included): 19.1% women
  • 2015: As announced at the opening: 20% women.

Or, in graph form:


  • the historical schedules up on
  • my brain guessing the gender of names. This is no doubt sometimes flawed.

Update/correction: lca2012 had around 20% women speakers at main conference (organizers gave numbers at opening) and 2006 had 3 at sysadmin miniconf and 1 in main conference. 2015 – Day 5 – Keynote/Panel

  • Everybody Sung Happy birthday to Baale
  • Bdale said he has a new house and FreedomBox 0.3 release this week
  • Rusty also on the panel
  • Questions:
    • Why is Linus so mean
    • Unified Storage/Memory machines – from HP
    • Young people getting into community
    • systemd ( I asked this)
    • Year of the Linux Desktop
    • Documentation & training material
    • Predict the security problems in next 12 month
    • Does NZ and Australia need a joint space agency
    • Will you be remembered more for Linux or Git?

Friday Session - Q&A with Linus Torvalds

Linus Torvalds

Way, way back in 2003, at LCA in Perth, there was a Q&A session with Linus Torvalds, Bdale Garbee and Andrew Tridgell. It’s time for a follow-up so at LCA 2015 in Auckland it’s going to happen!

The Q&A session is scheduled for 09:00 am Friday, 16 January 2015 and will be moderated by Bdale Garbee with the assistance of Andrew Tridgell.

Helsinki-born Linus, who simply calls himself a Software Engineer, was the principal force behind developing the Linux kernel. It all started from an initial usenet posting in August of 1991 and made what has proved to be a historic debut with the release of version 1.0 on March 14 1994.

In June 2003 Linus started working for Open Source Development Labs. After merging with the Free Standards Group it became the Linux Foundation where Linus continues to work as the project’s coordinator and is Chief Architect of the Linux kernel.

In 2005, after criticism for his use and alleged advocacy of BitKeeper, proprietary software for version-control in the Linux kernel, Linus wrote a free-software replacement for BitKeeper called GIT which is now the most widely-adopted version-control system for software development.

The LCA 2015 Auckland team would like to thank the Linux Foundation for their assistance in making this possible.

SWAG and sponsored items information page

Below is info and pictures of some of the amazing swag in this year's bag! If you want to take home some extra SWAG then go see the lovely volunteers at reception and you will be able to purchase some extras. The prices are below. We have only limited stock, so be quick!

The SWAG will be on sale after all people have completed registration on Wednesday morning.

Mi Power Bank 10400mAh

Mi Power Bank

This USB charger has rave reviews, due it's form factor and the amount of power it is able to pack into it's small size. The Mi Power Bank contains LG Lithium-ion batteries that can endure 500+ recharge cycles and a rated capacity of 3.6V/10400mAh (TYP). See for more details.

LCA Price: $NZ 40.00

  • The micro-USB port is used to recharge the power bank. It is best to use a 2.0A or higher charger for this.
  • The standard USB port is used to charge your target device (for example, your phone).
  • There are four white lights beside the power button used to indicate the power bank's charge. Each light represents 25% of the total charge available. For example, if all four lights are lit then the powerbank is 75-100% full.
  • To see the current charge in the power bank press-and-release the power button.
  • Plugging your target device into the standard USB port starts charging your target device automatically.
  • To briefly suspend charging of your target device without unplugging it from the power bank, hold down the "power" button on the power bank. Releasing the button will resume charging the target device.
  • When you disconnect your target device from the standard USB port the power bank will shut itself down automatically after 2 minutes.

LCA Bag (rucksack)


The LCA bag by Freeset Global is made under fair trade working conditions using sustainable or organic materials. Freeset Global are serious about bettering the lives of their producers and they also re-invest all profits back to the communities that create our products.

Price: $NZ 10.00

The Coffee Cup

Coffee Cup

The trendy coffee cups are made in New Zealand by CUPPACOFFEECUP. These recyclable coffee cups are made from food-grade polypropylene, which means they are hardy enough for you to reuse them many times, and when you do dispose of them they can be recycled into new consumer goods.

If wish a different design for your coffee cup, take it back to the registration desk and we can exchange it.

Price: $NZ 10.00


We should probably mention the stickers and toiletries - they're free - please take them - we have hundreds of them. :-)



A good day of solid technical stuff today, with no CoC problems (that I saw at least).

Paul McKenny and Matthew Garrett in one day means a lot of knowledge and enjoyment.

Astronomy BOF that night at the Auckland Stardome, where because we were early enough and there was enough room, we were let in to see two shows for the price of one.

Filed under: diary

Linux.conf.ay 2015 – Day 4 – Session 3

Drupal8 outta the box – Donna Benjamin

  • I went to the first half of this but wanted to catch the talk below so I missed the 2nd part


Connecting Containers: Building a PaaS with Docker and Kubernetes – Katie Miller

  • co-presented with Steve Pousty
  • Plugs their OpenShift book, they are re-archetecturing the whole thing based on what in the book
  • Platform as a service
    • dev tooling, runtime, OS , App server, middleware.
    • everything except the application itself
    • Openshift is an example
  • Reasons to rebuild
    • New tech
    • Lessons learned from old deploy
  • Stack
    • Atomic + docker + Kubeneties
  • Atomic
    • Redhat’s answer of CoreOS
    • RPM-OSTree – atomic update to the OS
    • Minimal System
    • Fast boot, container mngt, Good Kernel
  • Containers
    • Docker
    • Nice way of specifying everything
    • Pros – portable, easy to create, fast boot
    • Cons – host centric, no reporting
    • Wins – BYOP ( each container brings all it’s dependencies ) , Standard way to make containers , Big eco-system
  • Kubernetes
    • system managing containerize maps across multiple hosts
    • declarative model
    • open source by google
    • pod + service + label + replication controller
    • cluster = N*nodes + master(s) + etcd
    • Wins: Runtime and operation management + management related containers as a unit, container communication, available, scalable, automated, across multiple hosts
  • Rebuilding Openshift
    • Kubernetes provides container runtime
    • Openshift provides devops and team enviroment
  • Concepts
    • application = multiple pods linked togeather (front + back + db ) managed as a unit, scald independantly
    • config
    • template
    • build config = source + build -> image
    • deployment = image and settings for it
  • This is OpenShift v3 – things have been moving very fast so some docs are out of date
  • Slides 2015 – Day 4 – Session 2

Tunnels and Bridges: A drive through OpenStack Networking – Mark McClain

  • Challenges with the cloud
    • High density multi-tenancy
    • On demand provisioning
    • Need to place / move workloads
  • SDN , L2 fabric, network virtualisation Overlay tunneling
  • The Basics
    • The user sees the API, doesn’t matter too much what is behind
    • Neutron = Virtual subnet + L2 virtual network + virtual port
    • Nova = Server + interface on the server
  • Design Goals
    • Unified API
    • Small Core. Networks + Subnets + Ports
    • Plugable open archetecture
  • Features
    • Overlapping IPs
    • Configuration DHCP/Metadata
    • Floating IPs
    • Security Groups ( Like AWS style groups ) . Ingress/egress rules, IPv6 . VMs with multiple VIFS
  • Deployment
    • Database + Neutron Server + Message Queue
    • L2 Agent , L3 agent + DHCP Agent
  • Server
    • Core
    • Plugins types =  Proxy (proxy to backend) or direct control (login instide plugin)
    • ML2 – Modular Layer 2 plugin
  • Plugin extensions
    • Add to REST API
    • dpch, l3, quota, security group, metering, allowed addresses
  • L2 Agent
    • Runs on a hypervisor
    • Watch and notify when devices have been added/removed
  • L3 agent – static routing only for now
  • Load balancing as a service, based on haproxy
  • VPN as a service , based on openswan, replicates AWS VPC.
  • What is new in Juno?
    • IPv6
    • based on Radbd
    • Advised to go dual-stack
  • Look ahead to Kilo
    • Paying down technical debt
    • IPv6 prefix delegation, metadata service
    • IPAM – hook into external systems
    • Facilitate dynamic routing
    • Enabling NFV Applications
  • See Cloud Administrators Guide


Crypto Won’t Save You Either – Peter Gutmann

  • US Govt has capabilities against common encryption protocols
  • Example Games consoles
    • Signed executables
    • encrypted storage
    • Full media and memory encryption
    • All of these have been hacked
  • Example – Replaced signature checking code
  • Example – Hacked “secure” kernel to attack the application code
  • Example – Modify firmware to load over the checking code
  • Example – Recover key from firmware image
  • Example – Spoof on-air update
  • LOTS of examples
  • Nobody noticed bunch of DKIM keys were bad, cause all attackers had bypassed encryption rather than trying to beat the crypto
  • No. of times crypto broken: 0, bypassed: all the rest
  • National Security Letters – The Legalised form of rubber-hose cryptanalysis
  • Any well design crypto is NSA-proof
  • The security holes are sitting right next to the crypto


January 14, 2015 2015 – Day 4 – Session 1

8 writers in under 8 months: from zero to a docs team in no time flat – Lana Brindley

  • Co Presenting with Alexandra Settle
  • 8 months ago online 1 documentation person at rackspace
  • Hired a couple people
  • Horrible documentation suite
  • Hired some more
  • 4 in Australia, 4 in the US
  • Building a team fast without a terrible culture
    • Management by MEME – everybody had a meme created for them when they started
    • Not all work and No play. But we still get a lot of work done
    • Use tech to overcome geography
    • Treat people as humans not robots
    • Always stay flexible. Couch time, Gym time
  • Finding the right people
    • Work your network , job is probably not going to be advertise on linkedin, bad for diversity
    • Find great people, and work out how to hire them
    • If you do want a job, network
  • Toolchains and Systems
    • Have a vision and work towards it
    • acknowledge imperfection. If you can’t fix, ack and just move forward anyway
  • You can maintain crazy growth forever. You have to level off.
  • Pair US person with AU person for projects
  • Writers should attend Docs summit and encouraged to attend at least one Openstack summit




Bob Young keynote was a bit blah.

Dinner at Motat was great, I took maybe thirty photos. Lots of Melbourne trams for some reason.

Filed under: diary 2015 – Day 4 – Keynotes

Cooper Lees – Facebook

  • Open Source at facebook
  • Increase in pull requests, not just pushing out stuff or throwing over the wall anymore
  • Focussing on full life-cycle of opensource
  • Big Projects: react , hhvm , asyncdisplaykit , presto
  • Working on other projects and sending to upstream
  • Network Switches and Open Compute
    • Datacentre in NZ using open compute designs
  • Open source Switch
    • Top of rack switch
    • Want to be the open compute of network switches
    • Installer, OS, API to talk to asic that runs ports
    • Switches = Servers. running chef
  • Wedge
    • 16-32 of 40GE ports
    • Internal facebook design
    • 1st building block for disaggregated switching technology
    • Contributed to OCP project
    • Micro Server + Switchports

Carol Smith – Google

  • Works in Google Open Source office
  • Google Summer of code
    • Real world experience
    • Contacts and references
  • 11th year of the program
  • 8600 participated over last 10 years
  • Not enough people in office to do southern hemisphere programme. There is “Google code-in” though

Mark McLoughlin – Red Hat

  • Open Source and the datacenter
  • iaas, paas, microservices, etc
  • The big guys are leading (amazon, google). They are building on open source
  • Telcos
    • Squeezed and scrambling
    • Not so “special” anymore
    • Need to be agile and responsive
    • Telecom datacentre – filled with big, expensive, proprietary boxes
    • opposite of agile
  • OPNFV reference architecture
  • OpenStack, Open vswitch, etc
  • Why Open Source? – collaboration and coopetition , diversity drives innovation , sustainability


There was a Q&A. Mostly questions about diversity at the companies and grumps about having to move to US/Sydney for peopl eto work for them

Some Fun

It's been a while since we've done one of these...

Animals in Africa get drunk by eating ripe Marula fruit

Alcoholic Vervet Monkeys! - Weird Nature - BBC animals
Mourinho on Setanta - Gangsta Sven
ATM Theft Backfires as Explosion Knocks Down Robber

Some articles...

Some quotes...
  •  "Two friends are talking: "Say, buddy, could you loan me 100 Euros?" "Well, you know I only have 60 on me." "Ok, give me what you've got and you'll only owe me 40."
  • A young teacher is interviewing for a position. He is asked: "Can you give me three reasons why you wanted to be a teacher?" The interviewee promptly answers: "December, June, and July. 
  • "An attacker could simply download the My Satis application and use it to cause the toilet to repeatedly flush, raising the water usage and therefore utility cost to its owner," it says in its report. "Attackers could [also] cause the unit to unexpectedly open/close the lid, activate bidet or air-dry functions, causing discomfort or distress to [the] user." – Day 3 – Lightning talks


  • Clinton Roy + Tom Eastman – Python Conference Australia 2015 + Kiwi PyCon 2015
    • Brisbane , late July 2015
    • Similar Structure to LCA
    • Christchurch – Septemberish
  • Daniel Bryan – Comms for Camps
    • Detention camps for Australian boats people camps
    • Please contact if you can offer technical help
  • Phil Ingram – Beernomics
    • Doing stuff for people in return for beer
    • Windows reinstall = a Keg
    • Beercoin
  • Patrick Shuff – Open sourcing proxygen
    • C++ http framework. Built own webserver
    • Features they need, monitoring, fast, easy to add new features
    • github -> /facebook/progen
  • Nicolás Erdödy – Multicore World 2015 & the SKA.
    • Multicore World – 17-18 Feb 2015 Wellington
  • Paul Foxworthy – Open Source Industry Australia (OSIA)
    • Industry Body
    • Govt will consult with industry bodies but won’t listen to individual companies
    • Please join
  • Francois Marier – apt-get remove –purge skype
    • Web RTC
    • Now usable to replace skype
    • Works in firefox and chrome. Click link, no account, video conversation
    • Firefox Hello
  • Tobin Harding – Central Coast LUG
    • Update on Central Coast of NSW LUG
    • About 6 people regularly
  • Mark Smith – Failing Gracefully At 10,000ft
    • Private pilot
    • Aircrafts have 400+ page handbooks
    • Things will fail…
    • Have procedures…
    • Before the engine is on fire
    • test
    • The most important task is to fly the plane
  • Tim Serong – A very short song about memory management
    • 1 verson song
  • Angela Brett – Working at CERN and why you should do it
    • Really Really awesome
    • Basic I applied, lots of fellowship
    • Meet someone famous
    • Lectures online from famous people
  • Donna Benjamin – The D8 Chook Raffle
    • $125k fund to get Drupal8 out
    • Raffle. google it
  • Matthew Cengia/maia sauren – What is the Open Knowledge Foundation?
    • Open govt/ data / tech / jouralism / etc
    • govHack
    • Open Knowledge Brisbane Meetup Govt
  • Florian Forster – noping
    • Pretty graphs and output on command line ping
  • Jan Schmidt – Supporting 3D movies in GStreamer
    • A brief overview of it all
  • Justin Clacherty ORP – An open hardware, open software router
    • PowerPC 1-2G RAM
    • Package based updates
    • Signed packages
    • 2015 – Day 3 – Session 2

EQNZ – crisis response, open source style – Brenda Wallace

  • Started with a Trigger warning and “fucker”
  • First thing posted – “I am okay” , one tweet, one facebook
  • State of Scial Media
    • Social media not as common, SMS king, not many smartphones
    • Google Buzz, twitter, Facebook
    • Multiple hashtags
  • Questions people asked on social media
  • Official info was under strain, websites down due to bad generators
  • Crisis Commons
  • Skype
    • Free
    • Multi-platform
    • Txt based
    • Battery Drain very bad
    • Bad internet in Chc hard to use, no mobile, message reply for minutes on join
  • Things pop up within an hour
    • Pirate Pad
    • Couch apps
    • Wikis
    • WordPress installs
  • Short code 4000 for non-urgent help live by 5pm
    • Volenteers processing the queue
  • All telcos agree to coordinate their social media effort
  • Civil defence didn’t have site ready and refused offers, people decided to do independantly
  • Ushahidi instance setup
    • Google setup people finder app
    • Moved into ec2 cluther
    • hackfest, including added mobile
    • Some other Ushidis, in the end newspaper sites enbedded
  • Council
    • chc council wordpress for info
    • Very slow and bad UI
    • Hit very hard, old information from the previous earthquake
    • staff under extreme pressure
  • Civil Defence
    • Official info only
    • Falls over
    • Caught by DDOS against another govt site
  • Our reliability
    • Never wen tdown
    • contact and reassured some authorities
    • After 24h . 78k page impressions
  • Skype
    • 100+ chatting. limitations
    • IRC used by some but many no common enough
    • Gap for something common. cross platform, easy to use
  • Hashtag
    • twitter to SMS notifications to add stuff to website
  • Maps were a new thing
    • None of the authorities knew them
  • Council and DHB websites did not work on mobile and were not updating
  • Government
    • Govt officers didn’t talk – except NZ Geospacial office
    • Meeting that some people attended
  • Wrap up after 3 weeks
    • Redirected website
    • Anonymous copy of database
  • Pragmatic
    • Used closed source where we had too (eg skype)
    • But easier with OS could quick to modify
    • Closed source people could install webserver, use git, etc. Hard to use contributions
  • Burned Bridges
    • Better jobs with Gov agencies
  • These days
    • Tablets
    • Would use EC2 again
    • phones have low power mode
    • more open street maps


collectd in dynamic environments – Florian Forster

  • Started collectd in 2005
  • Dynamic environments – Number and location of machines change frequently – VM or job management system
  • NOTE: I use collectd so my notes are a little sparse here cause I knew most of it already
  • Collects timeseries data, does one thing well.
  • agent runs on each host, plugins mostly in C for lots of things or exec plug to run random stuff.
  • Read Plugins to get metrics from system metrics, applications, other weird stuff
  • Write plugs – Graphite, RRD, Reimann, MongoDB
  • Virtual machine Metrics
    • libvirt plugin
    • Various metrics, cpu, memory, swap, disk ops/bytes, network
    • GenericJMX plugin – connects to JVM. memory and garbage collection, threads
  • Network plugin
    • sends and receives metric
    • Effecient binary protocol. 50-100 byte UDP multicast/unicast protocol
    • crypto available
    • send, receive, forward packets
  • Aggregation
    • Often more useful for alerting
  • Aggregation plugin
    • Subscribes to metric
    • aggregates and forwards
    • Limitation, no state, eg medium, mean are missing
    • only metrics with one value
    • can be aggregated at any level
    • eg instead of each CPU then total usage of all your CPUS
  • Reimann
    • Lots of filters and functions
    • can aggregate, many otions
  • Bosum
    • Monitoring and alert language
  • Storage
    • Graphite
    • OpenTSDB based on hadoop
    • InfluxDB – understand collectd protocol native (and graphite).
    • Vaultaire ( no collectd integration but… )
  • New Dishboard –

January 13, 2015 2015 – Day 3 – Session 1

CoreOS: an introduction – Brandon Philips

  • Reference to the “Datacenter as a Computer Paper
  • Intro to containers
  • cAdvisor – API of what resources are used by a container
  • Rocket
    • Multiple implementations of container spec , rocket is just one implementation
  • Operating system is able to make less promises to applications
  • Kernel API is really stable
  • Making updates easy
    • Based on ChromeOS
    • Update one partition with OS version. Then flip over to that.
    • Keep another partition/version ready to fail back if needed
    • Safer to update the OS seperated from the app
    • Just around 100MB in size. Kernel, very base OS, systemd
  • etcd
    • Key value store over http (see my notes from yesterday)
    • multiple, leader election etc
    • Individual server less critical since data across multiple hosts
  • Scheduling stuff to servers
    • fleet – very simple, kinda systemd looking
    • fleetctl start foo.service   – sends it off to some machine
    • meso, kubernetes, swam other alternative scedulers
  • Co-ordination
    • locksmith
  • Service discover
    • skydns, discoverd, conf
    • Export location of application to DNS or http API
    • Need proxies to forward request to the right place (for apps not able to query service discovery directly)
  • It is all pretty much a new way of thinking about problems


Why you should consider using btrfs, real COW snapshots and file level incremental server OS upgrades like Google does. – Marc Merlin

  • Worked at netapp, hooked on snapshots, lvm snapshots never worked too well , also lvm partitions not too good
  • Switched laptop to btrfs to 3 years ago
  • Why you should consider btrfs
    • Copy on Write
    • Snapshots
    • cp -reflink=always
    • metadata is redundant and checksummed, data checksummed too
    • btrfs underlying filesystem [for now]
    • RAID 0, 1, 5, 6 built in
    • file compression is also built in
    • online background scrub (partial fsck)
    • block level filesystem diff backups(instead of a slow rsync)
    • convert difectly from ext3 (fails sometimes)
  • Why not use ZFS instead
    • ZFS more mature than ZFS
    • Same features plus more
    • Bad license. Oracle not interested in relicensing. Either hard to do or prfer btrfs
    • Netapp sued sun for infringing patents with ZFS. Might be a factor
    • Hard to ship a project with it due to license condistions
  • Is it safe now?
    • Use new kernels. 3.14.x works okay
    • You have to manually balance sometimes
    • snapshots, raid 0 , raid 1 mostly stable
    • Send/receive mostly works reliably
  • Missing
    • btrfs incomplete, but mostly not needed
    • file encryption not supported yet
    • dedup experimental
  • Who use it
    • openSUSE 13.2 ships with it by default
  • File System recovery
    • Good entry on bfrfs wiki
    • btrfs scrub, run weekly
    • Plan for recovery though, keep backups, not as mature as ext4/ext3 yet, prepare beforehand
    • btrfs-tools are in the Ubuntu initrd
  • Encryption
    • Recommends setup encryption on md raid device if using raid
  • Partitions
    • Not needed anymore
    • Just create storage pools, under them create sub volumes which can be mounted
    • boot: root=/dev/sda1  rootflags=solvol=root
  • Snapshots
    • Works using subvolumes
    • Read only or read-write
    • noatime is strongly recommended
    • Can sneakily fill up your disk “btrfs fi show” tells you real situation. Hard to tell what snapshots to delete to reclaim space
  • Compression
    • Mount option
    • lzo fast, zlib slower but better
    • if change option then files changed from then on use new option
  • Turn off COW for big files with lots of random rights in the middle. eg DBs and virtual disk images
  • Send/receive
    • rsync very slow to scan many files before copy
    • initial copy, then only the diffs. diff is computed instantly
    • backup up ssd to hard drive hourly. very fast
  • You can make metadata of file system at a different raid level than the the data
  • Talk slides here. Lots of command examples 2015 – Day 3 – Keynote

Bob Young

  • Warns that some stories might not be 100% true
  • ”  Liked about Early Linux – Nobody was very nice to each other but everybody was very respectful of the Intel Microprocessor “
  • CEO of Redhat 1992 – 2000
  • Various stories, hard to take notes from
  • One person said they walked out of the Keynote when they heard the quote “it was a complete meritocracy” re the early days of Linux.
  • Others didn’t other parts of the talk. General tone and some statements similar to the one above.
  • “SuSe User Loser” proviked from laughs and a Suse Lizzard being thrown at the speaker
  • Reasons the publishing industry rejects books: 1. no good; 2. market not big enough; 3. They already publish one on the subject.

Wednesday Keynote Speaker - Bob Young

Bob Young

Our Wednesday Keynote speaker is Bob Young, founder and chairman of, co-founder of Red Hat and the Center for Public Domain.

Bob Young is the founder and chairman of, a premiere international marketplace for new digital content on the Internet, with more than 300,000 recently published titles and more than 15,000 new creators from 80 different countries joining each week., founded in 2002, is Young's most recent endeavour. The success of this company has earned Young notable recognition; he was named one of the "Top 50 Agenda-Setters in the Technology Industry in 2006" and was ranked as the fourth "Top Entrepreneur for 2006," both by

In 1993, Young co-founded Red Hat (NYSE: RHT), the open-source software company that gives hardware and software vendors a standard platform on which to certify their technology. Red Hat is a Fortune 500 company and chief rival to Microsoft. His success at Red Hat won him industry accolades, including nomination as one of Business Week's "Top Entrepreneurs" in 1999

Before founding Red Hat, Young spent 20 years at the helm of two computer-leasing companies that he founded. His experiences as a high tech entrepreneur combined with his innate marketing savvy led to Red Hat's success. His book, "Under the Radar", chronicles how Red Hat's open source strategy successfully won wide industry acceptance in a market previously dominated by proprietary binary-only systems. Young has also imparted the lessons learned from his entrepreneurial experiences through his contributions to the books to "You've GOT to Read This Book!" and "Chicken Soup for the Entrepreneur's Soul."

In 2000, Young co-founded the Center for Public Domain, a non-profit foundation created to bolster healthy conversation of intellectual property, patent and copyright law, and the management of the public domain for the common good. Grant recipients included the Electronic Frontier Foundation, the Creative Commons, the Free Software Foundation, and the Future of Music Coalition.

In addition to enjoying fly fishing, Young collects calculators and antique typewriters, a nod to his beginnings as a typewriter salesman and can usually be found sporting a pair of red socks. However, instead of red on his head, Young now tips his orange hat.

The LCA 2015 Auckland Team

Printing, Re-Spinning, and Musical Experimentation

I've been meaning to purchase a new toner cartridge for my Brother HL-2140 laser printer for a short while now but noticed that the price of cartridges are multiples of their cheapest laser printer at 'Officeworks'.

The only problem is that you may need to update your drivers. I wasn't able to find any relevant Debian packages after a quick search online. I converted from what was available of RPM packages online. The existing driver for the Brother HL-1110 prints nothing but blanks at this stage on some version of Linux.

root@system:~/CUPS# alien *.rpm --scripts --to-deb

hl1110cupswrapper_3.0.1-2_i386.deb generated

hl1110lpr_3.0.1-2_i386.deb generated

root@system:~/CUPS# ls

hl1110cupswrapper-3.0.1-1.i386.rpm  hl1110cupswrapper_3.0.1-2_i386.deb  hl1110lpr-3.0.1-1.i386.rpm  hl1110lpr_3.0.1-2_i386.deb

Download my Debian packages from here...

Information on the difference between the 'Sampler' and 'Simpler' on Ableton.

I've been looking at these (online payment systems) for a while now to see what ways there were of efficiently transferring currency across the Internet for various projects I've been working on.

I've been investigating some of these as a means of online distribution of larger content (re-spun Linux distributions and other content). Note, that I do not have the ability to be able to use P2P style technologies because of various limitations at this moment in time..

There's been some work that I've been meaning to upload to GitHub for a while now. Surprised how much it has been streamlined...

Locations for free wallpapers that I've been looking at for re-spun Linux distributions.

These are some of my first attempts at Reaktor Software Synthesisers. It's interesting how much customisability there is within the software actually. I thought I may have to venture into other software (or direct programming) to be able achieve this level of power.

Download my Reaktor Synthesisers from here...

This is where I learnt how to build them. It'll be intresting to see whether I can build anything worth selling/purchasing...

Ever wondered how those URL breakdown systems work and who actually supplies such services? So did I?  Curious to know though whether they can be used for downloads. Think of the difference between a stealthy, a semi-stealthy, and a transparent proxy for an idea of what I mean...

I've been looking at some young House producers/composers of late looking for furher indications into how they actually build up/layer a track. If you've spent enough time looking around then it's clear that the sound of many of these young producers is quite immature. In terms of sounds intermingling it's quite complex but with regards to actual structure it's quite simplistic. Think of the song 'Icarus' from 'Madeon' as a good example (very reminiscent of Daft Punk's 'Around the World', in terms of structure)(it exudes innocence, young, vibrancy, and is 'poppy'. It's not exactly my cup of tea but hey it works right?). It basically has a bunch of clips goign over the top of one another and doesn't quite complement but actually builds up. It has an introduction, has a period in which it sets the stage, the main storyline, without much of an outro, it simply just fades out... Moreover, the main storyline is just like a chorus/choir joining in on a solo. It's not quite as difficult as you think to build something like this.

Terraform Presentation

Here are the slides from my Terrraform presentation at the Sydney Puppet Meetup.



Morning keynote by Eben, that’s going to take a few viewings to understand.

Spent most of the day at the Community Leaders Summit thingy Donna was running, ended up taking notes for both sessions, it took more out of me than I thought it would.

A quiet evening as I still haven’t got my sleep schedule sorted out over here.

Filed under: diary



First day of miniconfs, I spent some of my time at the kernel miniconf and some at the Debian miniconf.

That night the ghosts dinner was on, caught up with a couple of Melbourne friends.

Filed under: diary 2015 – Day 2 – Session 3 – Sysadmin

Alerting Husbandry – Julien Goodwin

  • Obsolete alerts
    • New staff members won’t have context to know was is obsolete and should have been removed (or ignorened)
  • Unactionable alerts – It is managed by another team but thought you’d like to be woken up
  • SLA Alerts – can I do something about that?
  • Bad thresholds ( server with 32 cores had load of 4 , that is not load ), Disk space alerts either too much or not enough margin
  • Thresholds only redo after complete monitoring rebuilds
  • Hair trigger alerts ( once at 51ms not 50ms )
  • Not impacting redundancy ( only one of 8 web servers is down )
  • Spamming alerts, things is down for the 2925379857 time. Even if important you’ve stopped caring
  • Alerts for something nobody cares about, eg test servers
  • Most of earlier items end up in “don’t care” bucket
  • Emails bad, within a few weeks the entire team will have a filter to ignore it.
  • Undocumented alerts – If it is broken, what am I supposed to do about it?
  • Document actions to take in  “playbook”
  • Alert acceptance practice, only oncallers should e accepting alerts
  • Need a way to silence it
  • Production by Fiat



Managing microservices effectively – Daniel Hall

  • Step one – write your own apps
  • keep state outside apps
  • not nanoservices, not milliservices
  • Each should be replaceable, independantly deployable , have a single capability
  • think about depandencies, especially circular
  • Packaging
    • small
    • multiple versions on same machine
    • in dev and prod
    • maybe use docker, have local registry
    • Small performance hit compared to VMs
    • Docker is a little immature
  • Step 3 deployment
    • Fast in and out
    • Minimal human interaction
    • Recovery from failures
    • Less overhead requires less overhead
    • We use Meso and marathon
    • Marathon handles switches from old app to new, task failure and recover
    •  Early on the Hype Cycle
  • Extra Credit Sceduling
    • Chronos within Mesos
    • A bit newish


Corralling logs with ELK – Mark Walkom

  • You don’t want to be your bosses grep
  • Cluster Elastisearch, single master at any point
  • Sizing best to determine with single machine, see how much it can hadle. Keep Java heap under 31GB
  • Lots of plugins and clients
  • APIs return json. ?pretty makes it looks nicer. The ” _cat/* ” api is more command line
  • new node scales, auto balancers and grows automatic
  • Logstash. lots of filters, handles just about any format, easy to setup.
  • Kibana – graphical front end for elastisearch
  • Curator, logstash-forwarder, grokdebugger

FAI — the universal deployment tool – Thomas Lange

  • From power off to applications running
  • It is all about installing software packages
  • Central administration and control
  • no master or golden image
  • can be expanded by hooks
  • plan your installation and FAI installs the plan
  • Boot up diskless client via PXE/tftp
  • creates partitions, file systems, installs, reboots
  • groups hosts by classes, mutiple classes per host etc
  • Classes can be executables, writeing to standard output, can be in shell, pass variables
  • partitioning, can handle LVM, RAID
  • Projected started in 1999
  • Supports debian based distributions including ubuntu
  • Supports bare metal, VM, chroot, LiveCD, Golden image


Documentation made complicated – Eric Burgueno

  • Incomplete, out of date, inconsistent
  • Tools – Word, LibreOffice  -> Sharepoint
  • Sharepoint = lets put this stuff over here so nobody will read it ever again
  • txt , markdown, html. Need to track changes
  • Files can be put in version control.
  • Mediawiki
  • Wiki – uncontrolled proliferation of pages, duplicate pages
  • Why can’t documentation be mixed in with the configuration management
  • Documentation snippits
    • Same everywhere (mostly)
    • Reusable
  • Transclusion in mediawiki (include one page install another)
  • Modern version of mediawiki have parser functions. display different content depending on a condition
  • 2015 – Day 2 – Session 2 – Sysadmin Miniconf

Mass automatic roll out of Linux with Windows as a VM guest – Steven Sykes

  • Was late and missed the start of the talk

etcd: distributed locking and service discovery – Brandon Philips

  • /etc distributed
  • open source, failure tolerant, durable, watchable, exposed via http, runtime configurable
  • API – get/put/del  basics plus some extras
  • Applications
    • Locksmith, distributed locks used when machines update
    • Vulcan http load balancer
  • Leader Election
    • TTL and atomic operations
    • Magical stuff explained faster than I can type it.
    • Just one leader cluster-wide
  • Aims for consistence ahead of raw performance


Linux at the University – Randy Appleton

  • No numbers on how many students use Linux
  • Peninsula Michigan
  • 3 schools
  • Michigan Tech
    • research, 7k students, 200CS Students, Sysadmin Majors in biz school
    • Linux used is Sysadmin courses, one of two main subjects
    • Research use Linux “alot”
    • Inactive LUG
    • Scripting languages. Python, perl etc
  • Northern Michigan
    • 9k students, 140 CS Majors
    • Growing CIS program
    • No Phd Programs
    • Required for sophomore and senior network programming course
    • Optional Linux sysadmin course
    • Inactive LUG
    • Sysadmin course: One teacher, app of the week (Apache, nfs, email ), shell scripting at end, big project at the end
    • No problem picking distributions, No problem picking topics, huge problem with desperate incoming knowledge
    • Kernel hacking. Difficult to do, difficult to teach, best students do great. Hard to teach the others
  • Lake Superior State
    • 2600 students
    • 70 CS Majors
    • One professor teaches Sysadmin and PHP/MySQL
    • No LUG
    • Not a lot of research
  • What is missing
    • Big power Universities
    • High Schools – None really
    • Community college – None really
  • Usage for projects
    • Sometimes, not for video games
  • Usage for infrastructure
    • Web sites, ALL
    • Beowuld Clusters
    • Databases – Mostly
  • Obstacles
    • Not in High Schools
    • Not on laptops, not supported by Uni
    • Need to attract liberal studies students
    • Is Sysadmin a core concept – not academic enough
  • What would make it better
    • Servers but not desktops
    • Not a edu distribution
    • Easier than Eclispe , better than visual studio

Untangling the strings: Scaling Puppet with inotify – Steven McDonald

  • Around 1000 nodes at site
  • Lots of small changes, specific to one node that we want to happen quickly
  • Historically restarting the puppet master after each update
  • Problem is the master gets slow as you scale up
  • 1300 manifests, takes at least a minute to read each startup
  • Puppet internal caching very coarse, per environment basis (and they have only one prod one)
  • Multiple environments doesn’t work well at site
  • Ideas – tell puppet exactly what files have changed with each rollout (via git, inotify). But puppet doesn’t support this
  • I missed the explan of exactly how puppet parses the change. I think it is “import” which is getting removed in the future
  • Inotify seemed to be more portable and simpler
  • Speed up of up to 5 minutes for nodes with complex catalogs, 70 seconds off average agent run
  • implementation doesn’t support the future parser, re-opening the class in a seperate file is not supported
  • Available on github. Doesn’t work with current ruby-inotify ( in current master branch )



Systemd Notes

A few months ago I gave a lecture about systemd for the Linux Users of Victoria. Here are some of my notes reformatted as a blog post:

Scripts in /etc/init.d can still be used, they work the same way as they do under sysvinit for the user. You type the same commands to start and stop daemons.

To get a result similar to changing runlevel use the “systemctl isolate” command. Runlevels were never really supported in Debian (unlike Red Hat where they were used for starting and stopping the X server) so for Debian users there’s no change here.

The command systemctl with no params shows a list of loaded services and highlights failed units.

The command “journalctl -u UNIT-PATTERN” shows journal entries for the unit(s) in question. The pattern uses wildcards not regexs.

The systemd journal includes the stdout and stderr of all daemons. This solves the problem of daemons that don’t log all errors to syslog and leave the sysadmin wondering why they don’t work.

The command “systemctl status UNIT” gives the status and last log entries for the unit in question.

A program can use ioctl(fd, TIOCSTI, …) to push characters into a tty buffer. If the sysadmin runs an untrusted program with the same controlling tty then it can cause the sysadmin shell to run hostile commands. The system call setsid() to create a new terminal session is one solution but managing which daemons can be started with it is difficult. The way that systemd manages start/stop of all daemons solves this. I am glad to be rid of the run_init program we used to use on SE Linux systems to deal with this.

Systemd has a mechanism to ask for passwords for SSL keys and encrypted filesystems etc. There have been problems with that in the past but I think they are all fixed now. While there is some difficulty during development the end result of having one consistent way of managing this will be better than having multiple daemons doing it in different ways.

The commands “systemctl enable” and “systemctl disable” enable/disable daemon start at boot which is easier than the SysVinit alternative of update-rc.d in Debian.

Systemd has built in seat management, which is not more complex than consolekit which it replaces. Consolekit was installed automatically without controversy so I don’t think there should be controversy about systemd replacing consolekit.

Systemd improves performance by parallel start and autofs style fsck.

The command systemd-cgtop shows resource use for cgroups it creates.

The command “systemd-analyze blame” shows what delayed the boot process and

systemd-analyze critical-chain” shows the critical path in boot delays.

Sysremd also has security features such as service private /tmp and restricting service access to directory trees.


For basic use things just work, you don’t need to learn anything new to use systemd.

It provides significant benefits for boot speed and potentially security.

It doesn’t seem more complex than other alternative solutions to the same problems.

January 12, 2015 – Day 2 – Session 1 – Sysadmin Miniconf

Configuration Management – A love Story – Javier Turegano

  • June 2008 – Devs want to deploy fast
  • June 2009 – git -> jenkins -> Puppet master
  • But things got pretty complicated and hard to maintain
  • Remove puppet master, puppet noop, but only happens now and then lots of changes but a couple of errors
  • Now doing manual changes
  • June 2010 – Thngs turned into a mess.
  • June 2011 – Devs want prod-like development
  • Cloud! Tooling! Chef! – each dev have their own environment
  • June 2012 – dev environments for all working in ec2
  • dev no longer prod-like. cloud vs datacentre, puppet vs chef , debian vs centos, etc
  • June 2013 – More into cloud, teams re-arranged
  • Build EC2 images and deploy out of jenkins. Eaither as AMI or as rpm
  • Each team fairly separate, doing thing different ways. Had guilds to share skills and procedures and experience
  • June 2014 – Cloudformation, Ansible used by some groups, random

Healthy Operations – Phil Ingram

  • Acquia – Enterprise Drupal as a service. GovCMS Australian Federal Government. 1/4 are remote
  • Went from working in office to working from home
  • Every week had phone call with boss
  • Talk about thing other than with work, ask home people are going, talk to people.
  • Not sleep, waking up at night, not exercising, quick to anger and negative thinking, inability to concentrate
  • Hadn’t taken more than 1 week off work, let exercise work, hobbies was computer stuff
  • In general being in Ops not as much of an option to take time off. Things stay broke until fix
  • Unable to learn via Osmosis, Timing of handing over between shifts
  • People do not understand that computers are run by people not robots
  • Methods: Turn work off at the end of the day, Rubber Ducking, exercise

Developments in PCP (Performance Co-Pilot) : Nathan Scott

  • See my slides from yesterday for intro to PCP
  • Stuff in last 12 months
    • Included in supported in RHEL 6.6 and RHEL 7
    • Regular stable releases
    • Better out of the box experience
    • Tackling some long-standing problems
  • JSON access – pmwebd , interactive web charts ( Graphite, grafana )
  • zero-install look-inside containers
  • Docker support but written to allow use by others
  • Collectors
    • Lots of new kernel metrics additions
    • New applications from web devs (memcached, DNS, web )
    • DB server additions
    • Python PMDA interfaces
  • Monitor work
    • Reporting tools
    • Web tools, GUIs
  • Also improving ease of setup
  • Getting historical data from sar, iostat

Security options for container implementations – Jay Coles

  • What doesn’t work: rlimits, quotas, blacklisting via ACLs
  • Capabilities: Big list that containers probably shouldn’t have
  • Cgroups – Accounting, Limiting resource usage, tracking of processes, preventing/allowing device access
  • App Armor vs selinux – Use at least one, selinux a little more featured

Kilo Nova deploy recommendations

What would a Nova developer tell a deployer to think about before their first OpenStack install? This was the question I wanted to answer for my OpenStack miniconf talk, and writing this essay seemed like a reasonable way to take the bullet point list of ideas we generated and turn it into something that was a cohesive story. Hopefully this essay is also useful to people who couldn't make the conference talk.

Please understand that none of these are hard rules -- what I seek is for you to consider your options and make informed decisions. Its really up to you how you deploy Nova.

Operating environment

  • Consider what base OS you use for your hypervisor nodes if you're using Linux. I know that many environments have standardized on a given distribution, and that many have a preference for a long term supported release. However, Nova is at its most basic level a way of orchestrating tools packaged by your distribution via APIs. If those underlying tools are buggy, then your Nova experience will suffer as well. Sometimes we can work around known issues in older versions of our dependencies, but often those work-arounds are hard to implement (and therefore likely to be less than perfect) or have performance impacts. There are many examples of the problems you can encounter, but hypervisor kernel panics, and disk image corruption are just two examples. We are trying to work with distributions on ensuring they back port fixes, but the distributions might not be always willing to do that. Sometimes upgrading the base OS on your hypervisor nodes might be a better call.
  • The version of Python you use matters. The OpenStack project only tests with specific versions of Python, and there can be bugs between releases. This is especially true for very old versions of Python (anything older than 2.7) and new versions of Python (Python 3 is not supported for example). Your choice of base OS will affect the versions of Python available, so this is related to the previous point.
  • There are existing configuration management recipes for most configuration management systems. I'd avoid reinventing the wheel here and use the community supported recipes. There are definitely resources available for chef, puppet, juju, ansible and salt. If you're building a very large deployment from scratch consider triple-o as well. Please please please don't fork the community recipes. I know its tempting, but contribute to upstream instead. Invariably upstream will continue developing their stuff, and if you fork you'll spend a lot of effort keeping in sync.
  • Have a good plan for log collection and retention at your intended scale. The hard reality at the moment is that diagnosing Nova often requires that you turn on debug logging, which is very chatty. Whilst we're happy to take bug reports where we've gotten the log level wrong, we haven't had a lot of success at systematically fixing this issue. Your log infrastructure therefore needs to be able to handle the demands of debug logging when its turned on. If you're using central log servers think seriously about how much disks they require. If you're not doing centralized syslog logging, perhaps consider something like logstash.
  • Pay attention to memory usage on your controller nodes. OpenStack python processes can often consume hundreds of megabytes of virtual memory space. If you run many controller services on the same node, make sure you have enough RAM to deal with the number of processes that will, by default, be spawned for the many service endpoints. After a day or so of running a controller node, check in on the VMM used for python processes and make any adjustments needed to your "workers" configuration settings.

  • Estimate your final scale now. Sure, you're building a proof of concept, but these things have a habit of becoming entrenched. If you are planning a deployment that is likely to end up being thousands of nodes, then you are going to need to deploy with cells. This is also possibly true if you're going to have more than one hypervisor or hardware platform in your deployment -- its very common to have a cell per hypervisor type or per hardware platform. Cells is relatively cheap to deploy for your proof of concept, and it helps when that initial deploy grows into a bigger thing. Should you be deploying cells from the beginning? It should be noted however that not all features are currently implemented in cells. We are working on this at the moment though.
  • Consider carefully what SQL database to use. Nova supports many SQL databases via sqlalchemy, but are some are better tested and more widely deployed than others. For example, the Postgres back end is rarely deployed and is less tested. I'd recommend a variant of MySQL for your deployment. Personally I've seen good performance on Percona, but I know that many use the stock MySQL as well. There are known issues at the moment with Galera as well, so show caution there. There is active development happening on the select-for-update problems with Galera at the moment, so that might change by the time you get around to deploying in production. You can read more about our current Galera problems on Jay Pipe's blog .
  • We support read only replicas of the SQL database. Nova supports offloading read only SQL traffic to read only replicas of the main SQL database, but I do no believe this is widely deployed. It might be of interest to you though.
  • Expect a lot of SQL database connections. While Nova has the nova-conductor service to control the number of connections to the database server, other OpenStack services do not, and you will quickly out pace the number of default connections allowed, at least for a MySQL deployment. Actively monitor your SQL database connection counts so you know before you run out. Additionally, there are many places in Nova where a user request will block on a database query, so if your SQL back end isn't keeping up this will affect performance of your entire Nova deployment.
  • There are options with message queues as well. We currently support rabbitmq, zeromq and qpid. However, rabbitmq is the original and by far the most widely deployed. rabbitmq is therefore a reasonable default choice for deployment.

  • Not all hypervisor drivers are created equal. Let's be frank here -- some hypervisor drivers just aren't as actively developed as others. This is especially true for drivers which aren't in the Nova code base -- at least the ones the Nova team manage are updated when we change the internals of Nova. I'm not a hypervisor bigot -- there is a place in the world for many different hypervisor options. However, the start of a Nova deploy might be the right time to consider what hypervisor you want to use. I'd personally recommend drivers in the Nova code base with active development teams and good continuous integration, but ultimately you have to select a driver based on its merits in your situation. I've included some more detailed thoughts on how to evaluate hypervisor drivers later in this post, as I don't want to go off on a big tangent during my nicely formatted bullet list.
  • Remember that the hypervisor state is interesting debugging information. For example with the libvirt hypervisor, the contents on /var/lib/instances is super useful for debugging misbehaving instances. Additionally, all of the existing libvirt tools work, so you can use those to investigate as well. However, I strongly recommend you only change instance state via Nova, and not go directly to the hypervisor.

  • Avoid new deployments of nova-network. nova-network has been on the deprecation path for a very long time now, and we're currently working on the final steps of a migration plan for nova-network users to neutron. If you're a new deployment of Nova and therefore don't yet depend on any of the features of nova-network, I'd start with Neutron from the beginning. This will save you a possible troublesome migration to Neutron later.

Testing and upgrades
  • You need a test lab. For a non-trivial deployment, you need a realistic test environment. Its expected that you test all upgrades before you do them in production, and rollbacks can sometimes be problematic. For example, some database migrations are very hard to roll back, especially if new instances have been created in the time it took you to decide to roll back. Perhaps consider turning off API access (or putting the API into a read only state) while you are validating a production deploy post upgrade, that way you can restore a database snapshot if you need to undo the upgrade. We know this isn't perfect and are working on a better upgrade strategy for information stored in the database, but we will always expect you to test upgrades before deploying them.
  • Test database migrations on a copy of your production database before doing them for real. Another reason to test upgrades before doing them in production is because some database migrations can be very slow. Its hard for the Nova developers to predict which migrations will be slow, but we do try to test for this and minimize the pain. However, aspects of your deployment can affect this in ways we don't expect -- for example if you have large numbers of volumes per instance, then that could result in database tables being larger than we expect. You should always test database migrations in a lab and report any problems you see.
  • Think about your upgrade strategy in general. While we now support having the control infrastructure running a newer release than the services on hypervisor nodes, we only support that for one release (so you could have your control plane running Kilo for example while you are still running Juno on your hypervisors, you couldn't run Icehouse on the hypervisors though). Are you going to upgrade every six months? Or are you going to do it less frequently but step through a series of upgrades in one session? I suspect the latter option is more risky -- if you encounter a bug in a previous release we would need to back port a fix, which is a much slower process than fixing the most recent release. There are also deployments which choose to "continuously deploy" from trunk. This gets the access to features as they're added, but means that the deployments need to have more operational skill and a closer association with the upstream developers. In general continuous deployers are larger public clouds as best as I can tell.

libvirt specific considerations
  • For those intending to run the libvirt hypervisor driver, not all libvirt hypervisors are created equal. libvirt implements pluggable hypervisors, so if you select the Nova libvirt hypervisor driver, you then need to select what hypervisor to use with libvirt as well. It should be noted however that some hypervisors work better than others, with kvm being the most widely deployed.
  • There are two types of storage for instances. There is "instance storage", which is block devices that exist for the life of the instance and are then cleaned up when the instance is destroyed. There is also block storage provided Cinder, which is persistent and arguably easier to manage than instance storage. I won't discuss storage provided by Cinder any further however, because it is outside the scope of this post. Instance storage is provided by a plug in layer in the libvirt hypervisor driver, which presents you with another set of deployment decisions.
  • Shared instance storage is attractive, but it comes at a cost. Shared instance storage is an attractive option, but isn't required for live migration of instances using the libvirt hypervisor. Think about the costs of shared storage though -- for example putting everything on network attached storage is likely to be expensive, especially if most of your instances don't need the facility. There are other options such as Ceph, but the storage interface layer in libvirt is one of the areas of code where we need to improve testing so be wary of bugs before relying on those storage back ends.

Thoughts on how to evaluate hypervisor drivers

As promised, I also have some thoughts on how to evaluate which hypervisor driver is the right choice for you. First off, if your organization has a lot of experience with a particular hypervisor, then there is always value in that. If that is the case, then you should seriously consider running the hypervisor you already have experience with, as long as that hypervisor has a driver for Nova which meets the criteria below.

What's important is to be looking for a driver which works well with Nova, and a good measure of that is how well the driver development team works with the Nova development team. The obvious best case here is where both teams are the same people -- which is true for drivers that are in the Nova code base. I am aware there are drivers that live outside of Nova's code repository, but you need to remember that the interface these drivers plug into isn't a stable or versioned interface. The risk of those drivers being broken by the ongoing development of Nova is very high. Additionally, only a very small number of those "out of tree" drivers contribute to our continuous integration testing. That means that the Nova team also doesn't know when those drivers are broken. The breakages can also be subtle, so if your vendor isn't at the very least doing tempest runs against their out of tree driver before shipping it to you then I'd be very worried.

You should also check out how many bugs are open in LaunchPad for your chosen driver (this assumes the Nova team is aware of the existence of the driver I suppose). Here's an example link to the libvirt driver bugs currently open. As well as total bug count, I'd be looking for bug close activity -- its nice if there is a very small number of bugs filed, but perhaps that's because there aren't many users. It doesn't necessarily mean the team for that driver is super awesome at closing bugs. The easiest way to look into bug close rates (and general code activity) would be to checkout the code for Nova and then look at the log for your chosen driver. For example for the libvirt driver again:

$ git clone
$ cd nova/nova/virt/driver/libvirt
$ git log .

That will give you a report on all the commits ever for that driver. You don't need to read the entire report, but it will give you an idea of what the driver authors have recently been thinking about.

Another good metric is the specification activity for your driver. Specifications are the formal design documents that Nova adopted for the Juno release, and they document all the features that we're currently working on. I write summaries of the current state of Nova specs regularly, which you can see posted at with this being the most recent summary at the time of writing this post. You should also check how much your driver authors interact with the core Nova team. The easiest way to do that is probably to keep an eye on the Nova team meeting minutes, which are posted online.

Finally, the OpenStack project believes strongly in continuous integration testing. It (s/It/Testing) has clear value in the number of bugs it finds in code before our users experience them, and I would be very wary of driver code which isn't continuously integrated with Nova. Thus, you need to ensure that your driver has well maintained continuous integration testing. This is easy for "in tree" drivers, as we do that for all of them. For out of tree drivers, continuous integration testing is done with a thing called "third party CI".

How do you determine if a third party CI system is well maintained? First off, I'd start by determining if a third party CI system actually exists by looking at OpenStack's list of known third party CI systems. If the third party isn't listed on that page, then that's a very big warning sign. Next you can use Joe Gordon's lastcomment tool to see when a given CI system last reported a result:

$ git clone
$ ./ --name "DB Datasets CI"
last 5 comments from 'DB Datasets CI'
[0] 2015-01-07 00:46:33 (1:35:13 old) 'Ignore 'dynamic' addr flag on gateway initialization' 
[1] 2015-01-07 00:37:24 (1:44:22 old) 'Use session with neutronclient' 
[2] 2015-01-07 00:35:33 (1:46:13 old) 'libvirt: Expanded test libvirt driver' 
[3] 2015-01-07 00:29:50 (1:51:56 old) 'ephemeral file names should reflect fs type and mkfs command' 
[4] 2015-01-07 00:15:59 (2:05:47 old) 'Support for ext4 as default filesystem for ephemeral disks' 

You can see here that the most recent run is 1 hour 35 minutes old when I ran this command. That's actually pretty good given that I wrote this while most of America was asleep. If the most recent run is days old, that's another warning sign. If you're left in doubt, then I'd recommend appearing in the OpenStack IRC channels on freenode and asking for advice. OpenStack has a number of requirements for third party CI systems, and I haven't discussed many of them here. There is more detail on what OpenStack considers a "well run CI system" on the OpenStack Infrastructure documentation page.

General operational advice

Finally, I have some general advice for operators of OpenStack. There is an active community of operators who discuss their use of the various OpenStack components at the openstack-operators mailing list, if you're deploying Nova you should consider joining that mailing list. While you're welcome to ask questions about deploying OpenStack at that list, you can also ask questions at the more general OpenStack mailing list if you want to.

There are also many companies now which will offer to operate an OpenStack cloud for you. For some organizations engaging a subject matter expert will be the right decision. Probably the most obvious way to evaluate which of those companies to use is to look at their track record of successful deployments, as well as their overall involvement in the OpenStack community. You need a partner who can advocate for you with the OpenStack developers, as well as keeping an eye on what's happening upstream to ensure it meets your needs.


Thanks for reading so far! I hope this document is useful to someone out there. I'd love to hear your feedback -- are there other things we wished deployers considered before committing to a plan? Am I simply wrong somewhere? Finally, this is the first time that I've posted an essay form of a conference talk instead of just the slide deck, and I'd be interested in if people find this format more useful than a YouTube video post conference. Please drop me a line and let me know if you find this useful!

Tags for this post: openstack nova

Related posts: One week of Nova Kilo specifications; Specs for Kilo; Juno nova mid-cycle meetup summary: nova-network to Neutron migration; Juno Nova PTL Candidacy; Juno nova mid-cycle meetup summary: scheduler; Juno nova mid-cycle meetup summary: ironic

Comment – Day 2 – Keynote by Eben Moglen

Last spoke 10 years ago in Canberra

Things have improved in the last ten years

  • $10s of billions of value have been lost in software patent war
  • But things have been so bad that some help was acquired, so worst laws have been pushed back  a little
  • “Fear of God” in industry was enough to push open Patent pools
  • Judges determined that Patent law was getting pathological, 3 wins in Supreme court
  • Likelihood worst patent laws will be applied against free software devs has decreased
  • “The Nature of the problem has altered because the world has altered”

The Next 10 years

  • Most important Patent system will be China’s
  • Lack of rule of law in China will cause problems in environment of patents
  • Too risky for somebody too try and stop a free software project. We have “our own baseball bat” to spring back at them

The last 10 years

  • Changes in Society more important changes in software
  • 21st century vs 20th century social organisations
    • Less need for hierarchy and secrecy
    • Transparency, Participation, non-hierarchical interaction
  • OS invented that organisation structure
  • Technology we made has taken over the creation of software
  • “Where is BitKeeper now?” – Eben Moglen
  • Even Microsoft reorganises that our way of software making won
  • Long term the organisation structure change everywhere will be more important than just it’s application in Software
  • If there has been good news about politics = “we did it”, bad news = “we tried”

Our common Values

  • “Bridge entire environment between vi and emacs”


  • Without PGP and free software then things could have been worse
  • The world would be a far more despotic place if PGP was driven underground back in 1993. Imagine today’s Net without HTTPS or SSH!
  • “We now live in the world we are afraid of”
  • “What stands between them and us is our inventions”
  • “Freedom itself depends on how we make use of the technologies we are creating.” – Eben Moglen
  • “You can’t trust what you can’t read”
  • Big power in the wrong is committed against the first law of robotics, they what technology to work for it.
  • From guy in twitter – “You can’t trust what you can’t read.” True, but if OpenSSL teaches us anything you can’t necessarily trust what you can
  • Attitudes in under-18s are a lot more positive towards him than those who are older (not just cause he looks like Harry Potter)
  • GNU Project is 30 years old, almost same age is Snowden


  • We can’t control the net but opportunity to prevent others from controlling it
  • Opportunity to prevent failure of freedom
  • Society is changing, demographics under control
  • But 1.6 billion people live in China, America is committed to spying, consumer companies are committed to collecting consumer information
  • Collecting everything is not the way we want the net to work
  • We are playing for keeps now.



Tuesday Keynote Speaker - Professor Eben Moglen

Eben Moglen

Today we have our first Keynote speaker - Professor Eben Moglen, Executive Director of the Software Freedom Law Center and professor of Law and Legal History at Columbia University Law School.

Professor Moglen's presentation is scheduled for 09:00 am Tuesday, 13 January 2015 so don't be late.

Professor Moglen has represented many of the world's leading free software developers. He earned his PhD in History and his law degree at Yale University during what he sometimes calls his “long, dark period” in New Haven.

After law school he clerked for Judge Edward Weinfeld of the United States District Court in New York City and for Justice Thurgood Marshall of the United States Supreme Court. He has taught at Columbia Law School since 1987 and has held visiting appointments at Harvard University, Tel Aviv University and the University of Virginia.

In 2003 he was given the Electronic Frontier Foundation's Pioneer Award for efforts on behalf of freedom in the electronic society.

The LCA 2015 Auckland Team

Floppy drive music

Some time in 2013 I set up a rig to play music with a set of floppy drives. At 2015 in Auckland I gave a brief lightning talk about this, and here is a set of photos and some demo music to accompany.

IMG_2001 IMG_2003 IMG_2014 IMG_2013

The hardware consists of six 3.5″ floppy drives connected to a LeoStick (Arduino) via custom vero board that connects the direction and step pins (18 and 20, respectively) as well as permanently grounding the select pin A (14).

The LeoStick is then connected via USB to a laptop, where the Moppy software (not written by me) is loaded onto the LeoStick and its companion Java software is run on the laptop.

The Moppy software expects MIDI format music as input. For me, I use Rosegarden to create and edit the MIDI files.

Music must be specially arranged for use with the floppies. Floppy drives have a useful range of only one octave, and notes must typically be shortened to be distinguished. As a general rule, halve the length of the note that would otherwise be played. In some cases if a particular part is not loud enough, I double it with a second floppy drive (usually playing an octave above/below rather than in unison, as balance issues are usually fundamental to the particular octave).

Each MIDI channel means a separate floppy drive. In Rosegarden, even though it is possible to play multiple notes simultaneously on a single MIDI channel, this is not supported by Moppy. MIDI channels must be numbered sequentially (this is a drop-down option in Rosegarden) unless my patch is used.

It is possible to arrange music by starting with a ready-made MIDI file obtained from public domain sources such as Mutopia Project or IMSLP. In this case it is vital to reorchestrate the music.

Rosegarden’s “Split by pitch” feature makes it much more efficient to transcribe piano music (or music that contains multiple parts in one MIDI channel) by allowing you to separate the top-most or bottom-most notes into separate channels. Rinse and repeat until you have filled all your floppy MIDI channels.

And here are the recordings:

Please be aware that I had a very limited recording setup, and the above recordings are (a) full of noise, and (b) incredibly soft. Crank that volume up! – Day 1 – Session 3 – Containers

Building a PaaS with Docker, Kubernetes, and Hard Work – Steven Pousty

  • Slides –
  • All about Openshift
  • So why a new Paas?
  • Project Atomic – stripped down RHEL install, everything else as a container. ostree file system, same kernel as RHEL
  • Kubernetes intro
    • Kubernetes Daemon – Routing for services
    • Sceduler etc
  • Openshift
    • Built-in software defined networking – OpenVSwith , HAPRoxy load balancing etc
  • Takeaway
    • PAAS seems to be cool again


Galera with Docker: How Synchronous Replication and Linux Containers Mesh Together – Raghavendra Prabhu

  • I got lost in the talk


Cloud, Containers, and Orchestration Panel -  Katie Miller

  • Steven Pousty , Bran Philips ,
    Tycho Andersen
    Tycho Andersen
    Tycho Andersen
    Tycho Andersen

    Tycho Andersen

  • Standard is Dockers to lose and they might manage it
  • 3-4 years before we should standardise them. Need to experiment first.
  • The kernel API imposes some limits on diversity
  • Lots of other stuff 2015 – Day 1 – Session 2 – Containers

AWS OpsWorks Orchestration War Stories – Andrew Boag

  • Autoscaling too slow since running build-from-scratch every time
  • Communications dependencies
  • Full stack rebuild in 20-40 minutes to use data currently in production
  • A bit longer in a different region
  • Great for load testing
  • If we ere doing again
    • AMI-based better
    • OPSWorks not suitable for all AWS stacks
    • Golden master for flexable
  • Auto-Scaling
    • Not every AMI instance is Good to Go upon provisioning
    • Not a magic bullet, you can’t broadly under-provision
    • needs to be throughly load-tested
  • Tips
    • Dual factor authentication
    • No single person / credentials should be able to delete all cloud-hosted copies of your data
  • Looked at Cloudformation at start, seemed to be more work
  • Fallen out of love with OpsWorks
  • Nice distinction by Andrew Boag: he doesn’t talk about “lock-in” to cloud providers, but about “cost to exit”.   – Quote from Paul


Slim Application Containers from Source – Sven Dowideit

  • Choose a base image and make a local version (so all your stuff uses the same one)
  • I’d pick debian (a little smaller) unless you can make do with busybox or scratch
  • Do I need these files? (check though the Dockerfile) eg remove docs files, manpages, timezones
  • Then build, export, import and it comes all clean with just one layer.
  • If all your images use same base, only on the disk once
  • Use related images with all your tools, related to deployment image but with the extra dev, debug, network tools
  • Version the dev images
  • Minimise to 2 layers
    • look at docker-squash
    • Get rid of all the sourc code from your image, just end up with whats need, not junk hidden in layers
  • Static micro-container nginx
    • Build as container
    • export as tar , reimport
    • It crashes :(
    • Use inotifywait to find what extra files (like shared libraries) it needs
    • Create new tarball with those extra files and “docker import” again
    • Just 21MB instead of 1.4GB with all the build fragments and random system stuff
    • Use docker build as last stage rather than docker import and you can run nginx from docker command line
    • Make 2 tar files, one for each image, one in libs/etc, second is nginx


Containers and PCP (Performance Co-Pilot) -  Nathan Scott

  • Been around for 20+ years, 11 years open source, Not a big mindshare
  • What is PCP?
    • Toolkit, System level analysis, live and historical, Extensible, distributed
    • pmcd daemon on each server, plus for various functions (bit of like collectd model)
    • pmlogger, pmchart, pmie, etc talk (pull or poll) to pmcd to get data
  • With Containers
    • Use –container=  to grab info inside a container/namespace
    • Lots of work still needed. Metrics inside containers limited compared to native OS


The Challenges of Containerizing your Datacenter – Daniel Hall

  • Goals at LIFX
    • Apps all stateless, easy to dockerize
    • Using mesos, zookeeper, marathon, chronos
    • Databases and other stuff outside that cloud
  • Mesos slave launches docker containers
  • Docker Security
    • chroot < Docker < KVM
    • Running untrusted Docket containers are a BAD IDEA
    • Don’t run apps as root inside container
    • Use a recent kernel
    • Run as little as possible in container
    • Single static app if possible
    • Run SELinux on the host
  • Finding things
    • Lots of micoroservices, marathon/mesos moves things all over the place
    • Whole machines going up and down
    • Marathon comes with a tool that pushes it’s state into HAProxy, works fairly well, apps talk to localhost on each machines and haproxy forwards
    • Use custom script for this
  • Collecting Logs
    • Not a good solution
    • can mount /dev/log but don’t restart syslog
    • Mesos collects stdout/stderror , hard to work with and no timestamps
    • Centralized logs
    • rsyslog log to -> haproxy -> contral machine
    • Sometimes needs to queue/drop if things take a little while to start
    • rsyslog -> logstash
    • elasticsearch on mesos
    • nginx tasks running kibana
  • Troubleshooting
    • Similar to service discover problem
    • Easier to get into a container than getting out
    • Find a container in marathon
    • Use docker exec to run a shell, doesn’t work so well on really thin containers
    • So debugging tolls can work from outside, pprof or jsonsole can connect to exposed port/pid of container

Is depression a kind of allergic reaction? | The Guardian

January 11, 2015 2015 – Day 1 – Session 1 – Containers

Clouds, Containers, and Orchestration Miniconf


Cloud Management and ManageIQ – John Mark Walker

  • Who needs management – Needs something to tie it all together
  • New Technology -> Adoption -> Proliferation -> chaos -> Control -> New Technology
  • Many technologies follow this, flies under the radar, becomes a problem to control, management tools created, management tools follow the same pattern
  • Large number of customers using hybrid cloud environment ( 70% )
  • Huge potential complexity, lots of requirements, multiple vendors/systems to interact with
  • ManageIQ
    • Many vendor managed open source products fail – open core, runt products
    • Better way – give more leeway to upstream developers
    • Article about taking it opensource on Took around a year from when decision was made
    • Lots of work to create a good open source project that will grow
    • Release named after Chess Grandmasters
    • Rails App


LXD: The Container-Based Hypervisor That Isn’t -  Tycho Andersen

  • Part of Openstack
  • Based on LXC , container based hypervisor
  • Secure by default: user namespaces, cgroups, Apparmor, etc
  • A daemon that doesn’t hypervisory things
  • A framework for maintaining container based applications
  • It Isn’t
    • No network configuration
    • No storage management – But storage aware
    • Not an application container tool
    • handwavy difference between it and docker, I’m sure it makes sense to some people. Something about running an init/systemd rather than the app directly.
  • Features
    • Snapshoting – eg something that is slow to start, snapshot after just starts and deploy it in that state
    • Injection – add files into the container for app to work on.
    • Migration – designed to go fairly fast with low downtime
  • Image
    • Public and private images
    • can be published
  • Roadmap
    • MVP 0.1 released late January 2015
    • container management only


Rocket and the App Container Spec – Brandon Philips

  • Single binary – rkt – runs everywhere, systemd not required
  • rkt fetch – downloads and discovers images ( can run as non-root user )
  • bash -> rkt -> application
  • upstart -> rkt -> application
  • rkt run
  • multiple processes in container common. Multiple can be run from command line or specified in json file of spec.
  • Steps in launch
    • stage 0 – downloads images, checks it
    • Stage 1 – Exec as root, setup namespaces and cgroups, run systemd container
    • Stage 2 – runs actual app in container. Things like policy to restart the app
    • rocket-gc garbage collects stuff , runs periodicly. no managmanent daemon
  • App Container spec is work in progress
    • images, files, compressed, meta-data, dependencies on other images
    • runtime , restarts processes, run multiple processes, run extra procs under specified conditions
    • metadata server
    • Intended to be built with test suite to verify 2015 is low live #BeHere #lca2015

Registration is downstairs (level 0) at the Owen G Glenn Business School at the University of Auckland.

We look forward to seeing you there, and we even have a coffee cart in the vicinity to keep those caffeine levels up.

Map of Owen G Glenn Building (OGGB)

Look for redshirts around the University - they will be happy to point you in the right direction!

Minions in Red Shirt

Twitter posts: 2015-01-05 to 2015-01-11



Spent the better part of the morning buying shorts that somehow didn’t make the trip with me…

Registered for, looks like I might get two t-shirts if the second batch of corrected printing comes in on time.

Dinner with old friends at an Indian place here. Managed to thank Paul McKenny for librcu.

Filed under: diary



Spent most of the day at the Auckland War memorial/museum. There’s an emphasis on natural history, Maori culture and conflict history. There’s a lot of stuff in the museum and I was running out of puff by the end of it.

Filed under: diary



Flew to Auckland for my fourteenth

Somehow ended up with a ticket that didn’t give me a meal during the flight.

Filed under: diary

January 10, 2015

Conference registration opens 2pm today #BeRegistered

Registration this year will be downstairs (level 0) at the Owen G Glenn Business School at the University of Auckland.

We look forward to seeing you there, and we even have a coffee cart in the vicinity to keep those caffeine levels up.

Map of Owen G Glenn Building (OGGB)

Follow the yellow path starting on Level 1 (see map below) to the rego desk which is on level 0 (downstairs).

The map below shows the level 0 (downstairs) of the OGGB building

Look for redshirts around the University - they will be happy to point you in the right direction!

Minions in Red Shirt

January 09, 2015

Terry 2.0: The ROS armada begins!

It all started with wanting to use a Kinect or other RGBD (Depth sensing) camera to do navigation... Things ended up slowly but surely with moving from a BeagleBone Black and custom nodejs script that I created as the heart to a quad core atom running ROS and many ROS nodes that I created ;)

The main gain to ROS is the nodes that other people have written. If you want to convert RGBD to a simulated laser scan in order to do 2d navigation then that's already available. If you want to make a map and then use it then that code is already there for you. And the visualization for these things. I'm not sure I'd have the time to write from scratch a 3d robot viewer and visualize my cut down 'fake' 2d laser scan data from the Kinect in OpenGL. But with ROS I got the joy of seeing the scan change in real time as Terry sensed me move in front of it.

I now have 3d control of the robot arm happening, including optional sinusoidal encoding of movements. The fun part is that the use of sinusoidal can be enabled or disabled without any code changes. I wrote that part as a JointTrajectory shim. For something to use smoother movement all it has to do is publish to that shim instead of directly to the servo controller itself. The publish and subscribe parts of the IPC that ROS has are very easy to get used to and allow breaking up the functionality into rather small pieces if desired.

The arm is one area that is ROS controlled, but not quite the way I want. It seems that using MoveIt is indicated for arm control but I didn't manage to get that to work as yet. The wizard only produced an arm that would articulate on one joint, so more tinkering is needed in that area. Instead I wrote my own ROS node to control the arm. It's all fairly basic trig to get the gripper at an x,y,z relative to the base of the arm. And an easy carry over to fix the gripper at a horizontal to the base no matter what position the arm is moved to. But in the future the option to MoveIt will be considered, can't hurt to have two codepaths to choose from for arm control.

As part of the refresh I updated the pan unit for the camera platform.Previously I used a solid 1/4 inch shaft with the load taken by a bearing and the gearmotor turning the shaft directly from below it. Unfortunately that setup has many drawbacks; no ability to use a slip ring, no torque multiplication, difficulty using an axle end rotary encoder IC to gain real world position feedback. The updated setup uses a 6 rpm gearmotor offset with a variable motor mount to drive a 24 tooth brass gear. That mates with an 80 tooth gear which is affixed to a hollow 1/2 inch alloy tube. As you can see at the top of the image, I've fed the tilt servo cable directly into the inside of that tube. No slip ring right now, but it is all set to allow the USB cable to slip through to the base and enable continuous rotation of the pan subsystem. So the Kinect becomes a radar style. One interesting aside is that you can no longer manually rotate the pan system because the gearmotor, even unpowered, will stop you. The grub screw will slip before the axle turns.

As shown below, the gearmotor is driven by an Arduino which is itself connected to a SparkFun breakout of the TB6612FNG HBridge IC. This combo is attached using double sided 3M tape to a flat bit of channel. Then the flat bit of channel is bolted to Terry. I've used this style a few times now and quite like it. A single unit and all it's wires can be attached and moved fairly quickly.

At first I thought the Arudino gearmotor control and the Web interface would be a bit outside the bounds of ROS. But there is an API for Arduino which gives the nice publish and subscribe with messages that one would expect on the main ROS platform. A little bit of python glue takes the ttyUSB right out of your view and you are left with a little extension from the main ROS right into the MCU. I feel that my 328 screen multiplexer will be updated to use this ROS message API. Reimplementing packeting and synchronization at the serial port level becomes a little less exciting after a while, and not having to even think about that with ROS is certainly welcome.

Below is the motherboard setup for all this. Unfortunately many of the things I wanted to attach used TTL serial, so I needed a handful of USB to TTL bridges. The IMU uses I2C, so its another matter of shoving a 328 into the mix to publish the ROS messages with the useful information for the rest of the ROS stack on the main machine to use at its will.

The web interface has been resurrected and extended from the old BBB driven Terry. This is the same Bootstrap/jQuery style interface but now using roslibjs to communicate from the browser to Terry. I'm using WebSockets to talk back, which is what I was doing manually from the BBB, but with ROS that is an implementation choice that gets hidden away and you again get a nice API to talk ROS like things such as publishing and subscribing standard and custom messages.

The below javascript code sends an array of 4 floats back to Terry to tell it where you want to have the arm (x,y,z,claw) to be located. The 4th number allows you to open and close the claw in the same command. The wrist is held horizontal to the ground for you. Notice that this message is declared to be a Float32MultiArray which is a standard message type.The msg and topic can be reused, so an update is just a prod to an array and a publish call. You can fairly easily publish these messages from the command line too for brute force testing.

var topic_arm_xyz = new ROSLIB.Topic({

   ros  : ros,

   name : '/arm/xyzc',

   messageType : 'std_msgs/Float32MultiArray'


var msg = new ROSLIB.Message({

  data : [ x,y,z, claw ]


topic_arm_xyz.publish( msg );

The learning curve is a bit sharp for some parts of ROS. Navigation requires many subsystems to be brought up, and at first I had a case that the robot model was visualized 90 degrees out of phase to reality. Most of the stuff is already there, but you need to have a robot base controller that is compatible. It is also a trap for the new players not to have a simple robot model urdf file. Without a model some parts of the system didn't work for me. I'd have liked to have won with the MoveIt control, and will get back to trying to do just that in the future. I think I'll dig around for shoe string examples, something like building a very basic three servo arm with ice cream sticks and $5 servos would make for an excellent example of MoveIt for hobby ROS folk. Who knows, maybe that example will appear here in a future post.


Walked to work.

There’s a construction site across the road from work with a huge rigged jackhammer, the noise is bad but the vibrations through the ground are worse.

Caught up with a friend for dinner in the city.

Bought some plastic containers for the storage cleanup in the study, they look like they’re going to work nicely, I just need to buy a few more.

Filed under: diary

January 08, 2015

Thursday Plenary: What the Foo?

On Thursday, as part of our taking a different approach to keynote sessions for 2015, we’re offering representatives from 3 leaders in open source to give us their “15 minutes of foo” to start our day. We expect this will be interesting, enlightening and entertaining.

Cooper Lees - Facebook

Cooper Lees

Representing Facebook we’ve got Cooper Lees providing an insight into the numerous open source projects they leverage every day, how they are working upstream, and innovations like the Open Compute Projects and FBOSS (their linux switch/router project).

Cooper is a Production Engineer in the Network PE team at Facebook in Menlo Park, CA, originally hailing from Wollongong, NSW, Australia. Network PE help the ‘true’ Network Engineers automate the network, not re-invent the wheel and take on existing Facebook technologies that solve their problem. Outside of being a nerd, Cooper enjoys playing Cricket, AFL (yes, in the Bay area) and Snowboarding in Tahoe as much as possible.

Carol Smith - Google

Carol Smith

Google Summer of code is now 10 years old, and Carol Smith from Google will be provide an update on the project and how this is benefiting the open source community in our part of the world.

Carol Smith is an Open Source Programs Manager at Google. She has managed the Google Summer of Code program for 5 years. She has a degree in Journalism from California State University, Northridge.

Mark McLoughlin - Red Hat

Mark McLoughlin

Then from Red Hat we’ve Mark McLoughlin with some insight on some of the innovation we should see in 2015. Mark will give a whirlwind tour of recent technology shifts - covering everything from virtualization, containers, IaaS, PaaS, SDN, scale-out storage, Big Data, DevOps and more - and how they are all coming together to form the basis of today's agile data center.

Mark McLoughlin is a consulting engineer at Red Hat and has spent over a decade contributing to and leading open source projects like GNOME, Fedora, KVM, qemu, libvirt, oVirt and, of course, OpenStack. Mark is a member of OpenStack’s technical committee and the OpenStack Foundation board of directors. He contributes mostly to Oslo, Nova and TripleO but will happily dive in to any project.

Conference Suggestions

LCA 2015 is next week so it seems like a good time to offer some suggestions for other delegates based on observations of past LCAs. There’s nothing LCA specific about the advice, but everything is based on events that happened at past LCAs.

Don’t Oppose a Lecture

Question time at the end of a lecture isn’t the time to demonstrate that you oppose everything about the lecture. Discussion time between talks at a mini-conf isn’t a time to demonstrate that you oppose the entire mini-conf. If you think a lecture or mini-conf is entirely wrong then you shouldn’t attend.

The conference organisers decide which lectures and mini-confs are worthy of inclusion and the large number of people who attend the conference are signalling their support for the judgement of the conference organisers. The people who attend the lectures and mini-confs in question want to learn about the topics in question and people who object should be silent. If someone gives a lecture about technology which appears to have a flaw then it might be OK to ask one single question about how that issue is resolved, apart from that the lecture hall is for the lecturer to describe their vision.

The worst example of this was between talks at the Haecksen mini-conf last year when an elderly man tried at great length to convince me that everything about feminism is wrong. I’m not sure to what degree the Haecksen mini-conf is supposed to be a feminist event, but I think it’s quite obviously connected to feminism – which is of course was why he wanted to pull that stunt. After he discovered that I was not going to be convinced and that I wasn’t at all interested in the discussion he went to the front of the room to make a sexist joke and left.

Consider Your Share of Conference Resources

I’ve previously written about the length of conference questions [1]. Question time after a lecture is a resource that is shared among all delegates. Consider whether you are asking more questions than the other delegates and whether the questions are adding benefit to other people. If not then send email to the speaker or talk to them after their lecture.

Note that good questions can add significant value to the experience of most delegates. For example when a lecturer appears to be having difficulty in describing their ideas to the audience then good questions can make a real difference, but it takes significant skill to ask such questions.

Dorm Walls Are Thin

LCA is one of many conferences that is typically held at a university with dorm rooms offered for delegates. Dorm rooms tend to have thinner walls than hotel rooms so it’s good to avoid needless noise at night. If one of your devices is going to make sounds at night please check the volume settings before you start it. At one LCA I was startled at about 2AM but the sound of a very loud porn video from a nearby dorm room, the volume was reduced within a few seconds, but it’s difficult to get to sleep quickly after that sort of surprise.

If you set an alarm then try to avoid waking other people. If you set an early alarm and then just get up then other people will get back to sleep, but pressing “snooze” repeatedly for several hours (as has been done in the past) is anti-social. Generally I think that an alarm should be at a low volume unless it is set for less than an hour before the first lecture – in which case waking people in other dorm rooms might be doing them a favor.

Phones in Lectures

Do I need to write about this? Apparently I do because people keep doing it!

Phones can be easily turned to vibrate mode, most people who I’ve observed taking calls in LCA lectures have managed this but it’s worth noting for those who don’t.

There are very few good reasons for actually taking a call when in a lecture. If the hospital calls to tell you that they have found a matching organ donor then it’s a good reason to take the call, but I can’t think of any other good example.

Many LCA delegates do system administration work and get calls at all times of the day and night when servers have problems. But that isn’t an excuse for having a conversation in the middle of the lecture hall while the lecture is in progress (as has been done). If you press the green button on a phone you can then walk out of the lecture hall before talking, it’s expected that mobile phone calls sometimes have signal problems at the start of the call so no-one is going to be particularly surprised if it takes 10 seconds before you say hello.

As an aside, I think that the requirement for not disturbing other people depends on the number of people who are there to be disturbed. In tutorials there are fewer people and the requirements for avoiding phone calls are less strict. In BoFs the requirements are less strict again. But the above is based on behaviour I’ve witnessed in mini-confs and main lectures.


It is the responsibility of people who consume substances to ensure that their actions don’t affect others. For smokers that means smoking far enough away from lecture halls that it’s possible for other delegates to attend the lecture without breathing in smoke. Don’t smoke in the lecture halls or near the doorways.

Also using an e-cigarette is still smoking, don’t do it in a lecture hall.


Unwanted photography can be harassment. I don’t think there’s a need to ask for permission to photograp people who harass others or break the law. But photographing people who break the social agreement as to what should be done in a lecture probably isn’t. At a previous LCA a man wanted to ask so many questions at a keynote lecture that he had a page of written notes (seriously), that was obviously outside the expected range of behaviour – but probably didn’t justify the many people who photographed him.

A Final Note

I don’t think that LCA is in any way different from other conferences in this regard. Also I don’t think that there’s much that conference organisers can or should do about such things.

ARM v8 (64-bit) developer boxes

Looks like things are moving along in the world of 64-bit ARM, systems aimed at early adopting developers are now around. For instance APM have their X-C1 Development Kit Plus which has 8 x 2.4GHz ARMv8 cores, 16GB RAM, 500GB HDD, 1x10gigE, 3x1gigE for ~US$2,500 (or a steep discount if you qualify as a developer). Oh, and it ships with Linux by default of course.

Found via a blog post by Steve McIntyre about bringing up Debian Jessie on ARMv8 (it’ll be a release architecture for it) which has the interesting titbit that (before ARM had their Juno developer boxes):

Then Chen Baozi and the folks running the Tianhe-2 supercomputer project in Guangzhou, China contacted us to offer access to some arm64 hardware

So it looks like (I presume) NUDT are paying it some attention & building/acquiring their own ARMv8 systems.

This item originally posted here:

ARM v8 (64-bit) developer boxes

[life] Day 344: Lego Discovery Centre, shopping in the city

We had a really nice, busy today, much more so than I'd envisaged when we set off in the morning.

Zoe woke up at 2am and ended up in bed with me. I forgot to open her bedroom door when I went to bed, so I have no idea if that was a contributing factor or not. Her room was 26°C, so she may have been too hot. She then proceeded to have a pretty decent sleep in in my bed and not wake up until around 7am.

The plan today had been to check out the Lego Discovery Centre. It's been something I've wanted to take Zoe to for some time now, and I finally got around to booking in for a 45 minute session at 10am.

We made a pretty quick departure after breakfast, and caught the bus in, and arrived with plenty of time up our sleeves. Zoe didn't have a particularly good breakfast, and was hungry, so we hunted around for a croissant nearby.

Zoe was initially apprehensive about me leaving her there (it was a parent-less activity), but once we browsed the store before it started, she quickly became excited.

I went for a bit of a wander through Southbank and ended up in a deck chair by the river watching the world go by for half an hour. It was nice.

I went back to collect Zoe, and found her playing with a little Duplo-style remote controlled car that she'd built, and having a ball. It turned out that there were three different 45 minute sessions, all back to back. They had capacity in the next two sessions, and Zoe was keen, so I figured she could do all three. I just wish I'd remembered to bring a book. I ended up wandering over to the art gallery to amuse myself.

I came back a bit before 12:45pm to pick her up. I managed to stay out of sight for a while and observe her without her seeing me. All the kids were playing happily with a massive amount of Lego. She definitely looked like she had a good time. For a total of $36, it definitely seemed worth it. Apparently they run the sessions every school holidays, and change the theme every time. The >10 year olds were doing full on robotics with Mindstorms, which seemed very cool. I'm really excited that Zoe seems into Lego. I'm looking forward to doing lots of it with her as she gets older.

We grabbed some lunch from a convenience store next door, and then walked over to Southbank. I wanted to do a spot of shopping in the city, and for something different, I thought we could take one of the bicycle taxis over to the city. Zoe thought that was pretty cool, and it was nice to not be the one pedalling her around for a change.

We did a spot of shopping in the Myer Centre, before heading back to the bus and going home.

By the time we got home, it was time for me start dinner. Zoe watched a bit of TV, and then it was bed time. And I'd been thinking we'd be scrounging for something to do by 11:45am.

The Great D8 Chook Raffle

The Drupal Association board approved a new initiative to help get Drupal 8 done.  It's called the D8 Accelerate fund. We also agreed to personally help do fundraising to support the program.  So I'm running a chook raffle.  For those of you who don't know what that is, Wikipedia gives a decent introduction.
The Drupal Association is working with the Drupal 8 branch maintainers to provide Drupal 8 Acceleration Grants. The goal is to fund work that will positively impact the release date. Drupal 8 has had over 2,400 contributors to date, which is more than any release so far. We hope that this initiative will encourage even more people to join the effort to get D8 done.
Please check out our Pozible campaign and make a pledge 

January 07, 2015

[life] Day 343: Yet another doctor visit and The Workshops Rail Museum

I started the day with a 7.5 km run, the longest distance I've managed to run lately. I'm slowly clawing my way back to 10 km.

After Sarah dropped Zoe off, I prepared a take away lunch, and we headed over to the doctor for another round of freezing the wart on her hand. She's getting really good about it now. This is one persistent wart though.

I'd made plans with Mel to go to The Workshops Rail Museum with Matthew and his brother and sister. Matthew had wanted to ride in our car, so after the doctor, I swung by Mel's place to pick him up.

We had an uneventful drive out there, and it was lunchtime by the time we arrived, so we had lunch first.

Matthew's older brother brought a friend with him, so we had five kids in total, in three different age brackets, so it was somewhat challenging keeping them all together and interested. Zoe was used to getting to go where she wanted, when she wanted, so had to learn to compromise a bit.

She was dying to get to the Nipper's Railway section and also the dining car play area and do a heap of role playing, so once we finally made it over there, she was in her element. Matthew played well with her as well.

It turned out to be a great day for going, because it was grey and drizzly outside all day.

Matthew wanted to come back to our place for a bit of a play afterwards, so we drove directly home. Both kids fell asleep on the way home, so to stretch their naps out a bit, I swung by the Valley to clear my PO box.

By the time we got home, there was less than an hour before Mel was going to pick up Matthew, and they mostly just watched a bit of TV. I used the down time to prep dinner.

After Matthew left and we had dinner, we went for a walk around the block to pick up some fruit from the Hawthorne Garage and kill some time before bedtime.

It was a nice, if somewhat tiring, day.

A quick walk to William Farrer's grave

This was a Canberra Bushwalking Club walk lead by John Evans. Not very long, but I would never have found this site without John's leadership, so much appreciated.


Interactive map for this route.

Tags for this post: blog pictures 20150107-william_farrers_grave photo canberra tuggeranong bushwalk historical grave

Related posts: A walk around Mount Stranger; Another lunch time walk; Taylor Trig; Lunchtime geocaching; Urambi Trig; Walk up Tuggeranong Hill



Walked to work.

Managed to finally start the Learning to Learn course! I’ve now done all the week one videos and quizzes, which I’m quite happy with. Unfortunately the first piece of assessment is due half way through LCA which is going to be..difficult.

Filed under: diary



Finally Mitre Ten was open when I was going past, picked up some lumber so I had something to mount the tool rack to.

Caught up with a friend who had spent Christmas and new years down in Melbourne.

Spent most of the night mounting the took rack, it took much longer than it should have. I need to reorganise one of the bookshelves, and start using it as a storage rack I think.

Filed under: diary

January 06, 2015

BlueHackers at LCA 2015 (Auckland NZ)

BlueHackers will have a presence at (this year in Auckland NZ, 12-16 Jan 2015), the awesome John Dalton is organising the BoF (Birds-of-a-Feather) session one evening, and he’ll also have a stash the little BlueHackers stickers that you can put on your laptop to show your support and understanding for mental health. Some stickers may also be available at the LCA registration desk.

Have an awesome time there – unfortunately I can’t make it this year.

Depression Doesn’t Make You Sad All the Time |

Embedded Linux Conference - submit a paper!

There are only a few days left to submit a paper for the Embedded Linux Conference in San Jose in March. This is the first conference with a Drone specific track organised under

Lorenz Meier and myself will both be presenting at the conference, and it will be a great opportunity for technical discussions within the DroneCode community. I'm really looking forward to meeting other members of the ArduPilot and DroneCode community and hearing about their work.

Call for Papers

The CE Workgroup of the Linux Foundation would like to invite you to make a presentation at our upcoming Embedded Linux Conference.  The conference will be held March 23 - 25 in San Jose, California.  The theme for this year is "Drones, Things and Automobiles", and we're excited to discuss some new areas where embedded Linux is really taking

off! (pun intended)

For general information about the conference, See

For information about the call for participation, see

Please note the guidelines on the CFP page.  It's usually pretty obvious when we're reviewing the abstracts, as a program committee, who has read the guidelines and who hasn't.

ELC is the premier vendor-neutral technical conference for embedded Linux developers. The conference is open to the public.


Presentations should be of a technical nature, covering topics related to use of Linux in embedded systems.  Topics related to consumer electronics are particularly encouraged, but any proposals about Linux that are of general relevance to most embedded developers are welcome.

Presentations that are commercial advertisements or sales pitches are not appropriate for this conference.

Especially encouraged this year are talks in the following areas:

  • Linux in Automotive
  • Drones and Robots
  • Linux in the Internet of Things

And we'd also love to hear your proposals in the following topic areas as well:

  • Audio, Video, Streaming Media, and Graphics
  • Security
  • System size, Boot speed, and Real-Time Performance
  • Flash Memory Devices and Filesystems
  • Build Systems, Embedded Distributions, and Development tools
  • Mobile Phones, DVRs, TVs, Cameras, etc.
  • Practical Experiences and War Stories
  • Standards

Most presentation slots will be 50 minutes long, including time for questions.

Tutorials, demos, and Birds-of-a-Feather sessions may also be proposed.

The deadline for submissions is midnight January 9, 2015 PDT.

To repeat, for additional info and details for making a proposal, see

January 05, 2015


Walked to and from work.

Got grumpy at Mitre 10 again for closing earlier than their website says.

Metered internet at home, so can’t get much done at all.

Filed under: diary

Speaker Feature: Florian Forster, Ian Romanick, Jean-Baptiste Kempf

Florian Forster

Florian Forster

collectd in dynamic environments

2:15pm Wednesday 14th January 2015

Florian started his first free software project in 2001 and has been active in the open source community ever since. In 2005 he started the collectd project and is still one of the project maintainers. His interests lie mainly with low-level backends and infrastructure services, though he has contributed to various window managers over the years. In his day job, he is a Site Reliability Engineer at Google.

For more information on Florian and his presentation, see here.

You can follow him as @flocto and don’t forget to mention #lca2015.

Ian Romanick

Ian Romanick

Reducing GLSL Compiler Memory Usage (or Fitting 5kg of Potatoes in a 2kg Bag)

11:35am Thursday 15th January 2015

Ian Romanick is the software architect for Intel's open-source OpenGL driver, and is currently Intel’s representative to the Khronos Board. Since 2001 he has been dedicated to OpenGL on Linux. He has been doing graphics programming for 23 years, having released his first Amiga demo in 1991. Ian holds a Bachelors Degree in Computer Science from Portland State University, and is just about finished with a Masters program there. He previoulsy taught graphics programming in the Visual and Game Programming department at the Art Institute of Portland for 7 years.

For more information on Ian and his presentation, see here.

You can follow him as @IanRomanick and don’t forget to mention #lca2015.

Jean-Baptiste Kempf

Jean-Baptiste Kempf

VideoLAN and VLC

2:15pm Thursday 15th January 2015

Jean-Baptiste Kempf is the president of the VideoLAN non-profit organization and one of the lead developers of the open source VLC media player.

Jean-Baptiste is a 31-year old French engineer and has been part of the VideoLAN community since 2005. Since then, Jean-Baptiste has worked or lead most VideoLAN related projects, including VLC for desktop, the relicensing of libVLC, the ports to mobile operating systems, and various multimedia libraries like libdvdcss or libbluray.

Jean-Baptiste has also been working in various video-related startups, and founded VideoLabs, a company focusing on open source multimedia technologies.Jean-Baptiste is the president of the VideoLAN non-profit organization and one of the lead developers of the open source VLC media player.

For more information on Jean-Baptiste and his presentation, see here.

Useless Gate Memeware

There are plenty of useless gate images out there in the wild. Many of these are licensed restrictively, or it’s unclear what terms apply, making honourable use in memes problematic. To rectify this situation, I hereby offer the following four images under CC0. Enjoy.

Useless Gate 1 (CC0) Useless Gate 2 (CC0) Useless Gate 3 (CC0) Useless Gate 4 (CC0)It seems appropriate to link to a Pirate Party Australia press release here too.

Cyanogenmod with encryption on a Sony Z1 Compact

So, new personal surveillance device (it’s pink! No more BORING black phone!).

Needed to be able to load my own firmware on it and have encryption. It turns out I had to go and do things like this: which is, in fact, repartitioning my phone.

It’s been a while since I’ve had to do math on partitions to get a Linux installed somewhere… but if you don’t change where the filesystem is, you can’t run with encryption.

Basically, if enabling encryption isn’t working, run “adb logcat” on your computer and look for ” E/Cryptfs ( 1890): Orig filesystem overlaps crypto footer region.  Cannot encrypt in place.” if you see that, you’re going to need to boot into recovery and ” adb shell” before finding what block device /data is (check the output of “mount”) and then check the number of blocks it is in /proc/partitions before running mkfs.ext4 on it but with a device size of a few kb less than the device (I think I picked 16kb less. After doing that, everything “just worked”.

how do I deal with OTA updates? Quite easily – copy the zip to the SD card and install from there.

January 04, 2015

[life] Day 338: A funeral and a shave

Zoe slept well. Opening up her bedroom door when I go to bed seems to be helping prevent the heat build up enough for her not to wake up as often.

It was Bryce's funeral today, and Anshu kindly agreed to look after Zoe for me so I could go on my own.

It was a nice service, and the photo slide show that they played at the end was really lovely. They even included the photo of the three of us that we got when we went to Underwater World, which I thought was a nice inclusion.

I briefly dropped into the wake at the pub nearby before heading home.

Zoe had had a nice morning with Anshu, doing some painting and going to the park.

In the afternoon, we all headed over to Bou's for Men at Portside to officially remove my Decembeard. I didn't get around to blogging about it, but I grew a beard for Decembeard to raise money and awareness for bowel cancer, since my cousin is currently battling it. It was a spur of the moment decision on the first of December. I managed to raise almost $500, which I was very happy with.

I had a lovely cutthroat shave, and I'm still trying to get used to my reflection.