Planet Linux Australia
Celebrating Australians & Kiwis in the Linux and Free/Open-Source community...

November 23, 2014

You stay classy, Uber

You may have heard that Uber has been under a bit of fire lately for its desires to hire private investigators to dig up “dirt” on journalists who are critical of Uber. From using users’ ride data for party entertainment, putting the assistance dogs of blind passengers in the trunk, adding a surcharge to reduce the number of dodgy drivers, or even booking rides with competitors and then cancelling, or using the ride to try and convince the driver to change teams, it’s pretty clear that Uber is a pretty good example of how companies are inherently sociopathic.

However, most of those examples are internal stupidities that happened to be made public. It’s a very rare company that doesn’t do all sorts of shady things, on the assumption that the world will never find out about them. Uber goes quite a bit further, though, and is so out-of-touch with the world that it blogs about analysing people’s sexual activity for amusement.

You’ll note that if you follow the above link, it sends you to the Wayback Machine, and not Uber’s own site. That’s because the original page has recently turned into a 404. Why? Probably because someone at Uber realised that bragging about how Uber employees can amuse themselves by perving on your one night stands might not be a great idea. That still leaves the question open of what sort of a corporate culture makes anyone ever think that inspecting user data for amusement would be a good thing, let alone publicising it? It’s horrific.

Thankfully, despite Uber’s fairly transparent attempt at whitewashing (“clearwashing”?), the good ol’ Wayback Machine helps us to remember what really went on. It would be amusing if Uber tried to pressure the Internet Archive to remove their copies of this blog post (don’t bother, Uber; I’ve got a “Save As” button and I’m not afraid to use it).

In any event, I’ve never used Uber (not that I’ve got one-night stands to analyse, anyway), and I’ll certainly not be patronising them in the future. If you’re not keen on companies amusing themselves with your private data, I suggest you might consider doing the same.

November 22, 2014

[life] Day 296: The day of walking errands

We did a rather huge amount of pedestrian travel today.

I had the car booked in for a service, so after Sarah dropped Zoe off, and she'd watched a bit of TV, we drove over to Newstead to drop the car off.

I'd packed Zoe's scooter in the boot, and once we left the car dealership, we headed over to the Teneriffe cross-river ferry, which is currently conveniently depositing passengers at Hawthorne. Even more conveniently, the ferry was waiting for us as we arrived.

I'd booked haircuts for us at 10am, and we comfortably made it to the hairdresser with about 10 minutes to spare.

After that, it was time to head over to Tumble Tastics, which was quite close to the hairdresser's. We ended up getting there about 20 minutes early, but that was fine.

After Tumble Tastics, we headed home for lunch, and the car was ready to be picked up, so after a brief rest, we headed out again.

This time, Zoe said she wanted to walk, rather than ride the scooter, so we headed out on foot, reversing our trip.

We were in no particular hurry, so we stopped for a little play in a park over at Newstead that we'd discovered in the morning, and then picked up the car. It was a very hot day, so it was nice to get out of the heat.

On the way home, I discovered that the Hawthorne Markets were on. I had some paperwork to drop off to Zoe's school, so after I filled that out, we walked over to her school, dropped it off, and then walked back to the Hawthorne Markets.

I bumped into one of my fellow Thermomix Consultants, Katia, and got introduced to one of her friends, who it turns out, was at the very first trial Tumble Tastics class we went to. She also had a daughter named Zoe. So my Zoe knocked around with this Zoe and Katia's kids, and we grabbed some dinner there. It was a nice night out.

I love the feeling of community that I have now. I don't think I've had this feeling of being so well established in a place, within such a walking distance, ever before. I am truly grateful for living in such a wonderful neighbourhood and community.

A GnuCash Tutorial

Tutorial presentation of GnuCash given to the CPA Young Professionals group at Victoria University, 19th November, 2014

November 21, 2014

Craige McWhirter: An Unexpected Journey

Earlier this year I was braced for a hard and personally gruelling year. What I didn't expect however, was that after my return to Sydney that an old friend would reveal how she truly felt about me. It was a brave moment for her but fortunately for us both I'd harboured the same feelings toward her.

How was I to know,

That you would rise,

Like a burning angel in my eyes

As expected, this year has certainly lived up to and exceeded those difficult expectations to be undoubtedly the most challenging year of my life. However I've been fortunate to balance that by now having the most amazing woman by my side.

Fiona's love, support, advice and humour has been an unprecedented experience in my life. I've found a lover and a partner in crime with whom I've formed an indomitable team as we've had each others backs through some rather unbelievable trials.

Which brings me to Paris. We walked to Pont des Arts, the bridge across the Seine and added our padlock at the centre of the bridge, amongst the thousands of others and made a wish.

Then we kissed.

I asked Fiona what she wished for but was politely told it was a secret.

I said I would tell her what I wished for, then dropped to one knee and paused for long enough to read the unmistakeable expression of "What are you doing? Get up you idiot!" written across Fiona's face before I produced an engagement ring and asked Fiona to marry me.

Fiona's Engagement Ring

Fiona said "yes!".

Before too long,

We'll be together and no one will tear us apart

Before too long,

The words will be spoken I know all the action by heart

Earlier in the night I'd slipped an engagement pendant into Fiona's pocket which she discovered and put around my neck before we celebrated with a meal opposite Notre Dame cathedral.

Craige's engagement pendant

I still shake my head in disbelief at how two such independent people have found themselves in a place where they cannot imagine their life without the other. Yet that's where we are.

Our life going forward is going to complicated and challenging, however there will be an awful lot of love and we'll have each other's backs all the way.

Thank you Fiona, for bringing such love and light into my life.

I've found the one I've waited for

All this time I've loved you

And never known your face

All this time I've missed you

And searched this human race

Here is true peace

Here my heart knows calm

Safe in your soul

Bathed in your sighs

Want to stay right here

Until the end of time

Sometimes, dreams do come true.

[life] Day 295: A big long play date

I met Kelley at the first P&C meeting I went to, and she immediately took me under her wing, and later gave me a bit of a tour of the school, and some tips on Prep teachers and whatnot. I then proceeded to run into her nearly every time I went near the school.

She has a daughter, Chloe, starting Prep next year, and an older daughter in Year 3, and she's fairly well entrenched in the school community.

I thought it'd be good for Zoe to get to know Chloe a bit better, so she's one more person she knows at the start of school next year, so we had a play date at her house.

The girls seem to get along well, and Kelley's really nice. We have similar views in a lot of areas, and her husband works in IT security, so I think this could be the beginning of a beautiful friendship.

After lunch, due to the heat, we decided to bike down to the Colmslie Pool together. Kelley has a bike adapter trailer thing that couples a normal kid's bike to the back of hers, with the front wheel slightly elevated.

Zoe made me very proud at the pool, doing a kneeling dive into the water and swimming half the length of the indoor pool. Her swimming continues to progress in leaps and bounds.

We had a good time at the pool, and then biked back to school so Kelley could pick up her other daughter. We just hung out at the pool a bit early for swim class, and then biked home afterwards.

Sarah picked up Zoe, and then I headed out for the second Thermomix cooking class I've had to help out with. This one was a bit more fun for me because we had a great number of consultants on hand to share the workload, and I wasn't on washing up duties this time.

[life] Day 294: Babysitting play date, final Prep introductory day and an afternoon play date

Wednesday was yet another full day. It's no wonder I'm feeling so tired, and have a backlog of blogging.

Mel had asked me if I could look after Matthew and Olivia for a couple of hours in the morning. Matthew and Zoe get along fabulously, and the time worked well, so I was happy to help out.

Zoe seems to be going through a bit of a nightmare phase at the moment. I'm sure the heat isn't helping. Zoe woke up with a nightmare about Smudge dying at 2am. Her room was 27°C at the time. 2am seems to be the nightmare time. I got her resettled within about half an hour. I really think I'm going to have to look into air-conditioning her bedroom sooner rather than later.

So I was a bit of a zombie when Mel dropped the kids off at 9am. Fortunately Matthew and Zoe just went off and played together, and Olivia was happy to just hang out with me. She's such a sweet little 2 and a half year old. She kept calling me "Lucy's Dad" or "Sophie's Dad" or something not quite right. It was very cute.

Mel was going to stay for lunch, and I'd been feeling adventurous, and made some hamburger buns and hamburger patties throughout the morning, with everyone running amok around me.

I improvised a bit on the hamburger buns, using a mix of baker's flour and whole-wheat flour and buckwheat. The result still turned out quite satisfactory.

After lunch, Zoe and I headed over to school for the final Prep introductory afternoon. Zoe wanted to walk today. It was a "best of" day for the fine motor skills activities, and Zoe was rather chuffed to get picked as a leader for the gross motor skills activities.

One of the Prep teachers (the one I hope Zoe gets next year) who had remarked on Zoe's timidity on the first day remarked today about what a different girl she was now.

Walking home, there were a ton of ibis on the football field we walk past, so Zoe had a great time running across the field chasing them all. She's getting a lot better about walking longer distances now.

Eva and Layla came over for a play with Tanya in tow after school, and the girls had a fun afternoon. A massive storm rolled in, and so I went and picked up Anshu from the ferry terminal. Once the storm abated, Tanya left with the girls, and then Sarah arrived to pick up Zoe.

Anshu tagged along with me to the P&C meeting. Not the most fun "date night", but I was glad to have another opportunity to attend a P&C meeting before the end of the school year.

Playing with the network

I'm in the position of needing to improve my internet connectivity, so one of the first steps is to decouple all the things that provide the services I rely upon.



Stage one is to turn my modem into just an ADSL endpoint, removing any DHCP, NAT, and PPPoE termination from the device so that it has a single function.


Fortunately my nb604n ADSL modem has a nice easy-to-follow guide for taking it into bridge mode: http://support.netcommwireless.com/sm/videos/nb604n/nb604n-bridge-mode-setup-guide


Now onto greater things!


Craige McWhirter: Deleting Root Volumes Attached to Non-Existent Instances

Let's say you've got an OpenStack build you're getting ready to go live with. Assume also that you're performing some, ahem, robustness testing to see what breaks and prevent as many surprises as possible prior to going into production. OpenStack controller servers are being rebooted all over the shop and during this background chaos, punters are still trying to launch instances with vary degrees of success.

Once everything has settled down, you may find that some lucky punters have deleted the unsuccessful instances but the volumes have been left behind. This isn't initially obvious from the cinder CLI without cross checking with nova:

$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+--
--------+--------------------------------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | B
ootable |             Attached to              |
+--------------------------------------+-----------+--------------+------+-------------+--
--------+--------------------------------------+
| 3e56985c-541c-4bdd-b437-16b3d96e9932 | in-use    |              |  3   |    block    |
 true   | 6e06aa0f-efa7-4730-86df-b32b47e53316 |
+--------------------------------------+-----------+--------------+------+-------------+--
--------+--------------------------------------+
$ nova show 6e06aa0f-efa7-4730-86df-b32b47e53316
ERROR (CommandError): No server with a name or ID of '6e06aa0f-efa7-4730-86df-b32b47e53316' exists.

It will manifest itself in Horizon like this:

Attached to None

Now trying to delete this volume is going to fail:

$ cinder delete 52aa706df17d-4599-948c-87ae46d945b2
Delete for volume 52aa706d-f17d-4599-948c-87ae46d945b2 failed: Invalid volume:
Volume status must be available or error, but current status is: creating (HTTP 400)
(Request-ID: req-f45671de-ed43-401c-b818-68e2a9e7d6cb)
ERROR: Unable to delete any of the specified volumes.

As will an attempt to detach it from the non-existent instance:

$ nova volume-detach 6e06aa0f-efa7-4730-86df-b32b47e53316 093f32f6-66ea-451b-bba6-7ea8604e02c6
ERROR (CommandError): No server with a name or ID of '6e06aa0f-efa7-4730-86df-b32b47e53316' exists.

and no, force-delete does not work either.

Here's my approach for resolving this problem:

SSH onto your MariaDB server for OpenStack and open MariaDB to the cinder database:

$ mysql cinder

Unset the attachment in the volumes table by repeating the below command for each volume that requires detaching from a non-existent instance:

MariaDB [cinder]> UPDATE volumes SET attach_status='detached', instance_uuid=NULL, \
attach_time=NULL, status="available" WHERE id='3e56985c-541c-4bdd-b437-16b3d96e9932';
Query OK, 1 row affected (0.01 sec)
Rows matched: 1  Changed: 1  Warnings: 0

Back on your OpenStack client workstations you should now be able to delete the offending volumes:

$ cinder delete 3e56985c-541c-4bdd-b437-16b3d96e9932

Happy housekeeping :-)

November 20, 2014

linux.conf.au 2015 Diversity Fund Announcement

AUCKLAND, New Zealand – Friday 21st November 2014 – linux.conf.au 2015 Organisers are proud to announce our funding programme!



InternetNZ Diversity Programme

LCA 2015 and InternetNZ are proud to support diversity. The InternetNZ Diversity Programme is one way we ensure that LCA 2015 continues to be an open and welcoming conference for everyone. Together with InternetNZ this program has been created to assist under-represented delegates who contribute to the Open Source community but, without financial assistance, would not be able to attend LCA 2015.

For more information please see our funding registration page.



About linux.conf.au

linux.conf.au is one of the world's best conferences for free and open source software! The coming linux.conf.au; LCA 2015 will be held at the University of Auckland, New Zealand from Monday 12 January to Saturday 16 January 2015. LCA 2015 will be fun, informal and seriously technical, bringing together Free and Open Source developers, users and community champions from around the world. LCA 2015 is the third time linux.conf.au has been held in New Zealand. The first was in Dunedin in 2006 and the second was in Wellington in 2010.

For more information please visit our website

About Linux Australia

Linux Australia is the peak body for Linux User Groups (LUGs) around Australia, and as such represents approximately 5000 Australian Linux users and developers. Linux Australia facilitates the organisation of this international Free Software conference in a different Australasian city each year.

For more information see: http://www.linux.org.au/

Emperor Penguin Sponsors

LCA 2015 is proud to acknowledge the support of our Emperor Penguin Sponsors, Catalyst IT, HP and IBM, and our diversity sponsor Internet NZ.

For more information about our sponsors click below -

        

Multi-level prefix delegation is not a myth! I've seen it!

Unless you’ve been living under a firewalled rock, you know that IPv6 is coming. There’s also a good chance that you’ve heard that IPv6 doesn’t have NAT. Or, if you pay close attention to the minutiae of IPv6 development, you’ve heard that IPv6 does have NAT, but you don’t have to (and shouldn’t) use it.

So let’s say we’ll skip NAT for IPv6. Fair enough. However, let’s say you have this use case:

  1. A bunch of containers that need Internet access…

  2. That are running in a VM…

  3. On your laptop…

  4. Behind your home router!

For IPv4, you’d just layer on the NAT, right? While SIP and IPsec might have kittens trying to work through three layers of NAT, for most things it’ll Just Work.

In the Grand Future of IPv6, without NAT, how the hell do you make that happen? The answer is “Prefix Delegation”, which allows routers to “delegate” management of a chunk of address space to downstream routers, and allow those downstream routers to, in turn, delegate pieces of that chunk to downstream routers.

In the case of our not-so-hypothetical containers-in-VM-on-laptop-at-home scenario, it would look like this:

  1. My “border router” (a DNS-323 running Debian) asks my ISP for a delegated prefix, using DHCPv6. The ISP delegates a /561. One /64 out of that is allocated to the network directly attached to the internal interface, and the rest goes into “the pool”, as /60 blocks (so I’ve got 15 of them to delegate, if required).

  2. My laptop gets an address on the LAN between itself and the DNS-323 via stateless auto-addressing (“SLAAC”). It also uses DHCPv6 to request one of the /60 blocks from the DNS-323. The laptop puts one /64 from that block as the address space for the “virtual LAN” (actually a Linux bridge) that connects the laptop to all my VMs, and puts the other 15 /64 blocks into a pool for delegation.

  3. The VM that will be running the set of containers under test gets an address on the “all VMs virtual LAN” via SLAAC, and then requests a delegated /64 to use for the “all containers virtual LAN” (another bridge, this one running on the VM itself) that the containers will each connect to themselves.

Now, almost all of this Just Works. The current releases of ISC DHCP support prefix delegation just fine, and a bit of shell script plumbing between the client and server seals the deal – the client needs to rewrite the server’s config file to tell it the netblock from which it can delegate.

Except for one teensy, tiny problem – routing. When the DHCP server delegates a netblock to a particular machine, the routing table needs to get updated so that packets going to that netblock actually get sent to the machine the netblock was delegated to. Without that, traffic destined for the containers (or the VM) won’t actually make it to its destination, and a one-way Internet connection isn’t a whole lot of use.

I cannot understand why this problem hasn’t been tripped over before. It’s absolutely fundamental to the correct operation of the delegation system. Some people advocate running a dynamic routing protocol, but that’s a sledgehammer to crack a nut if ever I saw one.

Actually, I know this problem has been tripped over before, by OpenWrt. Their solution, however, was to use a PHP script to scan logfiles and add routes. Suffice it to say, that wasn’t an option I was keen on exploring.

Instead, I decided to patch ISC DHCP so that the server can run an external script to add the necessary routes, and perhaps modify firewall rules – and also to reverse the process when the delegation is released (or expired). If anyone else wants to play around with it, I’ve put it up on Github. I don’t make any promises that it’s the right way to do it, necessarily, but it works, and the script I’ve added in contrib/prefix-delegation-routing.rb shows how it can be used to good effect. By the way, if anyone knows how pull requests work over at ISC, drop me a line. From the look of their website, they don’t appear to accept (or at least encourage) external contributions.

So, that’s one small patch for DHCP, one giant leap for my home network.

  1. The standard recommendation is for ISPs to delegate each end-user customer a /48 (giving 65,536 /64 networks); my ISP is being a little conservative in “only” giving me 256 /64s. It works fine for my purposes, but if you’re an ISP getting set for deploying IPv6, make life easy on your customers and give them a /48.

A benefit of running an alternate init in Debian Jessie

If you’re someone who doesn’t like Debian’s policy of automatically starting on install (or its heinous cousin, the RUN or ENABLE variable in /etc/default/<service>), then running an init system other than systemd should work out nicely.

November 19, 2014

DrupalSouth - Call for sessions open!! (closes 30 Nov 2014)

DrupalSouth is the biggest Drupal gathering in the Antipodes.

We'll be at the Melbourne Convention and Exhibition Centre over three days in early March 2015. March 5-7 to be exact.

Find out more at the website

https://melbourne2015.drupal.org.au/

The call for sessions is open, and we're trying hard to get the word out wide and far, to whisper in new ears, and encourage people of all sorts to share their ideas for sessions so we can create a truly wonderful, inspiring, engaging and fun program for this conference!

For those who may not know, Drupal is an open source content management system. It's used by people and organisations all around the world, for all sorts of web sites. It's also being used as back end application framework for mobile apps! It's amazing what Drupal can do.

Drupal events are the heart and soul of the community that makes Drupal. Bringing people together drives the project forward, and forges friendships.

But we're also part of the wider web. So we want to hear from all sorts of web specialists, not just Drupalists.

Please, submit a session, or simply help us spread the word. The deadline is looming and won't be extended. Get that proposal in by 30 November 2014. https://melbourne2015.drupal.org.au/program/session-submission

Speaker Feature: Andrew McDonnell, Jim Cheetham

Andrew McDonnell

Andrew McDonnell

Reverse engineering embedded software using Radare2

1:20pm Thursday 15th January 2015

Andrew McDonnell is a professional software engineer with two decades experience, having spent many years before that hacking code after receiving a Commodore 64 for Christmas at age 12. He has significant experience programming in C++, Java and Python and a multitude of scripting languages. Outside of family and work he sometimes has time to play with his collection of 8-bit and PC/XT-vintage computers; computing and electronics has always been his passion. He intermittently maintains a blog at http://blog.oldcomputerjunk.net sometimes posting how he solved a problem in the hope it may be useful to someone else.

For more information on Andrew and his presentation, see here. You can follow him as @pastcompute and don’t forget to mention #lca2015.



Jim Cheetham

Jim Cheetham

OneRNG - An Open and Verifiable hardware random number generator

1:20pm Thursday 15th January 2015

Jim works in Information Security, and has a long background in Unix/Linux and Open Source/Free software systems.

For more information on Jim and his presentation, see here. You can follow him as @onerng and don’t forget to mention #lca2015.

Unquestionably bad

Question 5:

Consider the following 6 data structures:

  • Stack
  • Queue
  • Hash table
  • Doubly-linked list
  • Binary search tree
  • Directed acyclic graph

Using these as the subject matter, construct 6 really good puns.

 

Answers:

After receiving a range of questions from different sources, I was unsure which to answer first — I was stack as to where to begin. And so because this was the last question that I received, it became the first that I answered.

Don’t get me wrong — I did appreciate the question. The capacity of my gratitude is, theoretically, unbounded. Thanqueue.

We have a cuckoo aviary. I keyp a record of each birth in a hatch table.

I noticed that I was leaning to one side. I spoke to a physician about it — he told me I was overweight because I was eating too much bread. My list, it seems, is linked to my dough-belly.

On a school trip to a pickle factory, my daughter went missing. I was able to climb the brinery search tree and spot her, though it took longer than I had hoped due to my poor balance.

While out walking, I deflected a cyclist’s gaffe, knocking him aside as he rode the wrong way down a one-way street. I looked down my nose at him and gave a topological snort to help him on his way.

 


The reader may decide whether the answers satisfy the requirements of the question.

November 18, 2014

Speaker Feature: Katie McLaughlin, Andrew Bartlett

Katie McLaughlin

Katie McLaughlin

Before All Else, Be Graphed

3:40pm Wednesday 14th January 2015

Katie is a part of the Engineering team at Anchor Systems, working to improve *all* the things. She has a history of enterprise development and Windows system administration, but has been successfully converted to the ways of the penguin in recent years.

When she's not changing the world, she enjoys making tapestries, cooking, and yelling at JavaScript and it's attempts at global variables.

For more information on Katie and her presentation, see here. You can follow her as @glasnt and don’t forget to mention #lca2015.



Andrew Bartlett

Andrew Bartlett

Pushing users into the pit of success - stories from the Samba 3 -> Samba 4 transition

3:40pm Thursday 15th January 2015

Andrew Bartlett is a Samba Developer currently employed by Catalyst in Wellington, NZ. Andrew has been developing Samba since 2001, and has had a strong focus on the Active Directory DC project for the past decade or so. He is passionate about authentication systems and making Samba a great, interoperable alternative to the dominant implementation from Microsoft.

For more information on Andrew and his presentation, see here.

[life] Day 293: Kindergarten, Property Occupations Act Roadshow

Zoe woke up at some point in the night. I have a vague recollection of a conversation with her, and lacking the willpower to get out of bed to put her back to bed in her own bed. The next thing it was 5:30am and she was sleeping sideways in bed with me.

Despite all that, I felt more rested this morning, which was good. We managed to get going quite early as well, without really trying. I had to be out at the Sleeman Sports Complex at 9am for a roadshow by the REIQ about the new Property Occupations Act, which kicks in on December 1 to replace the current Property Agents and Motor Dealers Act.

It also rained this morning, which doubly made it necessary to go to Kindergarten by car. We were actually running so early that we got there before opening time, which I've only managed to do a few times all year.

I ended up getting to the Sleeman Sports Complex about 15 minutes early. It was fun playing "spot the real estate agent's car".

I didn't learn anything earthshattering in the briefing, but it was useful to get fully up to speed on the new legislation. I just hope that being half way through a course that has covered the old legislation isn't going to be a problem.

I got home from that with enough time to just chill out for a bit (I ended up doing a bit of tinkering) before it was time to pick up Zoe. The weather was still a bit questionable, so I picked her up in the car.

Zoe wanted to watch Megan's tennis lesson again, and I had to be at home for a 3pm video chat, so I left her with Jason and popped home.

After my video chat, I went around to Jason's and helped with a bit of painting before heading home to start on dinner.

I had enough for Jason, Megan and Megan's little sister, so they came over for dinner as well.

I got Zoe down to bed at the normal time, but her bedroom is ridiculously hot. I'm not terribly confident I won't get another uninterrupted night's sleep.

One week with the Nexus 5

My ageing Motorola Milestone finally received a kick to the bucket last week when my shiny new Nexus 5 phone arrived.

Though fantastic by 2009 standards, the Milestone could only officially run Android 2.2, and 2.3 with the help of an unofficial CyanogenMod port. Having been end-of-lifed for some time now, and barely being able to render a complex web page without running out of memory, it was time for me to move on.

I was adamant that I would only buy a Nexus phone. Vendors that ship OEM customisations to the Android image are the spawn of the devil, and I wasn’t interested in buying a device that would be abandoned after the next model came out. After all, I’m not a gadget person. This is a big deal for me, and I hope this phone lasts me four years, just like my Milestone did.

Can I just say how fantastic the hardware is. The case is much more aesthetically pleasing than most of the Android phones I’ve had the (dis)pleasure of trying out, the screen is beautiful, and the software keyboard is smooth, accurate, and responsive.

On the screen. I think five inches is the maximum size I can cope with. I must say, being a person with small hands, I am not a large screen person. I can only just reach the opposite X axis with my thumb, and I need to reposition my hand (or use a second hand) to reach the opposite X and Y points. So yes, that’s why I didn’t get a Nexus 6.

On the software, I am thoroughly impressed by Android 4.4. Thoroughly. Google have done just about everything right. Nearly anything bad I have ever said about Android in the past either doesn’t apply to Android 4.4, or only applies to customised OEM builds.

Everything I would have wanted to root my phone to do previously is totally unnecessary.

Out of the box, FLAC audio and IPsec Xauth VPNs (main mode only, not aggressive mode) are supported. Just by installing an app, I can get my strongSwan IKEv2 VPN working.

Interestingly enough, this phone constantly bombards me with security warnings as a result of the fact that I have installed my own certificate authorities. I think this is an interesting development, and is probably a proactive stance against the possibilities that ISPs and/or governments may encourage you to allow them to perform SSL man-in-the-middle attacks on your connection in future for tracking and advertising purposes.

Hopefully warnings appearing on users’ phones worded such as “your network may be monitored” is enough to scare off those who may have such evil intentions.

The phone is amazingly responsive. Not only that, it multitasks with ease, and the user interface is smooth.

One minor criticism is that Google Maps appears to be capped at around 15 frames per second. This is odd, as similar apps such as Google Earth run at a much more pleasing framerate.

It is probably an unfair comparison, as the Nexus 5 is so much higher specced, but overall I am finding the device much faster and more responsive (and therefore I’m more likely to grab it and use it for quick tasks) than my iPhone 4S.

Ever since the release of iOS 7, my iPhone has been frustratingly slow and unstable. Sadly, apps crashing due to low memory conditions are an almost daily occurrence.

It is unclear to me whether this is a deliberate decision by Apple in order to make their later model iPhones look better, but I find it fascinating that I find my Nexus 5 being more pleasurable to use than my iPhone 4S. Something I would not have thought possible a fortnight ago.

I’m so impressed by Android 4.4 that I’m almost dreading the impending 5.0 upgrade in the fear that Google will “do an iOS 7″ — i.e. make the device significantly less useful by making it slower and less stable.

November 17, 2014

Speaker Feature: Marc Merlin, Jussi Pakkanen

Marc Merlin

Marc Merlin

Why you should consider using btrfs, real COW snapshots and file level incremental server OS upgrades like Google does

11:35am Wednesday 14th January 2015

Marc has been using linux since 0.99pl15f (slackware 1.1.2, 1994), both as a sysadmin and userland contributor. He has worked for various tech companies in the Silicon Valley, including Network Appliance, SGI, VA Linux, Sourceforge.net, and now Google since 2002, both a server sysadmin and software engineer.

He has done hacking in various areas like mail with exim, mailman, SpamAssassin and SA-Exim, as well as maintained various linux distributions at Google and elsewhere, and given talks about some of those projects, and others at linux conferences since 2001 (LCA, OLS, Linuxcon, Usenix/LISA).

For more information on Marc's presentation, see here.



Jussi Pakkanen

Jussi Pakkanen

Making build systems not suck

2:15pm Thursday 15th January 2015

Jussi got his doctoral degree in computer science in 2006. Since then he has worked in various problem fields including mail sorting. He is currently employed by Canonical where he has worked on various parts of Ubuntu desktop and phone. In his free time he dabbles with drawing, creating computer games, photography and whatever else might catch his fancy.

For more information on Jussi and his presentation, see here. You can follow him as @jpakkane and don’t forget to mention #lca2015.

[life] Day 292: Kindergarten, return from Sydney, groceries and general malaise

I felt pretty exhausted this morning. In fact, I felt exhausted before I flew to Sydney for the weekend. It hasn't gotten any better.

I managed to get on an earlier (by an hour) flight back, which gave me a comfortable amount of time to unpack, put away the laundry and generally tidy up before picking up Zoe from Kindergarten.

After I picked her up, we popped over to the supermarket to do some grocery shopping and escape the heat, before heading home.

I had a crack at making sushi for dinner tonight. It turned out so-so. I'll tweak it a bit more next time.

Zoe seemed pretty worn out by bedtime too, and went to bed easily. I'm looking forward to a long night's sleep.

November 16, 2014

Speaker Feature: Paul Foxworthy, Keith Packard

Paul Foxworthy

Paul Foxworthy

Auprefs.info – A vain attempt to rescue the Australian democracy with a few hundred lines of Java Script

11:35am Wednesday 14th January 2015

Paul is an open source developer and trainer. He is a committer to the Apache OFBiz project and also currently serves as a director of Open Source Industry Australia. He is very pleased at this, his eighth LCA, to have finally done something just possibly cool enough to talk about.

Away from work, Paul tries to get away to his bush block in eastern Victoria, and to teach agile techniques to Ilke the German Shepherd.

For more information on Paul's presentation, see here. You can follow him as @ConcreteGannet and don’t forget to mention #lca2015.



Keith Packard

Keith Packard

Putting the Polish on Glamor

10:40am Wednesday 14th January 2015

Keith Packard has been developing open source software since 1986, focusing on the X Window System since 1987, designing and implementing large parts of the current implementation. He is currently a Principal Engineer with Intel's Open Source Technology Center. Keith received a Usenix Lifetime Achievement award in 1999, an O'Reilly Open Source award in 2011, sits on the X.org foundation board and is a member of the Debian Technical Committee.

For more information on Keith and his presentations, see here.

Twitter posts: 2014-11-10 to 2014-11-16

Fast Food Nation







ISBN: 9780547750330

LibraryThing

I don't read a lot of non-fiction, but I decided to finally read this book having had it sit on the shelf for a few years. I'm glad I read it, but as someone who regularly eats in the US I am not sure if I should be glad or freaked out. The book is an interesting study in how industrialization without proper quality controls can have some pretty terrible side effects. I'm glad to live in a jurisdiction where we actively test for food quality and safety.



The book is a good read, and I'd recommend it to people without weak stomaches.



Tags for this post: book eric_schlosser food quality meat fast industrialized

Related posts: Dinner; Dishwasher Trout; Yum; 14 November 2003; Food recommendation; Generally poor audio quality on pod casts?
Comment Recommend a book

Salt and Pepper Squid with Fresh Greens

A few days ago I told Andrew Wafaa I’d write up some notes for him and publish them here. I became hungry contemplating this work, so decided cooking was the first order of business:

Salt and Pepper Squid with Fresh Greens

It turned out reasonably well for a first attempt. Could’ve been crispier, and it was quite salty, but the pepper and chilli definitely worked (I’m pretty sure the chilli was dried bhut jolokia I harvested last summer). But this isn’t a post about food, it’s about some software I’ve packaged for managing Ceph clusters on openSUSE and SUSE Linux Enterprise Server.

Specifically, this post is about Calamari, which was originally delivered as a proprietary dashboard as part of Inktank Ceph Enterprise, but has since been open sourced. It’s a Django app, split into a backend REST API and a frontend GUI implemented in terms of that backend. The upstream build process uses Vagrant, and is fine for development environments, but (TL;DR) doesn’t work for building more generic distro packages inside OBS. So I’ve got a separate branch that unpicks the build a little bit, makes sure Calamari is installed to FHS paths instead of /opt/calamari, and relies on regular packages for all its dependencies rather than packing everything into a Python virtualenv. I posted some more details about this to the Calamari mailing list.

Getting Calamari running on openSUSE is pretty straightforward, assuming you’ve already got a Ceph cluster configured. In addition to your Ceph nodes you will need one more host (which can be a VM, if you like), on which Calamari will be installed. Let’s call that the admin node.

First, on every node (i.e. all Ceph nodes and your admin node), add the systemsmanagement:calamari repo (replace openSUSE_13.2 to match your actual distro):

# zypper ar -f http://download.opensuse.org/repositories/systemsmanagement:/calamari/openSUSE_13.2/systemsmanagement:calamari.repo

Next, on your admin node, install and initialize Calamari. The calamari-ctl command will prompt you to create an administrative user, which you will use later to log in to Calamari.

# zypper in calamari-clients
# calamari-ctl initialize

Third, on each of your Ceph nodes, install, configure and start salt-minion (replace CALAMARI-SERVER with the hostname/FQDN of your admin node):

# zypper in salt-minion
# echo "master: CALAMARI-SERVER" > /etc/salt/minion.d/calamari.conf
# systemctl enable salt-minion
# systemctl start salt-minion

Now log in to Calamari in your web browser (go to http://CALAMARI-SERVER/). Calamari will tell you your Ceph hosts are requesting they be managed by Calamari. Click the “Add” button to allow this.

calamari-authorize-hosts calamari-authorize-hosts-wait

Once that’s complete, click the “Dashboard” link at the top to view the cluster status. You should see something like this:

calamari-status

And you’re done. Go explore. You might like to put some load on your cluster and see what the performance graphs do.

Concerning ceph-deploy

The instructions above have you manually installing and configuring salt-minion on each node. This isn’t too much of a pain, but is even easier with ceph-deploy which lets you do the whole lot with one command:

ceph-deploy calamari connect --master <calamari-fqdn> <node1> [<node2> ...]

Unfortunately, at the time of writing, we don’t have a version of ceph-deploy on OBS which supports the calamari connect command on openSUSE or SLES. I do have a SUSE-specific patch for ceph-deploy to fix this (feel free to use this if you like), but rather than tacking that onto our build of ceph-deploy I’d rather push something more sensible upstream, given the patch as written would break support for other distros.

Distros systemsmanagement:calamari Builds Against

The systemsmanagement:calamari project presently builds everything for openSUSE 13.1, 13.2, Tumbleweed and Factory. You should be able to use the packages supplied to run a Calamari server on any of these distros.

Additionally, I’m building salt (which is how the Ceph nodes talk to Calamari) and diamond (the metrics collector) for SLE 11 SP3 and SLE 12. This means you should be able to use these packages to connect Calamari running on openSUSE to a Ceph cluster running on SLES, should you so choose. If you try that and hit any missing Python dependencies, you’ll need to get these from devel:languages:python.

Disconnecting a Ceph Cluster from Calamari

To completely disconnect a Ceph cluster from Calamari, first, on each Ceph node, stop salt and diamond:

# systemctl disable salt-minion
# systemctl stop salt-minion
# systemctl disable diamond
# systemctl stop diamond

Then, make the Calamari server forget the salt keys, ceph nodes and ceph cluster. You need to use the backend REST API for this. Visit each of /api/v2/key, /api/v2/server and /api/v2/cluster in your browser. Look at the list of resources, and for each item to be deleted, construct the URL for that and click “Delete”. John Spray also mentioned this on the mailing list, and helpfully included a couple of screenshots.

Multiple Cluster Kinks

When doing development or testing, you might find yourself destroying and recreating clusters on the same set of Ceph nodes. If you keep your existing Calamari instance running through this, it’ll still remember the old cluster, but will also be aware of the new cluster. You may then see errors about the cluster state being stale. This is because the Calamari backend supports multiple clusters, but the frontend doesn’t (this is planned for version 1.3), and the old cluster obviously isn’t providing updates any more, as it no longer exists. To cope with this, on the Calamari server, run:

# calamari-ctl clear --yes-i-am-sure
# calamari-ctl initialize

This will make Calamari forget all the old clusters and hosts it knows about, but will not clear out the salt minion keys from the salt master. This is fine if you’re reusing the same nodes for your new cluster.

Sessions to Attend at SUSECon

SUSECon starts tomorrow (or the day after, depending on what timezone you’re in). It would be the height of negligence for me to not mention the Ceph related sessions several of my esteemed colleagues are running there:

  • FUT7537 – SUSE Storage – Software Defined Storage Introduction and Roadmap: Getting your tentacles around data growth
  • HO8025 – SUSE Storage / Ceph hands-on session
  • TUT8103 – SUSE Storage: Sizing and Performance
  • TUT6117 – Quick-and-Easy Deployment of a Ceph Storage Cluster with SLES – With a look at SUSE Studio, Manager and Build Service
  • OFOR7540 – Software Defined Storage / Ceph Round Table
  • FUT8701 – The Big Picture: How the SUSE Management, Cloud and Storage Products Empower Your Linux Infrastructure
  • CAS7994 – Ceph distributed storage for the cloud, an update of enterprise use-cases at BMW

Update: for those who were hoping for an actual food recipe, please see this discussion.

November 15, 2014

Writing eBooks For Profit

Over the years it's been clear that I've had a propensity for writing. What hasn't been so clear was how to monetise this. Recent research has indicated that if you're a writer it isn't as difficult as you think. If you work with standard word processors and office suites then it becomes clear that it's possible basically to just type things up, export to PDF, and then publish this.


- a good example of this are the 'Building a Cloud Computing Service', 'Convergence Effect', and 'Cloud and Internet Security' (has been cleared by Australian Intelligence Services for sensitive material so it's not a problem if you're curious) reports which are now available via Amazon and Google Play Book stores for 5 USD each (pretty decent content and research for the price to be honest. Will be curious to see how this experiment goes...)

For those who are curious here are some interesting notes:
- there are some plugins and standalone applications which will allow for this but at the end of the day you need to be able to run your book through the automated checkers to be able to get anything actually posted on to the online store

- another option could be paying someone to manually convert your chosen file. The problem is that you never know the quality of the work that you're going to get so I suggest going on your own
- most if not all stores will take a cut of what you sell
https://kdp.amazon.com/

https://play.google.com/books/publish/

- though there are other options out there if you want to sell in a different way
http://www.cnet.com/how-to/how-to-self-publish-an-ebook/

- some sites will ask for ISBN details while others will supply them for you for free

- depending on your status you may need to sign up to have a Tax File Number in the United States. There are often taxation agreements with more developed countries though
http://www.irs.gov/Individuals/International-Taxpayers/Taxpayer-Identification-Numbers-%28TIN%29

- many book stores will require you to use specific file formats or applications
- note that there are are many options/programs out there that will let you preview, manage, and convert your eBooks

November 14, 2014

OpenRadio Part 2 – Prototype Works!

Since the first post on the OpenRadio project Mark has been moving ahead and leaps and bounds. In just a few late nights work he has assembled and tested the radio, managed to receive off air signals, and even tested the PSK31 transmitter! Fine business Mark.

Mark writes:

Hooked it up to a real antenna tonight:

That’s me decoding actual 20m PSK31 signals!

Signal path is:

Antenna —RF—-> OpenRadio SDR —-IQ–>|Laptop|—-IQ—> Spectravue

(IQ Demod) —SSB—> fldigi

Mark managed to build the radio in 1-2 hours, including taking plenty of photos to document assembly and aid others. Now not everyone will have Mark’s radio assembly skills. However even allowing for a learning curve and a few coffee breaks we are on track for a one-day (say 6 hour) mini-conf assembly time.

A working prototype verifies the hardware design, so we are now getting ready to re-spin the PCB and start putting the kits together.

In other news Edwin from Dragino has added the OpenRadio kit to his store. We estimate the kits will be available for shipping in December. Kim, Mark and I, are still deciding if we will bring a bunch of kits to LCA, or have delegates pre-order them from Dragino. More on that shortly.

Links

OpenRadio Wiki

LCA 2015 Penguin Dinner - The Theme

We're not sure if you've noticed but we all tend to have more than just a trifling interest in things technological and futuristic and, well... How do we put this? Some people have even felt moved to describe us as geeks. Can you believe it? Us?!

We thought for the 2015 Penguin Dinner maybe we should put that myth to rest, finally, and show that we do, indeed, conform to the norm. We're going to do something more traditional, more what you'd expect of a respectable, conservative "black tie dinner".

So it is with some seriousness and gravity, Lords, Ladies and Gentleman, that we announce, for the first time the LCA 2015 Penguin Dinner will have a theme:

Steampunk!

We would love it if you would dress for the occasion. It adds a certain air to the evening, however it is not mandatory (we wouldn't do that to you!). Costumes are suitable, gadgets will be welcome, the more quirky and fantastical the better! We know you're among the cleverest, most inventive and imaginative people to be found (not that we're biased!) so we can't wait to see you as futuristic and anachronistic characters and marvel at your latest inventions and innovations!



Steampunk1 man with camera Steampunk3

C bitfields considered harmful

In C (and C++) you can specify that a variable should take a specific number of bits of storage by doing “uint32_t foo:4;” rather than just “uint32_t foo”. In this example, the former uses 4 bits while the latter uses 32bits. This can be useful to pack many bit fields together.

Or, that’s what they’d like you to think.

In reality, the C spec allows the compiler to do just about anything it wants with these bitfields – which usually means it’s something you didn’t expect.

For a start, in a struct -e.g. “struct foo { uint32_t foo:4; uint32_t blah; uint32_t blergh:20; }” the compiler could go and combine foo and blergh into a single uint32_t and place it somewhere… or it could not. In this case, sizeof(struct foo) isn’t defined and may vary based on compiler, platform, compiler version, phases of the moon or if you’ve washed your hands recently.

Where this can get interesting is in network protocols (OMG DO NOT DO IT), APIs (OMG DO NOT DO IT), protecting different parts of a struct with different mutexes (EEP, don’t do it!) and performance.

I recently filed MySQL bug 74831 which relates to InnoDB performance on POWER8. InnoDB uses C bitfields which are themselves bitfields (urgh) for things like “flag to say if this table is compressed”. At various parts of the code, this flag is checked.

When you apply this simple patch:

--- mysql-5.7.5-m15.orig/storage/innobase/include/dict0mem.h
+++ mysql-5.7.5-m15/storage/innobase/include/dict0mem.h
@@ -1081,7 +1081,7 @@ struct dict_table_t {
        Use DICT_TF_GET_COMPACT(), DICT_TF_GET_ZIP_SSIZE(),
        DICT_TF_HAS_ATOMIC_BLOBS() and DICT_TF_HAS_DATA_DIR() to parse this
        flag. */
-       unsigned                                flags:DICT_TF_BITS;
+       unsigned                                flags;

I get 10,000 key lookups/sec more than without it!

Why is this? If you go and read the bug, you’ll see that the amount of CPU time spent on the instruction checking the bit flag is actually about the same… and this puzzled me for a while. That is, until Anton reminded me that the PMU can be approximate and perhaps I should look at the loads.

Sure enough, the major difference is that with the bitfield in place (i.e. MySQL 5.7.5 as it stands today), there is a ld instruction doing the load – which is a 64bit load. In my patched version, it’s a lwx instruction – which is a 32bit load.

So, basically, we were loading 8 bytes instead of 4 every time we were checking if it was a compressed table.

So, along with yesterday’s lesson of never, ever, ever use volatile, today’s lesson is never, ever, ever use bitfields.

November 13, 2014

[life] Day 288: A day of vomit, canceled play dates and bus excursions

I'd made plans with Kelley, the mum of Chloe, who is also starting Prep next year, to go on a day trip over to the Redcliffe lagoon for a swim and a picnic. Despite it raining overnight, and it being overcast in the morning, I was confident we could still make a good day of it.

Then Zoe threw up rather spectacularly before breakfast. At least she had the decency to do it in the kitchen on the tiles, so it was easy to clean up.

At that stage, I wasn't sure if it was a gastro bug or what, so I canceled the play date to be on the safe side. Zoe kept breakfast down, and after doing a bit of craft (we made a really awesome little castle out of a tissue box and some toilet rolls), I decided it was okay to venture out.

Zoe's Kindergarten is doing some sort of Christmas production, and she was asked to provide a plain green t-shirt for it. Big W was apparently selling such t-shirts for the bargain price of $3, so I thought we might as well head out there. Zoe was keen to take a bus. We've never tried a multiple bus excursion before, but I figured it would be a good way to pass the day. There was a bus due in 10 minutes of hatching this plan, so we hastily headed out the door.

We took the usual city-bound 230 bus from Hawthorne, and had to change for a Carindale-bound bus at Woolloongabba, which struck me as a really inefficient way to travel, but it was what Google Maps said to do, so we did it. On the bus to Carindale, my friend Steve called me up because he had an hour to kill, and we arranged to meet up at Carindale for a coffee.

By the time we got there it was pretty much time for an early lunch, so we grabbed some lunch in the Glasshouse precinct of Westfield and Zoe had a play in the play area. I also tried out the selfie stick I'd picked up a few weeks ago. Zoe thought it was pretty cool.

After Steve departed, we bought the t-shirt, and headed back to the buses. It was nice to just wander around Carindale without a particular agenda for a change.

Having been the first time we'd traveled to Carindale by bus, we got on the 204 bus going in the wrong direction, and I didn't realise until we reached the end of the line. We had a brief wait for the next bus to leave going back the right way. I had a bit of a chat with the bus driver, who had a granddaughter about Zoe's age. I learned that 6 bus drivers a week get bashed, which seems rather dreadful. I also learned that they've been told it causes less fuss to just let passengers travel with a Go card with insufficient credit, so I'm not sure why I'm bothering to auto top up mine.

With the unplanned extra delay, we got home running late for swim class, so we drove to swim class, and Zoe had her swimming lesson.

After a brief side trip to the Hawthorne Garage to grab some stuff for dinner, we headed home, and Zoe watched a bit of TV while I put dinner on.

Bedtime was nice and uneventful.

linux.conf.au 2015 OpenStack miniconf call for presentations

linux.conf.au will be hosting the fourth OpenStack mini-conference to run in the Australia / New Zealand region. Excitingly, this is the first OpenStack mini-conference to be run in New Zealand.
The idea for next years linux.conf.au miniconf is to “pivot” from previous miniconfs and present something which is clearly targetted at deployers and system admins. This is based on feedback that conference attendees believed the miniconf to be a developer meetup, and therefore weren’t attending. The plan is for the miniconf to follow a logical flow of things a deployer would be thinking about as they install OpenStack in their existing environment. The miniconf will be recorded, so hopefully this will form a useful resource for others.
The mini-conference is therefore calling for proposals for content. Speakers at the mini-conference must be registered for linux.conf.au 2015 as delegates, or discuss their needs with the mini-conference organizers if that isn’t possible.
Proposed miniconf flow:
  • What are the components of OpenStack and what order should I deploy them in? What problems do each component solve?
    • Layer 1: keystone, glance, nova, neutron
    • Layer 2: cinder, ironic
    • Layer 3: horizon, ceilometer
    • Swift (its not clear what layer this will end up in, we will handle it separately)
  • Configuration management options:
    • puppet
    • chef
    • ansible
    • juju
  • Keystone:
    • integrating auth with your existing auth systems (LDAP, Active Directory)
  • Nova:
    • hypervisor options
    • instance storage options
  • Neutron:
    • network deployment options
    • gotchas
  • Cinder
  • Ironic
  • Horizon
    • Theming to match your corporate look
    • Adding panels
  • Ceilometer
  • Swift
  • Operational matters:
    • monitoring
    • logging
    • debugging
    • where to get help
For your proposed talk, please specify where in this flow it would fit.

Some important details:

  • linux.conf.au runs from 12 to 16 January 2015 in Auckland, New Zealand
  • the mini-conference will be on Tuesday the 13th of January
  • proposals are due to the mini-conference organiser no later than 4 December 2014
  • there are two types of talks — full length (45 minutes) and half length (20 minutes), because of the intention to follow a logical flow some negotiation is expected around the content and duration of talks proposed
CFP submissions are made by completing this online form: https://docs.google.com/forms/d/1pNXwmw8iwNFRTp_yZq0yVvyA6_KRyrp-hU9PITJCqNs/viewform
If you have questions about this call for presentations, please contact Michael Still at openstack-lca2015@lists.stillhq.com for more details.

volatile considered harmful

While playing with MySQL 5.7.5 on POWER8, I came across a rather interesting bug (74775 - and this is not the only one… I think I have a decent amount of auditing and patching to do now) which made me want to write a bit on memory barriers and the volatile keyword.

Memory barriers are hard.

Like, super hard. It’s the kind of thing that makes you curse hardware designers, probably because they’re not magically solving all your problems for you. Basically, as you get more CPU cores and each of them have caches, it gets more expensive to keep everything in sync. It’s quite obvious that with *ahem* an eventually consistent model, you could save a bunch of time and effort at the expense of shifting some complexity into software.

Those in the MySQL world should recognize this – we’ve been dealing with asynchronous replication for well over a decade as a good way to scale.

On some CPU architectures (POWER for example) not all loads are created equal. When you load a value from memory, it will be consistent with your thread of execution. That is, with any stores that you have done in this thread of execution. If another thread updates that memory location you may not see that update even if your load occurs after that thread updates that memory location. Think eventually consistent.

If you want up to date reads (and not clobber writes), then you get to do memory barriers! (a topic for elsewhere – the PowerISA document has good explanations of what we have on POWER though, and how load with reserve works).

What the volatile keyword does is generate load and store instructions. It is useful when talking to hardware, as the load and store instructions are actually doing something there that the compiler doesn’t know about and thus shouldn’t optimize away.

The volatile keyword does not add any memory barriers. This is important to realize – volatile just makes loads and stores happen for your thread, not in relation to any other threads of execution. Thus, you cannot use volatile as a thread synchronization mechanism at all. It is completely and totally wrong.

Basically, if you have a volatile variable and you do stores to it in one thread and loads in another, after the store happens, it could be quite a long time before the thread doing the loads sees it! For some applications this may be okay (although I can’t really think of any beyond very very inaccurate status variables)… but if it matters at all for application correctness, volatile is the wrong thing to use.

Further reading:

November 12, 2014

Speaker Feature: Leslie Hawthorn, Richard Jones

Leslie Hawthorn

Leslie Hawthorn

Checking Your Privilege: A How-To for Hard Things

2:15pm Wednesday 14th January 2015

An internationally known community manager, speaker and author, Leslie has spent the past decade creating, cultivating and enabling open source communities. She created the world’s first initiative to involve pre-university students in open source software development, launched Google’s #2 Developer Blog, received an O’Reilly Open Source Award in 2010 and gave a few great talks on many things open source.

In August 2013, she joined Elasticsearch as Community Manager, where she leads Developer Relations. She works from Elasticsearch’s EU HQ in Amsterdam, The Netherlands – when not out and about gathering user praise and pain points.

For more information on Leslie's presentation, see here.



Richard Jones

Richard Jones

The future of Python packaging

1:20pm Thursday 15th January 2015

Richard has been a Python programmer and part of the Python community for a very long time now. He created and administers pypi.python.org, though he has help from infrastructure elves these days.

For more information on Richard and his presentation, see here. You can follow him as @r1chardj0n3s and don’t forget to mention #LCA2015.

[life] Day 287: Kindergarten, another Prep day and a big dinner

I was up and about very early again this morning. I started the day off with my chiropractic adjustment and got stuck into the last part of the current unit of my real estate licence course.

The cleaners were about 15 minutes late arriving, and I had to race out the door as soon as they arrived to have a one-on-one with my group leader.

That went okay, and I got home and did some more of my coursework before heading out for my massage. I was feeling really sore after my run and my yoga class, so I really needed it this week.

After that, I had enough time to grab some lunch and race over to pick up Zoe from Kindergarten early to take her to Prep for her third introductory session. We introduced ourselves to the Mum of a little boy who was also heading to school and recognised Zoe in her Prep t-shirt.

The session went much better than the first one, probably because we were less rushed getting there on time and Zoe was well and truly with the program by now.

After school, we went back to the uniform shop to return one dress too many, and then ducked over to the supermarket to get some stuff for dinner. I'd arranged for Megan and her sister and Dad to come for dinner, as Laura was out of town. At school, I ran into Eva and Layla's Mum, Tanya, and Justin was out of town for a funeral, so I invited them over for dinner as well.

Five kids in a small apartment was quite the zoo, but everyone had a good dinner, play and bath (I need a few more towels to deal with that many bathing guests). Zoe was pretty tired, and with all the excitement I didn't get her down to bed until a little bit late.

Unfortunately, I didn't quite get the unit of study completed that I was working on today. I'll have to try and finish it off over the next couple of days.

Preliminary MySQL Cluster benchmark results on POWER8

Yesterday, I got the basics going for MySQL Cluster on POWER. Today, I finished up a couple more patches to improve performance and ran some benchmarks.

This is on a 3.7Ghz POWER8 machine with non-balanced memory (only 2 of the 4 NUMA nodes have memory, so we have less total memory bandwidth than we could have, plus I’m going to bind ndbmtd to the CPUs in these NUMA nodes)

With a setup of a single replica and two data nodes on the one machine (each bound to a specific NUMA node), running the flexAsync benchmark on MySQL Cluster 7.3.7, I could get around:

  • 3.2 million reads/sec
  • 2.6 million deletes/sec
  • 2.4 million updates/sec
  • 2.4 million inserts/sec.

So, that’s at least in the right ballpark for a first go.

(I’m running this on a big endian host kernel, some random kernel I booted on the box and built with gcc 4.8 with whatever build options the MySQL Cluster cmake foo chooses by default)

November 11, 2014

Speaker Feature: Deb Nicholson, Julian Simpson

Deb Nicholson

Deb Nicholson

Software Patents: Trolls and Other Bullies

11:35am Wednesday 14th January 2015

Deb wants to make the world a better place with technology and social justice for all. After many years of local political organizing, she started handling outreach for the Free Software Foundation and became an enthusiastic free software activist. She likes talking to developers about software patents, to project maintainers about leadership and to activists about free software. She is currently the Community Outreach Director at the Open Invention Network and the Community Manager at GNU MediaGoblin. She also serves on the board at Open Hatch, a.k.a. Free Software's Welcoming Committee. She lives in Cambridge, Massachusetts surrounded by a community of food nerds and noisy musicians.

Deb won the O’Reilly Open Source Award, one of the most recognized awards in the FLOSS world, for her work on GNU MediaGoblin and OpenHatch.

For more information on Deb and her presentation, see here. You can follow her as @mediagoblin and don’t forget to mention #LCA2015.



Julian Simpson

Julian Simpson

Surrounded by Graphs - a short introduction to Graph Databases and Neo4j

1:20pm Friday 16th January 2015

Julian has had a long and interesting career in the very comfortable trenches of IT. From a very entry-level position in the industry, he's worked in Support, Unix Systems Administration, Build and Release Management, and Development. He has worked in New Zealand, the USA and the UK where he was foresighted enough to buy his EuroStar ticket to the first DevOpsDays conference in 2009.

Julian has worked at many companies as an employee and as a consultant. Most of his experience is in Telecommunications, Media, and Finance. For the last 3 years he's worked for Neo Technology, the people who make Neo4j. He lives in Auckland with his wife and family.

For more information on Julian and his presentation, see here. You can follow her as @builddoctor and don’t forget to mention #LCA2015.

MySQL Cluster on POWER8

So, I’ve written previously on MySQL on POWER, and today is a quick bit of news about MySQL Cluster on POWER – specifically MySQL Cluster 7.3.7.

I ran into three main issues in getting some flexAsync benchmark results. One of them was the fact that I wanted to do this in the middle of all the POWER8 machines I usually use moving buildings (hard to run benchmarks when computers are packed up in boxes on a truck).

The next issue was that ndbmtd (the multi-threaded data node) needs memory barriers for the magic message passing stuff between threads. So, that’s pretty easy (about an eight line patch).

The next issue was in the results from flexAsync, it turns out 32bit math is a bad idea with results from my POWER8 box.

My preliminary performance numbers are fairly promising (actually… what is the world record for a single machine and NDB these days? Single data node?). I think there’s a bit more low hanging fruit and a couple more things that are a bit more involved.

Bugs with patches:

  • Bug 74782 – compile fix (memory barriers for POWER)
  • Bug 74781 – flexAsync uses 32bit math, leading to incorrect summary on POWER8

[life] Day 286: Kindergarten, startup stuff, uniform shopping

This morning was a bit cooler on account of it being overcast, so I managed to leap out of bed and go for a run. I realised afterwards that it's been quite a while since I've gone for one. I mustered 7 km this morning before blowing up, mostly due to a lack of willpower to keep slogging on in the heat. I was happy I lasted that long.

I made up for last week's lost progress on my real estate licence course and knocked over a unit and put it in the mail. I managed to get through the first two parts of another unit, and hopefully I can finish off the third part and get it into the mail tomorrow.

I also did some productive procrastinating and may actually have a successful backup of daedalus currently getting written out to virtual "tape". Who would have thought the TCP keepalive interval would be the cause of all the problems?

I biked to Kindergarten to pick Zoe up, and then we went to the post office. I thought we should get Zoe's school uniform shopping out of the way, so under protest, we headed to the uniform shop on the way home from the post office.

For some reason, Zoe hadn't been excited about the prospect of going uniform shopping. Every time I'd asked her if she wanted to do it, she'd declined. Once we got into the store and were trying on uniforms though, she wanted to wear one home on the bike. So that's the uniform shopping out of the way, we just have to get some shoes, which I'll leave until the last minute in case her feet grow.

After we got home, Zoe watched some TV and I had a crack at making a herb and garlic pull-apart that I've been wanting to try and make for a while. It's smelling delicious in the oven as I write.

I'm looking forward to closing out the day with a yoga class.

USB Vendor ID for documentation

If you are writing documentation then you don't want to use an assigned magic number, like a real IP address or a real DNS name. That can readily lead to: misunderstandings; operational difficulties for the vendor's equipment if the number escapes from documentation into production; and difficulties for the author because of the risk of defamation and trademark infringement.

For these reasons standards associations commonly issue a range of their magic numbers for documentation purposes. For example, the IETF issued magic numbers for documentation in RFC2606 for DNS names, in RFC5737 for IPv4 addresses and in RFC3849 for IPv6 addresses.

I was writing some documentation for using udev, and rather than defame some vendor by suggesting that their product may need a workaround, I asked the USB Implementors' Forum if there is a USB Vendor ID for documentation purposes.

Sadly, there is not:

From: USB-IF Administration <redacted>

Subject: RE: Vendor-ID for use in documentation

Date: 11 November 2014 2:34:21 PM ACDT

To: Glen Turner <redacted>

Dear Glen,

Thank you for your message. Vendor IDs (VIDs) are owned by the vendor company and are assigned and maintained by the USB-IF only. We do not have a generic VID for documentation.

Regards, redacted

OpenRadio – a one day Software Defined Radio project

For the 2015 Linux Conference, I am working with Kim Hawtin and Mark Jessop on a 1 day Open Radio Mini-conference.

In this mini-conf a classroom of people will solder together their very own software defined radio (SDR) transceivers in just a few hours. It will be capable of receiving signals on the HF radio bands (3 to 30 MHz), and short range transmission of FSK/PSK data on the 13.5 and 27 MHz ISM bands (no license required).

The project is being documented on our OpenRadio Wiki. It’s completely open source and we have published the PCB CAD files, and the parts list with Digikey/Element14 catalogue part numbers. It’s based on the soft-rock radio designs.

We have put a lot of effort into making the radio easy to build. For example a minimum of (large footprint) surface mount parts, and a simple, fast to assemble design. We have intentionally included one or two inductors and transformers to wind to give people a taste of the complete radio assembly experience. With a little supervision, the project is quite suitable for radio/electronics beginners or school age children. It’s a “crystal set” for the 21st century.

Mark has done a great job designing the radio, and we have just received the prototype PCBs:

This week we will assemble and test the first prototypes, measure how long they take to build, and noting possible snags for inexperienced builders. Then our good friend Edwin from Dragino will prepare and ship kits for the mini-conf.

The resources we create for this project (wiki, CAD files, software, kits from Dragino) will remain available after LCA. So you, your radio club, hackerspace, or even school class will have access to an easy to build a Software Defined Radio (SDR).

November 10, 2014

Speaker Feature: Fraser Tweedale, Peter Chubb

Fraser Tweedale

Fraser Tweedale

FreeIPA: Open Source Identity Management

2:15pm Friday 16th January 2015

Fraser is a developer at Red Hat, where he works on the FreeIPA identity management solution and Dogtag Certificate System. He is passionate about security and privacy. In his spare time, Fraser writes a lot of Haskell and patiently awaits the strongly-typed functional programming revolution.

For more information on Fraser and his presentation, see here. You can follow his as @hackuador and don’t forget to mention #LCA2015.



Peter Chubb

Peter Chubb

SD Cards and filesystems for Embedded Systems

2:15pm Friday 16th January 2015

Peter has been hacking on UNIX since 1979, and has never used Windows. He currently does system (kernel and low-level) programming in a Linux environment for NICTA.

Peter's research interests include operating system algorithms for scalability, including storage, scheduling, memory management, and locking. He is also interested in systems performance measurement and optimisation.

Related hobbies include music, photography and fine wines, these also occasionally lead to research.

For more information on Peter and his presentation, see here.

[life] Day 285: Kindergarten and some tinkering

I've felt exceptionally flat today, despite having a good night's sleep. I'm blaming the heat, or treking around Saint Helena Island in the hot sun yesterday.

We biked to Kindergarten this morning for drop off, and I left the trailer there. I was feeling pretty flat just after biking there and back in the heat.

I'm trying to rig up something to bring to Kindergarten one day, which uses a Raspberry Pi and the MaKey MaKey. I spent the day fooling around with it, and not much else.

I biked back to Kindergarten to pick Zoe up. Zoe and Megan wanted to have a play date, and Jason had to run some errands, so we biked home, and he dropped Megan off.

The girls had a good time running amok, and I made a start on my first batch of fruit mince for mince pies of this Christmas season.

Jason came over to pick up Megan, and Sarah arrived not long afterwards.

LUV Beginners November Meeting: Developing GUI applications

Nov 15 2014 12:30
Nov 15 2014 16:30
Nov 15 2014 12:30
Nov 15 2014 16:30
Location: 

RMIT Building 91, 110 Victoria Street, Carlton South

There are a few useful tools/IDEs available on Linux to develop GUI applications. They are all similar in features. In this talk Daniel Jitnah will briefly talk about how GUI applications work, and what are the toolkits available: GTK, QT and Tk as examples. He will also demonstrate how a very simple GUI application can be built. The IDEs used will be QTDesigner, Lazarus, Anjuta+Glade and Netbeans.

LUV would like to acknowledge Red Hat for their help in obtaining the Buzzard Lecture Theatre venue and VPAC for hosting.

Linux Users of Victoria Inc., is an incorporated association, registration number A0040056C.

November 15, 2014 - 12:30

read more

November 09, 2014

Speaker Feature: Selena Deckelmann, Thomas Sprinkmeier

Selena Deckelmann

Selena Deckelmann

What technical learners need

3:40pm Thursday 15th January 2015

Selena is a major contributor to PostgreSQL and a data architect at Mozilla. She is a director of the Python Software Foundation.

She's been involved with free and open source software since 1995 and began running conferences for PostgreSQL in 2007. In 2012, she founded PyLadiesPDX, a portland chapter of PyLadies. She founded Open Source Bridge, Postgres Open and speaks internationally about open source, databases and community. She also keeps chickens and gives a lot of technical talks.

For more information on Selena and her presentation, see here. You can follow her as @selenamarie and don’t forget to mention #LCA2015.



Thomas Sprinkmeier

Thomas Sprinkmeier

How to train your Minions

3:40pm Friday 16th January 2015

Thomas graduated from UniSA in 1992 as an Electronic Engineer where he was seduced by PC's early in first year.

He's been working as a Software Engineer ever since for Ebor Computing in a variety of projects, usually with heavy mathematical, signal processing and networking components, occasionally interfacing to the 'real world'. Most recently he has been working at making cars smarter an safer, on the assumption that this might be easier than upgrading drivers.

Thomas started embarassing his kids at school by taking over the class and teaching about things from pulleys to railguns, paper planes to robot programming, conducting playdough to tidal locks. Most recently he has been teaching on weekends about Raspberry Pi, Arduino and 3D printing.

For more information on Thomas and his presentation, see here.

Twitter posts: 2014-11-03 to 2014-11-09

November 07, 2014

[life] Day 282: A day with a nanny, REIQ course, family fun night

The REIQ was offering a one day "Starting an agency" course, which seemed too good to pass up. Unfortunately, being on a Friday, and not wanting to cause Zoe to miss out on Tumble Tastics, I had to get a nanny in, instead of just sticking her in day care for the day.

After getting some recommendations from friends on Facebook, and doing price comparisons, I went with Nanny's R Us.

They had to do a last minute swap, but everything worked out fine, and even with a 30 minute late start due to traffic, I got to my course on time.

The course was well worth it, and I got some useful information that I'd been lacking to date. The course instructor also happened to be the trainer who is marking my next four units of my real estate license course, so it was nice to meet him in the flesh.

The course finished a bit earlier than the advertised time, so I used the time to run some errands I was planning on doing on Monday.

Zoe's future primary school had a "family fun night", which started at 3pm, so I'd asked the nanny to take Zoe to that and met them there at 5pm. It was a pretty good night, and Zoe had lots of fun on the rides.

[life] Day 281: Movieworld

My friends Chris and Kelly were holidaying on the Gold Coast with their daughter Alyssa, and I made plans to catch up with them at Movieworld on Thursday.

The day went super well. We timed our arrival for right on opening time, and quickly found them. Zoe and Alyssa got along really well.

The highlight of the day (for me) was that Zoe managed to be allowed onto the Scooby Doo ride in the morning. I think technically she was still a tiny bit under the height restriction, as a different attendant didn't let her on again in the afternoon.

Zoe was initially too scared by the dark in the queuing area and wanted to leave again, but the attendant talked her into going on the ride, and we had a great time. It's my favourite ride so far, in terms of length. Zoe turned around and went on it again straight away.

The other ride that we got heaps of mileage out of was the Hall of Justice ride, which Zoe has been on the first time we went, but the second time was too scared to go on. This time, her and Alyssa probably went on it 10 times in a row.

It was a really great day out, and I even got to go on the Green Lantern ride with Kelly, which was all over surprisingly quickly.

I expected Zoe to fall asleep instantly in the car on the way back, but she stayed away chattering away and playing with the Scooby Doo toy that Alyssa had bought her.

We got back with enough time to do a couple of post office runs and pick up Anshu from the ferry terminal, before Sarah arrived to pick up Zoe.

November 05, 2014

New podcasts

I've enjoyed a couple of new podcasts lately:

Slate's Working podcast finds people in interesting jobs and interviews them about their workdays. It's brand new. The first episode – with Stephen Colbert – was fantastic. The show is short and dense. David Plotz as the host (along with some helpful editing, I'm sure) gets the guest talking (they usually have a lot to say) and then gets out of the way. I've appreciated finding another non-tech podcast to keep in my roster.

I still have plenty of room for good tech podcasts, though. Thoughtbot have just launched another new podcast, called The Bike Shed, covering their general experiences in web development. This looks like it will be a discussion show with regular hosts Sean Griffin and Derek Prior. They seem humble and grounded, and the first show on Sandi Metz' rules was thoughtful, and directly applicable to my work as a web developer. I'm still thinking over what they shared. I'm also appreciative they've kept the show to under 30 minutes. This makes it easy to cover on a walk into work!

Speaker Feature: Timothy Jones, Tom Clark

Timothy Jones

Timothy Jones

Hop, Skip, Jump: Implementing a concurrent interpreter with Promises

3:40pm Wednesday 14th January 2015

Tim is a PhD student at Victoria University involved in programming language design and type theory. He got his start in open-source and the PL world as a core contributor to the CoffeeScript project before becoming enamoured with Haskell, and now teaches advanced programming language concepts, has produced a few papers in the field, and occasionally writes in Agda. He also organises Pixel Jam, an annual 48-hour game jam in Wellington, during which he always builds novelty fishing simulators.

Tim has worked as part of the Grace language project over the last three years, contributing to its open-source implementation and spawning several smaller projects of his own. His most recent work has been on Hopper, a concurrent JavaScript interpreter.

For more information on Tim and his presentation, see here. You can follow him as @zmthy and don’t forget to mention #lca2015.



Tom Clark

Tom Clark

What should a Systems Administration Student's Homework Look Like?

1:20pm Wednesday 14th January 2015

Tom Clark studied mathematics and computer science at Seattle University and Dartmouth College before going on to work in various software development, IT operations, and tertiary teaching roles. Tom joined the staff of Dunedin, New Zealand's Otago Polytechnic in July of 2013 to teach in the Bachelor of Information Technology programme. He teaches papers in programming, networking, and systems administration.

For more information on Tom and his presentation, see here. You can follow him as @tom_clark and don’t forget to mention #lca2015.

[life] Day 280: A fairly quiet day, some exploration, and another Prep day

I felt absolutely trashed this morning. I was completely oblivious to Zoe coming in and sleeping in my bed at 1:20am. The first thing I knew of her being there was at around 5am when she woke me up.

We eventually got going in the late morning. I wanted to go check out a wholesale kitchen place I'd heard about, Kitchen Discounts. Like every other wholesale direct to the public place I've found out about recently, this place was also awesome. Zoe and I had a great time browsing the store.

We had lunch when we got home, and then started pulling apart the old garbage disposal, which ended up being a bit of a job. They really weren't kidding when they said it wasn't serviceable. It had some weird screws that I couldn't undo, so I drilled them all out, and we got as far as pulling the motor apart. It was definitely leaking, the bottom of the grinding plate had corroded through.

It was then time to head to Zoe's future primary school for her second Prep introductory session. Zoe scootered to school with lots of time to spare so we wouldn't be late, and I made sure Zoe was fed and watered, and things went much better than last week.

The format was pretty much the same as last week, just different fine and gross motor skills. Zoe participated much better than last week, so the challenge will be to have a smoother early departure from Kindergarten for next week's session.

On the way back home, we stopped off at the park for a play, and then continued home in time for Sarah to pick up Zoe.

[life] Day 279: Kindergarten, Grandparents Day, lots of minor annoyances

Yesterday was just full of annoyances.

Zoe had a brief wake up at 11pm. I think it was a bad dream.

I woke up at 5:30am with the light and decided to get up and get things going. Zoe on the other hand, decided to sleep in until 7:20am.

We got going to Kindergarten on the bike. The sky looked really ominous, and sure enough, it started to rain by the time we'd gotten half way up the Hawthorne Road hill. I figured we should just cut our losses and turn around and go in the car instead.

So we eventually got to Kindergarten. It was Grandparents Day, and all of Zoe's living grandparents and great-grandparents were going to make an appearance.

I got home and made a start on the current unit I'm working on for my real estate licence. After finishing Part A, I took a break, and doing some cleaning up, caused the garbage disposal to trip the earth-leakage circuit breaker, killing power to my unit and killing a multi-day backup of daedalus that I had running.

I discovered the garbage disposal has been leaking water under the sink, and that was probably what caused the circuit breaker to trip. The model I have isn't serviceable, so it needed to be replaced. After some calling around, Bunnings were the cheapest, so I picked one up and proceeded to spend the rest of the day installing it.

After much swearing, I got it installed successfully. It was not how I wanted to be spending the day.

I got it finished with enough time to clean up before driving over to Kindergarten to pick up Zoe.

Zoe wanted to watch Megan's tennis class, so we stuck around for that, and then popped into Megan's for a little bit before heading home.

Dinner was quick and easy tonight, so we used the extra time to go out for a babyccino before bedtime.

November 04, 2014

Speaker Feature: Rafael Wysocki, Stewart Smith

Rafael Wysocki

Rafael Wysocki

Getting more out of System Suspend in Linux

11:35am Thursday 15th January 2015

Rafael is the maintainer of the Linux kernel's core ACPI and power management code, including the core infrastructure for runtime PM, system suspend and hibernation, cpuidle and cpufreq. He works at Intel Open Source Technology Center as a Software Engineer with focus on the Linux kernel. Rafael has been actively contributing to Linux since January 2005, in particular to the kernel's suspend and hibernate subsystem, power management in general (runtime PM, PM QoS, wakeup framework etc.), hot-plug infrastructure, ACPI core and PCI core. Rafael received an MSc from the University of Warsaw, Faculty of Physics, in 1996 and a PhD from that faculty in 2002.

For more information on Rafael and his presentation, see here.



Stewart Smith

Stewart Smith

Towards One MILLION SQL Queries Per Second

3:40pm Wednesday 14th January 2015

Stewart currently works for IBM in the Linux Technology Center on KVM on POWER, giving him a job that is even harder to explain to non-Linux geek people than ever before. Previously he worked for Percona as Director of Server Development where he oversaw development of many of Percona’s software products. He comes from many years of experience in databases and free and open source software development. He’s often found hacking on the Drizzle database server, taking photos, running, brewing beer and cycling (yes, all at the same time).

For more information on Stewart and his presentation, see here.

Stewart is also one of our wonderful Miniconf organisers; running the Developer, Testing, Release and Continuous Integration Automation Miniconf on Tuesday 13th.

November 03, 2014

[life] Day 278: Kindergarten, a trip to QCAT, and a very social swim class

Again, I failed to start the day with a run, despite that being my intention. I think I'm officially out of the running habit now, so it's going to take some work to get back into it.

I'd decided that since I'd just finished the dispute management unit of my real estate licence course, that it might be educational to go and sit in on some residential tenancy dispute hearings at the Queensland Civil and Administrative Tribunal, so I donned my suit, hopped on a bus and rocked up at the hearing rooms in the city.

As they only publish their daily hearing list the night before, I kind of had to wing it as to the suitability. As luck would have it, Tuesday's list looks way better than Monday's list.

I ended up sitting in on one residential tenancy hearing, where the respondent was a no show, but they still proceeded with terminating the tenancy due to massive arrears. The next two were public housing hearings, which weren't relevant to me. One, where the respondent did show up, was particularly messy.

I didn't get as much of an educational benefit out of it as I would have hoped, but I did get to see how the process worked, so if I ever wind up there it won't be totally foreign to me. I may go again another time for the heck of it.

I caught a bus back home with enough time to bike down to the post office to collect some mail before biking to Kindergarten to pick Zoe up.

I'd switched Zoe's swim class from Thursday to Monday this week to free up Thursday for a trip to Movieworld to catch up with some friends visiting from Adelaide, so we biked directly to swim class, getting there a bit early. We hung out and had a snack before her class started.

One of Zoe's friends from Kindergarten was having a class at the same time, so Zoe got to say hi to her before their classes started, and then Eva and Layla arrived for classes after Zoe's. Zoe got to hang out with them individually, because they had back to back classes, and Justin and I got to have a chat. Zoe had a great old time playing with a bunch of kids, and we didn't end up leaving the pool until 4:30pm. It was a really nice afternoon.

osquery is neat

Facebook recently made opensource, osquery. It gives you operating system data via SQL queries! Its very neat, and you can test this even on MacOSX (it works on that platform & Linux). It is by far the project with the most advanced functionality, linked here in this post.

I noticed that rather quickly, there was a PostgreSQL project, called pgosquery, based on Foreign Data Wrappers with a similar idea. (apparently it was written in less than 15 minutes; so a much lower learning curve than the regular MySQL storage engine interface)

I immediately thought about an older MySQL project, by Chip Turner (then at Google, now at Facebook), called mysql-filesystem-engine. This idea was kicking around in 2008. I was intrigued by hearing about this at a talk (probably at the MySQL Conference & Expo); it’s a pity no one took this further.

On a similar tangent, did you also know that there is the option to use MySQL as storage via FUSE (see: mysqlfs)? An article by Ben Martin shows some practical examples.

At its heyday, MySQL had many storage engines (maybe around 50). Wikipedia has an incomplete list. I see some engines on that list, and think that some of these folk are also creating MongoDB backends — competition. At MariaDB we are probably shipping the most storage engines of any MySQL-based distribution, however I think we could be doing an even better job at working with upstream vendors, and figuring out how to support & augment business around it.

November 02, 2014

Call for Presentations for Clouds, Containers, and Orchestration Miniconf

"Software defined everything," DevOps, and cloud are driving open source further and faster than we might have imagined possible just a decade ago. Most recently, Docker containers and orchestration have opened up all kinds of new opportunities to develop, deploy, and manage software from the developer's desktop well into production.

Call for Presentations

The miniconf will focus on the open source tools and best practices for working with cloud tools, containers, and orchestration software (e.g., Kubernetes, Apache Mesos, and others). We want the leading developers working on those tools, as well as users who are deploying them in real production environments to share their knowledge and show where tools will be going in 2015.

We welcome talks on container security, creating complex applications in the cloud, working with open source Platforms-as-a-Service, container orchestration, packaging applications in containers, single-purpose operating systems, and presentations on the state and future of these applications from their developers.

Presentations should be useful to practitioners, and technical in nature. Talks should not be promotional in nature.

Format for Presentations

As we only have one day, presentation slots will be shorter than many speakers may be used to. Most talks will be 20 minutes, with the possibility for one or two longer slots for exceptionally interesting and involved topics.

Please indicate which type of presentation you're seeking:

  • 20 minute full presentation.
  • 10 minute "case study" or "state of project" presentation.
  • 40 minute double-length presentation.
Final configuration of talk/times is pending submissions received.

Submitting Talks

Please see the Submission Template and submit your proposal to miniconf [at] dissociatedpress.net. Please include [LCA15 CFP] and talk title in your subject line. (For example: [LCA15 CFP] Whiz-Bang Container Wrangling with Docker and Acme Widgets.)

Note: In order to present at the miniconf, you must be registered for the main Linux.conf.au conference, and presenting at the miniconf does not entitle speakers to registration at the main conference, or any travel sponsorship.

Important Dates

  • 2014-11-15 Deadline for early submissions
  • 2014-11-20 Early submissions confirmation
  • 2014-11-25 Deadline for all submissions
  • 2014-12-1 Confirmation for submissions
  • 2014-12-2 Final speaker confirmation required
  • 2014-12-4 Final schedule announced
  • 2015-01-12 Miniconf (first day of Linux.conf.au 2015)

Schedule

The schedule will be announced on 4 December 2014.

Code of Conduct

The Clouds, Containers, and Orchestration Miniconf follows the Linux.conf.au Code of Conduct and values statement. Please do read the full Code of Conduct and Values Statement, but the bottom line is that Linux.conf.au is meant to be an event for a diverse community. Anyone – regardless of age, race, gender identity or expression, background, disability, appearance, sexuality, walk of life, or religion – should be able to attend, learn from and be inspired by other people in the Free and Open Source community.

It is important that everyone be courteous and respectful to other attendees. All public presentations should be suitable for people 12-years-old and above. Presentations may not contain:

  • Sexual or violent imagery
  • Exclusionary language
  • Language that is not appropriate for an all-ages audience

If you're in doubt, you are encouraged to confer with conference speaker liaiso

Current CFPs for lca2015 miniconfs

n or mailing list/forum moderators before your session.

Questions?

If you have questions that aren't covered here, please contact Joe Brockmeier at miniconf [at] dissociatedpress.net. For fastest response, please include [LCA15 CFP] in your subject line.

Call for Papers for Developer, Testing, Release and Continuous Integration Automation Miniconf

This is the Call for Papers for the Developer, Testing, Release and Continuous Integration Automation Miniconf at linux.conf.au 2015 in Auckland. The CFP closes at midnight on 21st November 2014.

This miniconf is all about improving the way we produce, collaborate, test and release software.

We want to cover tools and techniques to improve the way we work together to produce higher quality software:

  • code review tools and techniques (e.g. gerrit)
  • continuous integration tools (e.g. jenkins)
  • CI techniques (e.g. gated trunk, zuul)
  • testing tools and techniques (e.g. subunit, fuzz testing tools)
  • release tools and techniques: daily builds, interacting with distributions, ensuring you test the software that you ship.
  • applying CI in your workplace/project

We’re looking for talks about open source technology and the human side of things.

Speakers at this miniconf must be registered for the main conference (although there are a limited number of miniconf only tickets available for miniconf speakers if required).

There will be a projector, and there is a possibility the talk will be recorded (depending on if the conference A/V is up and running) – if recorded, talks will be posted with the same place with the same CC license as main LCA talks are.

CFP is open until midnight November 21st 2014.

By submitting a presentation, you’re agreeing to the following:

I allow Linux Australia to record my talk.

I allow Linux Australia to release any recordings of my presentations, tutorials and minconfs under the Creative Commons Attribution-Share Alike License

I allow Linux Australia to release any other material (such as slides) from my presentations, tutorials and minconfs under the Creative Commons Attribution-Share Alike License.

I confirm that I have the authority to allow Linux Australia to release the above material. i.e., if your talk includes any information about your employer, or another persons copyrighted material, that person has given you authority to release this information.

Please use this form http://goo.gl/forms/KZI1YDDw8n to submit your presentation.

Any questions? Contact miniconf organiser, Stewart Smith: stewart@flamingspork.com

Call for Presentations for Open Radio Miniconf

This is an invitation for proposals for presentations for the second Open Radio Miniconf at linux.conf.au in Auckland, NZ from Kim Hawtin, the miniconf's organiser. Proposals must be received by 14 December 2014.

Hi, this is Kim VK5FJ.

In early January, I’ll be kicking off the second one day Open Radio Miniconf, in Auckland, NZ.

The Open Radio miniconf is about:

  • exploring the open source hardware of software defined radio (SDR),
  • understanding the open source software used in software defined radio, and
  • exploring the open source protocols used over the air.

We’ll start off with a build-a-thon and a little theory.

We’re using an established SDR design, reworked by Mark VK5QI and Codec2 author David VK5DGR.

We will cover the how and why of SDR, and look at encoding and decoding some old and new modes.

Later in the day we will have a session for short talks on these topics, each around 10-15 minutes.

So if you are interested in presenting, please send an email to vk5fj@wia.org.au. Please submit your proposal to present at this miniconf by 14 December 2014.

More information on registering for Linux Conf in Auckland can be found at lca2015.linux.org.au.

73 from Kim VK5FJ

Audio of Open Radio Miniconf Call for Presentations (mp3)

Twitter posts: 2014-10-27 to 2014-11-02

More Stories

Sorry it’s been so long, I’m still working on them! I work on around eight at a time so that’s why it’s taking so long! I will hopefully have one up in a few days!

Related Posts:

  • No related posts

[life] Day 275: Baking fail, Water beads, TumbleTastics, Halloween

I forget what time Zoe woke up, but I'm sure it was earlier than I would have liked.

I've been wanting to do some baking for a while to get away from Zoe snacking on store-bought processed snacks, so I thought I'd have a crack at making some "40 second cookies" using a combination of macadamia nuts and almonds. I also tried milling some whole wheat into flour, but I didn't do it up front, I did it after I already had the nuts in the bowl. Long story short, it was a total fail. The dough came out too moist, and the wheat didn't mill down completely, so when I baked them they all just ran together into a large tray-sized slab of cookie. I cut out a few "cookies" with a scone cutter after they'd baked, and Zoe liked them anyway.

I had a few packets of water beads that I'd acquired recently, and we had a few errands to run. I got Zoe to put a few of the water beads in some water before we did the baking, and then headed out. After we got back from running our errands, they'd started swelling up, but not to their full size.

We got ready for TumbleTastics early for a change, and Zoe scootered down to TumbleTastics with a good 15 minutes to spare. It was nice to not be in a mad rush for once.

After TumbleTastics we came back home, and had some lunch, and the water beads had swollen up to their full capacity. Zoe had a bit of a play around with them.

The big highlight of the day was our local Councillor, Shayne Sutton, had her Hocus Pocus Halloween in the Park event, which started at 3pm. Sarah had bought an Incredibles costume for Zoe, which she was very excited about. She'd managed to break the elastic strap at Kindergarten on Wednesday, so after lunch I did a bit of a repair job on it.

Zoe was dying to wear her costume, and I needed to grab a few things from the supermarket for dinner, so I let her wear her costume to the supermarket. She thought that was pretty awesome.

After we got back from the supermarket, we headed down to the park, a little bit early, so Zoe had a play in the playground.

Once things kicked off in earnest, we did the rounds, picked up candy from all the stations, and went on the various rides. It was a good free outing. I do like Shayne Sutton's family activities. She spends her allotted funds well.

I never got into Halloween in a really big way when we lived in the US. I'd carve a pumpkin every year, for the novelty value, but after Zoe was born, I realised how much fun Halloween was for kids. One of the things I miss from the US are all the traditional holidays. Halloween is getting quite big in Australia, but it's just not the same in mid-spring, which lately seems to be becoming an extended summer.

Sarah met us at the park to pick up Zoe, so I left them to it, and headed home to start dinner.

October 31, 2014

JackPair – Secure Phone Calls using Codec 2

I’ve just found out about a new Kickstarter for JackPair, a device that enables secure phone calls over a mobile phone. It uses Codec 2.

Over the past 12 months I have been approached by a couple of groups interested in building a similar product (but not JackPair). These groups asked me to develop a modem that could pass data through a cell phone voice codec. Given I know modems and codecs it was a good fit. Quite a challenge too, to get 1200 – 2400 bit/s through a voice codec. To both groups I said I would only do the job if it was open source, and it never went any further.

I feel a product like this must be open source, in order to audit it and know it is really secure. So the software should be GPL and the hardware open. An end user must be able to (re)flash from blank silicon using their own trusted firmware. The paranoid could even do this every time they use it. Or solder their own device from scratch. That’s where I’m heading with my open source radio work – make the radio hardware trivial, and the software open and capable of running on commodity CPU.

The SM1000 has the hardware to build a JackPair type product, e.g Codec 2, DSP capability, microphone and speaker amps, and line audio interfaces. It would need a different firmware load (modem, crypto). The SM1000 is open hardware, so a good starting point.

Clearly the JackPair is a product whose time has come. I support this sort of project (secure telephony for everybody) as I feel my governments response to terrorism as more of a concern than terrorism itself. Good to see it happening, and nice to see Codec 2 helping make the world a better place.

Centrelink's PLAID broken

Jean Paul Degabriele, Victoria Fehr, Marc Fischlin, Tommaso Gagliardoni, Felix Günther, Giorgia Azzurra Marson, Arno Mittelbach, Kenneth G. Paterson. Unpicking PLAID. A cryptographic analysis of an ISO-standards-track authentication protocol.

Upon public release in 2009 PLAID was claimed to have been the subject of three years' cryptanalysis by the then Defence Signals Directorate. With that in mind the sections at the end of the paper about misuse of CBC are more concerning than the exploitation of shrill keys.

Identically partition disks.. the easy way!

Was just looking into a software RAID howto.. for no reason really, but kinda glad I did! When you set up software raid you want to make sure all disks are partitioned the same, right. so check this out:

3. Create partitions on /dev/sda identical to the partitions on /dev/sdb:

sfdisk -d /dev/sdb | sfdisk /dev/sda

That’s a much easier way ;)

This gem is thanks to: http://www.howtoforge.com/how-to-create-a-raid1-setup-on-an-existing-centos-redhat-6.0-system

NTLM Authentication in Squid using Winbind.

Some old windows servers require authentication through the old NTLM protocol, luckily with the help from squid, samba and winbind we can do this under Linux.

Some URLs a much of this information was gathered from are:

  • http://wiki.squid-cache.org/ConfigExamples/Authenticate/NtlmCentOS5
  • http://wiki.squid-cache.org/ConfigExamples/Authenticate/Ntlm

HOW TO

In order to authenticate through winbind we will be using that and samba to connect to a windows domain, so you will need to have a domain and the details for it or all this will be for naught. I’ll use some fake credentials for this post.

Required Packages

Let’s install all the required packages:



yum install squid krb5-workstation samba-common ntp samba-winbind authconfig

NTP (Network Time Protocol)

Kerberos and windbind can be a little thingy about date and time, so its a good idea to use NTP for your network, I’ll assume your domain controller (DC) will be also your NTP server in which case lets set it up.

Comment out any lines that begin with server and create only one that points to your Active Directory PDC.



# vim /etc/ntp.conf

server pdc.test.lan

Now add it to the default runlevels and start it.



chkconfig ntpd on

/etc/init.d/ntpd start

Samba, Winbind and Kerberos

We will the use the authconfig package/command we installed earlier to configure Samba, Winbind and perform the join in one step, this makes things _SO_ much

easier!!!

NOTE: If you don’t have DNS set up then you will need to add the DC to your hosts file, and it is important to use the name the DC machine knows itself as in AD.



authconfig --enableshadow --enablemd5 --passalgo=md5 --krb5kdc=pdc.test.lan \

--krb5realm=TEST.LAN --smbservers=pdc.test.lan --smbworkgroup=TESTLAN \

--enablewinbind --enablewinbindauth --smbsecurity=ads --smbrealm=TEST.LAN \

--smbidmapuid="16777216-33554431" --smbidmapgid="16777216-33554431" --winbindseparator="+" \

--winbindtemplateshell="/bin/false" --enablewinbindusedefaultdomain --disablewinbindoffline \

--winbindjoin=administrator --disablewins --disablecache --enablelocauthorize --updateall

NOTE: Replace pdc.test.lan with that of your FQDN of your DC server, TESTLAN with your domain, TEST.LAN with the full name of the domain/realm, and make sure you set ‘–winbindjoin’ with a domain admin.

If that succeeds lets test it:



# wbinfo -u

# wbinfo -g



If you are able to enumerate your Active Directory Groups and Users, everything is working.

Next lets test that we can authenticate with winbind:



# wbinfo -a



E.G:



# wbinfo -a testuser

Enter testuser's password:

plaintext password authentication succeeded

Enter testuser's password:

challenge/response password authentication succeeded

Great, we have been added to the domain, so now we can setup squid for NTLM authentication.

SQUID Configuration

Squid comes with its own ntlm authentication binary (/usr/lib64/squid/ntlm_smb_lm_auth) which uses winbind, but as of Samba 3.x, samba bundle their own which is the recommended binary to use (according to the squid and samba projects). So the binary we use comes from the samba-winbind package we installed earlier:



/usr/bin/ntlm_auth

Add the following configuration elements to the squid.conf to enable NTLM authentication:



#NTLM

auth_param ntlm program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp

auth_param ntlm children 5

auth_param ntlm keep_alive on

acl ntlm proxy_auth REQUIRED

http_access allow ntlm



NOTE: The above is allowing anyone access as long as they authenticate themselves via NTLM, you could use further acl's to restrict this more.

The ntlm_auth binary has other switches that might be of use, such as restricting users by group membership:



auth_param ntlm program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp --require-membership-of=EXAMPLE+ADGROUP

Before we are complete there is one more thing we need to do, for squid to be allowed to use winbind, the squid user (which was created when the squid package was installed) needs to be a member of a wbpriv group:



gpasswd -a squid wbpriv

IMPORTANT!

NTLM authentication WILL FAIL if you have "cache_effective_group squid" set, if you do then remove it! As this overrides the effective group and squid then isn't seen as part of the 'wbpriv' group which breaks authentication!!!

/IMPORTANT!

Add squid to the runlevels and start it:



# chkconfig squid on

# /etc/init.d/squid start

Trouble shooting

Make sure you open the port in iptables, if squid is listening on 3128 then:



# iptables -I INPUT 1 -p tcp --dport 3128 -j ACCEPT

# /etc/init.d/iptables save

NOTE: The '/etc/init.d/iptables save' command saves the current running configuration so the new rule will be applied on reboot.

Happy squid-ing.

Reverse proxy using squid + Redirection

Squid – Reverse Proxy

In computer networks, a reverse proxy is a type of proxy server that retrieves resources on behalf of a client from one or more servers. These resources are then returned to the client as though it originated from the reverse proxy itself. While a forward proxy is usually situated between the client application (such as a web browser) and the server(s) hosting the desired resources, a reverse proxy is usually situated closer to the server(s) and will only return a configured set of resources.

See: http://en.wikipedia.org/wiki/Reverse_proxy

Configuration

Squid should already be installed, if not then install it:



yum install squid

Then we edit squid config:



vim /etc/squid/squid.conf

Add we add the following to the top of the file:



http_port 80 vhost

https_port 443 cert=/etc/squid/localhost.crt key=/etc/squid/localhost.key vhost

cache_effective_user squid

cache_effective_group squid

cache_peer 1.2.3.4 parent 80 0 no-query originserver login=PASS name=site1-http

cache_peer 1.2.3.5 parent 443 0 no-query originserver login=PASS ssl sslflags=DONT_VERIFY_PEER name=site2-ssl

cache_peer_domain site1-http site1.example.lan

cache_peer_domain site2-ssl site2.anotherexample.lan

acl bad_requests urlpath_regex -i cmd.exe \/bin\/sh \/bin\/bash default\.ida?XXX insert update delete select

http_access deny bad_requests

Now I’ll walk us through the above configuration.



http_port 80 vhost

https_port 443 cert=/etc/squid/localhost.crt key=/etc/squid/localhost.key vhost

This sets the http and https ports squid is listening on. Note the cert options for https, we can get squid to use https up to the proxy and unencrytped link to the last hop if we want.. which is cool. If for some reason the server doesn’t support https.



cache_effective_user squid

cache_effective_group squid



Set the effective user and group for squid.. this may not be required, but doesn’t hurt.



cache_peer 1.2.3.4 parent 80 0 no-query originserver name=site1-http

cache_peer 1.2.3.5 parent 443 0 no-query originserver ssl sslflags=DONT_VERIFY_PEER name=site2-ssl

cache_peer_domain site1-http site1.example.lan

cache_peer_domain site2-ssl site2.anotherexample.lan

This is the magic, the first two lines, tell squid which peer to reverse proxy for and what port to use. Note if you use ssl the ‘sslflags=DONT_VERIFY_PEER’ is useful otherwise if your using a self signed cert you’ll have certificate errors.

IMPORTANT: If you want to allow http authentication (auth handled by the web server, such as htaccess) then you need to add ‘login=PASS’ otherwise squid will try and authenticate to squid rather than the http server.

The last two lines, reference the first two and tell squid the domains to listen to, so if someone connects to squid looking for that domain it knows where to go/cache.



acl bad_requests urlpath_regex -i cmd.exe \/bin\/sh \/bin\/bash default\.ida?XXX insert update delete select

http_access deny bad_requests



NOTE: The acl line has been cut over two lines, this should be on one. There should be the ACL line and the http_access line.

These lines set up some bad requests to which we deny access to, this is to help prevent SQL injection, and other hack attempts, etc.

That’s it, after a (re)start to squid you it will be reverse proxying the domains.

Redirect to SSL

We had a requirement to automatically redirect to https if someone came in on http. Squid allows redirecting through a variety of ways, you can write a redirect script at get squid to use it, but there is a simpler way, using all squid internals and acls.

Add the following to the entries added in the last section:



acl port80 myport 80

acl site1 dstdomain site1.example.lan

http_access deny port80 site1

deny_info https://site1.example.lan/ site1

acl site2 dstdomain site2.anotherexample.lan

http_access deny port80 site2

deny_info https://site2.anotherexample.lan/ site2

We create an acl for the squids port 80 and then one for the domain we want to redirect. We then use "http_access deny" to cause squid to deny access to that domain coming in on port 80 (http). This causes a deny which is caught by the deny_info which redirects it to https.

The order used of the acl's in the http_access and the deny_info is important. Squid only remembers the last acl used by a http_access command and will look for a corresponding deny_info matched to that acl. So make sure the last acl matches the acl used in the deny_info statement!

NOTE: See http://www.squid-cache.org/Doc/config/deny_info/

Appendix

The following is the configuration all put together now.

Reverse proxy + redirection:



http_port 80 vhost

https_port 443 cert=/etc/squid/localhost.crt key=/etc/squid/localhost.key vhost

cache_effective_user squid

cache_effective_group squid

cache_peer 1.2.3.4 parent 80 0 no-query originserver login=PASS name=site1-http

cache_peer 1.2.3.5 parent 443 0 no-query originserver login=PASS ssl sslflags=DONT_VERIFY_PEER name=site2-ssl

cache_peer_domain site1-http site1.example.lan

cache_peer_domain site2-ssl site2.anotherexample.lan

acl bad_requests urlpath_regex -i cmd.exe \/bin\/sh \/bin\/bash default\.ida?XXX insert update delete select

http_access deny bad_requests

acl port80 myport 80

acl site1 dstdomain site1.example.lan

http_access deny port80 site1

deny_info https://site1.example.lan/ site1

acl site2 dstdomain site2.anotherexample.lan

http_access deny port80 site2

deny_info https://site2.anotherexample.lan/ site2

Posfix – Making sense of delays in mail

The maillog

The maillog is easy enough to follow, but when you understand what all the delay and delays numbers mean then this may help really understand what is going on!

A standard email entry in postfix looks like:



Jan 10 10:00:00 testmtr postfix/smtp[20123]: 34A1B160852B: to=, relay=mx1.example.lan[1.2.3.4]:25, delay=0.49, delays=0.2/0/0.04/0.25, dsn=2.0.0, status=sent

Pretty straight forward: date, email identifier in the mailq (34A1B160852B), recipient, which server the email is being sent to (relay). It is the delay and delays I’d like to talk about.

Delay and Delays

If we take a look at the example email from above:



Jan 10 10:00:00 testmtr postfix/smtp[20123]: 34A1B160852B: to=, relay=mx1.example.lan[1.2.3.4]:25, delay=0.49, delays=0.2/0/0.04/0.25, dsn=2.0.0, status=sent

The delay parameter (delay=0.49) is fairly self explanatory, it is the total amount of time this email (34A1B160852B) has been on this server. But what is the delays parameter all about?



delays=0.2/0/0.04/0.25



NOTE: Numbers smaller than 0.01 seconds are truncated to 0, to reduce the noise level in the logfile.

You might have guessed it is a break down of the total delay, but what do each number represent?

Well from the release notes we get:



delays=a/b/c/d:

a=time before queue manager, including message transmission;

b=time in queue manager;

c=connection setup time including DNS, HELO and TLS;

d=message transmission time.

There for looking at our example:

  • a (0.2): The time before getting to the queue manager, so the time it took to be transmitted onto the mail server and into postfix.
  • b (0): The time in queue manager, so this email didn’t hit the queues, so it was emailed straight away.
  • c (0.04): The time it took to set up a connection with the destination mail relay.
  • d (0.25): The time it took to transmit the email to the destination mail relay.

However if the email is deferred, then when the email is attempted to be sent again:



Jan 10 10:00:00 testmtr postfix/smtp[20123]: 34A1B160852B: to=, relay=mx1.example.lan[1.2.3.4]:25, delay=82, delays=0.25/0/0.5/81, dsn=4.4.2, status=deferred (lost connection with mx1.example.lan[1.2.3.4] while sending end of data -- message may be sent more than once)

Jan 10 testmtr postfix/smtp[20123]: 34A1B160852B: to=, relay=mx1.example.lan[1.2.3.4]:25, delay=1092, delays=1091/0.2/0.8/0.25, dsn=2.0.0, status=sent

This time the first entry shows how long it took before the destination mail relay took to time out and close the connection:



delays=0.25/0/0.5/81

Therefore: 81 seconds.

The email was deferred then about 15 minutes later (1009 seconds [delays - <total delay from last attempt> ]) another attempt is made.

This time the delay is a lot larger, as the total time this email has spent on the server is a lot longer.

delay=1092, delays=1091/0.2/0.8/0.25



What is interesting though is the value of ‘a’ is now 1091, which means when an email is resent the ‘a’ value in the breakdown also includes the amount of time this email has currently spend on the system (before this attempt).

So there you go, those delays values are rather interesting and can really help solve where bottlenecks lie on your system. In the above case we obviously had some problem communicating to the destination mail relay, but worked the second time, so isn’t a problem with our system… or so I’d like to think.

Use xmllint and vim to format xml documents

If you want vim to nicely format an XML file (and a xena file in this example, 2nd line) then add this to your ~/.vimrc file:

" Format *.xml and *.xena files by sending them to xmllint

au FileType xml exe ":silent 1,$!xmllint --format --recover - 2>/dev/null"

au FileType xena exe ":silent 1,$!xmllint --format --recover - 2>/dev/null"



This uses the xmllint command to format the xml file.. useful on xml docs that aren’t formatted in the file.

Debian 6 GNU/KFreeBSD Grub problems on VirtualBox

Debian 6 was released the other day, with this release they not only released a Linux kernel version but they now support a FreeBSD version as well!

So I decided to install it under VirtualBox and check it out…

The install process went smoothly until I got to the end when it was installing and setting up grub2. It installed ok on the MBR but got an error in the installer while trying to set it up. I jumped into the console to take a look around.

I started off trying to run the update-grub command which fails silently (checking $? shows the return code of 1). On closer inspection I noticed the command created an incomplete grub config named /boot/grub/grub.cfg.new

So all we need to do is finish off this config file. So jump back into the installer and select continue without boot loader, this will pop up a message about what you must set the root partition as when you do set up a boot loader, so take note of it.. mine was /dev/ad0s5.

OK, with that info we can finish off our config file. Firstly lets rename the incomplete one:

cp /boot/grub/grub.cfg.new /boot/grub/grub.cfg

Now my /boot/grub/grub.cfg ended like:

### BEGIN /etc/grub.d/10_kfreebsd ###

menuentry 'Debian GNU/kFreeBSD, with kFreeBSD 8.1-1-amd64' --class debian --class gnu-kfreebsd --class gnu --class os {

insmod part_msdos

insmod ext2




set root='(hd0,1)'

search --no-floppy --fs-uuid --set dac05f8a-2746-4feb-a29d-31baea1ce751

echo 'Loading kernel of FreeBSD 8.1-1-amd64 ...'

kfreebsd /kfreebsd-8.1-1-amd64.gz

So I needed to add the following to finish it off (note this I’ll repeat that last part):

### BEGIN /etc/grub.d/10_kfreebsd ###

menuentry 'Debian GNU/kFreeBSD, with kFreeBSD 8.1-1-amd64' --class debian --class gnu-kfreebsd --class gnu --class os {

insmod part_msdos

insmod ext2

insmod ufs2




set root='(hd0,1)'

search --no-floppy --fs-uuid --set dac05f8a-2746-4feb-a29d-31baea1ce751

echo 'Loading kernel of FreeBSD 8.1-1-amd64 ...'

kfreebsd /kfreebsd-8.1-1-amd64.gz

set kFreeBSD.vfs.root.mountfrom=ufs:/dev/ad0s5

set kFreeBSD.vfs.root.mountfrom.options=rw

}

Note: My root filesytem was UFS, thus the ‘ufs:/dev/ad0s5′ in the mountfrom option.

That’s it, you Debian GNU/kFreeBSD should now boot successfully :)

Links October 2014

The Verge has an interesting article about Tim Cook (Apple CEO) coming out [1]. Tim says “if hearing that the CEO of Apple is gay can help someone struggling to come to terms with who he or she is, or bring comfort to anyone who feels alone, or inspire people to insist on their equality, then it’s worth the trade-off with my own privacy”.

Graydon2 wrote an insightful article about the right-wing libertarian sock-puppets of silicon valley [2].

George Monbiot wrote an insightful article for The Guardian about the way that double-speak facilitates killing people [3]. He is correct that the media should hold government accountable for such use of language instead of perpetuating it.

Anne Thériault wrote an insightful article for Vice about the presumption of innocence and sex crimes [4].

Dr Nerdlove wrote an interesting article about Gamergate as the “extinction burst” of “gamer culture” [5], we can only hope.

Shweta Narayan wrote an insightful article about Category Structure and Oppression [6]. I can’t summarise it because it’s a complex concept, read the article.

Some Debian users who don’t like Systemd have started a “Debian Fork” project [7], which so far just has a web site and nothing else. I expect that they will never write any code. But it would be good if they did, they would learn about how an OS works and maybe they wouldn’t disagree so much with the people who have experience in developing system software.

A GamerGate terrorist in Utah forces Anita Sarkeesian to cancel a lecture [8]. I expect that the reaction will be different when (not if) an Islamic group tries to get a lecture cancelled in a similar manner.

Model View Culture has an insightful article by Erika Lynn Abigail about Autistics in Silicon Valley [9].

Katie McDonough wrote an interesting article for Salon about Ed Champion and what to do about men who abuse women [10]. It’s worth reading that while thinking about the FOSS community…

Samsung Galaxy Note 3

In June last year I bought a Samsung Galaxy Note 2 [1]. Generally I was very happy with that phone, one problem I had is that less than a year after purchasing it the Ingress menus burned into the screen [2].

2 weeks ago I bought a new Galaxy Note 3. One of the reasons for getting it is the higher resolution screen, I never realised the benefits of a 1920*1080 screen on a phone until my wife got a Nexus 5 [3]. I had been idly considering a Galaxy Note 4, but $1000 is a lot of money to pay for a phone and I’m not sure that a 2560*1440 screen will offer much benefit in that size. Also the Note 3 and Note 4 both have 3G of RAM, as some applications use more RAM when you have a higher resolution screen the Note 4 will effectively have less usable RAM than the Note 3.

My first laptop cost me $3,800 in 1998, that’s probably more than $6,000 in today’s money. The benefits that I receive now from an Android phone are in many ways greater than I received from that laptop and that laptop was definitely good value for money for me. If the cheapest Android phone cost $6,000 then I’d pay that, but given that the Note 3 is only $550 (including postage) there’s no reason for me to buy something more expensive.

Another reason for getting a new phone is the limited storage space in the Note 2. 16G of internal storage is a limit when you have some big games installed. Also the recent Android update which prevented apps from writing to the SD card meant that it was no longer convenient to put TV shows on my SD card. 32G of internal storage in the Note 3 allows me to fit everything I want (including the music video collection I downloaded with youtube-dl). The Note 2 has 16G of internal storage and an 8G SD card (that I couldn’t fully use due to Android limitations) while the Note 3 has 32G (the 64G version wasn’t on sale at any of the cheap online stores). Also the Note 3 supports an SD card which will be good for my music video collection at some future time, this is a significant benefit over the Nexus 5.

In the past I’ve written about Android service life and concluded that storage is the main issue [4]. So it is a bit unfortunate that I couldn’t get a phone with 64G of storage at a reasonable price. But the upside is that getting a cheaper phone allows me to buy another one sooner and give the old phone to a relative who has less demanding requirements.

In the past I wrote about the warranty support for my wife’s Nexus 5 [5]. I should have followed up on that before, 3 days after that post we received a replacement phone. One good thing that Google does is to reserve money on a credit card to buy the new phone and then send you the new phone before you send the old one back. So if the customer doesn’t end up sending the broken phone they just get billed for the new phone, that avoids excessive delays in getting a replacement phone. So overall the process of Google warranty support is really good, if 2 products are equal in other ways then it would be best to buy from Google to get that level of support.

I considered getting a Nexus 5 as the hardware is reasonably good (not the greatest but quite good enough) and the price is also reasonably good. But one thing I really hate is the way they do the buttons. Having the home button appear on the main part of the display is really annoying. I much prefer the Samsung approach of having a hardware button for home and touch-screen buttons outside the viewable area for settings and back. Also the stylus on the Note devices is convenient on occasion.

The Note 3 has a fake-leather back. The concept of making fake leather is tacky, I believe that it’s much better to make honest plastic that doesn’t pretend to be something that it isn’t. However the texture of the back improves the grip. Also the fake stitches around the edge help with the grip too. It’s tacky but utilitarian.

The Note 3 is slightly smaller and lighter than the Note 2. This is a good technical achievement, but I’d rather they just gave it a bigger battery.

Update USB 3

One thing I initially forgot to mention is that the Note 3 has USB 3. This means that it has a larger socket which is less convenient when you try and plug it in at night. USB 3 seems unlikely to provide any benefit for me as I’ve never had any of my other phones transfer data at rates more than about 5MB/s. If the Note 3 happens to have storage that can handle speeds greater than the 32MB/s a typical USB 2 storage device can handle then I’m still not going to gain much benefit. USB 2 speeds would allow me to transfer the entire contents of a Note 3 in less than 20 minutes (if I needed to copy the entire storage contents). I can’t imagine myself having a real-world benefit from that.

The larger socket means more fumbling when charging my phone at night and it also means that the Note 3 cable can’t be used in any other phone I own. In a year or two my wife will have a phone with USB 3 support and that cable can be used for charging 2 phones. But at the moment the USB 3 cable isn’t useful as I don’t need to have a phone charger that can only charge one phone.

Conclusion

The Note 3 basically does everything I expected of it. It’s just like the Note 2 but a bit faster and with more storage. I’m happy with it.

Terry 2.0 includes ROS!

What started as a little tinker around the edges has resulted in many parts of Terry being updated. The Intel j1900 motherboard is now mounted in the middle of the largest square structure, and SSD is mounted (the OCZ black drive at the bottom), yet another battery is mounted which is a large external laptop supply, the Kinect is now mounted on the pan and tilt mechanism along with the 1080p webcam that was previously there. The BeagleBone Black is moved to its own piece of channel and a breadboard is sunk into the main 2nd top level channel.





I haven't cabled up the j1900 yet. On the SSD is Ubuntu and ROS including a working DSLAM (strangely some fun and games getting that to compile and then to not segv right away).



I used 3 Actobotics Beams, one IIRC is a 7.7 incher and two shorter ones. The long beam actually lines on for the right side of the motherboard that you see in the image. The beam is attached with nylon bolts and has a 6.6mm standoff between the motherboard and the beam to avoid any undesired electrical shorts. With the two shorter beams on the left side of the motherboard it is rather securely attached to Terry now. The little channel you see on the right side up a little from the bottom is there for the 7.7 inch beam to attach to (behind the motherboard) and there is a shorter beam on this side to secure the floating end of the channel to the base channel.







The alloy structure at the top of the pan and tilt now has a Kinect attached. I used a wall mount plastic adaptor which with great luck and convenience the nut traps lined up to the actobotics holes. I have offset the channel like you see so that the center of gravity is closer to directly above the pan and tilt. Perhaps I will have to add some springs to help the tilt servo when it moves the Kinect back too far from the centre point. I am also considering a counter balance weight below the tilt which would also work to try to stabilize the Kinect at the position shown.







I was originally planning to put some gripper on the front of Terry. But now I'm thinking about using the relatively clean back channel to attach a threaded rod and stepper motor so that the gripper can have access to the ground and also table top. Obviously the cameras would have to rotate 180 degrees to be able to see what the gripper was up to. Also for floor pickups the tilt might have to be able to handle a reasonable downward "look" without being too hard on the servo.



There were also some other tweaks. A 6 volt regulator is now inlined into a servo extension cable and the regulator is itself bolted to some of the channel. Nice cooling, and it means that the other end of that servo extension can take something like 7-15v and it will give the servo the 6v it wants. That is actually using the same battery pack as the drive wheels (8xAA).



One thing that might be handy for others who find this post, the BeagleBone Black Case from sparkfun attaches to Actobotics channel fairly easily. I used two cheesehead m3 nylocks and had to force them into the enclosure. The nylocks lined up to the Actobotics channel and so the attachment was very simple. You'll want a "3 big hole" or more bit of channel to attach the enclosure to. I attached it to a 3 bit holer and then attaced that channel to the top of Terry with a few threaded standoffs. Simplifies attach and remove should that ever be desired.



I know I need slip rings for the two USB cameras up top. And for the tilt servo as well. I can't use a USB hub up top because both the USB devices can fairly well saturate a USB 2.0 bus. I use the hardware encoded mjpeg from the webcam which helps bandwidth, but I'm going to give an entire USB 2.0 bus to the Kinect.



Keynote Speaker - Professor Eben Moglen

Eben Moglen

The LCA 2015 team is honoured to announce our first Keynote speaker - Professor Eben Moglen, Executive Director of the Software Freedom Law Center and professor of Law and Legal History at Columbia University Law School.

Professor Moglen's presentation is scheduled for 09:00 am Tuesday, 13 January 2015

Professor Moglen has represented many of the world's leading free software developers. He earned his PhD in History and his law degree at Yale University during what he sometimes calls his “long, dark period” in New Haven.

After law school he clerked for Judge Edward Weinfeld of the United States District Court in New York City and for Justice Thurgood Marshall of the United States Supreme Court. He has taught at Columbia Law School since 1987 and has held visiting appointments at Harvard University, Tel Aviv University and the University of Virginia.

In 2003 he was given the Electronic Frontier Foundation's Pioneer Award for efforts on behalf of freedom in the electronic society.

We are especially grateful to Michael Davies for his efforts in bringing Professor Moglen to LCA 2015 in Auckland for us - thank you Michael!

The LCA 2015 Auckland Team

October 30, 2014

2014 GStreamer Conference

I’ve been home from Europe over a week, after heading to Germany for the annual GStreamer conference and Linuxcon Europe.

We had a really great turnout for the GStreamer conference this year

GstConf2k14

as well as an amazing schedule of talks. All the talks were recorded by Ubicast, who got all the videos edited and uploaded in record time. The whole conference is available for viewing at http://gstconf.ubicast.tv/channels/#gstreamer-conference-2014

I gave one of the last talks of the schedule – about my current work adding support for describing and handling stereoscopic (3D) video. That support should land upstream sometime in the next month or two, so more on that in a bit.

elephant

There were too many great talks to mention them individually, but I was excited by 3 strong themes across the talks:

  • WebRTC/HTML5/Web Streaming support
  • Improving performance and reducing resource usage
  • Building better development and debugging tools

I’m looking forward to us collectively making progress on all those things and more in the upcoming year.

[life] Day 274: Errands, friends old and new, and swim class

In researching ways to try and help Zoe sleep for longer, I learned that there's basically two triggers for waking up in the morning: light and heat. Because Queenslanders hate daylight saving, the sun gets up ridiculously early in summer. Because Queensland is hot, it also gets very hot pretty early. Our bedrooms are on the eastern side of the apartment to boot.

I already have nice blackout curtains, and I had pelmets installed last summer to try and reduce the light leakage around the curtains. I also had reflective window film put on our bedroom windows last summer in an effort to reduce the morning heat when the sun rose, but I don't think it's made a massive difference to a closed up bedroom. I think Zoe woke up at about 5:40am this morning. I'm not sure what the room temperature was, because the Twine in her room decided not to log it this morning. Air conditioning is the next thing to try.

After breakfast, we ran a few errands, culminating at a trip to the carwash for babyccino. After that, we headed over to Toowong to pick up Geneal, who was a friend of my biological mother that I've kept in loose contact since I've been an adult. We went over to the Toowong Bowls Club for lunch, and had a nice catch up.

The Toowong Bowls Club has a rather disturbing line on the wall showing the height of the 2011 floods. It's probably taller than my raised arm from the ground level of the building.

After lunch, and dropping Geneal home, we headed over for a play date at the home of Chloe, who will be starting Prep next year at Zoe's school. I met Chloe's Mum, Kelley, at the P&C meeting I went to earlier in the year, and then proceeded to continue to bump into her at numerous school-related things ever since. She's been a good person to know, having an older daughter at the school as well, and has given me lots of advice.

Zoe and Chloe got along really well, and Chloe seems like a nice kid. After the play date, we walked to school to collect Chloe's older sister, and then to swim class. We were early, but Zoe was happy to hang out.

I am just so loving the vibe I'm getting about the school, and really loving the school community itself. I'm really looking forward to the next seven years here.

After swim class, we walked back to Chloe's house to retrieve the car, and say goodbye to Chloe, and headed home. It was another nice full, but not too full day.

LUV Main November 2014 Meeting: Raspberry Pi update + systemd

Nov 5 2014 19:00
Nov 5 2014 21:00
Nov 5 2014 19:00
Nov 5 2014 21:00
Location: 

The Buzzard Lecture Theatre. Evan Burge Building, Trinity College, Melbourne University Main Campus, Parkville.

Please note that the November meeting is on Wednesday night rather than Tuesday night due to the Melbourne Cup.

Alec Clews, Raspberry Pi update

Russell Coker, systemd

The Buzzard Lecture Theatre, Evan Burge Building, Trinity College Main Campus Parkville Melways Map: 2B C5

Notes: Trinity College's Main Campus is located off Royal Parade. The Evan Burge Building is located near the Tennis Courts. See our Map of Trinity College. Additional maps of Trinity and the surrounding area (including its relation to the city) can be found at http://www.trinity.unimelb.edu.au/about/location/map

Parking can be found along or near Royal Parade, Grattan Street, Swanston Street and College Crescent. Parking within Trinity College is unfortunately only available to staff.

For those coming via Public Transport, the number 19 tram (North Coburg - City) passes by the main entrance of Trinity College (Get off at Morrah St, Stop 12). This tram departs from the Elizabeth Street tram terminus (Flinders Street end) and goes past Melbourne Central Timetables can be found on-line at:

http://www.metlinkmelbourne.com.au/route/view/725

Before and/or after each meeting those who are interested are welcome to join other members for dinner. We are open to suggestions for a good place to eat near our venue. Maria's on Peel Street in North Melbourne is currently the most popular place to eat after meetings.

LUV would like to acknowledge Red Hat for their help in obtaining the Buzzard Lecture Theatre venue and VPAC for hosting, and BENK Open Systems for their financial support of the Beginners Workshops

Linux Users of Victoria Inc., is an incorporated association, registration number A0040056C.

November 5, 2014 - 19:00

read more

October 29, 2014

New libeatmydata release: 105

Over on the project page and on launchpad you can now download libeatmydata 105.

This release fixes a couple of bugs that came in via the Debian project, including a rather interesting one about some binaries not running .so ctors to properly init libeatmydata and the code path in the libeatmydata open() not really dealing with being called first in this situation.

Enjoy!

Speaker Feature: Meg Howie, Joshua Hesketh

Meg Howie

Meg Howie

Ask Away: Staking Out the Stakeholders

11:35am Friday 16th January 2015

Meg is a designer and thinker whose practice spans graphic, interactive, film, service and performance design. She is currently undertaking a Master of Design at Massey University and her research explores the influence of open source culture and participatory democracy on civic engagement. Meg’s work is deeply social, and draws from human-centred design, behavioural psychology and collaborative modes of working.

For more information on Meg and her presentation, see here. You can follow her as @howiemeg and don’t forget to mention #LCA2015.



Joshua Hesketh

Joshua Hesketh

Who is Linux Australia?

3:40pm Thursday 15th January 2015

Joshua is a software developer for Rackspace Australia working on upstream OpenStack. He works from his home in Hobart, Tasmania. Joshua is currently President of Linux Australia, previously the co-chair for PyCon Australia and a key organiser for linux.conf.au. He has an interest in robotics having recently completed a degree in mechatronic engineering. Josh is an active contributor to the openstack-infra and nova projects.

For more information on Josh and his presentation, see here.

[life] Day 273: Kindergarten, more startup stuff, and another Prep day

I had another busy day today. I've well and truly falled off the running wagon, which I really need to fix rather urgently. I would have liked to have gone for a run this morning, but it didn't happen.

I started off with a chiropractic adjustment, and then a bit of random cooking to use up some perishables, before the cleaners arrived.

While the cleaners were here, I managed to knock over another unit of my real estate course, which I was pretty stoked about. I'll try and get it in the mail tomorrow, and that's the last one from the first half of the course done.

I grabbed a massage, and then headed over to pick up Zoe early from Kindergarten to take her to school for another Prep introduction session. I really like Zoe's school. This year for the first time they're running a four week program where the kids can come for a couple of hours.

Today it was fine and gross motor skills. They divided the group in half, and Zoe's half did fine motor skills first. The kids rotated through three different stations, which all had three or four activities each. Zoe did pretty well with these.

Then the groups swapped over, and we returned to the hall where we started, to do some gross motor skills. I would have thought this would have been right up Zoe's alley, since a lot of it was similar to TumbleTastics, but she was very clingy, and they kept rotating between stations faster than she got warmed up to the activity.

She was a bit overwhelmed in the larger group setting in general. Hopefully next week with a bit of preparation before we come (and no Kindergarten) she'll do better.

After we got home, I showed Zoe a balloon full of water that I'd put in the freezer. She had a great time smashing it on the balcony. I'll have to do that again.

It's a hot night tonight, I hope Zoe sleeps okay. It was definitely time to bust out the fan.

Training and Education in High Performance Computing for eReseachers

"Big data" requires processing. Processing requires HPC. Increased processing results in increased research output. Research organisations that do not increase HPC usage will fall behind. HPC requires either 'dumb down the interface or skill up the user'. Making "user friendly" interfaces may not be the right path to take as HPC use will always have a minimum level of complexity. Training courses that use andragogical technqiues correlate with increased HPC use.

Presentation to eResearch Australasia, Melbourne, October 28, 2014

October 28, 2014

Speaker Feature: Christoph Lameter, Brandon Philips

Christoph Lameter

Christoph Lameter

SL[AUO]B:Kernel memory allocator design and philosophy

12:15pm Friday 16th January 2015

Christoph specializes in High Performance Computing and High Frequency Trading technologies. As an operating system designer and kernel developer he has been developing memory management technologies for Linux to enhance performance and reduce latencies. He is fond of new technologies and new ways of thinking that disrupt existing industries and causes new development communities to emerge.

For more information on Christoph and his presentation, see here. You can follow him as @qant and don’t forget to mention #lca2015.



Brandon Philips

Brandon Philips

CoreOS: An introduction

10:40 am Wednesday 14th January 2015

Brandon Philips is helping to build modern Linux server infrastructure at CoreOS. Prior to CoreOS, he worked at Rackspace hacking on cloud monitoring and was a Linux kernel developer at SUSE. In addition to his work at CoreOS, Brandon sits on Docker's governance board and is one of the top contributors to Docker. As a graduate of Oregon State's Open Source Lab he is passionate about open source technologies.

Brandon has also been a speaker at many conferences including Open Source Bridge 2012 and Open Source Conference 2012.

For more information on Brandon and his presentation, see here. You can follow him as @BrandonPhilips and don’t forget to mention #lca2015.

[life] Day 272: Kindergarten, startup stuff

I had a great, productive day today.

I got stuck into my real estate licence coursework this morning, and finished off a unit. I biked down to the post office to mail it off, and picked up the second half of my coursework. After I finish the unit I started today, I'll have 8 more units to go. Looking at the calendar, if I can punch out a unit a week (which is optimistic, particularly considering that school holidays are approaching) I could be finished by the end of the year. More realistically, I can try to be finished by the time Zoe starts school, which will be perfect, and well inside the 12 month period I'm supposed to get it done in. We shall see how things pan out.

I biked to Kindergarten to pick up Zoe, and she wanted to watch Megan's tennis class for a while, so we hung around. She was pretty wiped out from a water play day at Kindergarten today. We biked home, and then she proceeded to eat everything in the house that wasn't tied down until Sarah arrived to pick her up.

I used the rest of the afternoon to do some more administrative stuff and tidy up a bit, before heading off to my yoga class. I had a really lovely stretch class with just me and my yoga teacher, so we spent the whole class chatting and having a great catch up. It was a great way to end the day.

[life] Day 271: Kindergarten, lots of administrivia and some tinkering

Zoe woke up at about 6am, which gave us a bit of extra time to get moving in the morning, or so I thought.

We biked over to the Kindergarten for drop off, and I left the trailer there to make biking back in the afternoon heat easier.

I had a pretty productive day. It was insanely hot, so I figured I could run the air conditioning more or less guilt (and expense free) courtesy of my solar power. I should check just how much power it draws to see how "free" it is to run.

I mostly cleared lots of random stuff off my to do list, and made a few lengthy phone calls. I also did some more tinkering with my BeagleBone Black, trying to get it set up so I can back up daedalus. It's been fun playing with Puppet again. I now have a pretty nice set up where I can wipe the BeagleBone Black and get it back to how I want it configured in about 5 minutes, thanks to Puppet.

I biked over to Kindergarten to pick up. I got there a few minutes early, and received a very heartening phone call regarding an issue I'd been working on earlier.

Zoe and Megan wanted to have a play date, and since it was hot and I'd left the air conditioning on, I suggested it be at our place. I biked home, and Jason dropped Megan around.

The girls played inside for a bit, but then wanted to do some more craft on the balcony, so I let them get to it, with instructions to put stuff away before they took more stuff out, and the balcony ended up significantly cleaner as a result. I used the time to do some more tinkering with my backups and to book a flight down to Sydney to help a friend out with some stuff.

A massive storm rolled in, not long after Anshu arrived, so we all went out on the balcony to watch the lightning, and then Sarah arrived to pick up Zoe. Megan hung out for a bit longer until Jason arrived to pick her up.

October 27, 2014

Speaker Feature: Lillian Grace, David Rowe

Lillian Grace

Lillian Grace

Wiki New Zealand: Winning through collaboration

4:35pm Thursday 15th January 2015

Lillian is the founder and chief of Wiki New Zealand.

Wiki New Zealand is a collaborative website making data about New Zealand visually accessible to everyone. The site presents data in simple, visual form only, so that it remains as unbiased and as accessible to everyone as possible. The content is easy to understand and digest, and is presented from multiple angles, wide contexts and over time, inviting users to compare, contrast and interpret. Lillian is an accomplished presenter who was invited to speak at OSDC 2013, was a keynote speaker at Gather 2014 and a speaker at TEDx Auckland 2013.

For more information on Lillian and her presentation, see here. You can follow her as @GracefulLillian and don’t forget to mention #LCA2015.



David Rowe

David Rowe

The Democratisation of Radio

10:40am Thursday 15th January 2015

David is an electronic engineer living in Adelaide, South Australia. His mission is to improve the world – just a little bit, through designing open hardware and writing open source software for telephony.

In January 2006 David quit corporate life as an Engineering Manager to become an open source developer. He now develops open telephony hardware and software full time. David likes to build advanced telephony technology – then give it away.

For more information on David and his presentation, see here. You can follow him as @davidgrowe67 and don’t forget to mention #LCA2015.

Linux Security Summit 2014 Wrap-Up

The slides from the 2014 Linux Security Summit in August may be found linked at the schedule.

LWN covered both the James Bottomley keynote, and the SELinux on Android talk by Stephen Smalley.

We had an engaging and productive two days, with strong attendance throughout.  We’ll likely follow a similar format next year at LinuxCon.  I hope we can continue to expand the contributor base beyond mostly kernel developers.  We’re doing ok, but can certainly do better.  We’ll also look at finding a sponsor for food next year.

Thanks to those who contributed and attended, to the program committee, and of course, to the events crew at Linux Foundation, who do all of the heavy lifting logistics-wise.

See you next year!

Speaker Feature: Lana Brindley & Alexandra Settle, Olivier Bilodeau

Lana Brindley and Alexandra Settle

Alexandra Settle Lana Brindley

8 writers in under 8 months: from zero to a docs team in no time flat

11:35am Thursday 15th January 2015

Lana and Alexandra are both technical writers at Rackspace, the open Cloud Company.

Lana has been writing open source technical documentation for about eight years, and right now I’m working on documenting OpenStack with Rackspace, she does a lot of speaking, mostly about writing. She also talks about other topics from open source software to geek feminism and working in IT.

Lana is also involved in several volunteer projects including linux.conf.au, Girl Geek Dinners, LinuxChix, OWOOT (Oceania Women of Open Tech), and various Linux Users Groups (LUGs). Alexandra is a technical writer with the Rackspace Cloud Builders Australia team. She began her career as a writer for the cloud documentation team at Red Hat, Australia. Alexandra prefers Fedora over other Linux distributions.

Recently she was part of a team that authored the OpenStack Design Architecture Guide, and hopes to further promote involvement in the OpenStack community within Australia.

For more information on Lana and Alexandra and their presentation, see here. You can follow them as @Loquacities (Lana) or @dewsday (Alexandra) and don’t forget to mention #LCA2015.



Olivier Bilodeau

Olivier Bilodeau

Advanced Linux Server-Side, Threats: How they work and what you can do about them

1:20pm Friday 16th January 2015

Olivier is an engineer that loves technology, software, security, open source, linux, brewing beer, travels and android.

Coming from the dusty Unix server room world, Olivier evolved professionally in networking, information security and open source software development to finally become malware researcher at ESET Canada. Presenting at Defcon, publishing in (In)secure Mag, teaching infosec to undergrads (ÉTS), driving the NorthSec Hacker Jeopardy and co-organizer of the MontréHack training initiative are among its note-worthy successes.

For more information on Olivier and his presentation, see here. You can follow him as @obilodeau and don’t forget to mention #LCA2015.

October 26, 2014

Twitter posts: 2014-10-20 to 2014-10-26

October 25, 2014

Craige McWhirter: Automating Building and Synchronising Local & Remote Git Repos With Github

I've blogged about some git configurations in the past. In particular working with remote git repos.

I have a particular workflow for most git repos

  • I have a local repo on my laptop
  • I have a remote git repo on my server
  • I have a public repo on Github that functions as a back up.

When I push to my remote server, a post receive hook automatically pushes the updates to Github. Yay for automation.

However this wasn't enough automation, as I found myself creating git repos and running through the setup steps more often than I'd like. As a result I created gitweb_repo_build.sh which takes all the manual steps I go through to setup my workflow and automates it.

The script currently does the following:

  • Builds a git repo locally
  • Adds a README.mdwn and a LICENCE. Commits the changes.
  • Builds a git repo hosted via your remote git server
  • Adds to the remote server, a git hook for automatically pushing to github
  • Adds to the remote server, a git remote for github.
  • Creates a repo at GitHub a via API 3
  • Pushes the READEME and LICENCE to the remote, which pushes to github.

It's currently written in bash and has no error handling.

I've planned a re-write in Haskell which will have error handling.

If this is of use to you, enjoy :-)

That rare feeling …

… of actually completing things.

Upon reflection, it appears to have been a sucessful week.

Work – We relocated offices (including my own desk (again)) over the previous week from one slightly pre-used office building to another more well-used office building. My role as part of this project was to ensure that the mechanics of the move as far as IT and Comms occured and proceed smoothly. After recabling the floor, working with networks, telephones and desktops staff it was an almost flawless move, and everyone was up and running easily on Monday morning. I received lots of positive feedback which was good.

Choir – The wrap up SGM for the 62nd Australian Intervarsity Choral Festival Perth 2011, Inc happened. Pending the incorporation of the next festival, it is all over bar a few cheques and paperwork. Overall it was a great festival and as Treasurer was pleased with the final financial result (positive).

Hacking – This weeks little project has been virtualsnack. This is a curses emulator of the UCC Snack Machine and associated ROM. It is based on a previous emulator written with PyGTK and Glade that had bitrotted in the past ten years to be non-functioning and not worth the effort to ressurect. The purpose of the emulator is enable development of code to speak to the machine without having to have the real machine available to test against.

I chose to continue to have the code in python and used npyscreen as the curses UI library. One of the intermediate steps was creating a code sample, EXAMPLE-socket.py, which creates a daemon that speaks to a curses interfaces.

I hereby present V1.0 “Gobbledok” of virtualsnack. virtualsnack is hosted up on Github for the moment, but may move in future. I suspect this item of software will only be of interest to my friends at UCC.

October 24, 2014

[life] Day 268: Science Friday, TumbleTastics, haircuts and a big bike outing

I didn't realise how jam packed today was until we sat down at dinner time and recounted what we'd done today.

I started the day pretty early, because Anshu had to be up for an early flight. I pottered around at home cleaning up a bit until Sarah dropped Zoe off.

After Zoe had watched a bit of TV, I thought we'd try some bottle rocket launching for Science Friday. I'd impulse purchased an AquaPod at Jaycar last year, and haven't gotten around to using it yet.

We wandered down to Hawthorne Park with the AquaPod, an empty 2 litre Sprite bottle, the bicycle pump and a funnel.

My one complaint with the AquaPod would have to be that the feet are too smooth. If you don't tug the string strongly enough you end up just dragging the whole thing across the ground, which isn't really what you want to be doing. Once Zoe figured out how to yank the string the right way, we were all good.

We launched the bottle a few times, but I didn't want to waste a huge amount of water, so we stopped after about half a dozen launches. Zoe wanted to have a play in the playground, so we wandered over to that side of the park for a bit.

It was getting close to time for TumbleTastics, and we needed to go via home to get changed, so we started the longish walk back home. It was slow going in the mid-morning heat and no scooter, but we got there eventually. We had another mad rush to get to TumbleTastics on time, and miraculously managed to make it there just as they were calling her name.

Lachlan wasn't there today, and I was feeling lazy, and Zoe was keen for a milkshake, so we dropped into Ooniverse on the way home. Zoe had a great old time playing with everything there.

After we got home again, we biked down to the Bulimba post office to collect some mail, and then biked over for a haircut.

After our haircuts, Zoe wanted to play in Hardcastle Park, so we biked over there for a bit. I'd been wanting to go and check out the newly opened Riverwalk and try taking the bike and trailer on a CityCat. A CityCat just happened to be arriving when we got to the park, but Zoe wasn't initially up for it. As luck would have it, she changed her mind as the CityCat docked, but it was too late to try and get on that one. We got on the next one instead.

I wasn't sure how the bike and the trailer were going to work out on the CityCat, but it worked out pretty well going from Hawthorne to New Farm Park. We boarded at Hawthorne from the front left hand side, and disembarked at New Farm Park from the front right hand side, so I basically just rolled the bike on and off again, without needing to worry about turning it around. It was a bit tight cornering from the pontoon to the gangway, but the deckhand helped me manoeuvre the trailer.

It was quite a nice little ride through the back streets of New Farm to get to the start of the Riverwalk, and we had a nice quick ride into the city. We biked all the way along the riverside through to the Old Botanic Gardens. We stopped for a little play in the playground that Zoe had played in the other weekend when we were wandering around for Brisbane Open House, and then continued through the gardens, over the Goodwill Bridge, and the bottom of the Kangaroo Point cliffs.

We wound our way back home through Dockside, and Mowbray Park and along the bikeway alongside Wynnum Road. It was a pretty huge ride, and I'm excited that it's opened up an easy way to access Southbank by bicycle. I'm looking forward to some bigger forays in the near future.

Watching Grass Grow

For Hackweek 11 I thought it’d be fun to learn something about creating Android apps. The basic training is pretty straightforward, and the auto-completion (and auto-just-about-everything-else) in Android Studio is excellent. So having created a “hello world” app, and having learned something about activities and application lifecycle, I figured it was time to create something else. Something fun, but something I could reasonably complete in a few days. Given that Android devices are essentially just high res handheld screens with a bit of phone hardware tacked on, it seemed a crime not to write an app that draws something pretty.

openSUSE wallpaperThe openSUSE desktop wallpaper, with its happy little Geeko sitting on a vine, combined with all the green growing stuff outside my house (it’s spring here) made me wonder if I couldn’t grow a little vine jungle on my phone, with many happy Geekos inhabiting it.

Android has OpenGL ES, so thinking that might be the way to go I went through the relevant lesson, and was surprised to see nothing on the screen where there should have been a triangle. Turns out the view is wrong in the sample code. I also realised I’d probably have to be generating triangle strips from curvy lines, then animating them, and the brain cells I have that were once devoted to this sort of graphical trickery are so covered in rust that I decided I’d probably be better off fiddling around with beziers on a canvas.

So, I created an app with a SurfaceView and a rendering thread which draws one vine after another, up from the bottom of the screen. Depending on Math.random() it extends a branch out to one side, or the other, or both, and might draw a Geeko sitting on the bottom most branch. Originally the thread lifecycle was tied to the Activity (started in onResume(), killed in onPause()), but this causes problems when you blank the screen while the app is running. So I simplified the implementation by tying the thread lifecycle to Surface create/destroy, at the probable expense of continuing to chew battery if you blank the screen while the app is active.

Then I realised that it would make much more sense to implement this as live wallpaper, rather than as a separate app, because then I’d see it running any time I used my phone. Turns out this simplified the implementation further. Goodbye annoying thread logic and lifecycle problems (although I did keep the previous source just in case). Here’s a screenshot:

Geeko Live Wallpaper

The final source is on github, and I’ve put up a release build APK too in case anyone would like to try it out – assuming of course that you trust me not to have built a malicious binary, trust github to host it, and trust SSL to deliver it safely ;-)

Enjoy!

Update 2014-10-27: The Geeko Live Wallpaper is now up on the Google Play store, although for some reason the “Live Wallpaper” category wasn’t available, so it’s in “Personalization” until (hopefully) someone in support gets back to me and tells me what I’m missing to get it into the right category.

Updated Update: Someone in support got back to me. “Live Wallpaper” can’t be selected as a category in the developer console, rather you have to wait for Google’s algorithms to detect that the app is live wallpaper and recategorize it automatically.

Specs for Kilo

Here's an updated list of the specs currently proposed for Kilo. I wanted to produce this before I start travelling for the summit in the next couple of days because I think many of these will be required reading for the Nova track at the summit.



API



  • Add instance administrative lock status to the instance detail results: review 127139 (abandoned).
  • Add more detailed network information to the metadata server: review 85673.
  • Add separated policy rule for each v2.1 api: review 127863.
  • Add user limits to the limits API (as well as project limits): review 127094.
  • Allow all printable characters in resource names: review 126696.
  • Expose the lock status of an instance as a queryable item: review 85928 (approved).
  • Implement instance tagging: review 127281 (fast tracked, approved).
  • Implement tags for volumes and snapshots with the EC2 API: review 126553 (fast tracked, approved).
  • Implement the v2.1 API: review 126452 (fast tracked, approved).
  • Microversion support: review 127127.
  • Move policy validation to just the API layer: review 127160.
  • Provide a policy statement on the goals of our API policies: review 128560.
  • Support X509 keypairs: review 105034.




Administrative



  • Enable the nova metadata cache to be a shared resource to improve the hit rate: review 126705 (abandoned).
  • Enforce instance uuid uniqueness in the SQL database: review 128097 (fast tracked, approved).




Containers Service







Hypervisor: Docker







Hypervisor: FreeBSD



  • Implement support for FreeBSD networking in nova-network: review 127827.




Hypervisor: Hyper-V



  • Allow volumes to be stored on SMB shares instead of just iSCSI: review 102190 (approved).




Hypervisor: Ironic







Hypervisor: VMWare



  • Add ephemeral disk support to the VMware driver: review 126527 (fast tracked, approved).
  • Add support for the HTML5 console: review 127283.
  • Allow Nova to access a VMWare image store over NFS: review 126866.
  • Enable administrators and tenants to take advantage of backend storage policies: review 126547 (fast tracked, approved).
  • Enable the mapping of raw cinder devices to instances: review 128697.
  • Implement vSAN support: review 128600 (fast tracked, approved).
  • Support multiple disks inside a single OVA file: review 128691.
  • Support the OVA image format: review 127054 (fast tracked, approved).




Hypervisor: libvirt







Instance features







Internal



  • Move flavor data out of the system_metdata table in the SQL database: review 126620 (approved).
  • Transition Nova to using the Glance v2 API: review 84887.




Internationalization



  • Enable lazy translations of strings: review 126717 (fast tracked).




Performance



  • Dynamically alter the interval nova polls components at based on load and expected time for an operation to complete: review 122705.




Scheduler



  • Add an IOPS weigher: review 127123 (approved).
  • Add instance count on the hypervisor as a weight: review 127871 (abandoned).
  • Allow limiting the flavors that can be scheduled on certain host aggregates: review 122530 (abandoned).
  • Convert the resource tracker to objects: review 128964 (fast tracked, approved).
  • Create an object model to represent a request to boot an instance: review 127610.
  • Decouple services and compute nodes in the SQL database: review 126895.
  • Implement resource objects in the resource tracker: review 127609.
  • Isolate the scheduler's use of the Nova SQL database: review 89893.
  • Move select_destinations() to using a request object: review 127612.




Security



  • Provide a reference implementation for console proxies that uses TLS: review 126958 (fast tracked).
  • Strongly validate the tenant and user for quota consuming requests with keystone: review 92507.




Tags for this post: openstack kilo blueprint spec

Related posts: One week of Nova Kilo specifications; Compute Kilo specs are open; On layers; Juno nova mid-cycle meetup summary: slots; My candidacy for Kilo Compute PTL; Juno nova mid-cycle meetup summary: nova-network to Neutron migration



Comment

October 23, 2014

[life] Day 267: An outing to the Valley for lunch, and swim class

I was supposed to go to yoga in the morning, but I just couldn't drag my sorry arse out of bed with my man cold.

Sarah dropped Zoe around, and she watched a bit of TV while we were waiting for a structural engineer to come and take a look at the building's movement-related issues.

While I was downstairs showing the engineer around, Zoe decided she'd watched enough TV and, remembering that I'd said we needed to tidy up her room the previous morning, but not had time to, took herself off to her room and tidied it up. I was so impressed.

After the engineer was finished, we walked to the ferry terminal to take the cross-river ferry over to Teneriffe, and catch the CityGlider bus to the Valley for another one of the group lunches I get invited to.

After lunch, we reversed our travel, dropping into the hairdresser on the way home to make an appointment for the next day. We grabbed a few things from the Hawthorne Garage on the way through.

We pottered around at home for a little bit before it was time to bike to swim class.

After swim class, we biked home, and Zoe watched some TV while I got organised for a demonstration that night.

Sarah picked up Zoe, and I headed out to my demo. Another full day.

Call for Volunteers

The Earlybird registrations are going extremely well – over 50% of the available tickets have sold in just two weeks! This is no longer a conference we are planning – this is a conference that is happening and that makes the Organisation Team very happy!

Speakers have been scheduled. Delegates are coming. We now urgently need to expand our team of volunteers to manage and assist all these wonderful visitors to ensure that LCA 2015 is unforgettable – for all the right reasons.

Volunteers are needed to register our delegates, show them to their accommodation, guide them around the University and transport them here and there. They will also manage our speakers by making sure that their presentations don't overrun, recording their presentations and assisting them in many other ways during their time at the conference.

Anyone who has been a volunteer before will tell you that it’s an extremely busy time, but so worthwhile. It’s rewarding to know that you’ve helped everybody at the conference to get the most out of it. There's nothing quite like knowing that you've made a difference.

But there is more, membership has other privileges and advantages! You don't just get to meet the delegates and speakers, you get to know many of them while helping them as well. You get a unique opportunity to get behind the scenes and close to the action. You can forge new relationships with amazing, interesting, wonderful people you might not ever get the chance to meet any other way.

Every volunteer's contribution is valued and vital to the overall running and success of the conference. We need all kinds of skills too – not just the technically savvy ones (although knowing which is the noisy end of a walkie-talkie may help). We want you! We need you! It just wouldn't be the same without you! If you would like to be an LCA 2015 volunteer it's easy to register. Just go to our volunteer page for more information. We review volunteer registrations regularly and if you’re based in Auckland (or would like a break away from wherever you are) then we would love to meet you at one of our regular meetings. Registered volunteers will receive information about these via email.

Assembly Primer Part 7 — Working with Strings — ARM

These are my notes for where I can see ARM varying from IA32, as presented in the video Part 7 — Working with Strings.

I’ve not remotely attempted to implement anything approximating optimal string operations for this part — I’m just working my way through the examples and finding obvious mappings to the ARM arch (or, at least what seem to be obvious). When I do something particularly stupid, leave a comment and let me know :)

Working with Strings

.data
     HelloWorldString:
        .asciz "Hello World of Assembly!"
    H3110:
        .asciz "H3110"

.bss
    .lcomm Destination, 100
    .lcomm DestinationUsingRep, 100
    .lcomm DestinationUsingStos, 100

Here’s the storage that the provided example StringBasics.s uses. No changes are required to compile this for ARM.

1. Simple copying using movsb, movsw, movsl

    @movl $HelloWorldString, %esi
    movw r0, #:lower16:HelloWorldString
    movt r0, #:upper16:HelloWorldString

    @movl $Destination, %edi
    movw r1, #:lower16:Destination
    movt r1, #:upper16:Destination

    @movsb
    ldrb r2, [r0], #1
    strb r2, [r1], #1

    @movsw
    ldrh r3, [r0], #2
    strh r3, [r1], #2

    @movsl
    ldr r4, [r0], #4
    str r4, [r1], #4

More visible complexity than IA32, but not too bad overall.

IA32’s movs instructions implicitly take their source and destination addresses from %esi and %edi, and increment/decrement both. Because of ARM’s load/store architecture, separate load and store instructions are required in each case, but there is support for indexing of these registers:

ARM addressing modes

According to ARM A8.5, memory access instructions commonly support three addressing modes:

  • Offset addressing — An offset is applied to an address from a base register and the result is used to perform the memory access. It’s the form of addressing I’ve used in previous parts and looks like [rN, offset]
  • Pre-indexed addressing — An offset is applied to an address from a base register, the result is used to perform the memory access and also written back into the base register. It looks like [rN, offset]!
  • Post-indexed addressing — An address is used as-is from a base register for memory access. The offset is applied and the result is stored back to the base register. It looks like [rN], offset and is what I’ve used in the example above.

2. Setting / Clearing the DF flag

ARM doesn’t have a DF flag (to the best of my understanding). It could perhaps be simulated through the use of two instructions and conditional execution to select the right direction. I’ll look further into conditional execution of instructions on ARM in a later post.

3. Using Rep

ARM also doesn’t appear to have an instruction quite like IA32’s rep instruction. A conditional branch and a decrement will be the long-form equivalent. As branches are part of a later section, I’ll skip them for now.

    @movl $HelloWorldString, %esi
    movw r0, #:lower16:HelloWorldString
    movt r0, #:upper16:HelloWorldString

    @movl $DestinationUsingRep, %edi
    movw r1, #:lower16:DestinationUsingRep
    movt r1, #:upper16:DestinationUsingRep

    @movl $25, %ecx # set the string length in ECX
    @cld # clear the DF
    @rep movsb
    @std

    ldm r0!, {r2,r3,r4,r5,r6,r7}
    ldrb r8, [r0,#0]
    stm r1!, {r2,r3,r4,r5,r6,r7}
    strb r8, [r1,#0]

To avoid conditional branches, I’ll start with the assumption that the string length is known (25 bytes). One approach would be using multiple load instructions, but the load multiple (ldm) instruction makes it somewhat easier for us — one instruction to fetch 24 bytes, and a load register byte (ldrb) for the last one. Using the ! after the source-address register indicates that it should be updated with the address of the next byte after those that have been read.

The storing of the data back to memory is done analogously. Store multiple (stm) writes 6 registers×4 bytes = 24 bytes (with the ! to have the destination address updated). The final byte is written using strb.

4. Loading string from memory into EAX register

    @cld
    @leal HelloWorldString, %esi
    movw r0, #:lower16:HelloWorldString
    movt r0, #:upper16:HelloWorldString

    @lodsb
    ldrb r1, [r0, #0]

    @movb $0, %al
    mov r1, #0

    @dec %esi  @ unneeded. equiv: sub r0, r0, #1
    @lodsw
    ldrh r1, [r0, #0]

    @movw $0, %ax
    mov r1, #0

    @subl $2, %esi # Make ESI point back to the original string. unneeded. equiv: sub r0, r0, #2
    @lodsl
    ldr r1, [r0, #0]

In this section, we are shown how the IA32 lodsb, lodsw and lodsl instructions work. Again, they have implicitly assigned register usage, which isn’t how ARM operates.

So, instead of a simple, no-operand instruction like lodsb, we have a ldrb r1, [r0, #0] loading a byte from the address in r0 into r1. Because I didn’t use post indexed addressing, there’s no need to dec or subl the address after the load. If I were to do so, it could look like this:

    ldrb r1, [r0], #1
    sub r0, r0, #1

    ldrh r1, [r0], #2
    sub r0, r0, #2

    ldr r1, [r0], #4

If you trace through it in gdb, look at how the value in r0 changes after each instruction.

5. Storing strings from EAX to memory

    @leal DestinationUsingStos, %edi
    movw r0, #:lower16:DestinationUsingStos
    movt r0, #:upper16:DestinationUsingStos

    @stosb
    strb r1, [r0], #1
    @stosw
    strh r1, [r0], #2
    @stosl
    str r1, [r0], #4

Same kind of thing as for the loads. Writes the letters in r1 (being “Hell” — leftovers from the previous section) into DestinationUsingStos (the result being “HHeHell”). String processing on little endian architectures has its appeal.

6. Comparing Strings

    @cld
    @leal HelloWorldString, %esi
    movw r0, #:lower16:HelloWorldString
    movt r0, #:upper16:HelloWorldString
    @leal H3110, %edi
    movw r1, #:lower16:H3110
    movt r1, #:upper16:H3110

    @cmpsb
    ldrb r2, [r0,#0]
    ldrb r3, [r1,#0]
    cmp r2, r3

    @dec %esi
    @dec %edi
    @not needed because of the addressing mode used

    @cmpsw
    ldrh r2, [r0,#0]
    ldrh r3, [r1,#0]
    cmp r2, r3

    @subl $2, %esi
    @subl $2, %edi
    @not needed because of the addressing mode used
    @cmpsl
    ldr r2, [r0,#0]
    ldr r3, [r1,#0]
    cmp r2, r3

Where IA32’s cmps instructions implicitly load through the pointers in %edi and %esi, explicit loads are needed for ARM. The compare then works in pretty much the same way as for IA32, setting condition code flags in the current program status register (cpsr). If you run the above code, and check the status registers before and after execution of the cmp instructions, you’ll see the zero flag set and unset in the same way as is demonstrated in the video.

The condition code flags are:

  • bit 31 — negative (N)
  • bit 30 — zero (Z)
  • bit 29 — carry (C)
  • bit 28 — overflow (V)

There’s other flags in that register — all the details are on page B1-16 and B1-17 in the ARM Architecture Reference Manual.

And with that, I think we’ve made it (finally) to the end of this part for ARM.

Other assembly primer notes are linked here.

October 22, 2014

CFP for Developer, Testing, Release and Continuous Integration Automation Miniconf at linux.conf.au 2015

This is the Call for Papers for the Developer, Testing, Release and Continuous Integration Automation Miniconf at linux.conf.au 2015 in Auckland. See http://linux.conf.au

This miniconf is all about improving the way we produce, collaborate, test and release software.

We want to cover tools and techniques to improve the way we work together to produce higher quality software:

– code review tools and techniques (e.g. gerrit)

– continuous integration tools (e.g. jenkins)

– CI techniques (e.g. gated trunk, zuul)

– testing tools and techniques (e.g. subunit, fuzz testing tools)

– release tools and techniques: daily builds, interacting with distributions, ensuring you test the software that you ship.

– applying CI in your workplace/project

We’re looking for talks about open source technology *and* the human side of things.

Speakers at this miniconf must be registered for the main conference (although there are a limited number of miniconf only tickets available for miniconf speakers if required)

There will be a projector, and there is a possibility the talk will be recorded (depending on if the conference A/V is up and running) – if recorded, talks will be posted with the same place with the same CC license as main LCA talks are.

CFP is open until midnight November 21st 2014.

By submitting a presentation, you’re agreeing to the following:

I allow Linux Australia to record my talk.

I allow Linux Australia to release any recordings of my presentations, tutorials and minconfs under the Creative Commons Attribution-Share Alike License

I allow Linux Australia to release any other material (such as slides) from my presentations, tutorials and minconfs under the Creative Commons Attribution-Share Alike License.

I confirm that I have the authority to allow Linux Australia to release the above material. i.e., if your talk includes any information about your employer, or another persons copyrighted material, that person has given you authority to release this information.

Any questions? Contact me: stewart@flamingspork.com

 

http://goo.gl/forms/KZI1YDDw8n

[life] Day 266: Prep play date, shopping and a play date

Zoe's sleep seems a bit messed up lately. She yelled out for me at 3:53am, and I resettled her, but she wound up in bed with me at 4:15am anyway. It took me a while to get back to sleep, maybe around 5am, but then we slept in until about 7:30am.

That made for a bit of a mad rush to get out the door to Zoe's primary school for her "Prep Play Date" orientation. We managed to make it out the door by a bit after 8:30am.

15 minutes is what it appears to take to scooter to school, which is okay. With local traffic being what it is, I think this will be a nice way to get to and from school next year, weather permitting.

We signed in, and Zoe got paired up with an existing (extremely tall) Prep student to be her buddy. The other girl was very keen to hold Zoe's hand, which Zoe was a bit dubious about at first, but they got there eventually.

The kids spent about 20 minutes rotating through the three classrooms, with a different buddy in each classroom. They were all given a 9 station name badge when they signed in, and they got a sticker for each station that they visited in each classroom.

It was a really nice morning, and I discovered there's one other girl from Zoe's Kindergarten going to her school, so I made a point of introducing myself to her mother.

I've got a really great vibe about the school, and Zoe enjoyed the morning. I'm looking forward to the next stage of her education.

We scootered home afterwards, and Zoe got the speed wobbles going down the hill and had a spectacular crash, luckily without any injuries thanks to all of her safety gear.

Once we got home, we headed out to the food wholesaler at West End to pick up a few bits and pieces, and then I had to get to Kindergarten to chair the monthly PAG meeting. I dropped Zoe at Megan's place for a play date while I was at the Kindergarten.

After the meeting, I picked up Zoe and we headed over to Westfield Carindale to buy a birthday present for Zoe's Kindergarten friend, Ivy, who is having a birthday party on Saturday.

We got home from Carindale with just enough time to spare before Sarah arrived to pick Zoe up.

I then headed over to Anshu's place for a Diwali dinner.

Speaker Feature: Audrey Lobo-Pulo, Jack Moffitt

Audrey Lobo-Pulo

Audrey Lobo-Pulo

Evaluating government policies using open source models

10:40am Wednesday 14th January 2015

Dr. Audrey Lobo-Pulo is a passionate advocate of open government and the use of open source software in government modelling. Having started out as a physicist developing theoretical models in the field of high speed data transmission, she moved into the economic policy modelling sphere and worked at the Australian Treasury from 2005 till 2011.

Currently working at the Australian Taxation Office in Sydney, Audrey enjoys discussions on modelling economic policy.

For more information on Audrey and her presentation, see here. You can follow her as @AudreyMatty and don’t forget to mention #LCA2015.



Jack Moffitt

Jack Moffitt

Servo: Building a Parallel Browser

10:40am Friday 16th January 2015

Jacks current project is called Chesspark and is an online community for chess players built on top of technologies like XMPP (aka Jabber), AJAX, and Python.

He previously created the Icecast Streaming Media Server, spent a lot of time developing and managing the Ogg Vorbits project, and helping create and run the Xiph.org Foundation. All these efforts exist to create a common, royalty free, and open standard for multimedia on the Internet.

Jack is also passionate about Free Software and Open Source, technology, music, and photography.

For more information on Jack and his presentation, see here. You can follow him as @metajack and don’t forget to mention #LCA2015.

October 21, 2014

Speaker Feature: Denise Paolucci, Gernot Heiser

Denise Paolucci

Denise Paolucci

When Your Codebase Is Nearly Old Enough To Vote

11:35 am Friday 16th January 2015

Denise is one of the founders of Dreamwidth, a journalling site and open source project forked from Livejournal, and one of only two majority-female open source projects.

Denise has appeared at multiple open source conferences to speak about Dreamwidth, including OSCON 2010 and linux.conf.au 2010.

For more information on Denise and her presentation, see here.



Gernot Heiser

Gernot Heiser

seL4 Is Free - What Does This Mean For You?

4:35pm Thursday 15th January 2015

Gernot is a Scientia Professor and the John Lions Chair for operating systems at the University of New South Wales (UNSW).

He is also leader of the Software Systems Research Group (SSRG) at NICTA. In 2006 he co-founded Open Kernel Labs (OK Labs, acquired in 2012 by General Dynamics) to commercialise his L4 microkernel technology

For more information on Gernot and his presentation, see here. You can follow him as @GernotHeiser and don’t forget to mention #LCA2015.

OpenStack infrastructure swift logs and performance

Turns out I’m not very good at blogging very often. However I thought I would put what I’ve been working on for the last few days here out of interest.

For a while the OpenStack Infrastructure team have wanted to move away from storing logs on disk to something more cloudy – namely, swift. I’ve been working on this on and off for a while and we’re nearly there.

For the last few weeks the openstack-infra/project-config repository has been uploading its CI test logs to swift as well as storing them on disk. This has given us the opportunity to compare the last few weeks of data and see what kind of effects we can expect as we move assets into an object storage.

  • I should add a disclaimer/warning, before you read, that my methods here will likely make statisticians cringe horribly. For the moment though I’m just getting an indication for how things compare.

The set up

Fetching files from an object storage is nothing particularly new or special (CDN’s have been doing it for ages). However, for our usage we want to serve logs with os-loganalyze giving the opportunity to hyperlink to timestamp anchors or filter by log severity.

First though we need to get the logs into swift somehow. This is done by having the job upload its own logs. Rather than using (or writing) a Jenkins publisher we use a bash script to grab the jobs own console log (pulled from the Jenkins web ui) and then upload it to swift using credentials supplied to the job as environment variables (see my zuul-swift contributions).

This does, however, mean part of the logs are missing. For example the fetching and upload processes write to Jenkins’ console log but because it has already been fetched these entries are missing. Therefore this wants to be the very last thing you do in a job. I did see somebody do something similar where they keep the download process running in a fork so that they can fetch the full log but we’ll look at that another time.

When a request comes into logs.openstack.org, a request is handled like so:

  1. apache vhost matches the server
  2. if the request ends in .txt.gz, console.html or console.html.gz rewrite the url to prepend /htmlify/
  3. if the requested filename is a file or folder on disk, serve it up with apache as per normal
  4. otherwise rewrite the requested file to prepend /htmlify/ anyway

os-loganalyze is set up as an WSGIScriptAlias at /htmlify/. This means all files that aren’t on disk are sent to os-loganalyze (or if the file is on disk but matches a file we want to mark up it is also sent to os-loganalyze). os-loganalyze then does the following:

  1. Checks the requested file path is legitimate (or throws a 400 error)
  2. Checks if the file is on disk
  3. Checks if the file is stored in swift
  4. If the file is found markup (such as anchors) are optionally added and the request is served
    1. When serving from swift the file is fetched via the swiftclient by os-loganlayze in chunks and streamed to the user on the fly. Obviously fetching from swift will have larger network consequences.
  5. If no file is found, 404 is returned

If the file exists both on disk and in swift then step #2 can be skipped by passing ?source=swift as a parameter (thus only attempting to serve from swift). In our case the files exist both on disk and in swift since we want to compare the performance so this feature is necessary.

So now that we have the logs uploaded into swift and stored on disk we can get into some more interesting comparisons.

Testing performance process

My first attempt at this was simply to fetch the files from disk and then from swift and compare the results. A crude little python script did this for me: http://paste.openstack.org/show/122630/

The script fetches a copy of the log from disk and then from swift (both through os-loganalyze and therefore marked-up) and times the results. It does this in two scenarios:

  1. Repeatably fetching the same file over again (to get a good average)
  2. Fetching a list of recent logs from gerrit (using the gerrit api) and timing those

I then ran this in two environments.

  1. On my local network the other side of the world to the logserver
  2. On 5 parallel servers in the same DC as the logserver

Running on my home computer likely introduced a lot of errors due to my limited bandwidth, noisy network and large network latency. To help eliminate these errors I also tested it on 5 performance servers in the Rackspace cloud next to the log server itself. In this case I used ansible to orchestrate the test nodes thus running the benchmarks in parallel. I did this since in real world use there will often be many parallel requests at once affecting performance.

The following metrics are measured for both disk and swift:

  1. request sent – time taken to send the http request from my test computer
  2. response – time taken for a response from the server to arrive at the test computer
  3. transfer – time taken to transfer the file
  4. size – filesize of the requested file

The total time can be found by adding the first 3 metrics together.

 

Results

Home computer, sequential requests of one file

 

The complementary colours are the same metric and the darker line represents swift’s performance (over the lighter disk performance line). The vertical lines over the plots are the error bars while the fetched filesize is the column graph down the bottom. Note that the transfer and file size metrics use the right axis for scale while the rest use the left.

As you would expect the requests for both disk and swift files are more or less comparable. We see a more noticable difference on the responses though with swift being slower. This is because disk is checked first, and if the file isn’t found on disk then a connection is sent to swift to check there. Clearly this is going to be slower.

The transfer times are erratic and varied. We can’t draw much from these, so lets keep analyzing deeper.

The total time from request to transfer can be seen by adding the times together. I didn’t do this as when requesting files of different sizes (in the next scenario) there is nothing worth comparing (as the file sizes are different). Arguably we could compare them anyway as the log sizes for identical jobs are similar but I didn’t think it was interesting.

The file sizes are there for interest sake but as expected they never change in this case.

You might notice that the end of the graph is much noisier. That is because I’ve applied some rudimentary data filtering.

request sent (ms) – disk request sent (ms) – swift response (ms) – disk response (ms) – swift transfer (ms) – disk transfer (ms) – swift size (KB) – disk size (KB) – swift
Standard Deviation 54.89516183 43.71917948 56.74750291 194.7547117 849.8545127 838.9172066 7.121600095 7.311125275
Mean 283.9594368 282.5074598 373.7328851 531.8043908 5091.536092 5122.686897 1219.804598 1220.735632

 

I know it’s argued as poor practice to remove outliers using twice the standard deviation, but I did it anyway to see how it would look. I only did one pass at this even though I calculated new standard deviations.

 

request sent (ms) – disk request sent (ms) – swift response (ms) – disk response (ms) – swift transfer (ms) – disk transfer (ms) – swift size (KB) – disk size (KB) – swift
Standard Deviation 13.88664039 14.84054789 44.0860569 115.5299781 541.3912899 515.4364601 7.038111654 6.98399691
Mean 274.9291111 276.2813889 364.6289583 503.9393472 5008.439028 5013.627083 1220.013889 1220.888889

 

I then moved the outliers to the end of the results list instead of removing them completely and used the newly calculated standard deviation (ie without the outliers) as the error margin.

Then to get a better indication of what are average times I plotted the histograms of each of these metrics.

Here we can see a similar request time.

 

Here it is quite clear that swift is slower at actually responding.

 

Interestingly both disk and swift sources have a similar total transfer time. This is perhaps an indication of my network limitation in downloading the files.

 

Home computer, sequential requests of recent logs

Next from my home computer I fetched a bunch of files in sequence from recent job runs.

 

 

Again I calculated the standard deviation and average to move the outliers to the end and get smaller error margins.

request sent (ms) – disk request sent (ms) – swift response (ms) – disk response (ms) – swift transfer (ms) – disk transfer (ms) – swift size (KB) – disk size (KB) – swift
Standard Deviation 54.89516183 43.71917948 194.7547117 56.74750291 849.8545127 838.9172066 7.121600095 7.311125275
Mean 283.9594368 282.5074598 531.8043908 373.7328851 5091.536092 5122.686897 1219.804598 1220.735632
Second pass without outliers
Standard Deviation 13.88664039 14.84054789 115.5299781 44.0860569 541.3912899 515.4364601 7.038111654 6.98399691
Mean 274.9291111 276.2813889 503.9393472 364.6289583 5008.439028 5013.627083 1220.013889 1220.888889

 

What we are probably seeing here with the large number of slower requests is network congestion in my house. Since the script requests disk, swift, disk, swift, disk.. and so on this evens it out causing a latency in both sources as seen.

 

Swift is very much slower here.

 

Although comparable in transfer times. Again this is likely due to my network limitation.

 

The size histograms don’t really add much here.

 

Rackspace Cloud, parallel requests of same log

Now to reduce latency and other network effects I tested fetching the same log over again in 5 parallel streams. Granted, it may have been interesting to see a machine close to the log server do a bunch of sequential requests for the one file (with little other noise) but I didn’t do it at the time unfortunately. Also we need to keep in mind that others may be access the log server and therefore any request in both my testing and normal use is going to have competing load.

 

I collected a much larger amount of data here making it harder to visualise through all the noise and error margins etc. (Sadly I couldn’t find a way of linking to a larger google spreadsheet graph). The histograms below give a much better picture of what is going on. However out of interest I created a rolling average graph. This graph won’t mean much in reality but hopefully will show which is faster on average (disk or swift).

 

You can see now that we’re closer to the server that swift is noticeably slower. This is confirmed by the averages:

 

  request sent (ms) – disk request sent (ms) – swift response (ms) – disk response (ms) – swift transfer (ms) – disk transfer (ms) – swift size (KB) – disk size (KB) – swift
Standard Deviation 32.42528982 9.749368282 245.3197219 781.8807534 1082.253253 2737.059103 0 0
Mean 4.87337544 4.05191168 39.51898688 245.0792916 1553.098063 4167.07851 1226 1232
Second pass without outliers
Standard Deviation 1.375875503 0.8390193564 28.38377158 191.4744331 878.6703183 2132.654898 0 0
Mean 3.487575109 3.418433003 7.550682037 96.65978872 1389.405618 3660.501404 1226 1232

 

Even once outliers are removed we’re still seeing a large latency from swift’s response.

The standard deviation in the requests now have gotten very small. We’ve clearly made a difference moving closer to the logserver.

 

Very nice and close.

 

Here we can see that for roughly half the requests the response time was the same for swift as for the disk. It’s the other half of the requests bringing things down.

 

The transfer for swift is consistently slower.

 

Rackspace Cloud, parallel requests of recent logs

Finally I ran just over a thousand requests in 5 parallel streams from computers near the logserver for recent logs.

 

Again the graph is too crowded to see what is happening so I took a rolling average.

 

 

request sent (ms) – disk request sent (ms) – swift response (ms) – disk response (ms) – swift transfer (ms) – disk transfer (ms) – swift size (KB) – disk size (KB) – swift
Standard Deviation 0.7227904332 0.8900549012 434.8600827 909.095546 1913.9587 2132.992773 6.341238774 7.659678352
Mean 3.515711867 3.56191383 145.5941102 189.947818 2427.776165 2875.289455 1219.940039 1221.384913
Second pass without outliers
Standard Deviation 0.4798803247 0.4966553679 109.6540634 171.1102999 1348.939342 1440.2851 6.137625464 7.565931993
Mean 3.379718381 3.405770445 70.31323922 86.16522485 2016.900047 2426.312363 1220.318912 1221.881335

 

The averages here are much more reasonable than when we continually tried to request the same file. Perhaps we’re hitting limitations with swifts serving abilities.

 

I’m not sure why we have sinc function here. A network expert may be able to tell you more. As far as I know this isn’t important to our analysis other than the fact that both disk and swift match.

 

Here we can now see swift keeping a lot closer to disk results than when we only requested the one file in parallel. Swift is still, unsurprisingly, slower overall.

 

Swift still loses out on transfers but again does a much better job of keeping up.

 

Error sources

I haven’t accounted for any of the following swift intricacies (in terms of caches etc) for:

  • Fetching random objects
  • Fetching the same object over and over
  • Fetching in parallel multiple different objects
  • Fetching the same object in parallel

I also haven’t done anything to account for things like file system caching, network profiling, noisy neighbours etc etc.

os-loganalyze tries to keep authenticated with swift, however

  • This can timeout (causes delays while reconnecting, possibly accounting for some spikes?)
  • This isn’t thread safe (are we hitting those edge cases?)

We could possibly explore getting longer authentication tokens or having os-loganalyze pull from an unauthenticated CDN to add the markup and then serve. I haven’t explored those here though.

os-loganalyze also handles all of the requests not just from my testing but also from anybody looking at OpenStack CI logs. In addition to this it also needs to deflate the gzip stream if required. As such there is potentially a large unknown (to me) load on the log server.

In other words, there are plenty of sources of errors. However I just wanted to get a feel for the general responsiveness compared to fetching from disk. Both sources had noise in their results so it should be expected in the real world when downloading logs that it’ll never be consistent.

Conclusions

As you would expect the request times are pretty much the same for both disk and swift (as mentioned earlier) especially when sitting next to the log server.

The response times vary but looking at the averages and the histograms these are rarely large. Even in the case where requesting the same file over and over in parallel caused responses to go slow these were only in the magnitude of 100ms.

The response time is the important one as it indicates how soon a download will start for the user. The total time to stream the contents of the whole log is seemingly less important if the user is able to start reading the file.

One thing that wasn’t tested was streaming of different file sizes. All of the files were roughly the same size (being logs of the same job). For example, what if the asset was a few gigabytes in size, would swift have any significant differences there? In general swift was slower to stream the file but only by a few hundred milliseconds for a megabyte. It’s hard to say (without further testing) if this would be noticeable on large files where there are many other factors contributing to the variance.

Whether or not these latencies are an issue is relative to how the user is using/consuming the logs. For example, if they are just looking at the logs in their web browser on occasion they probably aren’t going to notice a large difference. However if the logs are being fetched and scraped by a bot then it may see a decrease in performance.

Overall I’ll leave deciding on whether or not these latencies are acceptable as an exercise for the reader.

[life] Day 265: Kindergarten and startup stuff

Zoe yelled out for me at 5:15am for some reason, but went back to sleep after I resettled her, and we had a slow start to the day a bit after 7am. I've got a mild version of whatever cold she's currently got, so I'm not feeling quite as chipper as usual.

We biked to Kindergarten, which was a bit of a slog up Hawthorne Road, given the aforementioned cold, but we got there in the end.

I left the trailer at the Kindergarten and biked home again.

I finally managed to get some more work done on my real estate course, and after a little more obsessing over one unit, got it into the post. I've almost got another unit finished as well. I'll try to get it finished in the evenings or something, because I'm feeling very behind, and I'd like to get it into the mail too. I'm due to get the second half of my course material, and I still have one more unit to do after this one I've almost finished.

I biked back to Kindergarten to pick up Zoe. She wanted to watch Megan's tennis class, but I needed to grab some stuff for dinner, so it took a bit of coaxing to get her to leave. I think she may have been a bit tired from her cold as well.

We biked home, and jumped in the car. I'd heard from Matthew's Dad that FoodWorks in Morningside had a good meat selection, so I wanted to check it out.

They had some good roasting meat, but that was about it. I gave up trying to mince my own pork and bought some pork mince instead.

We had a really nice dinner together, and I tried to get her to bed a little bit early. Every time I try to start the bed time routine early, the spare time manages to disappear anyway.

October 20, 2014

SM1000 Part 7 – Over the air in Germany

Michael Wild DL2FW in Germany recently attended a Hamfest where he demonstrated his SM1000. Michael sent me the following email (hint: I used Google translate on the web sites):

Here is the link to the review of our local hamfest.

At the bottom is a video of a short QSO on 40m using the SM-1000 over about 400km. The other station was Hermann (DF2DR). Hermann documented this QSO very well on his homepage also showing a snapshot of the waterfall during this QSO. Big selective fading as you can see, but we were doing well!

He also explains that, when switching to SSB at the same average power level, the voice was almost not understandable!

SM1000 Beta and FreeDV Update

Rick KA8BMA has been working hard on the Beta CAD work, and fighting a few Eagle DRC battles. Thanks to all his hard work we now have an up to date schematic and BOM for the Betas. He is now working on the Beta PCB layout, and we are refining the BOM with Edwin from Dragino in China. Ike, W3IKIE, has kindly been working with Rick to come up with a suitable enclosure. Thanks guys!

My current estimate is that the Beta SM1000s will be assembled in November. Once I’ve tested a few I’ll put them up on my store and start taking orders.

In the mean time I’ve thrown myself into modem simulations – playing with a 450 bit/s version of Codec 2, LPDC FEC codes, diversity schemes and coherent QPSK demodulation. I’m pushing towards a new FreeDV mode that works on fading channels at negative SNRs. More on that in later posts. The SM1000 and a new FreeDV mode are part of my goals for 2014. The SM1000 will make FreeDV easy to use, the new mode(s) will make it competitive with SSB on HF radio.

Everything is open source, both hardware and software. No vendor lock in, no software licenses and you are free to experiment and innovate.