Planet Linux Australia
Celebrating Australians & Kiwis in the Linux and Free/Open-Source community...

October 01, 2016

DevOpsDays Wellington 2016 – Day 2, Session 3

Ignites

Mrinal Mukherjee – How to choose a DevOps tool

Right Tool
– Does the job
– People will accept

Wrong tool
– Never ending Poc
– Doesn’t do the job

How to pick
– Budget / Licensing
– does it address your pain points
– Learning cliff
– Community support
– API
– Enterprise acceptability
– Config in version control?

Central tooling team
– Pro standardize, educate, education
– Constant Bottleneck, delays, stifles innovation, not in sync with teams

DevOps != Tool
Tools != DevOps

Tools facilitate it not define it.

Howard Duff – Eric and his blue boxes

Physical example of KanBan in an underwear factory

Lindsey Holmwood – Deepening people to weather the organisation

Note: Lindsey presents really fast so I missed recording a lot from the talk

His Happy, High performing Team -> He left -> 6 months later half of team had left

How do you create a resilient culture?

What is culture?
– Lots of research in organisation psychology
– Edgar Schein – 3 levels of culture
– Artefacts, Values, Assumptions

Artefacts
– Physical manifestations of our culture
– Standups, Org charts, desk layout, documentation
– actual software written
– Easiest to see and adopt

Values
– Goals, strategies and philosophise
– “we will dominate the market”
– “Management if available”
– “nobody is going to be fired for making a mistake”
– lived values vs aspiration values (People have good nose for bullshit)
– Example, cores values of Enron vs reality
– Work as imagined vs Work is actually done

Assumptions
– beliefs, perceptions, thoughts and feelings
– exist on an unconscious level
– hard to discern
– “bad outcomes come from bad people”
– “it is okay to withhold information”
– “we can’t trust that team”
– “profits over people”

If we can change our people, we can change our culture

What makes a good team member?

Trust
– Vulnerability
– Assume the best of others
– Aware of their cognitive bias
– Aware of the fundamental attribution error (judge others by actions, judge ourselves by our intentions)
– Aware of hindsight bias. Hindsight bias is your culture killer
– When bad things happen explain in terms of foresight
– Regular 1:1s
Eliminate performance reviews
Willing to play devils advocate

Commit and acting
– Shared goal settings
– Don’t solutioneer
– Provide context about strategy, about desired outcome
What makes a good team?

Influence of hiring process
– Willingness to adapt and adopt working in new team
– Qualify team fit, tech talent then rubber stamp from team lead
– have a consistent script, but be prepared to improvise
– Everyone has the veto power
– Leadership is vetoing at the last minute, thats a systemic problem with team alignment not the system
– Benefit: team talks to candidate (without leadership present)
– Many different perspectives
– unblock management bottlenecks
– Risk: uncovering dysfunctions and misalignment in your teams
– Hire good people, get out of their way

Diversity and inclusion
– includes: race, gender, sexual orientation, location, disability, level of experience, work hours
– Seek out diverse candidates.
– Sponsor events and meetups
– Make job description clear you are looking for diverse background
– Must include and embrace differences once they actually join
– Safe mechanism for people to raise criticisms, and acting on them

Leadership and Absence of leadership
– Having a title isn’t required
– If leader steps aware things should continue working right
– Team is their own shit umbrella
– empowerment vs authority
– empowerment is giving permission from above (potentially temporary)
– authority is giving power (granting autonomy)

Part of something bigger than the team
– help people build up for the next job
– Guilds in the Spotify model
– Run them like meetups
– Get senior management to come and observe
– What we’re talking about is tech culture

We can change tech culture
– How to make it resist the culture of the rest of the organisation
– Artefacts influence behaviour
– Artifact fast builds -> value: make better quality
– Artifact: post incident reviews -> Value: Failure is an opportunity for learning

Q: What is a pre-incident review
A: Brainstorm beforehand (eg before a big rollout) what you think might go wrong if something is coming up
then afterwards do another review of what just went wrong

Q: what replaces performance reviews
A: One on ones

Q: Overcoming Resistance
A: Do it and point back at the evidence. Hard to argue with an artifact

Q: First step?
A: One on 1s

Getting started, reading books by Patrick Lencioni:
– Solos, Politics and turf wars
– 5 Dysfunctions of a team

FacebookGoogle+Share

September 30, 2016

DevOpsDays Wellington 2016 – Day 2, Session 2

Troy Cornwall & Alex Corkin – Health is hard: A Story about making healthcare less hard, and faster!

Maybe title should be “Culture is Hard”

@devtroy @4lexNZ

Working at HealthLink
– Windows running Java stuff
– Out of date and poorly managed
– Deployments manual, thrown over the wall by devs to ops

Team Death Star
– Destroy bad processes
– Change deployment process

Existing Stack
– VMware
– Windows
– Puppet
– PRTG

CD and CI Requirements
– Goal: Time to regression test under 2 mins, time to deploy under 2 mins (from 2 weeks each)
– Puppet too slow to deploy code in a minute or two. App deply vs Conf mngt
– Can’t use (then) containers on Windows so not an option

New Stack
– VMware
– Ubuntu
– Puppet for Server config
– Docker
– rancher

Smashed the 2 minute target!

But…
– We focused on the tech side and let the people side slip
– Windows shop, hard work even to get a Linux VM at the start
– Devs scared to run on Linux. Some initial deploy problems burnt people
– Lots of different new technologies at once all pushed to devs, no pull from them.

Blackout where we weren’t allowed to talk to them for four weeks
– Should have been a warning sign…

We thought we were ready.
– Ops was not ready

“5 dysfunctions of a team”
– Trust as at the bottom, we didn’t have that

Empathy
– We were aware of this, but didn’t follow though
– We were used to disruption but other teams were not

Note: I’m not sure how the story ended up, they sort of left it hanging.

Pavel Jelinek – Kubernetes in production

Works at Movio
– Software for Cinema chains (eg Loyalty cards)
– 100million emails per month. million of SMS and push notifications (less push cause ppl hate those)

Old Stack
– Started with mysql and php application
– AWS from the beginning
– On largest aws instance but still slow.

Decided to go with Microservices
– Put stuff in Docker
– Used Jenkins, puppet, own docker registery, rundeck (see blog post)
– Devs didn’t like writing puppet code and other manual setup

Decided to go to new container management at start of 2016
– Was pushing for Nomad but devs liked Kubernetes

Kubernetes
– Built in ports, HA, LB, Health-checks

Concepts in Kub
– POD – one or more containers
– Deployment, Daemon, Pet Set – Scaling of a POD
– Service- resolvable name, load balancing
– ConfigMap, Volume, Secret – Extended Docker Volume

Devs look after some kub config files
– Brings them closer to how stuff is really working

Demo
– Using kubectl to create pod in his work’s lab env
– Add load balancer in front of it
– Add a configmap to update the container’s nginx config
– Make it public
– LB replicas, Rolling updates

Best Practices
– lots of small containers are better
– log on container stdout, preferable via json
– Test and know your resource requirements (at movio devs teams specify, check and adjust)
– Be aware of the node sizes
– Stateless please
– if not stateless than clustered please
– Must handle unexpected immediate restarts

FacebookGoogle+Share

Linux Security Summit 2016 Wrapup

Here’s a summary of the 2016 Linux Security Summit, which was held last month in Toronto.

Presentation slides are available at http://events.linuxfoundation.org/events/archive/2016/linux-security-summit/program/slides.

This year, videos were made of the sessions, and they may be viewed at https://www.linux.com/news/linux-security-summit-videos — many thanks to Intel for sponsoring the recordings!

LWN has published some excellent coverage:

This is a pretty good representation of the main themes which emerged in the conference: container security, kernel self-protection, and integrity / secure boot.

Many of the core or low level security technologies (such as access control, integrity measurement, crypto, and key management) are now fairly mature. There’s more focus now on how to integrate these components into higher-level systems and architectures.

One talk I found particularly interesting was Design and Implementation of a Security Architecture for Critical Infrastructure Industrial Control Systems in the Era of Nation State Cyber Warfare. (The title, it turns out, was a hack to bypass limited space for the abstract in the cfp system).  David Safford presented an architecture being developed by GE to protect a significant portion of the world’s electrical grid from attack.  This is being done with Linux, and is a great example of how the kernel’s security mechanisms are being utilized for such purposes.  See the slides or the video.  David outlined gaps in the kernel in relation to their requirements, and a TPM BoF was held later in the day to work on these.  The BoF was reportedly very successful, as several key developers in the area of TPM and Integrity were present.

Attendance at LSS was the highest yet with well over a hundred security developers, researchers and end users.

Special thanks to all of the LF folk who manage the logistics for the event.  There’s no way we could stage something on this scale without their help.

Stay tuned for the announcement of next year’s event!

 

DevOpsDays Wellington 2016 – Day 2, Session 1

Jethro Carr – Powering stuff.co.nz with DevOps goodness

Stuff.co.nz
– “News” Website
– 5 person DevOps team

Devops
– “Something you do because Gartner said it’s cool”
– Sysadmin -> InfraCoder/SRE -> Dev Shepherd -> Dev
– Stuff in the middle somewhere
– DevSecOps

Company Structure drives DevOps structure
– Lots of products – one team != one product
– Dev teams with very specific focus
– Scale – too big, yet to small

About our team
– Mainly Ops focus
– small number compared to developers
– Operate like an agency model for developers
– “If you buy the Dom Post it would help us grow our team”
– Lots of different vendors with different skill levels and technology

Work process
– Use KanBan with Jira
– Works for Ops focussed team
– Not so great for long running projects

War Against OnCall
– Biggest cause of burnout
– focus on minimising callouts
– Zero alarm target
– Love pagerduty

Commonalities across platforms
– Everyone using compute
– Most Java and javascript
– Using Public Cloud
– Using off the shelf version control, deployment solutions
– Don’t get overly creative and make things too complex
– Proven technology that is well tried and tested and skills available in marketplace
– Classic technologist like Nginx, Java, Varnish still have their place. Don’t always need latest fashion

Stack
– AWS
– Linux, ubuntu
– Adobe AEM Java CMS
– AWS 14x c4.2xlarge
– Varnish in front, used by everybody else. Makes ELB and ALB look like toys

How use Varnish
– Retries against backends if 500 replies, serve old copies
– split routes to various backends
– Control CDN via header
– Dynamic Configuration via puppet

CDN
– Akamai
– Keeps online during breaking load
– 90% cache offload
– Management is a bit slow and manual

Lamda
– Small batch jobs
– Check mail reputation score
– “Download file from a vendor” type stuff
– Purge cache when static file changes
– Lamda webapps – Hopefully soon, a bit immature

Increasing number of microservices

Standards are vital for microservices
– Simple and reasonable
– Shareable vendors and internal
– flexible
– grow organicly
– Needs to be detail
– 12 factor App
– 3 languages Node, Java, Ruby
– Common deps (SQL, varnish, memcache, Redis)
– Build pipeline standardise. Using Codeship
– Standardise server builds
– Everything Automated with puppet
– Puppet building docker containers (w puppet + puppetstry)
– Std Application deployment

Init systems
– Had proliferation
– pm2, god, supervisord, systemvinit are out
– systemd and upstart are in

Always exceptions
– “Enterprise ___” is always bad
– Educating the business is a forever job
– Be reasonable, set boundaries

More Stuff at
http://tinyurl.com/notclickbaithonest

Q: Pull request workflow
A: Largely replaced traditional review

Q: DR eg AWS outage
A: Documented process if codeship dies can manually push, Rest in 2*AZs, Snapshots

Q: Dev teams structure
A: Project specific rather than product specific.

Q: Puppet code tested?
A: Not really, Kinda tested via the pre-prod environment, Would prefer result (server spec) testing rather than low level testing of each line
A: Code team have good test coverage though. 80-90% in many cases.

Q: Load testing, APM
A: Use New Relic. Not much luck with external load testing companies

Q: What is somebody wants something non-standard?
A: Case-by-case. Allowed if needed but needs a good reason.

Q: What happens when automation breaks?
A: Documentation is actually pretty good.

FacebookGoogle+Share

September 29, 2016

DevOpsDays Wellington 2016 – Day 1, Session 3

Owen Evans – DevOps is Dead, long live DevOps

Theory: Devops is role that never existed.

In the old days
– Shipping used to be hard and expensive, eg on physical media
– High cost of release
– but everybody else was the same.
– Lots of QA and red tape, no second chances

Then we got the Internet
– Speed became everything
– You just shipped enough

But Hardware still was a limiting factor
– Virtual machines
– IaaS
– Containers

This led to complacency
– Still had a physical server under it all

Birth of devops
– Software got faster but still had to have hardware under their somewhere
– Disparity between operations cadence and devs cadence
– things got better
– But we didn’t free ourselves from hardware
– Now everything is much more complex

Developers are now divorced from the platform
– Everything is abstracted
– It is leaky buckets all the way down

Solutions
– Education of developers as to what happens below the hood
– Stop reinventing the where
– Harmony is much more productive
– Lots of tools means that you don’t have enough expertise on each
– Reduce fiefdoms
– Push responsibility but not ownership (you own it but the devs makes some of the changes)
– Live with the code
– Pit of success, easy ways to fail that don’t break stuff (eg test environments, by default it will do the right thing)
– Be Happy. Everybody needs to be a bit devops and know a bit of everything.

FacebookGoogle+Share

DevOpsDays Wellington 2016 – Day 1, Session 2

Martina Iglesias – Automatic Discovery of Service metadata for systems at scale

Backend developer at Spotify

Spotify Scale
– 100m active users
– 800+ tech employees
– 120 teams
– Microservices architecture

Walk though Sample artist’s page
– each component ( playlist, play count, discgraphy) is a seperate service
– Aggregated to send result back to client

Hard to co-ordinate between services as scale grows
– 1000+ services
– Each need to use each others APIs
– Dev teams all around the world

Previous Solution
– Teams had docs in different places
– Some in Wiki, Readme, markdown, all different

Current Solution – System Z
– Centralise in one place, as automated as possible
– Internal application
– Web app, catalog of all systems and its parts
– Well integrated with Apollo service

Web Page for each service
– Various tabs
– Configuration (showing versions of build and uptimes)
– API – list of all endpoints for service, scheme, errors codes, etc (automatically populated)
– System tab – Overview on how service is connected to other services, dependencies (generated automatically)

Registration
– System Z gets information from Apollo and prod servers about each service that has been registered

Apollo
– Java libs for writing microservices
– Open source

Apollo-meta
– Metadata module
– Exposes endpoint with metadata for each service
– Exposes
– instance info – versions, uptime
– configuration – currently loaded config of the service
– endpoints –
– call information – monitors service and learns and returns what incoming and outgoing services the service actually does and to/from what other services.
– Automatically builds dependencies

Situation Now
– Quicker access to relevant information
– Automated boring stuff
– All in one place

Learnings
– Think about growth and scaling at the start of the project

Documentation generators
-Apollo
– Swagger.io
– ralm.org

Blog: labs.spotify.com
Jobs: spotify.com/jobs

Q: How to handle breaking APIs
A: We create new version of API endpoint and encourage people to move over.

Bridget Cowie – The story of a performance outage, and how we could have prevented it

– Works for Datacom
– Consultant in Application performance management team

Story from Start of 2015

– Friday night phone calls from your boss are never good.
– Dropped in application monitoring tools (Dynatrace) on Friday night, watch over weekend
– Prev team pretty sure problem is a memory leak but had not been able to find it (for two weeks)
– If somebody tells you they know what is wrong but can’t find it, give details or fix it then be suspicious

Book: Java Enterprise performance

– Monday prod load goes up and app starts crashing
– Told ops team but since crash wasn’t visable yet, was not believed. waited

Tech Stack
– Java App, Jboss on Linux
– Multiple JVMs
– Oracle DBs, Mulesoft ESB, ActiveMQ, HornetQ

Ah Ha moment
– Had a look at import process
– 2.3 million DB queries per half hour
– With max of 260 users, seems way more than what is needed
– Happens even when nobody is logged in

Tip: Typically 80% of all issues can be detected in dev or test if you look for them.

Where did this code come from?
– Process to import a csv into the database
– 1 call mule -> 12 calls to AMQ -> 12 calls to App -> 102 db queries
– Passes all the tests… But
– Still shows huge growth in queries as we go through layers
– DB queries grow bigger with each run

Tip: Know how your code behaves and track how this behavour changes with each code change (or even with no code change)

Q: Why Dynatrace?
A: Quick to deploy, useful info back in only a couple of hours

FacebookGoogle+Share

DevOpsDays Wellington 2016 – Day 1, Session 1

Ken Mugrage – What we’re learning from burnout and how DevOps culture can help

Originally in the Marines, environment where burnout not tolerated
Works for Thoughtworks – not a mental health professional

Devops could make this worse
Some clichéd places say: “Teach the devs puppet and fire all the Ops people”

Why should we address burnout?
– Google found psychological safety was the number 1 indicator of an effective team
– Not just a negative, people do better job when feeling good.

What is burnout
– The Truth about burnout – Maslach and Leiter
– The Dimensions of Burnout
– Exhaustion
– Cynicism
– Mismatch between work and the person
– Work overload
– Lack of control
– Insufficient reward
– Breakdown of communication

Work overload
– Various prioritisation methods
– More load sharing
– Less deploy marathons
– Some orgs see devops as a cost saving
– There is no such thing as a full stack engineer
– team has skills, not a person

Lack of Control
– Team is ultimately for the decissions
– Use the right technolgy and tools for the team
– This doesnt mean a “Devops team” contolling what others do

Insufficient Reward
– Actually not a great motivator

Breakdown in communication
– Walls between teams are bad
– Everybody involved with product should be on the same team
– 2 pizza team
– Pairs with different skill sets are common
– Swarming can be done when required ( one on keyboard, everybody else watching and talking and helping on big screen)
– Blameless retrospectives are held
– No “Devops team”, creating a silo is not a solution for silos

Absence of Fairness
– You build it, you run it
– Everybody is responsible for quality
– Everybody is measured in the same way
– example Expedia – *everything* deployed has A/B tesing
– everybody goes to release party

Conflicting Values
– In the broadest possible sense
– eg Company industry and values should match your own

Reminder: it is about you and how you fit in with the above

Pay attention to how you feel
– Increase your self awareness
– Maslach Burnout inventory
– Try not to focus on the negative.

Pay attention to work/life balance
– Ask for it, company might not know your needs
– If you can’t get it then quit

Talk to somebody
– Professional help is the best
– Trained to identify cause and effect
– can recommend treatment
– You’d call them if you broke your arm

Friends and family
– People who care, that you haven’t even meet
– Empathy is great , but you aren’t a professional
– Don’t guess cause and effect
– Don’t recommend treatment if not a professional

Q: Is it Gender specific for men (since IT is male dominated) ?
– The “absence of fairness” problem is huge for women in IT

Q: How to promote Psychological safety?
– Blameless post-mortems

 

Damian Brady – Just let me do my job

After working in govt, went to work for new company and hoped to get stuff done

But whole dev team was unhappy
– Random work assigned
– All deadlines missed
– Lots of waste of time meetings

But 2 years later
– Hitting all deadlines
– Useful meetings

What changes were made?

New boss, protect devs for MUD ( Meetings, uncertainty, distractions )

Meetings
– In board sense, 1-1, all hands, normal meetings
– People are averaging 7.5 hours/week in meetings
– On average 37% of meeting time is not relevant to person ( ~ $8,000 / year )
– Do meetings have goals and do they achieve those goals?
– 38% without goals
– only half of remaining meet those goals
– around 40% of meetings have and achieve goals
– Might not be wasted. Look at “What has changed as result of this meeting?”

Meetings fixes
– New Boss went to meetings for us (didn’t need everybody) as a representative
– Set a clear goal and agenda
– Avoid gimmicks
– don’t default to 30min or 1h

Distractions
– 60% of people interrupted 10 or more times per day
– Good to stay in a “flow state”
– 40% people say they are regularly focussed in their work. but all are sometimes
– 35% of time loss focus when interrupted
– Study shows people can take up to 23mins to get focus back after interruption
– $25,000/year wasting according to interruptions

Distraction Fixes
– Allowing headphones, rule not to interrupt people wearing headphones
– “Do not disturb” times
– Little Signs
– Had “the finger” so that you could tell somebody your were busy right now and would come back to them
– Let devs go to meeting rooms or cafes to hide from interruptions
– All “go dark” where email and chat turned off

Uncertainty
– 82% in survey were clear
– nearly 60% of people their top priority changes before they can finish it.
– Autonomy, mastery, purpose

Uncertainty Fixes
– Tried to let people get clear runs at work
– Helped people acknowledge the unexpected work, add to Sprint board
– Established a gate – Business person would have to go through the manager
– Make the requester responsible – made the requester decide what stuff didn’t get done by physically removing stuff from the sprint board to add their own

FacebookGoogle+Share

September 26, 2016

MySQL removes the FRM (7 years after Drizzle did)

The new MySQL 8.0.0 milestone release that was recently announced brings something that has been a looooong time coming: the removal of the FRM file. I was the one who implemented this in Drizzle way back in 2009 (July 28th 2009 according to Brian)- and I may have had a flashback to removing the tentacles of the FRM when reading the MySQL 8.0.0 announcement.

As an idea for how long this has been on the cards, I’ll quote Brian from when we removed it in Drizzle:

We have been talking about getting rid of FRM since around 2003. I remember a drive up to northern Finland with Kaj Arnö, where we spent an hour talking about this. I, David, and MontyW have talked about this for years.

http://krow.livejournal.com/642329.html

Soo… it was a known problem for at least thirteen years. One of the issues removing it was how pervasive all of the FRM related things were. I shudder at the mention of “pack_flag” and Jay Pipes probably does too.

At the time, we tried a couple of approaches as to how things should look. Our philosophy with Drizzle was that it should get out of the way at let the storage engines be the storage engines and not try to second guess them or keep track of things behind their back. I still think that was the correct architectural approach: the role of Drizzle was to put SQL on top of a storage engine, not to also be one itself.

Looking at the MySQL code, there’s one giant commit 31350e8ab15179acab5197fa29d12686b1efd6ef. I do mean giant too, the diffstat is amazing:

 786 files changed, 58471 insertions(+), 25586 deletions(-)

How anyone even remotely did code review on that I have absolutely no idea. I know the only way I could get it to work in Drizzle was to do it incrementally, a series of patches that gradually chiseled out what needed to be taken out so I could put it an API and the protobuf code.

Oh, and in case you’re wondering:

- uint offset,pack_flag;
+ uint offset;

Thank goodness. Now, you may not appreciate that as much as I might, but pack_flag was not the height of design, it was… pretty much a catchalll for some kind of data about a field that wasn’t something that already had a field in the FRM. So it may include information on if the field could be null or not, if it’s decimal, how many bytes an integer takes, that it’s a number and how many oh, just don’t ask.

Also gone is the weird interval_id and a whole bunch of limitations because of the FRM format, including one that I either just discovered or didn’t remember: if you used all 256 characters in an enum, you couldn’t create the table as MySQL would pick either a comma or an unused character to be the separator in the FRM!?!

Also changed is how the MySQL server handles default values. For those not aware, the FRM file contains a static copy of the row containing default values. This means the default values are computed once on table creation and never again (there’s a bunch of work arounds for things like AUTO_INCREMENT and DEFAULT NOW()). The new sql/default_values.cc is where this is done now.

For now at least, table metadata is also written to a file that appears to be JSON format. It’s interesting that a SQL database server is using a schemaless file format to describe schema. It appears that these files exist only for disaster recovery or perhaps portable tablespaces. As such, I’m not entirely convinced they’re needed…. it’s just a thing to get out of sync with what the storage engine thinks and causes extra IO on DDL (as well as forcing the issue that you can’t have MVCC into the data dictionary itself).

What will be interesting is to see the lifting of these various limitations and how MariaDB will cope with that. Basically, unless they switch, we’re going to see some interesting divergence in what you can do in either database.

There’s certainly differences in how MySQL removed the FRM file to the way we did it in Drizzle. Hopefully some of the ideas we had were helpful in coming up with this different approach, as well as an extra seven years of in-production use.

At some point I’ll write something up as to the fate of Drizzle and a bit of a post-mortem, I think I may have finally worked out what I want to say…. but that is a post for another day.

Percona Live Europe Amsterdam PostgreSQL Day

This is my very first post on Planet PostgreSQL, so thank you for having me here! I’m not sure if you’re aware, but the PostgreSQL Events page lists the conference as something that should be of interest to PostgreSQL users and developers.

There is a PostgreSQL Day on October 4 2016 in Amsterdam, and if you’re planning on just attending a single day, use code PostgreSQLRocks and it will only cost €200+VAT.

I for one am excited to see Patroni: PostgreSQL High Availability made easy, Relational Databases at Uber: MySQL & Postgres, and Linux tuning to improve PostgreSQL performance: from hardware to postgresql.conf.

I’ll write notes here, if time permits we’ll do a database hackers lunch gathering (its good to mingle with everyone), and I reckon if you’re coming for PostgreSQL day, don’t forget to also signup to the Community Dinner at Booking.com.

September 22, 2016

First look at MySQL 8.0.0 Milestone

So, about ten days ago the MySQL Server Team released MySQL 8.0.0 Milestone to the world. One of the most unfortunate things about MySQL development is that it’s done behind closed doors, with the only hints of what’s to come arriving in maybe a note on a bug or such milestone releases that contain a lot of code changes. How much code change? Well, according to the text up on github for the 8.0 branch “This branch is 5714 commits ahead, 4 commits behind 5.7. ”

Way back in 2013, I looked at MySQL Code Size over releases, which I can again revisit and include both MySQL 5.7 and 8.0.0.

While 5.7 was a big jump again, we seem to be somewhat leveling off, which is a good thing. Managing to add features and fix long standing problems without bloating code size is good for software maintenance. Honestly, hats off to the MySQL team for keeping it to around a 130kLOC code size increase over 5.7 (that’s around 5%).

These days I’m mostly just a user of MySQL, pointing others in the right direction when it comes to some issues around it and being the resident MySQL grey(ing)beard(well, if I don’t shave for a few days) inside IBM as a very much side project to my day job of OPAL firmware.

So, personally, I’m thrilled about no more FRM, better Unicode, SET PERSIST and performance work. With my IBM hat on, I’m thrilled about the fact that it compiled on POWER out of the box and managed to work (I haven’t managed to crash it yet). There seems to be a possible performance issue, but hey, this is a huge improvement over the 5.7 developer milestones when run on POWER.

A lot of the changes are focused around usability, making it easier to manage and easier to run at at least a medium amount of scale. This is long overdue and it’s great to see even seemingly trivial things like SET PERSIST coming (I cannot tell you how many times that has tripped me up).

In a future post, I’ll talk about the FRM removal!

Inside North Korea, Russia Vs USA Part 3, and More

At times, you have to admit the international situation regarding North Korea is fairly comical. The core focus on it has basically been it's nuclear weapons programs but there's obviously there's a lot more to a country than just it's defense force. I wanted to take at what's happening inside. While they clearly have difficulties thing's don't seem entirely horrible? North Korea’s Nuclear

Lesson 124 in why scales on a graph matter…

The original article presented two graphs: one of MariaDB searches (which are increasing) and the other showing MySQL searches (decreasing or leveling out). It turns out that the y axis REALLY matters.

I honestly expected better….

Compiling your own firmware for Barreleye (OpenCompute OpenPOWER system)

Aaron Sullivan announced on the Rackspace Blog that you can now get your own Barreleye system! What’s great is that the code for the Barreleye platform is upstream in the op-build project, which means you can build your own firmware for them (just like garrison, the “IBM S822LC for HPC” system I blogged about a few days ago).

Remarkably, to build an image for the host firmware, it’s eerily similar to any other platform:

git clone --recursive https://github.com/open-power/op-build.git
cd op-build
. op-build-env
op-build barreleye_defconfig
op-build

…and then you wait. You can cross compile on x86.

You’ve been able to build firmware for these machines with upstream code since Feb/March (I wouldn’t recommend running with builds from then though, try the latest release instead).

Hopefully, someone involved in OpenBMC can write on how to build the BMC firmware.

September 19, 2016

LUV Main October 2016 Meeting: Sending Linux to Antarctica, 2012-2017 / Annual General Meeting

Oct 4 2016 18:30
Oct 4 2016 20:30
Oct 4 2016 18:30
Oct 4 2016 20:30
Location: 

6th Floor, 200 Victoria St. Carlton VIC 3053

Speakers:

• Scott Penrose, Sending Linux to Antarctica: 2012-2017
• Annual General Meeting and lightning talks

200 Victoria St. Carlton VIC 3053 (formerly the EPA building)

Before and/or after each meeting those who are interested are welcome to join other members for dinner. We are open to suggestions for a good place to eat near our venue. Maria's on Peel Street in North Melbourne is currently the most popular place to eat after meetings.

LUV would like to acknowledge Red Hat for their help in obtaining the venue.

Linux Users of Victoria Inc. is an incorporated association, registration number A0040056C.

October 4, 2016 - 18:30

read more

September 17, 2016

The Road to DrupalCon Dublin

DrupalCon Dublin is just around the corner. Earlier today I started my journey to Dublin. This week I'll be in Mumbai for some work meetings before heading to Dublin.

On Tuesday 27 September at 1pm I will be presenting my session Let the Machines do the Work. This lighthearted presentation provides some practical examples of how teams can start to introduce automation into their Drupal workflows. All of the code used in the examples will be available after my session. You'll need to attend my talk to get the link.

As part of my preparation for Dublin I've been road testing my session. Over the last few weeks I delivered early versions of the talk to the Drupal Sydney and Drupal Melbourne meetups. Last weekend I presented the talk at Global Training Days Chennai, DrupalCamp Ghent and DrupalCamp St Louis. It was exhausting presenting three times in less than 8 hours, but it was definitely worth the effort. The 3 sessions were presented using hangouts, so they were recorded. I gained valuable feedback from attendees and became aware of some bits of my talk needed some attention.

Just as I encourage teams to iterate on their automation, I've been iterating on my presentation. Over the next week or so I will be recutting my demos and polishing the presentation. If you have a spare 40 minutes I would really appreciate it if you watch one of the session recording below and leave a comment here with any feedback.

Global Training Days Chennai

Thumbnail frame from DrupalCamp Ghent presentation video

DrupalCamp Ghent

Thumbnail frame from DrupalCamp Ghent presentation video

Note: I recorded the audience not my slides.

DrupalCamp St Louis

Thumbnail frame from DrupalCamp St Louis presentation video

Note: There was an issue with the mic in St Louis, so there is no audio from their side.

September 15, 2016

Frankenwheezy! Keeping wheezy alive on a container host running libc6 2.24

It’s Alive!

The day before yesterday (at Infoxchange, a non-profit whose mission is “Technology for Social Justice”, where I do a few days/week of volunteer systems & dev work), I had to build a docker container based on an ancient wheezy image. It built fine, and I got on with working with it.

Yesterday, I tried to get it built on my docker machine here at home so I could keep working on it, but the damn thing just wouldn’t build. At first I thought it was something to do with networking, because running curl in the Dockerfile was the point where it was crashing – but it turned out that many programs would segfault – e.g. it couldn’t run bash, but sh (dash) was OK.

I also tried running a squeeze image, and that had the same problem. A jessie image worked fine (but the important legacy app we need wheezy for doesn’t yet run in jessie).

After a fair bit of investigation, it turned out that the only significant difference between my workstation at IX and my docker machine at home was that I’d upgraded my home machines to libc6 2.24-2 a few days ago, whereas my IX workstation (also running sid) was still on libc6 2.23.

Anyway, the point of all this is that if anyone else needs to run a wheezy on a docker host running libc6 2.24 (which will be quite common soon enough), you have to upgrade libc6 and related packages (and any -dev packages, including libc6-dev, you might need in your container that are dependant on the specific version of libc6).

In my case, I was using docker but I expect that other container systems will have the same problem and the same solution: install libc6 from jessie into wheezy. Also, I haven’t actually tested installing jessie’s libc6 on squeeze – if it works, I expect it’ll require a lot of extra stuff to be installed too.

I built a new frankenwheezy image that had libc6 2.19-18+deb8u4 from jessie.

To build it, I had to use a system which hadn’t already been upgraded to libc6 2.24. I had already upgraded libc6 on all the machines on my home network. Fortunately, I still had my old VM that I created when I first started experimenting with docker – crazily, it was a VM with two ZFS ZVOLs, a small /dev/vda OS/boot disk, and a larger /dev/vdb mounted as /var/lib/docker. The crazy part is that /dev/vdb was formatted as btrfs (mostly because it seemed a much better choice than aufs). Disk performance wasn’t great, but it was OK…and it worked. Docker has native support for ZFS, so that’s what I’m using on my real hardware.

I started with the base wheezy image we’re using and created a Dockerfile etc to update it. First, I added deb lines to the /etc/apt/sources.list for my local jessie and jessie-updates mirror, then I added the following line to /etc/apt/apt.conf:

APT::Default-Release "wheezy";

Without that, any other apt-get installs in the Dockerfile will install from jesssie rather than wheezy, which will almost certainly break the legacy app. I forgot to do it the first time, and had to waste another 10 minutes or so building the app’s container again.

I then installed the following:

apt-get -t jessie install libc6 locales libc6-dev krb5-multidev comerr-dev zlib1g-dev libssl-dev libpq-dev

To minimise the risk of incompatible updates, it’s best to install the bare minimum of jessie packages required to get your app running. The only reason I needed to install all of those -dev packages was because we needed libpq-dev, which pulled in all the rest. If your app doesn’t need to talk to postgresql, you can skip them. In fact, I probably should try to build it again without them – I added them after the first build failed but before I remembered to set Apt::Default::Release (OTOH, it’s working OK now and we’re probably better off with libssl-dev from jessie anyway).

Once it built successfully, I exported the image to a tar file, copied it back to my real Docker machine (co-incidentally, the same machine with the docker VM installed) and imported it into docker there and tested it to make sure it didn’t have the same segfault issues that the original wheezy image did. No problem, it worked perfectly.

That worked, so I edited the FROM line in the Dockerfile for our wheezy app to use frankenwheezy and ran make build. It built, passed tests, deployed and is running. Now I can continue working on the feature I’m adding to it, but I expect there’ll be a few more yaks to shave before I’m finished.

When I finish what I’m currently working on, I’ll take a look at what needs to be done to get this app running on jessie. It’s on the TODO list at work, but everyone else is too busy – a perfect job for an unpaid volunteer. Wheezy’s getting too old to keep using, and this frankenwheezy needs to float away on an iceberg.

Frankenwheezy! Keeping wheezy alive on a container host running libc6 2.24 is a post from: Errata

Moving to …

Last October data.gov.au was moved from the Department of Finance to the Department of Prime Minister and Cabinet (PM&C) and I moved with the team before going on maternity leave in January. In July of this year, whilst still on maternity leave, I announced that I was leaving PM&C but didn’t say what the next gig was. In choosing my work I’ve always tried to choose new areas, new parts of the broader system to better understand the big picture. It’s part of my sysadmin background – I like to understand the whole system and where the config files are so I can start tweaking and making improvements. These days I see everything as a system, and anything as a “config file”, so there is a lot to learn and tinker with!

Over the past 3 months, my little family (including new baby) has been living in New Zealand on a bit of a sabbatical, partly to spend time with the new bub during that lovely 6-8 months period, but partly for us to have the time and space to consider next steps, personally and professionally. Whilst in New Zealand I was invited to spend a month working with the data.govt.nz team which was awesome, and to share some of my thoughts on digital government and what systemic “digital transformation” could mean. It was fun and I had incredible feedback from my work there, which was wonderful and humbling. Although tempting to stay, I wanted to return to Australia for a fascinating new opportunity to expand my professional horizons.

Thus far I’ve worked in the private sector, non-profits and voluntary projects, political sphere (as an advisor), and in the Federal and State/Territory public sectors. I took some time whilst on maternity leave to think about what I wanted to experience next, and where I could do some good whilst building on my experience and skills to date. I had some interesting offers but having done further tertiary study recently into public policy, governance, global organisations and the highly complex world of international relations, I wanted to better understand both the regulatory sphere and how international systems work. I also wanted to work somewhere where I could have some flexibility for balancing my new family life.

I’m pleased to say that my next gig ticks all the boxes! I’ll be starting next week at AUSTRAC, the Australian financial intelligence agency and regulator where I’ll be focusing on international data projects. I’m particularly excited to be working for the brilliant Dr Maria Milosavljevic (Chief Innovation Officer for AUSTRAC) who has a great track record of work at a number of agencies, including as CIO of the Australian Crime Commission. I am also looking forward to working with the CEO, Paul Jevtovic APM, who is a strong and visionary leader for the organisation, and I believe a real change agent for the broader public sector.

It should be an exciting time and I look forward to sharing more about my work over the coming months! Wish me luck :)

September 12, 2016

Diplomacy Part 2, Russia Vs USA Part 2, and More

This is obviously a continuation of my last post, http://dtbnguyen.blogspot.com/2016/09/diplomacy-russia-vs-usa-and-more.html - if you've ever been exposed to communism then you'll realise that there are those who have been exposed to the 'benefits' of capitalism look back upon it harshly. It's taken me a long while but I get where they're coming from. Many times regional and the global

Pia, Thomas and Little A’s Excellent Adventure – Final days

Well, the last 3 months just flew past on our New Zealand adventure! This is the final blog post. We meant to blog more often but between limited internet access and being busy getting the most of our much needed break, we ended up just doing this final post. Enjoy!

Photos were added every week or so to the flickr album.
Our NZ Adventure

Work

I was invited to spend 4 weeks during this trip working with the Department of Internal Affairs in the New Zealand Government on beta.data.govt.nz and a roadmap for data.govt.nz. The team there were just wonderful to work with as were the various people I met from across the NZ public sector. It was particularly fascinating to spend some time with the NZ Head Statistician Liz MacPherson who is quite a data visionary! It was great to get to better know the data landscape in New Zealand and contribute, even in a small way, to where the New Zealand Government could go next with open data, and a more data-driven public sector. I was also invited to share my thoughts on where government could go next more broadly, with a focus on “gov as an API” and digital transformation. It really made me realise how much we were able to achieve both with data.gov.au from 2013-2015 and in the 8 months I was at the Digital Transformation Office. Some of the strategies, big picture ideas and clever mixes of technology and system thinking created some incredible outcomes, things we took for granted from the inside, but things that are quite useful to others and are deserving of recognition for the amazing public servants who contributed. I shared with my New Zealand colleagues a number of ideas we developed at the DTO in the first 8 months of the “interim DTO”, which included the basis for evidence based service design, delivery & reporting, and a vision for how governments could fundamentally change from siloed services to modular and mashable government. “Mashable government” enables better service and information delivery, a competitive ecosystem of products and services, and the capability to automate system to system transactions – with citizen permission of course – to streamline complex user needs. I’m going to do a dedicated blog post later on some of the reflections I’ve had on that work with both data.gov.au and the early DTO thinking, with kudos to all those who contributed.

I mentioned in July that I had left the Department of Prime Minister and Cabinet (where data.gov.au was moved to in October 2015, and I’ve been on maternity leave since January 2016). My next blog post will be about where I’m going and why. You get a couple of clues: yes it involves data, yes it involves public sector, and yes it involves an international component. Also, yes I’m very excited about it!! Stay tuned ;)

Fishing

When we planned this trip to New Zealand, Thomas has some big numbers in mind for how many fish we should be able to catch. As it turned out, the main seasonal run of trout was 2 months later than usual so for the first month and a half of our trip, it looked unlikely we would get anywhere near what we’d hoped. We got to about 100 fish, fighting for every single one (and keeping only about 5) and then the run began! For 4 weeks of the best fishing of the season I was working in Wellington Mon-Fri, with Little A accompanying me (as I’m still feeding her) leaving Thomas to hold the fort. I did manage to get some great time on the water after Wellington, with my best fishing session (guided by Thomas) resulting in a respectable 14 fish (over 2 hours). Thomas caught a lazy 42 on his best day (over only 3 hours), coming home in time for breakfast and a cold compress for his sprained arm. All up our household clocked up 535 big trout (mostly Thomas!) of which we only kept 10, all the rest were released to swim another day. A few lovely guests contributed to the numbers so thank you Bill, Amanda, Amelia, Miles, Glynn, Silvia and John who together contributed about 40 trout to our tally!

Studies

My studies are going well. I now have only 1.5 subjects left in my degree (the famously elusive degree, which was almost finished and then my 1st year had to be repeated due to doing it too long ago for the University to give credit for, gah!). To finish the degree, a Politics degree with loads of useful stuff for my work like public policy, I quite by chance chose a topic on White Collar Crime which was FASCINATING!

Visitors

Over the course of the 3 months we had a number of wonderful guests who contributed to the experience and had their own enjoyable and relaxing holidays with us in little Turangi: fishing, bushwalking, going to the hot pools and thermal walks, doing high tea at the Tongariro Chateau at Whakaapa Village, Huka Falls in Taupo, and even enjoying some excellent mini golf. Thank you all for visiting, spending time with us and sharing in our adventure. We love you all!

Little A

Little A is now almost 8 months old and has had leaps and bounds in development from a little baby to an almost toddler! She has learned to roll and commando crawl (pulling herself around with her arms only) around the floor. She loves to sit up and play with her toys and is eating her way through a broad range of foods, though pear is still her favourite. She is starting to make a range of noises and the race is on as to whether she’ll say ma or da first :) She has quite the social personality and we adore her utterly! She surprised Daddy with a number of presents on Father’s Day, and helped to make our first family Father’s Day memorable indeed.

Salut Turangi

And so it’s with mixed feelings that we bid adieu to the sleepy town of Turangi. It’s been a great adventure, with lots of wonderful memories and a much-needed chance to get off the grid for a while, but we’re both looking forward to re-entering respectable society, catching up with those of you that we haven’t seen for a while, and planning our next great adventure. We’ll be back in Turangi in February for a different adventure with friends of ours from the US, but that will be only a week or so. Turangi is a great place, and if you’re ever in the area stop into the local shopping centre and try one of the delicious pork and watercress or lamb, mint and kumara pies available from the local bakeries – reason enough to return again and again.

Speaking in September 2016

A few events, but mostly circling around London:

  • Open collaboration – an O’Reilly Online Conference, at 10am PT, Tuesday September 13 2016 – I’m going to be giving a new talk titled Forking Successfully. I’ve seen how the platform works, and I’m looking forward to trying this method out (its like a webminar but not quite!)
  • September MySQL London Meetup – I’m going to focus on MySQL, a branch, Percona Server and the fork MariaDB Server. This will be interesting because one of the reasons you don’t see a huge Emacs/XEmacs push after about 20 years? Feature parity. And the work that’s going into MySQL 8.0 is mighty interesting.
  • Operability.io should be a fun event, as the speakers were hand-picked and the content is heavily curated. I look forward to my first visit there.

Compiling your own firmware for the S822LC for HPC

IBM (my employer) recently announced  the new S822LC for HPC POWER8+NVLINK NVIDIA P100 GPUs server (press release, IBM Systems Blog, The Register). The “For HPC” suffix on the model number is significant, as the S822LC is a different machine. What makes the “for HPC” variant different is that the POWER8 CPU has (in addition to PCIe), logic for NVLink to connect the CPU to NVIDIA GPUs.

There’s also the NVIDIA Tesla P100 GPUs which are NVIDIA’s latest in an SXM2 form factor, but instead of delving into GPUs, I’m going to tell you how to compile the firmware for this machine.

You see, this is an OpenPOWER machine. It’s an OpenPOWER machine where the vendor (in this case IBM) has worked to get all the needed code upstream, so you can see exactly what goes into a firmware build.

To build the latest host firmware (you can cross compile on x86 as we use buildroot to build a cross compiler):

git clone --recursive https://github.com/open-power/op-build.git
cd op-build
. op-build-env
op-build garrison_defconfig
op-build

That’s it! Give it a while and you’ll end up with output/images/garrison.pnor – which is a firmware image to flash onto PNOR. The machine name is garrison as that’s the code name for the “S822LC for HPC” (you may see Minsky in the press, but that’s a rather new code name, Garrison has been around for a lot longer as a name).

September 10, 2016

Software Freedom Day Meeting 2016

Sep 17 2016 10:00
Sep 17 2016 16:30
Sep 17 2016 10:00
Sep 17 2016 16:30
Location: 

Electron Workshop 31 Arden Street, North Melbourne.

There will not be a regular LUV Beginners workshop for the month of September. Instead, you're going to be in for a much bigger treat!

This month, Free Software Melbourne[1], Linux Users of Victoria[2] and Electron Workshop[3] are joining forces to bring you the local Software Freedom Day event for Melbourne.

The event will take place on Saturday 17th September between 10am and 4:30pm at:

Electron Workshop
31 Arden Street, North Melbourne.
Map: http://www.sfd.org.au/melbourne/

Electron Workshop is on the south side of Arden Street, about half way between Errol Street and Leveson Street. Public transport: 57 tram, nearest stop at corner of Errol and Queensberry Streets; 55 and 59 trams run a few blocks away along Flemington Road; 402 bus runs along Arden Street, but nearest stop is on Errol Street. On a Saturday afternoon, some car parking should be available on nearby streets.

LUV would like to acknowledge Red Hat for their help in obtaining the Trinity College venue.

Linux Users of Victoria Inc., is an incorporated association, registration number A0040056C.

September 17, 2016 - 10:00

read more

September 09, 2016

Houndbot suspension test fit

I now have a few crossover plates in the works to hold the upgraded suspension in place. See the front wheel of the robot on your right. The bottom side is held in place with a crossover to go from the beam to a 1/4 inch bearing mount. The high side uses one of the hub mount brackets which are a fairly thick alloy and four pretapped attachment blocks. To that I screw my newly minted alloy blocks which have a sequence of M8 sized holes in them. I was unsure of the final fit on the robot so made three holes to give me vertical variance to help set the suspension in the place that I want.



Notice that the high tensile M8 bolt attached to the top suspension is at a slight angle. In the end the top of the suspension will be between the two new alloy plates. But to do that I need to trim some waste from the plates, but to do that I needed to test mount to see where and what needs to be trimmed. I now have an idea of what to trim for a final test mount ☺.

Below is a close up view of the coil over showing the good clearance from the tire and wheel assembly and the black markings on the top plate giving an idea of the material that I will be removing so that the top tension nut on the suspension clears the plate.


 The mounting hole in the suspension is 8mm diameter. The bearing blocks are for 1/4 inch (~6.35mm) diameters. For test mounting I got some 1/4 inch threaded rod and hacked off about what was needed to get clear of both ends of the assembly. M8 nylock nuts on both sides provide a good first mounting for testing. The crossover plate that I made is secured to the beam by two bolts. At the moment the bearing block is held to the crossover by JB Weld only, I will likely use that to hold the piece and drill through both chunks of ally and bolt them together too. It's somewhat interesting how well these sorts of JB and threaded rod assemblies seem to work though. But a fracture in the adhesive at 20km/h when landing from a jump without a bolt fallback is asking for trouble.


The top mount is shown below. I originally had the shock around the other way, to give maximum clearance at the bottom so the tire didn't touch the shock. But with the bottom mount out this far I flipped the shock to give maximum clearance to the top mounting plates instead.


So now all I need is to cut down the top plates, drill bolt holes for the bearing to crossover plate at the bottom, sand the new bits smooth, and maybe I'll end up using the threaded rod at the bottom with some JB to soak up the difference from 1/4 inch to M8.

Oh, and another order to get the last handful of parts needed for the mounting.

APM:Plane 3.7.0 released

The ArduPilot development team is proud to announce the release of version 3.7.0 of APM:Plane. This is a major update so please read the notes carefully.

The biggest changes in this release are:

  • more reliable recovery from inverted flight
  • automatic IC engine support
  • Q_ASSIST_ANGLE for stall recovery on quadplanes
  • Pixhawk2 IMU heater support
  • PH2SLIM support
  • AP_Module support
  • Parrot Disco support
  • major VRBrain support merge
  • much faster boot time on Pixhawk

I'll give a bit of detail on each of these changes before giving the more detailed list of changes.

More reliable recovery from inverted flight

Marc Merlin discovered that on some types of gliders that ArduPilot would not reliably recover from inverted flight. The problem turned out to be the use of the elevator at high bank angles preventing the ailerons from fully recovering attitude. The fix in this release prevent excessive elevator use when the aircraft is beyond LIM_ROLL_CD. This should help a lot for people using ArduPilot as a recovery system for manual FPV flight.

Automatic IC engine support

ArduPilot has supported internal combustion engines for a long time, but until now the pilot has had to control the ignition and starter manually using transmitter pass throughs. A new "ICE" module in ArduPilot now allows for fully automatic internal combustion engine support.

Coupled with an RPM sensor you can setup your aircraft to automatically control the ignition and starter motor, allowing for one touch start of the motor on the ground and automatic restart of the motor in flight if needed.

The IC engine support is also integrated into the quadplane code, allowing for automatic engine start at a specified altitude above the ground. This is useful for tractor engine quadplanes where the propeller could strike the ground on takeoff. The engine can also be automatically stopped in the final stage of a quadplane landing.

Q_ASSIST_ANGLE for stall recovery

Another new quadplane feature is automatic recovery from fixed wing stall. Previously the VTOL motors would only provide assistance in fixed wing modes when the aircraft airspeed dropped below Q_ASSIST_SPEED. Some stalls can occur with higher airspeed however, and this can result in the aircraft losing attitude control without triggering a Q_ASSIST_SPEED recovery. A new parameter Q_ASSIST_ANGLE allows for automatic assistance when attitude control is lost, triggering when the attitude goes outside the defined roll and pitch limits and is more than Q_ASSIST_ANGLE degrees from the desired attitude. Many thanks to Iskess for the suggestion and good discussion around this feature.

Pixhawk2 heated IMU support

This release adds support for the IMU heater in the upcoming Pixhawk2, allowing for more stable IMU temperatures. The Pixhawk2 is automatically detected and the heater enabled at boot, with the target IMU temperature controllable via BRD_IMU_TARGTEMP.

Using an IMU heater should improve IMU stability in environments with significant temperature changes.

PH2SLIM Support

This release adds support for the PH2SLIM variant of the Pixhawk2, which is a Pixhawk2 cube without the isolated sensor top board. This makes for a very compact autopilot for small aircraft. To enable PH2SLIM support set the BRD_TYPE parameter to 6 using a GCS connected on USB.

AP_Module Support

This is the first release of ArduPilot with loadable module support for Linux based boards. The AP_Module system allows for externally compiled modules to access sensor data from ArduPilot controlled sensors. The initial AP_Module support is aimed at vendors integrating high-rate digital image stabilisation using IMU data, but it is expected this will be expanded to other use cases in future releases.

Parrot Disco Support

This release adds support for the Parrot C.H.U.C.K autopilot in the new Disco airframe. The Disco is a very lightweight flying wing with a nicely integrated Linux based autopilot. The Disco flies very nicely with ArduPilot, bringing the full set of mission capabilities of ArduPilot to this airframe.

Major VRBrain Support Update

This release includes a major merge of support for the VRBrain family of autopilots. Many thanks to the great work by Luke Mike in putting together this merge!

Much Faster Boot Time

Boot times on Pixhawk are now much faster due to a restructuring of the driver startup code, with slow starting drivers not started unless they are enabled with the appropriate parameters. The restructuring also allows for support of a wide variety of board types, including the PH2SLIM above.

This release includes many other updates right across the flight stack, including several new features. Some of the changes include:

  • improved quadplane auto-landing
  • limit roll and pitch by Q_ANGLE_MAX in Q modes
  • improved ADSB avoidance and MAVLink streaming
  • smoother throttle control on fixed-wing to VTOL transition
  • removed "demo servos" movement on boot
  • fixed a problem with spurious throttle output during boot (thanks
  • to Marco for finding this)
  • support MAVLink SET_ATTITUDE_TARGET message
  • log all rally points on startup
  • fixed use of stick mixing for rudder with STICK_MIXING=0
  • fixed incorrect tuning warnings when vtol not active
  • support MAVLink based external GPS device
  • support LED_CONTROL MAVLink message
  • prevent baro update while disarmed for large height change
  • support PLAY_TUNE MAVLink message
  • added AP_Button support for remote button input reporting
  • support Ping2020 ADSB transceiver
  • fixed disarm by rudder in quadplanes
  • support 16 channel SERVO_OUTPUT_RAW in MAVLink2
  • added automatic internal combustion engine support
  • support DO_ENGINE_CONTROL MAVLink message
  • added ground throttle suppression for quadplanes
  • added MAVLink reporting of logging subsystem health
  • prevent motor startup on reboot in quadplanes
  • added quadplane support for Advanced Failsafe
  • added support for a 2nd throttle channel
  • fixed bug in crash detection during auto-land flare
  • lowered is_flying groundspeed threshold to 1.5m/s
  • added support for new FrSky telemetry protocol varient
  • added support for fence auto-enable on takeoff in quadplanes
  • added Q_ASSIST_ANGLE for using quadplane to catch stalls in fixed wing flight
  • added BRD_SAFETY_MASK to allow for channel movement for selected channels with safety on
  • numerous improvements to multicopter stability control for quadplanes
  • support X-Plane10 as SITL backend
  • lots of HAL_Linux improvements to bus and thread handling
  • fixed problem with elevator use at high roll angles that could
  • prevent attitude recovery from inverted flight
  • improved yaw handling in EKF2 near ground
  • added IMU heater support on Pixhawk2
  • allow for faster accel bias learning in EKF2
  • fixed in-flight yaw reset bug in EKF2
  • added AP_Module support for loadable modules
  • support Disco airframe from Parrot
  • use full throttle in initial takeoff in TECS
  • added NTF_LED_OVERRIDE support
  • added terrain based simulation in SITL
  • merged support for wide range of VRBrain boards
  • added support for PH2SLIM and PHMINI boards with BRD_TYPE
  • greatly reduced boot time on Pixhawk and similar boards
  • fixed magic check for signing key in MAVLink2
  • fixed averaging of gyros for EKF2 gyro bias estimate

Many thanks to the many people who have contributed to this release, and happy flying!

September 08, 2016

Speaking at Percona Live Europe Amsterdam

I’m happy to speak at Percona Live Europe Amsterdam 2016 again this year (just look at the awesome schedule). On my agenda:

I’m also signed up for the Community Dinner @ Booking.com, and I reckon you should as well – only 35 spots remain!

Go ahead and register now. You should be able to search Twitter or the Percona blog for discount codes :-)

September 06, 2016

Standard versus Reality

While dereferencing a NULL pointer may be undefined, there’s a legitimate reason to do so: you want to store something at address 0. Why? Well, not all of us are fancy and have an MMU turned on.

Intercepting hotplug on the Freecom FSG-3

The Freecom FSG-3 wireless storage router has four USB ports, and has support for hotplug built into the kernel.  This makes it ideal for use as a docking station for OpenMoko phones.

Unfortunately, it does not have the normal hotplug agent scripts that you expect to find on a desktop Linux distribution.

So you have to roll your own:

  1. Run “mv /sbin/hotplug /sbin/hotplug.freecom
  2. Create a new “/sbin/hotplug” shell script (the following is an example of how to automatically enable USB networking for an OpenMoko phone):
    #!/bin/sh
    
    case $1 in
      ( usb )
        case $PRODUCT/$INTERFACE in
          ( 1457/5122/212/2/6/0 ) # OpenMoko GTA01 cdc-ether
            case $ACTION in
              ( add )
                ifconfig usb0 192.168.0.200 up
                ;;
              ( remove )
                ifconfig usb0 192.168.0.200 down
                ;;
            esac
            ;;
        esac
        ;;
    esac
    
    /sbin/hotplug.freecom "$@"
    

  3. Run “chmod ugo+x /sbin/hotplug” to ensure that your new hotplug script is executable.
  4. See http://linux-hotplug.sourceforge.net/?selected=usb for the list of environment variables you can use to distinguish different devices.

Syncing Treo650 with Google Contacts using CompanionLink for Google

In preparation for a possible move from my Treo 650 to the new Palm Pre, I’ve decided to try and synchronise my contacts between the Treo and Google Contacts.

So I’m evaluating CompanionLink for Google as a possible tool to achieve this. Another option might be GooSync.

I tried syncing some sample contacts in both directions, with the following results:

  1. Google Contacts to PalmOS Contacts
  2. Syncing a new contact from Google Contacts to PalmOS Contacts results in the following fields being synched correctly:

    • Name
    • Title
    • Company
    • Home Phone
    • Work Phone
    • Mobile Phone
    • E-mail (synched with the Work Email field in Google Contacts)
    • Fax (synched with the Work Fax field in Google Contacts)
    • Home Address
    • Work Address
    • Other Address
    • Notes

    The following Google Contacts fields are not synched:

    • Home Email
    • Other Email
    • Home Fax
    • Pager
    • Other Phone
    • IM Fields (Google Talk, Skype, Jabber, etc)
    • Custom Field

  3. PalmOS Contacts to Google Contacts
  4. Syncing a new contact from PalmOS Contacts to Google Contacts results in the following fields being synched correctly:

    • Name
    • Title
    • Company
    • Work Email (synched with the first E-mail field in PalmOS Contacts)
    • Home Phone
    • Work Phone
    • Mobile Phone
    • Home Address
    • Work Address
    • Other Address
    • Google Talk (synched with the IM field in PalmOS Contacts)
    • Notes (synched with a combination of the Custom and Note fields in PalmOS Contacts)

    The following PalmOS Contacts fields are not synched:

    • Secondary E-mail entries
    • Other Phone
    • AIM
    • MSN
    • Web site

I then tried duplicating contacts to see if I could determine the primary synchronisation key. Duplicating a contact in PalmOS Contacts and then synchronising did not result in a duplicated contact in Google Contacts. However, changing the E-Mail field in the duplicated contact in PalmOS Contacts was enough to cause it to be created as a second separate record in Google Contacts. So it seems that the PalmOS E-Mail field (which syncs with the Google Work Email field) is the primary key.
Interestingly, even while the PalmOS Contacts HotSync conduit is set up to sync with Google Contacts, the syncing with the Palm Desktop still happens. Indeed, the deletion of a record in the Palm Desktop is reflected in PalmOS Contacts on each sync, but it seems does not trigger a corresponding deletion in Google Contacts (perhaps there is some QuickSync vs SlowSync thing happening here). Modifying the record in Google Contacts which had already been deleted in PalmOS Contacts (through the Palm Desktop) did cause it to be reinstated in PalmOS Contacts on the next sync.
Adding a new E-Mail field to the PalmOS Contacts record (before the existing field) causes that new field to be the one that is synched with the Google Contacts Work Email field. So it seems that synchronisation happens between the first E-Mail field in PalmOS Contacts and the Work Email field in Google Contacts, and that only one Email address is ever synchronised between the two. If there is no Work Email field in Google Contacts, then all E-Mail fields in PalmOS Contacts are deleted. Additional Email fields in Google Contacts are not replicated in PalmOS Contacts. If an additional E-Mail field is added to PalmOS Contacts, then synchronisation exits with an error on the first attempt (“Can have at most one primary email address, found 2″), and prevents other fields being synchronisedbut then succeeds on the second attempt.  As long as a Work Email field is synching properly, other non-synching Email fields on both sides are retained (but not synched, even though other non-Email fields are synched).

Five new NSLU2 firmware releases in five days

In the last five days, we have made five new NSLU2 firmware releases:

2007-12-31 – Unslung 6.10-beta Release
2007-12-30 – SlugOS 4.8-beta Release
2007-12-29 – OpenWrt/NSLU2 Kamikaze 7.09 Release
2007-12-28 – Angstrom/NSLU2 2007.12 Release
2007-12-27 – Debian/NSLU2 Stable 4.0r2 Release

All of these new releases are available at

http://www.slug-firmware.net/

See http://article.gmane.org/gmane.comp.misc.nslu2.linux/20610 for
an explanation of the pros and cons of each different firmware
distribution, and the installable packages available for each.

Thanks to everyone in the NSLU2-Linux, OpenWrt, Angstrom, OpenEmbedded
and Debian projects who contributed to these releases.

Remember, if you find any of the firmware or packages that the
NLSU2-Linux project provides useful, feel free to make a donation to
the project at

http://www.nslu2-linux.org/wiki/Main/Donate

We are currently in need of about $500 to buy a RAID controller card
and some disks for our autobuild machine to support all this new
firmware with up-to-date package feeds …

The Definitive Analysis of Palm Pre App Install Limits and the Palm App Catalog Hanging

After the Preware 0.9.4 release, which included Applications, Themes, and Patches, and offers over 670 homebrew packages for installation, we (webos-internals.org)  started getting reports of the Palm App Catalog “hanging” the user interface for 30 seconds or more when the installation of a new application is initiated, but only when the Package Manager Service (the service which does all the Linux-level work for the Preware front-end) was installed.

After some analysis, I found that disabling all the feeds in Preware reduced the “hang” from more than 30 seconds to less than a second.

Looking through the output of ‘dbus-util –capture’ showed that the “hang” was during the call to the queryInstallCapacity method of the com.palm.appinstaller service, the routine that the Palm App Catalog application uses to determine whether there is enough space to install a new application from the Palm App Catalog.  Yes, this is the method which is at the heart of the dreaded “Too many applications” errors that people are seeing when they have a number of homebrew applications installed and try to install a new application from the Palm App Catalog.

Watching the output of ‘ps -ef’ during calls to this method (you can call it manually using luna-send) showed that palm was calling “ipkg -o /var list”.  Curious.  Why would you want to know the *complete* list of all *available* applications when determining whether there is room to install one known new application.  I suspect that Palm should be calling “ipkg -o /var list_installed” instead (which just lists the installed applications).  Note that Palm doesn’t use feeds the way that Preware does, so for Palm’s official use of ipkg, list and list_installed would return the same thing in their testing, but list_installed is definitely what they should be using to determine the size of installed applications.

The plot thickens when you use strace (which Palm conveniently includes in the official firmware image) on the running LunaSysMgr process.

It seems that LunaSysMgr calls “ipkg -o /var list” to get a list of installed packages (the probably intend to just get the list of installed packages, but when you have Preware installed and have feed configuration files in /var/etc/ipkg/*.conf, it actually returns a list of all *available packages).

LunaSysMgr then does an execve of “/usr/bin/du -s /var/usr/palm/applications/package” for each package in that list.  (BTW Palm, you seem to have a bug in the logic of that code, cause it’s running du on random garbage strings after the end of the real package list)

Yes, that’s right.  A call to queryInstallCapacity spawns a new program (“du”) once for each package returned by “ipkg -o /var list”.  No wonder the UI hangs for 30 seconds or more!

A single “du -s /var/usr/palm/applications/*” would be a far more efficient way to get exactly the same information, but again, Palm would not see this difference in testing because they do not support the third-party Preware application usage of ipkg feeds.

You can imagine that this behaviour is probably related to the app install limit that many people are experiencing too.  Unfortunately, I’ll have to reduce my /var partition size down from it’s current 2GB size (courtesy of the WebOS Internals Meta-Doctor) to be able to investigate this one.

Now the webos-internals.org developers need to develop a new method of installing homebrew applications so that this bug in Palm’s appInstaller service is not triggered.

In the meantime, the work-around is to go into the Preware Preferences screen, hit the “Feeds” button in the top-right corner, and disable all the feeds while you use the Palm App Catalog application in another card.  No need to exit the Feeds screen, just toggle all the button to “Off”, and then toggle them back to “On” when you’re finished with the App Catalog.

For the solution to this problem, see Update #2, below.

I’ve created a thread on PreCentral where this issue can be discussed.  As I uncover more information, I’ll publish my finding here.

Update #1: I’ve now webOS Doctored my Pre in the name of science, and have tested the limits of installing applications.

If you run “du -s /var/usr/palm/applications/*”, and add up all the numbers in the first column, then as soon as you hit the 62367 1K blocks limit of the addition of the sizes reported by that “du” command and the size of the app you with to install, you will get the dreaded “Sorry, Not Enough Memory” error from the Palm App Catalog application (and any other installer, like fileCoaster or PreLoad, which uses the palm appInstaller API).  It doesn’t matter whether you have 192MB free in your /var partition, it will max out at just under 64MB of application usage.

Update #2: I have now created a Linux Application called “Fair Dinkum App Limit” (org.webosinternals.fairdinkum), which removes both the “hang” and the arbitrary application limit.  You can find it in Preware.  Just install it (no need to even run anything – if it is installed, it’s working), and you’re ready to install more applications than you can poke a stick at …

Fair Dinkum App Limit works by simply putting a couple of wrapper scripts in /usr/local/bin, which returns a size of zero when du is called, and returns the output of “ipkg -o /var list_installed” when “ipkg -o /var list” is called.  In the future, the wrappers will be made much more sophisticated than they are right now to prevent operation outside of the specific cases where they need to fool LunaSysMgr, and to also include a safety buffer so that users do not fill the /var partition.  This is a tactical response to a problem that people using homebrew applications are experiencing.  Hopefully, Palm will provide the long term solution for limits on application installation in a release in the near future.

Notes for Palm, if you happen to read this:

1) We fully appreciate that the usage of the ipkg database in /var for homebrew applications is a choice that the homebrew community has made, and is not supported by Palm.

2) We fully agree that the use of “ipkg -o /var list” instead of “ipkg -o /var list_installed” would work perfectly fine for the way that Palm is officially using the ipkg database in /var/usr/lib/ipkg, but we contend that the “list” and “list_installed” commands have clear intended usage, and the one to find the list of installed applications for checking available disk space should be “list_installed”.

3) We fully realise that the initial version of the FairDinkum scripts are unsafe.  Returning a zero value for du is a temporary solution while we work out how to achieve the same result safely.  The intention is to only return false values when du is being called from LunaSysMgr, and to make sure that a safety buffer is kept so that users do not fill the /var partition.

4) I would be very happy to discuss these issues with anyone at Palm (Chuq and Google both have my email address), and would hope that we can together architect a solution for supporting homebrew application installation which does not require these work-arounds.

5) We have designed these work-arounds in a way which does not collide with Palm OTA Updates, and in a way that we can update them automatically using the Preware installer application, and in a way that we can cause them to self-remove when Palm releases a long term solution.

Update #3:

It seems that there is yet another limit on application installation imposed by LunaSysMgr.  Once the used space on /var crosses around 60%, LunaSysMgr will once again refuse to install applications.

I’m going to need to webOS Doctor my Pre yet again (to reallocate 2GB to /var) to determine whether this limit is a fixed percentage, or a fixed number of blocks.

Update #4:

The limit is 102400 free blocks.  Mystery solved.  That also means the Fair Dinkum App Limit cannot cause your /var to be filled.

Update #5:

Thanks to Carl Thompson, an improved version of Fair Dinkum App Limit which does not alter the operation of ‘du’ has been released.

Replacing dropbear with openssh

I prefer to use OpenSSH rather than Dropbear on my devices.  The main reason is to get sftp support (which is required by sshfs).  Another reason is to get better support for agent forwarding (which is essential for bouncing from one machine to another without leaving your private keys all over the internet).

To do this on OpenMoko (or any other OpenEmbedded-based distribution for that matter, for instance SlugOS or Angstrom):

  1. Edit /etc/init.d/dropbear by replacing “DROPBEAR_PORT=22” with “DROPBEAR_PORT=2222” (or any other unused port).
  2. Run “ipkg install -force-depends openssh” to install openssh.
  3. Make sure you have set a root password before rebooting (use “passwd” to set it).
  4. Reboot (dropbear will restart on the new port, and openssh will start on the normal ssh port).
  5. Check that openssh is now serving on port 22 by logging into the device over ssh.
  6. Run “ipkg remove -force-depends dropbear” to remove dropbear.
  7. Then run “ipkg install openssh-sftp” to install support for the sftp protocol which sshfs uses.

Palm Pre GPS doesn’t like my hemisphere

It seems the Palm Pre GPS was never tested in the southern hemisphere – my new Pre’s GPS reports Lat: 394.6, Long: 138.6

24926.609       PUB     call    460             :1.26   com.palm.location       //getCurrentPosition  «string=“{}”, string=“com.sfmpllc.sendmylocation 1058”»
24926.641       PRV     call    238             :1.68   com.palm.phone  /com/palm/phone/tel_getradiotype      «»
24926.661       PRV     return  238     0.020   :1.26   :1.68           «string=“success”, string=“CDMA”»
24926.751       PRV     call    239             :1.68   com.palm.phone  /com/palm/phone/tel_getbsinfo «»
24926.786       PUB     call    461             com.palm.luna   org.freedesktop.DBus    /org/freedesktop/DBus/AddMatch        «string=“interface=org.freedesktop.DBus,member=NameOwnerChanged,arg0=com.palm.location”»
24926.920       PUB     return  460             com.palm.location       com.palm.luna        «string=“{“errorCode”:0,”timestamp”:1.254820510841E12,”latitude”:394.593215,”longitude”:138.681593,”horizAccuracy”:150,”heading”:0,”velocity”:0,”altitude”:0,”vertAccuracy”:0}”»

24926.609       PUB     call    460             :1.26   com.palm.location       //getCurrentPosition  «string=“{}”, string=“com.sfmpllc.sendmylocation 1058”»

24926.920       PUB     return  460             com.palm.location       com.palm.luna        «string=“{“errorCode”:0,”timestamp”:1.254820510841E12,”latitude”:394.xxxxxx,”longitude”:138.xxxxxx,”horizAccuracy”:150,”heading”:0,”velocity”:0,”altitude”:0,”vertAccuracy”:0}”»

The latitude value should be 34.6 degrees South (or -34.6 degrees).

That would explain why Google Maps isn’t working.

Now I need to work out how to replace the Coordinates java class in /usr/lib/luna/java/location.jar, so that the getLatitude method returns a number between -90 and +90 …

I wonder how many WebOS applications will then barf on a negative latitude value …

The PreCentral thread has more information on other GPS tweaks.

Connecting a Treo650 to a Freecom DataTank 2

  1. Install bluez2-utils from Optware
  2. Install the following kernel modules: bluetooth, hci_usb, l2cap, bnep, rfcomm, hidp
  3. Create /dev/rfcomm0 as follows:
    mknod /dev/rfcomm0 c 216 0
  4. Enable routing from ppp0 to eth1 (don’t do this if you use ppp for your internet connection):
    /etc/init.d # diff -u routing.orig routing
    --- routing.orig        Sat Mar 22 18:57:23 2008
    +++ routing     Sat Mar 22 15:14:29 2008
    @@ -37,6 +37,7 @@
    
            # lo & eth0 always accepted (also if WAN port IP not set)
            /sbin/iptables -A INPUT -i $INIF -j ACCEPT
    +       /sbin/iptables -A INPUT -i ppp0 -j ACCEPT
            /sbin/iptables -A INPUT -i lo -j ACCEPT
    
            # get IP address from WAN port
    @@ -150,6 +151,7 @@
              /sbin/iptables -A FORWARD -j TCPMSS -o $EXIF --clamp-mss-to-pmtu -p tcp --tcp-flags SYN,RST SYN
            fi
            /sbin/iptables -A FORWARD -i $EXIF -o $INIF -m state --state ESTABLISHED,RELATED -j ACCEPT
    +       /sbin/iptables -A FORWARD -i $EXIF -o ppp0 -m state --state ESTABLISHED,RELATED -j ACCEPT
    
    
            grep -q ^proxy_server=checked /etc/master.conf
    @@ -163,6 +165,7 @@
              fi
            else
              /sbin/iptables -A FORWARD -s $SUBNET -i $INIF -j ACCEPT
    +         /sbin/iptables -A FORWARD -s $SUBNET -i ppp0 -j ACCEPT
            fi
    
            # port forwarding
    

  5. Edit /etc/ppp/options as follows:

    noauth
    crtscts
    lock
    local
    proxyarp
    ktune
    192.168.1.XX:192.168.1.YY
    ms-dns 192.168.1.ZZ

    (edit the last two lines to suit your network topology, the first IP address
    is your gateway device, the second IP address will be assigned to the client,
    and the third IP address is your DNS server)
  6. Add the following line to /etc/dnsmasq.conf:
    interface=eth0,ppp0

The WebOS Internals Meta-Doctor

Palm supplies Palm Pre owners with this wonderful recovery tool called the webOS Doctor.  Part of the charter of the WebOS Internals project is to ensure that anything we (or anyone following instructions we publish or installing packages we develop) do can always be reverted using the webOS Doctor.

Usually, a Palm Pre is required to be activated on the Sprint network before it can be used.  This is not possible for a Palm Pre in Australia.

So we need to allow the Pre to be used without activation, and there are a number of information sources showing how this can be done.  There are also some dubious sites that redistribute modified versions of the webOS Doctor (which is a clear violation of copyright law, since it contains proprietary Palm software).  Note that WebOS Internals is always careful to comply with all copyright laws (copyright law is the foundation upon which open source licenses are based).

So we need a way for a Pre owner (who has the right to use the webOS Doctor on their own Pre) to modify the webOS Doctor that is specific to their particular version of the Palm Pre before using it to flash that modified firmware onto their Pre.

That’s where the WebOS Internals “Meta-Doctor” comes into play.

I have created a tool which will download, unpack, patch, and repack a webOS Doctor image, applying a number of transformations along the way to:

  1. Bypass the need for activation
  2. Enable Palm Profile access
  3. Set developer mode on by default
  4. Increase the size of the /var partition to 2GB

You can find this tool in the WebOS Internals source code repository at https://github.com/webos-internals/meta-doctor

Do not redistribute modified versions of the webOS Doctor created using this tool – it is for end-user use only.

I’ve created a forum thread on PreCentral for discussion about this tool.

Setting the OpenMoko timezone

If you want to set the timezone on your phone correctly, do the following:

  1. ipkg install tzdata
  2. ipkg install your desired tzdata-* packages.  For instance, I use “tzdata-australia“.
  3. Enable your desired timezone by symlinking it to “/etc/localtime“.  Adjust the following example command line for your locality.
    • ln -s /usr/share/zoneinfo/Australia/Adelaide /etc/localtime
  4. The “date” command should now show the correct time for your timezone.  If it is not correct, then install the “ntpclient” package, and use it to set your clock.

Note that this technique should work on any OpenEmbedded-based Linux distribution.

    The Palm Pre lands in Australia

    Thanks to the generosity of the PreCentral and WebOS Internals communities, I am now the proud owner of a Palm Pre.

    There is just one catch  – since I live in Australia (which uses a proper cellular communications standard), the CDMA Palm Pre that I was able to import from the USA will never work as a phone here (yes, I knew this before I purchased it).  I plan to also purchase an unlocked GSM/UMTS Pre when one becomes available (maybe I’ll buy a German one and then swap the two keyboards).

    After founding the WebOS Internals project, and using the Pre Emulator in the development of Preware, it is great to have a real device to get the full Pre experience.

    If you want to keep up to date with the activities of the WebOS Internals group, just follow @webosinternals on Twitter.  You can also find a list of articles about WebOS Internals in our Press_Room.  We hang out in the #webos-internals IRC channel on Freenode, and have a webos-internals-announce Google group.

    I still use my trusty old Treo 650 as my daily phone, which allows me to not have to worry about reflashing the Pre to try out things, as I don’t keep any real personal data on it.

    I guess this also closes the OpenMoko chapter of my open source development activities.  I was involved with OpenMoko from the very start, but always said that an OpenMoko device with a hardware keyboard was my preferred form factor, and the Pre seems to satisfy that personal hardware form factor preference whilst still being open enough on the software side to attract my interest.  I wish those who are continuing the OpenMoko path the best of success.

    I’ll document my experiences with the Pre in subsequent posts …

    September 04, 2016

    Diplomacy, Russia Vs USA, and More

    Over and over again, the US and Russia (and other countries) seem to get in each others way. The irony is that while there are many out there who believe that they can come to an 'agreement' of sorts the more I look the more difficult I find this proposition to occur. - first, let's introduce ourselves to modern diplomacy though some videos Diplomacy https://www.youtube.com/channel/

    Houndbot rolling stock upgrade

    After getting Terry the robot to navigate around inside with multiple Kinects as depth sensors I have now turned my attention to outdoor navigation using two cameras as sensors. The cameras are from a PS4 eye which I hacked to be able to connect to a normal machine. The robot originally used 5.4 inch wheels which were run with foam inside them. This sort of arrangement can be seen in many builds in the Radio Controlled (RC) world and worked well when the robot was simple and fairly light. Now that it is well over 10kg the same RC style build doesn't necessarily still work. Foam compresses a bit to easily.

    I have upgraded to 12 inch wheels with air tube tires. This jump seemed a bit risky, would the new setup overwhelm the robot? Once I modified the wheels and came up with an initial mounting scheme to test I think the 12 inch is closer to what the robot naturally wants to have. This should boost the maximum speed of the machine to around 20km/h which is probably as much as you might want on something autonomous. For example, if your robot can out run you things get interesting.




    I had to get the wheels attached in order to work out clearances for the suspension upgrade. While the original suspension worked great for a robot that you only add 1-2kg to, with an itx case, two batteries, a fused power supply etc things seem to have added up to too much weight for the springs to counter.

    I now have some new small 'coil overs' in hand which are taken from mini mountain bike suspension. They are too heavy for what I am using, with around 600lb/inch compression. I have in mind some places that use coil overs in between the RC ones and the push bike ones which I may end up using. Also with slightly higher travel distance.



    As the photo reveals, I don't actually have the new suspension attached yet. I'm thinking about a setup based around two bearing mounts from sparkfun. I'd order from servocity but sfe has cheaper intl shipping :o Anyway, two bearing mounts at the top, two at the bottom and a steel shaft that is 8mm in the middle and 1/4 inch (6.35mm) on the edges. Creating the shafts like that, with the 8mm part just the right length will trap the shaft between the two bearing mounts for me. I might tack weld on either side of the coil over mounts so there is no side to side movement of the suspension.

    Yes, hubs and clamping collars were by first thought for the build and would be nice, but a reasonable result for a manageable price is also a factor.

    September 03, 2016

    Websockets + socket.io on the ESP8266 w/ Micropython

    I recently learned about the ESP8266 while at Pycon AU. It’s pretty nifty: it’s tiny, it has wifi, a reasonable amount of RAM (for a microcontroller) oh, and it can run Python. Specifically Micropython. Anyway I purchased a couple from Adafruit (specifically this one) and installed the Micropython UNIX port on my computer (be aware with the cheaper ESP8266 boards, they might not be very reflashable, or so I’ve been told, spend the extra money for one with decent flash).

    The first thing you learn is that the ports are all surprisingly different in terms of what functionality they support, and the docs don’t make it clear like they do for CPython. I learned the hard way there is a set of docs per port, which maybe is why you the method you’re looking for isn’t there.

    The other thing is that even though you’re getting to write in Python, and it has many Pythonic abstractions, many of those abstractions are based around POSIX and leak heavily on microcontrollers. Still a number of them look implementable without actually reinventing UNIX (probably).

    The biggest problem at the moment is there’s no “platform independent” way to do asynchronous IO. On the microcontroller you can set top-half interrupt handlers for IO events (no malloc here, yay!), gate the CPU, and then execute bottom halfs from the main loop. However that’s not going to work on UNIX. Or you can use select, but that’s not available on the ESP8266 (yet). Micropython does support Python 3.5 asyncio coroutines, so hopefully the port of asyncio to the ESP8266 happens soon. I’d be so especially ecstatic if I could do await pin.trigger(Pin.FALLING).

    There’s a few other things that could really help make it feel like Python. Why isn’t disabling interrupts a context manager/decorator. It’s great that you can try/finally your interrupt code, but the with keyword is so much more Pythonic. Perhaps this is because the code is being written by microprocessor people… which is why they’re so into protocols like MQTT for talking to their devices.

    Don’t get me wrong, MQTT is a great protocol that you can cram onto all sorts of devices, with all sorts of crappy PHYs, but I have wifi, and working SSL. I want to do something more web 2.0. Something like websockets. In fact, I want to take another service’s REST API and websockets, and deliver that information to my device… I could build a HTTP server + MQTT broker, but that sounds like a pain. Maybe I can just build a web server with socket.io and connect to that directly from the device?!

    The ESP8266 already has some very basic websocket support for its WebREPL, but that’s not very featureful and seems to only implement half of the spec. If we’re going to have Python on a device, maybe we can have something that looks like the great websockets module. Turns out we can!

    socket.io is a little harder, it requires a handshake which is not documented (I reversed it in the end), and decoding a HTTP payload, which is not very clearly documented (had to read the source). It’s not the most efficient protocol out there, but the chip is more than fast enough to deal with it. Also fun times, it turns out there’s no platform independent way to return from waiting for IO. Basically it turned out there were a lot of yaks to shave.

    Where it all comes into its own though is the ability to write what is pretty much everyday, beautiful Python however, it’s worth it over Arduino sketches or whatever else takes your fancy.

    uwebsockets/usocketio on Github.

    Electronics breadboard with a project on it sitting on a laptop keyboard Electronics breadboard with a project on it sitting on a laptop keyboard

    LUV Main September 2016 Meeting: Spartan / The Future is Awesome

    Sep 6 2016 18:30
    Sep 6 2016 20:30
    Sep 6 2016 18:30
    Sep 6 2016 20:30
    Location: 

    6th Floor, 200 Victoria St. Carlton VIC 3053

    Speakers:

    • Lev Lafayette, Spartan: A Linux HPC/Cloud Hybrid
    • Paul Fenwick, The Future is Awesome (and what you can do about it)

    200 Victoria St. Carlton VIC 3053

    Late arrivals, please call (0490) 049 589 for access to the venue.

    Before and/or after each meeting those who are interested are welcome to join other members for dinner. We are open to suggestions for a good place to eat near our venue. Maria's on Peel Street in North Melbourne is currently the most popular place to eat after meetings.

    LUV would like to acknowledge Red Hat and Infoxchange for their help in obtaining the meeting venues.

    Linux Users of Victoria Inc. is an incorporated association, registration number A0040056C.

    September 6, 2016 - 18:30

    read more

    August 31, 2016

    Configuring QEMU bridge helper after “access denied by acl file” error

    QEMU has a neat bridge-helper utility which allows a non-root user to easily connect a virtual machine to a bridged interface. In Fedora at least, qemu-bridge-helper runs as setuid (any user can run as root) and privileges are immediately dropped to cap_net_admin. It also has a simple white/blacklist ACL mechanism in place which limits connections to virbr0, libvirt’s local area network.

    That’s all great, but often you actually want a guest to be a part of your real network. This means it must connect to a bridged interface (often br0) on a physical network device.

    If your user tries to kick up such a QEMU guest while specifying bridge,br=br0, something like this (although probably also with a disk or kernel and initramfs):

    qemu-system-x86_64 \
    -machine accel=kvm \
    -cpu host \
    -netdev bridge,br=br0,id=net0 \
    -device virtio-net-pci,netdev=net0

    You may run into the following error:

    access denied by acl file
    qemu-system-ppc64: -netdev bridge,br=br0,id=net0: bridge helper failed

    As mentioned above, this is the QEMU bridge config file /etc/qemu/bridge.conf restricting bridged interfaces to virbr0 for all users by default. So how to make this work more nicely?

    One way is to simply edit the main config file and change virbr0 to all, however that’s not particularly fine-grained.

    Instead, we could create a new config file for the user which specifies any (or all) bridge devices that this user is permitted to connect guests to. This way all other users are restricted to virbr0 while your user can connect to other bridges.

    This doesn’t have to be a user, it could also be a group (just substitute ${USER} for the group, below), and you can also add multiple files.

    Instead of allow you can use deny in the same way to prevent a user or group from attaching to any or all bridges.

    So, let’s create a new file for our user and give them access to all interfaces (requires sudo):

    echo "allow all" | sudo tee /etc/qemu/${USER}.conf
    echo "include /etc/qemu/${USER}.conf" | sudo tee --append /etc/qemu/bridge.conf
    sudo chown root:${USER} /etc/qemu/${USER}.conf
    sudo chmod 640 /etc/qemu/${USER}.conf

    This user should now be able to successfully kick up the guest connected to br0.
    qemu-system-x86_64 \
    -machine accel=kvm \
    -cpu host \
    -netdev bridge,br=br0,id=net0 \
    -device virtio-net-pci,netdev=net0

    August 29, 2016

    Monitoring of Monitoring

    I was recently asked to get data from a computer that controlled security cameras after a crime had been committed. Due to the potential issues I refused to collect the computer and insisted on performing the work at the office of the company in question. Hard drives are vulnerable to damage from vibration and there is always a risk involved in moving hard drives or systems containing them. A hard drive with evidence of a crime provides additional potential complications. So I wanted to stay within view of the man who commissioned the work just so there could be no misunderstanding.

    The system had a single IDE disk. The fact that it had an IDE disk is an indication of the age of the system. One of the benefits of SATA over IDE is that swapping disks is much easier, SATA is designed for hot-swap and even systems that don’t support hot-swap will have less risk of mechanical damage when changing disks if SATA is used instead of IDE. For an appliance type system where a disk might be expected to be changed by someone who’s not a sysadmin SATA provides more benefits over IDE than for some other use cases.

    I connected the IDE disk to a USB-IDE device so I could read it from my laptop. But the disk just made repeated buzzing sounds while failing to spin up. This is an indication that the drive was probably experiencing “stiction” which is where the heads stick to the platters and the drive motor isn’t strong enough to pull them off. In some cases hitting a drive will get it working again, but I’m certainly not going to hit a drive that might be subject to legal action! I recommended referring the drive to a data recovery company.

    The probability of getting useful data from the disk in question seems very low. It could be that the drive had stiction for months or years. If the drive is recovered it might turn out to have data from years ago and not the recent data that is desired. It is possible that the drive only got stiction after being turned off, but I’ll probably never know.

    Doing it Properly

    Ever since RAID was introduced there was never an excuse for having a single disk on it’s own with important data. Linux Software RAID didn’t support online rebuild when 10G was a large disk. But since the late 90’s it has worked well and there’s no reason not to use it. The probability of a single IDE disk surviving long enough on it’s own to capture useful security data is not particularly good.

    Even with 2 disks in a RAID-1 configuration there is a chance of data loss. Many years ago I ran a server at my parents’ house with 2 disks in a RAID-1 and both disks had errors on one hot summer. I wrote a program that’s like ddrescue but which would read from the second disk if the first gave a read error and ended up not losing any important data AFAIK. BTRFS has some potential benefits for recovering from such situations but I don’t recommend deploying BTRFS in embedded systems any time soon.

    Monitoring is a requirement for reliable operation. For desktop systems you can get by without specific monitoring, but that is because you are effectively relying on the user monitoring it themself. Since I started using mon (which is very easy to setup) I’ve had it notify me of some problems with my laptop that I wouldn’t have otherwise noticed. I think that ideally for desktop systems you should have monitoring of disk space, temperature, and certain critical daemons that need to be running but which the user wouldn’t immediately notice if they crashed (such as cron and syslogd).

    There are some companies that provide 3G SIMs for embedded/IoT applications with rates that are significantly cheaper than any of the usual phone/tablet plans if you use small amounts of data or SMS. For a reliable CCTV system the best thing to do would be to have a monitoring contract and have the monitoring system trigger an event if there’s a problem with the hard drive etc and also if the system fails to send a “I’m OK” message for a certain period of time.

    I don’t know if people are selling CCTV systems without monitoring to compete on price or if companies are cancelling monitoring contracts to save money. But whichever is happening it’s significantly reducing the value derived from monitoring.

    [mtb/events] Oxfam Trailwalker - Sydney 2016 - ARNuts


    A great day out on the trail with friends (fullsize)
    Though it did not really hit me in the lead up or during the event until half way that it was yet another 100km and these are indeed somewhat tough to get through. The day out in the bush with my friends Alex, David and Julie was awesome.

    As I say in the short report with the photos linked below, Oxfam is a great charity and that they have these trailwalker events in many places around the world to fundraise and get people to enjoy some quality outdoor time is pretty awesome. This is a hard course, that it took us 14h30m to get through it shows that but it sure is pretty, amazing native flowers, views (water ways and bush) and that it can get in to Manly with hardly realising you are in the middle of the biggest city in Australia is awesome.

    My words and photos are online in my Oxfam Trailalker - Sydney 2016 - ARnuts gallery. What a fun day out!.

    August 28, 2016

    fakecloud

    I wrote my first Mojolicious web app yesterday, a cloud-init meta-data server to enable running pre-built VM images (e.g. as provided by debian, ubuntu, etc) without having to install and manage a complete, full-featured cloud environment like openstack.

    I hacked up something similar several years ago when I was regularly building VM images at home for openstack at work, with just plain-text files served by apache, but that had pretty-much everything hard-coded. fakecloud does a lot more and allows per-VM customisation of user-data (using the IP address of the requesting host). Not bad for a day’s hacking with a new web framework.

    https://github.com/craig-sanders/fakecloud

    fakecloud is a post from: Errata

    August 26, 2016

    Live migrating Btrfs from RAID 5/6 to RAID 10

    Recently it was discovered that the RAID 5/6 implementation in Btrfs is broken, due to the fact that can miscalculate parity (which is rather important in RAID 5 and RAID 6).

    So what to do with an existing setup that’s running native Btfs RAID 5/6?

    Well, fortunately, this issue doesn’t affect non-parity based RAID levels such as 1 and 0 (and combinations thereof) and it also doesn’t affect a Btrfs filesystem that’s sitting on top of a standard Linux Software RAID (md) device.

    So if down-time isn’t a problem, we could re-create the RAID 5/6 array using md and put Btrfs back on top and restore our data… or, thanks to Btrfs itself, we can live migrate it to RAID 10!

    A few caveats though. When using RAID 10, space efficiency is reduced to 50% of your drives, no matter how many you have (this is because it’s mirrored). By comparison, with RAID 5 you lose a single drive in space, with RAID 6 it’s two, no-matter how many drives you have.

    This is important to note, because a RAID 5 setup with 4 drives that is using more than 2/3rds of the total space will be too big to fit on RAID 10. Btrfs also needs space for System, Metadata and Reserves so I can’t say for sure how much space you will need for the migration, but I expect considerably more than 50%. In such cases, you may need to add more drives to the Btrfs array first, before the migration begins.

    So, you will need:

    • At least 4 drives
    • An even number of drives (unless you keep one as a spare)
    • Data in use that is much less than 50% of the total provided by all drives (number of disks / 2)

    Of course, you’ll have a good, tested, reliable backup or two before you start this. Right? Good.

    Plug any new disks in and partition or luksFormat them if necessary. We will assume your new drive is /dev/sdg, you’re using dm-crypt and that Btrfs is mounted at /mnt. Substitute these for your actual settings.
    cryptsetup luksFormat /dev/sdg
    UUID="$(cryptsetup luksUUID /dev/sdg)"
    echo "luks-${UUID} UUID=${UUID} none" >> /etc/crypttab
    cryptsetup luksOpen luks-${UUID} /dev/sdg
    btrfs device add /dev/mapper/luks-${UUID} /mnt

    The migration is going to take a long time, so best to run this in a tmux or screen session.

    screen
    time btrfs balance /mnt
    time btrfs balance start -dconvert=raid10 -mconvert=raid10 /mnt

    After this completes, check that everything has been migrated to RAID 10.
    btrfs fi df /srv/data/
    Data, RAID10: total=2.19TiB, used=2.18TiB
    System, RAID10: total=96.00MiB, used=240.00KiB
    Metadata, RAID10: total=7.22GiB, used=5.40GiB
    GlobalReserve, single: total=512.00MiB, used=0.00B

    If you still see some RAID 5/6 entries, run the same migrate command and then check that everything has migrated successfully.

    Now while we’re at it, let’s defragment everything.
    time btrfs filesystem defragment /srv/data/ # this defrags the metadata
    time btrfs filesystem defragment -r /srv/data/ # this defrags data

    For good measure, let’s rebalance again without the migration (this will also take a while).
    time btrfs fi balance start --full-balance /srv/data/

    August 25, 2016

    Debugging gnome-session problems on Ubuntu 14.04

    After upgrading an Ubuntu 14.04 ("trusty") machine to the latest 16.04 Hardware Enablement packages, I ran into login problems. I could log into my user account and see the GNOME desktop for a split second before getting thrown back into the LightDM login manager.

    The solution I found was to install this missing package:

    apt install libwayland-egl1-mesa-lts-xenial
    

    Looking for clues in the logs

    The first place I looked was the log file for the login manager (/var/log/lightdm/lightdm.log) where I found the following:

    DEBUG: Session pid=12743: Running command /usr/sbin/lightdm-session gnome-session --session=gnome
    DEBUG: Creating shared data directory /var/lib/lightdm-data/username
    DEBUG: Session pid=12743: Logging to .xsession-errors
    

    This told me that the login manager runs the gnome-session command and gets it to create a session of type gnome. That command line is defined in /usr/share/xsessions/gnome.desktop (look for Exec=):

    [Desktop Entry]
    Name=GNOME
    Comment=This session logs you into GNOME
    Exec=gnome-session --session=gnome
    TryExec=gnome-shell
    X-LightDM-DesktopName=GNOME
    

    I couldn't see anything unexpected there, but it did point to another log file (~/.xsession-errors) which contained the following:

    Script for ibus started at run_im.
    Script for auto started at run_im.
    Script for default started at run_im.
    init: Le processus gnome-session (GNOME) main (11946) s'est achevé avec l'état 1
    init: Déconnecté du bus D-Bus notifié
    init: Le processus logrotate main (11831) a été tué par le signal TERM
    init: Le processus update-notifier-crash (/var/crash/_usr_bin_unattended-upgrade.0.crash) main (11908) a été tué par le signal TERM
    

    Seaching for French error messages isn't as useful as searching for English ones, so I took a look at /var/log/syslog and found this:

    gnome-session[4134]: WARNING: App 'gnome-shell.desktop' exited with code 127
    gnome-session[4134]: WARNING: App 'gnome-shell.desktop' exited with code 127
    gnome-session[4134]: WARNING: App 'gnome-shell.desktop' respawning too quickly
    gnome-session[4134]: CRITICAL: We failed, but the fail whale is dead. Sorry....
    

    It looks like gnome-session is executing gnome-shell and that this last command is terminating prematurely. This would explain why gnome-session exits immediately after login.

    Increasing the amount of logging

    In order to get more verbose debugging information out of gnome-session, I created a new type of session (GNOME debug) by copying the regular GNOME session:

    cp /usr/share/xsessions/gnome.desktop /usr/share/xsessions/gnome-debug.desktop
    

    and then adding --debug to the command line inside gnome-debug.desktop:

    [Desktop Entry]
    Name=GNOME debug
    Comment=This session logs you into GNOME debug
    Exec=gnome-session --debug --session=gnome
    TryExec=gnome-shell
    X-LightDM-DesktopName=GNOME debug
    

    After restarting LightDM (service lightdm restart), I clicked the GNOME logo next to the password field and chose GNOME debug before trying to login again.

    This time, I had a lot more information in ~/.xsession-errors:

    gnome-session[12878]: DEBUG(+): GsmAutostartApp: starting gnome-shell.desktop: command=/usr/bin/gnome-shell startup-id=10d41f1f5c81914ec61471971137183000000128780000
    gnome-session[12878]: DEBUG(+): GsmAutostartApp: started pid:13121
    ...
    /usr/bin/gnome-shell: error while loading shared libraries: libwayland-egl.so.1: cannot open shared object file: No such file or directory
    gnome-session[12878]: DEBUG(+): GsmAutostartApp: (pid:13121) done (status:127)
    gnome-session[12878]: WARNING: App 'gnome-shell.desktop' exited with code 127
    

    which suggests that gnome-shell won't start because of a missing library.

    Finding the missing library

    To find the missing library, I used the apt-file command:

    apt-file update
    apt-file search libwayland-egl.so.1
    

    and found that this file is provided by the following packages:

    • libhybris
    • libwayland-egl1-mesa
    • libwayland-egl1-mesa-dbg
    • libwayland-egl1-mesa-lts-utopic
    • libwayland-egl1-mesa-lts-vivid
    • libwayland-egl1-mesa-lts-wily
    • libwayland-egl1-mesa-lts-xenial

    Since I installed the LTS Enablement stack, the package I needed to install to fix this was libwayland-egl1-mesa-lts-xenial.

    I filed a bug for this on Launchpad.

    August 24, 2016

    Small fix for AMP WordPress plugin

    If you use AMP plugin for WordPress to make AMP (Accelerated Mobile Pages) version of your posts and have some troubles validating them on AMP validator, you may try this fix for AMP plugin to make those pages valid.

    August 20, 2016

    Basics of Backups

    I’ve recently had some discussions about backups with people who aren’t computer experts, so I decided to blog about this for the benefit of everyone. Note that this post will deliberately avoid issues that require great knowledge of computers. I have written other posts that will benefit experts.

    Essential Requirements

    Everything that matters must be stored in at least 3 places. Every storage device will die eventually. Every backup will die eventually. If you have 2 backups then you are covered for the primary storage failing and the first backup failing. Note that I’m not saying “only have 2 backups” (I have many more) but 2 is the bare minimum.

    Backups must be in multiple places. One way of losing data is if your house burns down, if that happens all backup devices stored there will be destroyed. You must have backups off-site. A good option is to have backup devices stored by trusted people (friends and relatives are often good options).

    It must not be possible for one event to wipe out all backups. Some people use “cloud” backups, there are many ways of doing this with Dropbox, Google Drive, etc. Some of these even have free options for small amounts of storage, for example Google Drive appears to have 15G of free storage which is more than enough for all your best photos and all your financial records. The downside to cloud backups is that a computer criminal who gets access to your PC can wipe it and the backups. Cloud backup can be a part of a sensible backup strategy but it can’t be relied on (also see the paragraph about having at least 2 backups).

    Backup Devices

    USB flash “sticks” are cheap and easy to use. The quality of some of those devices isn’t too good, but the low price and small size means that you can buy more of them. It would be quite easy to buy 10 USB sticks for multiple copies of data.

    Stores that sell office-supplies sell USB attached hard drives which are quite affordable now. It’s easy to buy a couple of those for backup use.

    The cheapest option for backing up moderate amounts of data is to get a USB-SATA device. This connects to the PC by USB and has a cradle to accept a SATA hard drive. That allows you to buy cheap SATA disks for backups and even use older disks as backups.

    With choosing backup devices consider the environment that they will be stored in. If you want to store a backup in the glove box of your car (which could be good when travelling) then a SD card or USB flash device would be a good choice because they are resistant to physical damage. Note that if you have no other options for off-site storage then the glove box of your car will probably survive if your house burns down.

    Multiple Backups

    It’s not uncommon for data corruption or mistakes to be discovered some time after it happens. Also in recent times there is a variety of malware that encrypts files and then demands a ransom payment for the decryption key.

    To address these problems you should have older backups stored. It’s not uncommon in a corporate environment to have backups every day stored for a week, backups every week stored for a month, and monthly backups stored for some years.

    For a home use scenario it’s more common to make backups every week or so and take backups to store off-site when it’s convenient.

    Offsite Backups

    One common form of off-site backup is to store backup devices at work. If you work in an office then you will probably have some space in a desk drawer for personal items. If you don’t work in an office but have a locker at work then that’s good for storage too, if there is high humidity then SD cards will survive better than hard drives. Make sure that you encrypt all data you store in such places or make sure that it’s not the secret data!

    Banks have a variety of ways of storing items. Bank safe deposit boxes can be used for anything that fits and can fit hard drives. If you have a mortgage your bank might give you free storage of “papers” as part of the service (Commonwealth Bank of Australia used to offer that). A few USB sticks or SD cards in an envelope could fit the “papers” criteria. An accounting firm may also store documents for free for you.

    If you put a backup on USB or SD storage in your waller then that can also be a good offsite backup. For most people losing data from disk is more common than losing their wallet.

    A modern mobile phone can also be used for backing up data while travelling. For a few years I’ve been doing that. But note that you have to encrypt all data stored on a phone so an attacker who compromises your phone can’t steal it. In a typical phone configuration the mass storage area is much less protected than application data. Also note that customs and border control agents for some countries can compel you to provide the keys for encrypted data.

    A friend suggested burying a backup device in a sealed plastic container filled with dessicant. That would survive your house burning down and in theory should work. I don’t know of anyone who’s tried it.

    Testing

    On occasion you should try to read the data from your backups and compare it to the original data. It sometimes happens that backups are discovered to be useless after years of operation.

    Secret Data

    Before starting a backup it’s worth considering which of the data is secret and which isn’t. Data that is secret needs to be treated differently and a mixture of secret and less secret data needs to be treated as if it’s all secret.

    One category of secret data is financial data. If your accountant provides document storage then they can store that, generally your accountant will have all of your secret financial data anyway.

    Passwords need to be kept secret but they are also very small. So making a written or printed copy of the passwords is part of a good backup strategy. There are options for backing up paper that don’t apply to data.

    One category of data that is not secret is photos. Photos of holidays, friends, etc are generally not that secret and they can also comprise a large portion of the data volume that needs to be backed up. Apparently some people have a backup strategy for such photos that involves downloading from Facebook to restore, that will help with some problems but it’s not adequate overall. But any data that is on Facebook isn’t that secret and can be stored off-site without encryption.

    Backup Corruption

    With the amounts of data that are used nowadays the probability of data corruption is increasing. If you use any compression program with the data that is backed up (even data that can’t be compressed such as JPEGs) then errors will be detected when you extract the data. So if you have backup ZIP files on 2 hard drives and one of them gets corrupt you will easily be able to determine which one has the correct data.

    Failing Systems – update 2016-08-22

    When a system starts to fail it may limp along for years and work reasonably well, or it may totally fail soon. At the first sign of trouble you should immediately make a full backup to separate media. Use different media to your regular backups in case the data is corrupt so you don’t overwrite good backups with bad ones.

    One traditional sign of problems has been hard drives that make unusual sounds. Modern drives are fairly quiet so this might not be loud enough to notice. Another sign is hard drives that take unusually large amounts of time to read data. If a drive has some problems it might read a sector hundreds or even thousands of times until it gets the data which dramatically reduces system performance. There are lots of other performance problems that can occur (system overheating, software misconfiguration, and others), most of which are correlated with potential data loss.

    A modern SSD storage device (as used in a lot of the recent laptops) doesn’t tend to go slow when it nears the end of it’s life. It is more likely to just randomly fail entirely and then work again after a reboot. There are many causes of systems randomly hanging or crashing (of which overheating is common), but they are all correlated with data loss so a good backup is a good idea.

    When in doubt make a backup.

    Any Suggestions?

    If you have any other ideas for backups by typical home users then please leave a comment. Don’t comment on expert issues though, I have other posts for that.

    August 19, 2016

    Speaking in August 2016

    I know this is a tad late, but there have been some changes, etc. recently, so apologies for the delay of this post. I still hope to meet many of you to chat about MySQL/Percona Server/MariaDB Server, MongoDB, open source databases, and open source in general in the remainder of August 2016.

    • LinuxCon+ContainerCon North America – August 22-24 2016 – Westin Harbour Castle, Toronto, Canada – I’ll be speaking about lessons one can learn from database failures and enjoying the spectacle that is the 25th anniversary of Linux!
    • Chicago MySQL Meetup Group – August 29 2016 – Vivid Seats, Chicago, IL – more lessons from database failures here, and I’m looking forward to meeting users, etc. in the Chicago area

    While not speaking, Vadim Tkachenko and I will be present at the @scale conference. I really enjoyed my time there previously, and if you get an invite, its truly a great place to learn and network.

    August 17, 2016

    Getting In Sync

    Since at least v1.0.0 Petitboot has used device-mapper snapshots to avoid mounting block devices directly. Primarily this is so Petitboot can mount disks and potentially perform filesystem recovery without worrying about messing it up and corrupting a host's boot partition - all changes happen to the snapshot in memory without affecting the actual device.

    This of course gets in the way if you actually do want to make changes to a block device. Petitboot will allow certain bootloader scripts to make changes to disks if configured (eg, grubenv updates), but if you manually make changes you would need to know the special sequence of dmsetup commands to merge the snapshots back to disk. This is particulary annoying if you're trying to copy logs to a USB device!

    Depending on how recent a version of Petitboot you're running, there are two ways of making sure your changes persist:

    Before v1.2.2

    If you really need to save changes from within Petitboot, the most straightforward way is to disable snapshots. Drop to the shell and enter

    nvram --update-config petitboot,snapshots?=false
    reboot
    

    Once you have rebooted you can remount the device as read-write and modify it as normal.

    After v1.2.2

    To make this easier while keeping the benefit of snapshots, v1.2.2 introduces a new user-event that will merge snapshots on demand. For example:

    mount -o remount,rw /var/petitboot/mnt/dev/sda2
    cp /var/log/messages /var/petitboot/mnt/dev/sda2/
    pb-event sync@sda2
    

    After calling pb-event sync@yourdevice, Petitboot will remount the device back to read-only and merge the current snapshot differences back to disk. You can also run pb-event sync@all to sync all existing snapshots if desired.

    What’s next

    I received an overwhelming number of comments when I said I was leaving MariaDB Corporation. Thank you – it is really nice to be appreciated.

    I haven’t left the MySQL ecosystem. In fact, I’ve joined Percona as their Chief Evangelist in the CTO Office, and I’m going to focus on the MySQL/Percona Server/MariaDB Server ecosystem, while also looking at MongoDB and other solutions that are good for Percona customers. Thanks again for the overwhelming response on the various social media channels, and via emails, calls, etc.

    Here’s to a great time at Percona to focus on open source databases and solutions around them!

    My first blog post on the Percona blog – I’m Colin Charles, and I’m here to evangelize open source databases!, the press release.

    August 15, 2016

    Neo-Colonialism and Neo-Liberalism, Intelligence Analysis, and More

    Watch a lot of media outlets and over and over again and you hear the terms 'Neocolonialism' and 'Free Trade' from time to time. Until fairly recently, I wasn't entirely aware of what exactly this meant and how it came to be. As indicated in my last post, up until a certain point wealth was distributed rather evenly throughout the world. Then 'colonialism' happened and the wealth gap between

    Changing of the guard

    I posted a message to the internal mailing lists at MariaDB Corporation. I have departed (I resigned) the company, but definitely not the community. Thank you all for the privilege of serving the large MariaDB Server community of users, all 12 million+ of you. See you on the mailing lists, IRC, and the developer meetings.

    The Japanese have a saying, “leave when the cherry blossoms are full”.

    I’ve been one of the earliest employees of this post-merge company, and was on the founding team of the MariaDB Server having been around since 2009. I didn’t make the first company meeting in Mallorca (August 2009) due to the chickenpox, but I’ve been to every one since.

    We made the first stable MariaDB Server 5.1 release in February 2010. Our first Linux distribution release was in openSUSE. Our then tagline: MariaDB: Community Developed. Feature Enhanced. Backward Compatible.

    In 2013, we had to make a decision: merge with our sister company SkySQL or take on investment of equal value to compete; majority of us chose to work with our family.

    Our big deal was releasing MariaDB Server 5.5 – Wikipedia migrated, Google wanted in, and Red Hat pushed us into the enterprise space.

    Besides managing distributions and other community related activities (and in the pre-SkySQL days Rasmus and I did everything from marketing to NRE contract management, down to even doing press releases – you wear many hats when you’re in a startup of less than 20 people), in this time, I’ve written over 220 blog posts, spoken at over 130 events (an average of 18 per year), and given generally over 250 talks, tutorials and keynotes. I’ve had numerous face-to-face meetings with customers, figuring out what NRE they may need and providing them solutions. I’ve done numerous internal presentations, audience varying from the professional services & support teams, as well as the management team. I’ve even technically reviewed many books, including one of the best introductions by our colleague, Learning MySQL & MariaDB.

    Its been a good run. Seven years. Uncountable amount of flights. Too many weekends away working for the cause. A whole bunch of great meetings with many of you. Seen the company go from bootstrap, merger, Series A, and Series B.

    It’s been a true privilege to work with many of you. I have the utmost respect for Team MariaDB (and of course my SkySQL brethren!). I’m going to miss many of you. The good thing is that MariaDB Server is an open source project, and I’m not going to leave the project or #maria. I in fact hope to continue speaking and working on MariaDB Server.

    I hope to remain connected to many of you.

    Thank you for this great privilege.

    Kind Regards,
    Colin Charles

    [mtb/events] Razorback Ultra - Spectacular run in the Victorian Alps


    Alex and another Canberran on the Razorback (fullsize)
    Alex and I signed up for the Razorback Ultra because it is in an amazing part of the country and sounded like a fun event to go do. I was heading into it a week after Six Foot, however this is all just training for UTA100 so why not. All I can say is every trail runner should do this event, it is amazing.

    The atmosphere at the race is laid back and it is all about heading up into the mountains and enjoying yourself. I will be back for sure.

    My words and photos are online in my Razorback Ultra 2016 gallery. This is truly one of the best runs in Australia.

    [mtb/events] Geoquest 2016 - Port Mac again with Resultz


    My Mirage 730 - Matilda, having a rest while we ran around (fullsize)
    I have fun at Goequest and love doing the event however have been a bit iffy about trying to organise a team for a few years. As many say one of the hardest things in the event is getting 4 people to the start line ready to go.

    This year my attitude was similar to last, if I was asked to join a team I would probably say yes. I was asked and thus ended up racing with a bunch of fun guys under the banner of Michael's company Resultz Racing. Another great weekend on the mid north NSW coast with some amazing scenery (the two rogaines were highlights, especially the punchbowl waterfall on the second one).

    My words and photos are online in my Geoquest 2016 gallery. Always good fun and a nice escape from winter.

    [mtb] The lots of vert lunch run, reasons to live in Canberra


    Great view of the lake from the single track on the steep side of BM (fullsize)
    This run that is so easy to get out for at lunch is a great quality climbing session and shows off canberra beautifully. What fun.

    Photos and some words are online on my Lots of vert lunch run page.

    [various] Vote Greens Maybe

    I have had the parodies of the Call me maybe song in my head again today (the Orica Green edge one was brilliant and there are some inspired versions out there). This had me thinking of different lyrics, maybe something to suggest people vote Green this Saturday for a better and fairer Australia.

    Vote Green Maybe
    I threw a wish in the well
    For a better Australia today
    I looked at our leaders today
    And now they're in our way
    
    I'll not trade my freedom for them
    All our dollars and cents to the rich
    I wasn't looking for this
    But now they're in our way
    
    Our democracy is squandered
    Broken promises
    Lies everywhere
    Hot nights
    Winds are blowing
    Freak weather events, climate change
    
    Hey I get to vote soon
    And this isn't crazy
    But here's my idea
    So vote Greens maybe
    It's hard to look at our future 
    But here's my idea
    So vote Greens maybe
    
    Hey I get to vote soon
    And this isn't crazy
    But here's my idea
    So vote Greens maybe
    And all the major parties
    Try to shut us up
    But here's my idea
    So vote Greens maybe
    
    Liberal and Labor think they should rule
    I take no time saying they fail
    They gave us nothing at all
    And now they're in our way
    
    I beg for a fairer Australia
    At first sight our policies are real
    I didn't know if you read them
    But it's the Greens way  
    
    Your vote can fix things
    Healthier people
    Childrens education
    Fairer policies
    A change is coming
    Where you think you're voting, Greens?
    
    Hey I get to vote soon
    And this isn't crazy
    But here's my idea
    So vote Greens maybe
    It's worth a look to a brighter future
    But here's my idea
    So vote Greens maybe
    
    Before this change in our lives
    I see children in detention
    I see humans fleeing horrors
    I see them locked up and mistreated
    Before this change in our lives
    I see a way to fix this
    And you should know that
    Voting Green can help fix this, Green, Green, Green...
    
    It's bright to look at our future 
    But here's my idea
    So vote Greens maybe
    
    Hey I get to vote soon
    And this isn't crazy
    But here's my idea
    So vote Greens maybe
    And all the major parties
    Try to shut us up
    But here's my idea
    So vote Greens maybe
    
    Before this change in our lives
    I see children in detention
    I see humans fleeing horrors
    I see them locked up and mistreated
    Before this change in our lives
    I see a way to fix this
    And you should know that
    So vote Green Saturday
    
    Call Me Maybe (Carly Rae Jepsen)
    I threw a wish in the well
    Don't ask me I'll never tell
    I looked at you as it fell
    And now you're in my way
    
    I trade my soul for a wish
    Pennies and dimes for a kiss
    I wasn't looking for this
    But now you're in my way
    
    Your stare was holding
    Ripped jeans
    Skin was showing
    Hot night
    Wind was blowing
    Where you think you're going baby?
    
    Hey I just met you
    And this is crazy
    But here's my number
    So call me maybe
    It's hard to look right at you baby
    But here's my number
    So call me maybe
    
    Hey I just met you
    And this is crazy
    But here's my number
    So call me maybe
    And all the other boys
    Try to chase me
    But here's my number
    So call me maybe
    
    You took your time with the call
    I took no time with the fall
    You gave me nothing at all
    But still you're in my way
    
    I beg and borrow and steal
    At first sight and it's real
    I didn't know I would feel it
    But it's in my way
    
    Your stare was holding
    Ripped jeans
    Skin was showing
    Hot night
    Wind was blowing
    Where you think you're going baby?
    
    Hey I just met you
    And this is crazy
    But here's my number
    So call me maybe
    It's hard to look right at you baby
    But here's my number
    So call me maybe
    
    Before you came into my life
    I missed you so bad
    I missed you so bad
    I missed you so so bad
    Before you came into my life
    I missed you so bad
    And you should know that
    I missed you so so bad, bad, bad, bad....
    
    It's hard to look right at you baby
    But here's my number
    So call me maybe
    
    Hey I just met you
    And this is crazy
    But here's my number
    So call me maybe
    And all the other boys
    Try to chase me
    But here's my number
    So call me maybe
    
    Before you came into my life
    I missed you so bad
    I missed you so bad
    I missed you so so bad
    Before you came into my life
    I missed you so bad
    And you should know that
    So call me, maybe
    

    [various] Safety Sewing


    No reflections (fullsize)

    None outside either (fullsize)

    Better when full/open (fullsize)

    Also better when closed, much brightness (fullsize)
    For over a year I have been planning to do this, my crumpler bag (the complete seed) which I bought in 2008 has been my primary commuting and daily use bag for stuff since that time and as much as I love the bag there is one major problem. No reflective marking anywhere on the bag.

    Some newer crumplers have reflective strips and other such features and if I really wanted to spend big I could get them to do a custom bag with whatever colours and reflective bits I can dream up. There are also a number of other brands that do a courier bag with reflective bits or even entire panels or similar that are reflective. However this is the bag I own and it is still perfectly good for daily use so no need to go buy something new.

    So I got a $4 sewing kit I had sitting around in the house, some great 3M reflective tape material and finally spent the time to rectify this feature missing from the bag. After breaking 3 needles and spending a while getting it done I now have a much safer bag especially commuting home on these dark winter nights. The sewing work is a bit messy however it is functional which is all that matters to me.

    August 14, 2016

    The rise and fall of the Gopher protocol | MinnPost

    Twenty-five years ago, a small band of programmers from the University of Minnesota ruled the internet. And then they didn’t.

    The committee meeting where the team first presented the Gopher protocol was a disaster, “literally the worst meeting I’ve ever seen,” says Alberti. “I still remember a woman in pumps jumping up and down and shouting, ‘You can’t do that!’ ”

    Among the team’s offenses: Gopher didn’t use a mainframe computer and its server-client setup empowered anyone with a PC, not a central authority. While it did everything the U (University of Minnesota) required and then some, to the committee it felt like a middle finger. “You’re not supposed to have written this!” Alberti says of the group’s reaction. “This is some lark, never do this again!” The Gopher team was forbidden from further work on the protocol.

    Read the full article (a good story of Gopher and WWW history!) at https://www.minnpost.com/business/2016/08/rise-and-fall-gopher-protocol

    Have It Your Way: Maximizing Drive-Thru Contributions - PyConAu 2016

    by VM (Vicky) Brasseur.

    Slides.

    Vicky talked about the importance non-committing contributors but the primary focus is on committing contributors due to time limits.

    Covered the different types of drive-thru contributors and why they show up.

    • Scratching an itch.
    • Unwilling / Unable to find an alternative to this project
    • They like you.

    Why do they leave?

    • Itch has been sratched.
    • Not enough time.
    • No longer using the project.
    • Often a high barrier to contribution.
    • Absence of appreciation.
    • Unpleasant people.
    • Inappropriate attribution.

    Disadvantages

    • It takes more time to help them land patches
      • Reluctance to help them "as they're not community".

    It appears to be that many project see community as the foundation but Vicky contended it is contributors.

    More drive-thru contributors are a sign of a healthy project and can lead to a larger community.

    Advantages:

    • Have better processes in place.
    • Faster patch and release times.
    • More eyes and shallower bugs
    • Better community, code and project reputation.

    Leads to a healthier overall project.

    Methods for Maxmising drive-thru contributions:

    Documentation!

    • give your project super powers.
    • Scales!
    • Ensures efficient and successful contributions.
    • Minimises questions.
    • Standardises processes.
    • Vicky provided a documentation quick start guide.

    Mentoring!

    • Code review.
    • "Office hours" for communication.
    • Hackfests.
    • New contributor events.

    Process improvements!

    • Tag starter bugs
    • Contributor SLA
    • Use containers / VM of dev environment

    Culture!

    • Value contributions and contributors
    • Culture of documentation
    • Default to assistance

    Outreach! * Gratitude * Recognition * Follow-up!

    Institute the "No Asshole" rule.

    PyConAu 2016

    Keynote - Python All the Things - PyConAu 2016

    by Russell Keith-Magee.

    Keith spoke about porting Python to mobile devices. CPython being written in C enables it to leverage the supported platforms of the C language and be compiled a wide range of platforms.

    There was a deep dive in the options and pitfalls when selecting a method to and implementing Python on Android phones.

    Ouroboros is a pure Python implementation of the Python standard library.

    Most of the tools discussed are at an early stage of development.

    Why?

    • Being able to run on new or mobile platforms addresses an existential threat.
    • The threat also presents an opportunity to grown, broaden and improve Python.
    • Wants Python to be a "first contact" language, like (Visual) Basic once was.
    • Unlike Basic, Python also support very complex concepts and operations.
    • Presents an opportunity to encourage broader usage by otherwise passive users.
    • Technical superiority is rarely enough to guarantee success.
    • A breadth of technical domains is required for Python to become this choice.
    • Technical problems are the easiest to solve.
    • Te most difficult problems are social and community and require more attention.

    Keith's will be putting his focus into BeeWare and related projects.

    Fortune favours the prepared mind

    (Louis Pasteur)

    PyConAu 2016

    August 13, 2016

    SSD and M.2

    The Need for Speed

    One of my clients has an important server running ZFS. They need to have a filesystem that detects corruption, while regular RAID is good for the case where a disk gives read errors it doesn’t cover the case where a disk returns bad data and claims it to be good (which I’ve witnessed in BTRFS and ZFS systems). BTRFS is good for the case of a single disk or a RAID-1 array but I believe that the RAID-5 code for BTRFS is not sufficiently tested for business use. ZFS doesn’t perform very well due to the checksums on data and metadata requiring multiple writes for a single change which also causes more fragmentation. This isn’t a criticism of ZFS, it’s just an engineering trade-off for the data integrity features.

    ZFS supports read-caching on a SSD (the L2ARC) and write-back caching (ZIL). To get the best benefit of L2ARC and ZIL you need fast SSD storage. So now with my client investigating 10 gigabit Ethernet I have to investigate SSD.

    For some time SSDs have been in the same price range as hard drives, starting at prices well below $100. Now there are some SSDs on sale for as little as $50. One issue with SATA for server use is that SATA 3.0 (which was released in 2009 and is most commonly used nowadays) is limited to 600MB/s. That isn’t nearly adequate if you want to serve files over 10 gigabit Ethernet. SATA 3.2 was released in 2013 and supports 1969MB/s but I doubt that there’s much hardware supporting that. See the SATA Wikipedia page for more information.

    Another problem with SATA is getting the devices physically installed. My client has a new Dell server that has plenty of spare PCIe slots but no spare SATA connectors or SATA power connectors. I could have removed the DVD drive (as I did for some tests before deploying the server) but that’s ugly and only gives 1 device while you need 2 devices in a RAID-1 configuration for ZIL.

    M.2

    M.2 is a new standard for expansion cards, it supports SATA and PCIe interfaces (and USB but that isn’t useful at this time). The wikipedia page for M.2 is interesting to read for background knowledge but isn’t helpful if you are about to buy hardware.

    The first M.2 card I bought had a SATA interface, then I was unable to find a local company that could sell a SATA M.2 host adapter. So I bought a M.2 to SATA adapter which made it work like a regular 2.5″ SATA device. That’s working well in one of my home PCs but isn’t what I wanted. Apparently systems that have a M.2 socket on the motherboard will usually take either SATA or NVMe devices.

    The most important thing I learned is to buy the SSD storage device and the host adapter from the same place then you are entitled to a refund if they don’t work together.

    The alternative to the SATA (AHCI) interface on an M.2 device is known as NVMe (Non-Volatile Memory Express), see the Wikipedia page for NVMe for details. NVMe not only gives a higher throughput but it gives more command queues and more commands per queue which should give significant performance benefits for a device with multiple banks of NVRAM. This is what you want for server use.

    Eventually I got a M.2 NVMe device and a PCIe card for it. A quick test showed sustained transfer speeds of around 1500MB/s which should permit saturating a 10 gigabit Ethernet link in some situations.

    One annoyance is that the M.2 devices have a different naming convention to regular hard drives. I have devices /dev/nvme0n1 and /dev/nvme1n1, apparently that is to support multiple storage devices on one NVMe interface. Partitions have device names like /dev/nvme0n1p1 and /dev/nvme0n1p2.

    Power Use

    I recently upgraded my Thinkpad T420 from a 320G hard drive to a 500G SSD which made it faster but also surprisingly quieter – you never realise how noisy hard drives are until they go away. My laptop seemed to feel cooler, but that might be my imagination.

    The i5-2520M CPU in my Thinkpad has a TDP of 35W but uses a lot less than that as I almost never have 4 cores in use. The z7k320 320G hard drive is listed as having 0.8W “low power idle” and 1.8W for read-write, maybe Linux wasn’t putting it in the “low power idle” mode. The Samsung 500G 850 EVO SSD is listed as taking 0.4W when idle and up to 3.5W when active (which would not be sustained for long on a laptop). If my CPU is taking an average of 10W then replacing the hard drive with a SSD might have reduced the power use of the non-screen part by 10%, but I doubt that I could notice such a small difference.

    I’ve read some articles about power use on the net which can be summarised as “SSDs can draw more power than laptop hard drives but if you do the same amount of work then the SSD will be idle most of the time and not use much power”.

    I wonder if the SSD being slightly thicker than the HDD it replaced has affected the airflow inside my Thinkpad.

    From reading some of the reviews it seems that there are M.2 storage devices drawing over 7W! That’s going to create some cooling issues on desktop PCs but should be OK in a server. For laptop use they will hopefully release M.2 devices designed for low power consumption.

    The Future

    M.2 is an ideal format for laptops due to being much smaller and lighter than 2.5″ SSDs. Spinning media doesn’t belong in a modern laptop and using a SATA SSD is an ugly hack when compared to M.2 support on the motherboard.

    Intel has released the X99 chipset with M.2 support (see the Wikipedia page for Intel X99) so it should be commonly available on desktops in the near future. For most desktop systems an M.2 device would provide all the storage that is needed (or 2*M.2 in a RAID-1 configuration for a workstation). That would give all the benefits of reduced noise and increased performance that regular SSDs provide, but with better performance and fewer cables inside the PC.

    For a corporate desktop PC I think the ideal design would have only M.2 internal storage and no support for 3.5″ disks or a DVD drive. That would allow a design that is much smaller than a current SFF PC.

    Playing with Shifter Part 2 – converted Docker containers inside Slurm

    This is continuing on from my previous blog about NERSC’s Shifter which lets you safely use Docker containers in an HPC environment.

    Getting Shifter to work in Slurm is pretty easy, it includes a plugin that you must install and tell Slurm about. My test config was just:

    required /usr/lib64/shifter/shifter_slurm.so shifter_config=/etc/shifter/udiRoot.conf

    as I was installing by building RPMs (out preferred method is to install the plugin into our shared filesystem for the cluster so we don’t need to have it in the RAM disk of our diskless nodes). One that is done you can add the shifter programs arguments to your Slurm batch script and then just call shifter inside it to run a process, for instance:

    #!/bin/bash
    
    #SBATCH -p debug
    #SBATCH --image=debian:wheezy
    
    shifter cat /etc/issue

    results in the following on our RHEL compute nodes:

    [samuel@bruce Shifter]$ cat slurm-1734069.out 
    Debian GNU/Linux 7 \n \l
    
    

    simply demonstrating that it works. The advantage of using the plugin and this way of specifying the images is that the plugin will prep the container for us at the start of the batch job and keep it around until it ends so you can keep running commands in your script inside the container without the overhead of having to create/destroy it each time. If you need to run something in a different image you just pass the --image option to shifter and then it will need to set up & tear down that container, but the one you specified for your batch job is still there.

    That’s great for single CPU jobs, but what about parallel applications? Well turns out that’s easy too – you just request the configuration you need and slap srun in front of the shifter command. You can even run MPI applications this way successfully. I grabbed the dispel4py/docker.openmpi Docker container with shifterimg pull dispel4py/docker.openmpi and tried its Python version of the MPI hello world program:

    #!/bin/bash
    #SBATCH -p debug
    #SBATCH --image=dispel4py/docker.openmpi
    #SBATCH --ntasks=3
    #SBATCH --tasks-per-node=1
    
    shifter cat /etc/issue
    
    srun shifter python /home/tutorial/mpi4py_benchmarks/helloworld.py

    This prints the MPI rank to demonstrate that the MPI wire up was successful and I forced it to run the tasks on separate nodes and print the hostnames to show it’s communicating over a network, not via shared memory on the same node. But the output bemused me a little:

    [samuel@bruce Python]$ cat slurm-1734135.out
    Ubuntu 14.04.4 LTS \n \l
    
    libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
    libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
    --------------------------------------------------------------------------
    [[30199,2],0]: A high-performance Open MPI point-to-point messaging module
    was unable to find any relevant network interfaces:
    
    Module: OpenFabrics (openib)
      Host: bruce001
    
    Another transport will be used instead, although this may result in
    lower performance.
    --------------------------------------------------------------------------
    libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
    libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
    Hello, World! I am process 0 of 3 on bruce001.
    libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
    --------------------------------------------------------------------------
    [[30199,2],1]: A high-performance Open MPI point-to-point messaging module
    was unable to find any relevant network interfaces:
    
    Module: OpenFabrics (openib)
      Host: bruce002
    
    Another transport will be used instead, although this may result in
    lower performance.
    --------------------------------------------------------------------------
    Hello, World! I am process 1 of 3 on bruce002.
    libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
    --------------------------------------------------------------------------
    [[30199,2],2]: A high-performance Open MPI point-to-point messaging module
    was unable to find any relevant network interfaces:
    
    Module: OpenFabrics (openib)
      Host: bruce003
    
    Another transport will be used instead, although this may result in
    lower performance.
    --------------------------------------------------------------------------
    Hello, World! I am process 2 of 3 on bruce003.

    It successfully demonstrates that it is using an Ubuntu container on 3 nodes, but the warnings are triggered because Open-MPI in Ubuntu is built with Infiniband support and it is detecting the presence of the IB cards on the host nodes. This is because Shifter is (as designed) exposing the systems /sys directory to the container. The problem is that this container doesn’t include the Mellanox user-space library needed to make use of the IB cards and so you get warnings that they aren’t working and that it will fall back to a different mechanism (in this case TCP/IP over gigabit Ethernet).

    Open-MPI allows you to specify what transports to use, so adding one line to my batch script:

    export OMPI_MCA_btl=tcp,self,sm

    cleans up the output a lot:

    Ubuntu 14.04.4 LTS \n \l
    
    Hello, World! I am process 0 of 3 on bruce001.
    Hello, World! I am process 2 of 3 on bruce003.
    Hello, World! I am process 1 of 3 on bruce002.

    This also begs the question then – what does this do for latency? The image contains a Python version of the OSU latency testing program which uses different message sizes between 2 MPI ranks to provide a histogram of performance. Running this over TCP/IP is trivial with the dispel4py/docker.openmpi container, but of course it’s lacking the Mellanox library I need and as the whole point of Shifter is security I can’t get root access inside the container to install the package. Fortunately the author of the dispel4py/docker.openmpi has their implementation published on Github and so I forked their repo, signed up for Docker and pushed a version which simply adds the libmlx4-1 package I needed.

    Running the test over TCP/IP is simply a matter of submitting this batch script which forces it onto 2 separate nodes:

    #!/bin/bash
    #SBATCH -p debug
    #SBATCH --image=chrissamuel/docker.openmpi:latest
    #SBATCH --ntasks=2
    #SBATCH --tasks-per-node=1
    
    export OMPI_MCA_btl=tcp,self,sm
    
    srun shifter python /home/tutorial/mpi4py_benchmarks/osu_latency.py

    giving these latency results:

    [samuel@bruce MPI]$ cat slurm-1734137.out
    # MPI Latency Test
    # Size [B]        Latency [us]
    0                        16.19
    1                        16.47
    2                        16.48
    4                        16.55
    8                        16.61
    16                       16.65
    32                       16.80
    64                       17.19
    128                      17.90
    256                      19.28
    512                      22.04
    1024                     27.36
    2048                     64.47
    4096                    117.28
    8192                    120.06
    16384                   145.21
    32768                   215.76
    65536                   465.22
    131072                  926.08
    262144                 1509.51
    524288                 2563.54
    1048576                5081.11
    2097152                9604.10
    4194304               18651.98

    To run that same test over Infiniband I just modified the export in the batch script to force it to use IB (and thus fail if it couldn’t talk between the two nodes):

    #!/bin/bash
    #SBATCH -p debug
    #SBATCH --image=chrissamuel/docker.openmpi:latest
    #SBATCH --ntasks=2
    #SBATCH --tasks-per-node=1
    
    export OMPI_MCA_btl=openib,self,sm
    
    srun shifter python /home/tutorial/mpi4py_benchmarks/osu_latency.py

    which then gave these latency numbers:

    [samuel@bruce MPI]$ cat slurm-1734138.out
    # MPI Latency Test
    # Size [B]        Latency [us]
    0                         2.52
    1                         2.71
    2                         2.72
    4                         2.72
    8                         2.74
    16                        2.76
    32                        2.73
    64                        2.90
    128                       4.03
    256                       4.23
    512                       4.53
    1024                      5.11
    2048                      6.30
    4096                      7.29
    8192                      9.43
    16384                    19.73
    32768                    29.15
    65536                    49.08
    131072                   75.19
    262144                  123.94
    524288                  218.21
    1048576                 565.15
    2097152                 811.88
    4194304                1619.22

    So you can see that’s basically an order of magnitude improvement in latency using Infiniband compared to TCP/IP over gigabit Ethernet (which is what you’d expect).

    Because there’s no virtualisation going on here there is no extra penalty to pay when doing this, no need to configure any fancy device pass through, no loss of any CPU MSR access, and so I’d argue that Shifter makes Docker containers way more useful for HPC than virtualisation or even Docker itself for the majority of use cases.

    Am I excited about Shifter – yup! The potential to allow users build and application stack themselves right down to the OS libraries and (with a little careful thought) having something that could get native interconnect performance is fantastic. Throw in the complexities of dealing with conflicting dependencies between Python modules, system libraries, bioinformatics tools, etc, etc, and needing to provide simple methods for handling these and the advantages seem clear.

    So the plan is to roll this out into production at VLSCI in the near future. Fingers crossed! 🙂

    This item originally posted here:

    Playing with Shifter Part 2 – converted Docker containers inside Slurm

    Microsoft Chicago – retro in qemu!

    So, way back when (sometime in the early 1990s) there was Windows 3.11 and times were… for Workgroups. There was this Windows NT thing, this OS/2 thing and something brewing at Microsoft to attempt to make the PC less… well, bloody awful for a user.

    Again, thanks to abandonware sites, it’s possible now to try out very early builds of Microsoft Chicago – what would become Windows 95. With the earliest build I could find (build 56), I set to work. The installer worked from an existing Windows 3.11 install.

    I ended up using full system emulation rather than normal qemu later on, as things, well, booted in full emulation and didn’t otherwise (I was building from qemu master… so it could have actually been a bug fix).

    chicago-launch-setupMmmm… Windows 3.11 File Manager, the fact that I can still use this is a testament to something, possibly too much time with Windows 3.11.

    chicago-welcome-setupchicago-setupUnfortunately, I didn’t have the Plus Pack components (remember Microsoft Plus! ?- yes, the exclamation mark was part of the product, it was the 1990s.) and I’m not sure if they even would have existed back then (but the installer did ask).

    chicago-install-dirObviously if you were testing Chicago, you probably did not want to upgrade your working Windows install if this was a computer you at all cared about. I installed into C:\CHICAGO because, well – how could I not!

    chicago-installingThe installation went fairly quickly – after all, this isn’t a real 386 PC and it doesn’t have of-the-era disks – everything was likely just in the linux page cache.

    chicago-install-networkI didn’t really try to get network going, it may not have been fully baked in this build, or maybe just not really baked in this copy of it, but the installer there looks a bit familiar, but not like the Windows 95 one – maybe more like NT 3.1/3.51 ?

    But at the end… it installed and it was time to reboot into Chicago:
    chicago-bootSo… this is what Windows 95 looked like during development back in July 1993 – nearly exactly two years before release. There’s some Windows logos that appear/disappear around the place, which are arguably much cooler than the eventual Windows 95 boot screen animation. The first boot experience was kind of interesting too:
    Screenshot from 2016-08-07 20-57-00Luckily, there was nothing restricting the beta site ID or anything. I just entered the number 1, and was then told it needed to be 6 digits – so beta site ID 123456 it is! The desktop is obviously different both from Windows 3.x and what ended up in Windows 95.

    Screenshot from 2016-08-07 20-57-48Those who remember Windows 3.1 may remember Dr Watson as an actual thing you could run, but it was part of the whole diagnostics infrastructure in Windows, and here (as you can see), it runs by default. More odd is the “Switch To Chicago” task (which does nothing if opened) and “Tracker”. My guess is that the “Switch to Chicago” is the product of some internal thing for launching the new UI. I have no ideawhat the “Tracker” is, but I think I found a clue in the “Find File” app:

    Screenshot from 2016-08-13 14-10-10Not only can you search with regular expressions, but there’s “Containing text”, could it be indexing? No, it totally isn’t. It’s all about tracking/reporting problems:

    Screenshot from 2016-08-13 14-15-19Well, that wasn’t as exciting as I was hoping for (after all, weren’t there interesting database like file systems being researched at Microsoft in the early 1990s?). It’s about here I should show the obligatory About box:
    Screenshot from 2016-08-07 20-58-10It’s… not polished, and there’s certainly that feel throughout the OS, it’s not yet polished – and two years from release: that’s likely fair enough. Speaking of not perfect:

    Screenshot from 2016-08-07 20-59-03When something does crash, it asks you to describe what went wrong, i.e. provide a Clue for Dr. Watson:

    Screenshot from 2016-08-13 12-09-22

    But, most importantly, Solitaire is present! You can browse the Programs folder and head into Games and play it! One odd tihng is that applications have two >> at the end, and there’s a “Parent Folder” entry too.

    Screenshot from 2016-08-13 12-01-24Solitair itself? Just as I remember.

    Screenshot from 2016-08-07 21-21-27Notably, what is missing is anything like the Start menu, which is probably the key UI element introduced in Windows 95 that’s still with us today. Instead, you have this:

    Screenshot from 2016-08-13 11-55-15That’s about the least exciting Windows menu possible. There’s the eye menu too, which is this:

    Screenshot from 2016-08-13 11-56-12More unfinished things are found in the “File cabinet”, such as properties for anything:
    Screenshot from 2016-08-13 12-02-00But let’s jump into Control Panels, which I managed to get to by heading to C:\CHICAGO\Control.sys – which isn’t exactly obvious, but I think you can find it through Programs as well.Screenshot from 2016-08-13 12-02-41Screenshot from 2016-08-13 12-05-40The “Window Metrics” application is really interesting! It’s obvious that the UI was not solidified yet, that there was a lot of experimenting to do. This application lets you change all sorts of things about the UI:

    Screenshot from 2016-08-13 12-05-57My guess is that this was used a lot internally to twiddle things to see what worked well.

    Another unfinished thing? That familiar Properties for My Computer, which is actually “Advanced System Features” in the control panel, and from the [Sample Information] at the bottom left, it looks like we may not be getting information about the machine it’s running on.

    Screenshot from 2016-08-13 12-06-39

    You do get some information in the System control panel, but a lot of it is unfinished. It seems as if Microsoft was experimenting with a few ways to express information and modify settings.

    Screenshot from 2016-08-13 12-07-13But check out this awesome picture of a hard disk for Virtual Memory:

    Screenshot from 2016-08-13 12-07-47The presence of the 386 Enhanced control panel shows how close this build still was to Windows 3.1:

    Screenshot from 2016-08-13 12-08-08At the same time, we see hints of things going 32 bit – check out the fact that we have both Clock and Clock32! Notepad, in its transition to 32bit, even dropped the pad and is just Note32!

    Screenshot from 2016-08-13 12-11-10Well, that’s enough for today, time to shut down the machine:
    Screenshot from 2016-08-13 12-15-45

    Python for science, side projects and stuff! - PyConAu 2016

    By Andrew Lonsdale.

    • Talked about using python-ppt for collaborating on PowerPoint presentations.
    • Covered his journey so far and the lessons he learned.
    • Gave some great examples of re-creating XKCD comics in Python (matplotlib_venn).
    • Claimed the diversion into Python and Matplotlib has helped is actual research.
    • Spoke about how using Python is great for Scientific research.
    • Summarised that side projects are good for Science and Python.
    • Recommended Elegant SciPy
    • Demo's using Emoji to represent bioinformatics using FASTQE (FASTQ as Emoji).

    PyConAu 2016

    MicroPython: a journey from Kickstarter to Space by Damien George - PyConAu 2016

    Damien George.

    Motivations for MicroPython:

    • To provide a high level language to control sophisticated micro-controllers.
    • Approached it as an intellectually stimulating research problem.
    • Wasn't even sure it was possible.
    • Chose Python because:
      • It was a high level language with powerful features.
      • Large existing community.
      • Naively thought it would be easy.
      • Found Python easy to learn.
      • Shallow but long learning curve of python makes it good for beginners and advanced programmers.
      • Bitwise operaitons make it usefult for micro-controllers.

    Why Not Use CPython?

    • CPython pre-allocates memory, resulting in inefficient memory usage which is problematic for low RAM devices like micro controllers.

    Usage:

    • If you know Python, you know MicroPython - it's implemented the same

    Kickstarter:

    Damien covered his experiences with Kickstarter.

    Internals of MicroPython:

    • Damien covered the parser, lexer, compiler and runtime.
    • Walked us through the workflows of the internals.
    • Spoke about object represntation and the three machine word object forms:
      • Integers.
      • Strings.
      • Objects.
    • Covered the emitters:
      • Bytecode.
      • Native (machine code).
      • Inline assembler.

    Coding Style:

    Coding was more based on a physicist trying to make things work, than a computer engineer.

    • There's a code dashboard
    • Hosted on GitHub
    • Noted that he could not have done this without the support of the community.

    Hardware:

    Listed some of the micro controller boards that it runs on ad larger computers that currently run OpenWRT.

    Spoke about the BBC micron:bit project. Demo'd speech synthesis and image display running on it.

    MicroPython in Space:

    Spoke about the port to LEON / SPARC / RTEMS for the European Space agency for satellite control, particularly the application layer.

    Damien closed with an overview of current applications and ongoing software and hardware development.

    Links:

    micropython.org forum.micropython.org github.com/micropython

    PyConAu 2016

    August 12, 2016

    Doing Math with Python - Amit Saha - PyConAu 2016

    Amit Saha.

    Slides and demos.

    Why Math with Python?

    • Provides an interactive learning experience.
    • Provides a great base for future programming (ie: data science, machine learning).

    Tools:

    • Python 3
    • SymPy
    • matplotlib

    Amit's book: Doing Math with Python

    PyConAu 2016

    The Internet of Not Great Things - Nick Moore - PyConAu 2016

    Nick Moore.

    aka "The Internet of (Better) Things".

    • Abuse of IoT is not a technical issue.
    • The problem is who controls the data.
    • Need better analysis of the was it is used that is bad.
    • "If you're not the customer, you're the product."
      • by accepting advertising.
      • by having your privacy sold.
    • Led to a conflation of IoT and Big Data.
    • Product end of life by vendors ceasing support.
    • Very little cross vendor compatibility.
    • Many devices useless if the Internet is not available.
    • Consumer grade devices often fail.
    • Weak crypto support.
    • Often due to lack of entropy, RAM, CPU.
    • Poorly thought out update cycles.

    Turning Complaints into Requirements:

    We need:

    • Internet independence.
    • Generic interfaces.
    • Simplified Cryptography.
    • Easier Development.

    Some Solutions:

    • Peer to peer services.
    • Standards based hardware description language.
    • Shared secrets, initialised by QR code.
    • Simpler development with MicroPython.

    PyConAu 2016

    OpenBMC - Boot your server with Python - Joel Stanley - PyConAu 2016

    Joel Stanley.

    • OpenBMC is a Free Software BMC
    • Running embedded Linux.
    • Developed an API before developing other interfaces.

    Goals:

    • A modern kernel.
    • Up to date userspace.
    • Security patches.
    • Better interfaces.
    • Reliable performance.
      • REST interface.
      • SSH instead of strange tools.

    The Future:

    • Support more home devices.
    • Add a web interface.
    • Secure boot, trusted boot, more security features.
    • Upstream all of the things.
    • Support more hardware.

    PyConAu 2016

    Teaching Python with Minecraft - Digital K - PyConAu 2016

    by Digital K.

    The video of the talk is here.

    • Recommended for ages 10 - 16
    • Why Minecraft?
      • Kids familiarity is highly engaging.
      • Relatively low cost.
      • Code their own creations.
      • Kids already use the command line in Minecraft
    • Use the Minecraft API to receive commands from Python.
      • Place blocks
      • Move players
      • Build faster
      • Build larger structures and shapes
      • Easy duplication
      • Animate blocks (ie: colour change)
      • Create games

    Option 1:

    How it works:

    • Import Minecraft API libraries to your code.
    • Push it to the server.
    • Run the Minecraft client.

    What you can Teach:

    • Co-ordinates
    • Time
    • Multiplications
    • Data
    • Art works with maths
    • Trigonometry
    • Geo fencing
    • Design
    • Geography

    Connect to External Devices:

    • Connect to Raspberry Pi or Arduino.
    • Connect the game to events in the real world.

    Other Resources:

    PyConAu 2016

    Scripting the Internet of Things - Damien George - PyConAu 2016

    Damien George

    Damien gave an excellent overview of using MicroPython with microcontrollers, particularly the ESP8266 board.

    Damien's talk was excellent and covered a broad and interesting history of the project and it's current efforts.

    PyConAu 2016

    ESP8266 and MicroPython - Nick Moore - PyConAu 2016

    Nick Moore

    Slides.

    • Price and feature set are a game changer for hobbyists.
    • Makes for a more playful platform.
    • Uses serial programming mode to flash memory
    • Strict power requirements
    • The easy way to use them is with a NodeMCU for only a little more.
    • Tool kits:
    • Lua: (Node Lua).
    • Javascript: Espruino.
    • Forth, Lisp, Basic(?!).
    • Mircopython works on the ESP8266:
      • Drives micro controllers.
      • The onboard Wifi.
      • Can run a small webserver to view and control devices.
      • WebRepl can be used to copy files, as can mpy-utils.
      • Lacks:
        • an operating system.
        • Lacks multiprocessing.
        • Debugger / profiler.
    • Flobot:
      • Compiles via MicroPython.
      • A visual dataflow language for robots.

    ES8266 and MicroPython provide an accessible entry into working with micro-crontrollers.

    PyConAu 2016

    August 10, 2016

    Command line password management with pass

    Why use a password manager in the first place? Well, they make it easy to have strong, unique passwords for each of your accounts on every system you use (and that’s a good thing).

    For years I’ve stored my passwords in Firefox, because it’s convenient, and I never bothered with all those other fancy password managers. The problem is, that it locked me into Firefox and I found myself still needing to remember passwords for servers and things.

    So a few months ago I decided to give command line tool Pass a try. It’s essentially a shell script wrapper for GnuPG and stores your passwords (with any notes) in individually encrypted files.

    I love it.

    Pass is less convenient in terms of web browsing, but it’s more convenient for everything else that I do (which is often on the command line). For example, I have painlessly integrated Pass into Mutt (my email client) so that passwords are not stored in the configuration files.

    As a side-note, I installed the Password Exporter Firefox Add-on and exported my passwords. I then added this whole file to Pass so that I can start copying old passwords as needed (I didn’t want them all).

    About Pass

    Pass uses public-key cryptography to encrypt each password that you want to store as an individual file. To access the password you need the private key and passphrase.

    So, some nice things about it are:

    • Short and simple shell script
    • Uses standard GnuPG to encrypt each password into individual files
    • Password files are stored on disk in a hierarchy of own choosing
    • Stored in Git repo (if desired)
    • Can also store notes
    • Can copy the password temporarily to copy/paste buffer
    • Can show, edit, or copy password
    • Can also generate a password
    • Integrates with anything that can call it
    • Tab completion!

    So it’s nothing super fancy, “just” a great little wrapper for good old GnuPG and text files, backed by git. Perfect!

    Install Pass

    Installation of Pass (and Git) is easy:
    sudo dnf -y install git pass

    Prepare keys

    You’ll need a pair of keys, so generate these if you haven’t already (this creates the keys under ~/.gnupg). I’d probably recommend RSA and RSA, 4096 bits long, using a decent passphrase and setting a valid email address (you can also separately use these keys to send signed emails and receive encrypted emails).
    gpg2 --full-gen-key

    We will need the key’s fingerprint to give to pass. It should be a string of 40 characters, something like 16CA211ACF6DC8586D6747417407C4045DF7E9A2.
    gpg2 --list-secret-keys

    Note: Your fingerprint (and public keys) can be public, but please make sure that you keep your private keys secure! For example, don’t copy the ~/.gnupg directory to a public place (even though they are protected by a nice long passphrase, right? Right?).

    Initialise pass

    Before we can use Pass, we need to initialise it. Put the fingerprint you got from the output of gpg2 –list-secret-keys above (e.g. 5DF7E9A2).
    pass init 5DF7E9A2

    This creates the basic directory structure in the .password-store directory in your home directory. At this point it just has a plain text file (.password-store/.gpg-id) with the fingerprint of the public key that it should use.

    Adding git backing

    If you haven’t already, you’ll need to tell Git who you are. Using the email address that you used when creating the GPG key is probably good.
    git config --global user.email "you@example.com"
    git config --global user.name "Your Name"

    Now, go into the password-store directory and initialise it as a Git repository.
    cd ~/.password-store
    git init
    git add .
    git commit -m "intial commit"
    cd -

    Pass will now automatically commit changes for you!

    Hierarchy

    As mentioned, you can create any hierarchy you like. I quite like to use subdirectories and sort by function first (like mail, web, server), then domains (like gmail.com, twitter.com) and then server or username. This seems to work quite nicely with tab completion, too.

    You can rearrange this at any time, so don’t worry too much!

    Storing a password

    Adding a password is simple and you can create any hierarchy that you want; you just tell pass to add a new password and where to store it. Pass will prompt you to enter the password.

    For example, you might want to store your password for a machine at server1.example.com – you could do that like so:
    pass add servers/example.com/server1

    This creates the directory structure on disk and your first encrypted file!
    ~/.password-store/
    └── servers
        └── example.com
            └── server1.gpg
     
    2 directories, 1 file

    Run the file command on that file and it should tell you that it’s encrypted.
    file ~/.password-store/servers/example.com/server1.gpg

    But is it really? Go ahead, cat that gpg file, you’ll see it’s encrypted (your terminal will probably go crazy – you can blindly enter the reset command to get it back).
    cat ~/.password-store/servers/example.com/server1.gpg

    So this file is encrypted – you can safely copy it anywhere (again, please just keep your private key secure).

    Git history

    Browse to the .password-store dir and run some git commands, you’ll see your history and showing will prompt for your GPG passphrase to decrypt the files stored in Git.

    cd ~/.password-store
    git log
    git show
    cd -

    If you wanted to, you could push this to another computer as a backup (perhaps even via a git-hook!).

    Storing a password, with notes

    By default Pass just prompts for the password, but if you want to add notes at the same time you can do that also. Note that the password should still be on its own on the first line, however.
    pass add -m mail/gmail.com/username

    If you use two-factor authentication (which you should be), this is useful for also storing the account password and recovery codes.

    Generating and storing a password

    As I mentioned, one of the benefits of using a password manager is to have strong, unique passwords. Pass makes this easy by including the ability to generate one for you and store it in the hierarchy of your choosing. For example, you could generate a 32 character password (without special characters) for a website you often log into, like so:
    pass generate -n web/twitter.com/username 32

    Getting a password out

    Getting a password out is easy; just tell Pass which one you want. It will prompt you for your passphrase, decrypt the file for you, read the first line and print it to the screen. This can be useful for scripting (more on that below).

    pass web/twitter.com/username

    Most of the time though, you’ll probably want to copy the password to the copy/paste buffer; this is also easy, just add the -c option. Passwords are automatically cleared from the buffer after 45 seconds.
    pass -c web/twitter.com/username

    Now you can log into Twitter by entering your username and pasting the password.

    Editing a password

    Similarly you can edit an existing password to change it, or add as many notes as you like. Just tell Pass which password to edit!
    pass edit web/twitter.com/username

    Copying and moving a password

    It’s easy to copy an existing password to a new one, just specify both the original and new file.
    pass copy servers/example.com/server1 servers/example.com/server2

    If the hierarchy you created is not to your liking, it’s easy to move passwords around.
    pass mv servers/example.com/server1 computers/server1.example.com

    Of course, you could script this!

    Listing all passwords

    Pass will list all your passwords in a tree nicely for you.
    pass list

    Interacting with Pass

    As pass is a nice standard shell program, you can interact with it easily. For example, to get a password from a script you could do something like this.
    #!/usr/bin/env bash
     
    echo "Getting password.."
    PASSWORD="$(pass servers/testing.com/server2)"
    if [[ $? -ne 0 ]]; then
        echo "Sorry, failed to get the password"
        exit 1
    fi
    echo "..and we got it, ${PASSWORD}"

    Try it!

    There’s lots more you can do with Pass, why not check it out yourself!

    August 08, 2016

    Setting up OpenStack Ansible All-in-one behind a proxy

    Setting up OpenStack Ansible (OSA) All-in-one (AIO) behind a proxy requires a couple of settings, but it should work fine (we’ll also configure the wider system). There are two types of git repos that we should configure for (unless you’re an OpenStack developer), those that use http (or https) and those that use the git protocol.

    Firstly, this assumes an Ubuntu 14.04 server install (with at least 60GB of free space on / partition).

    All commands are run as the root user, so switch to root first.

    sudo -i

    Export variables for ease of setup

    Setting these variables here means that you can copy and paste the relevant commands from the rest of this blog post.

    Note: Make sure that your proxy is fully resolvable and then replace the settings below with your actual proxy details (leave out user:password if you don’t use one).

    export PROXY_PROTO="http"
    export PROXY_HOST="user:password@proxy"
    export PROXY_PORT="3128"
    export PROXY="${PROXY_PROTO}://${PROXY_HOST}:${PROXY_PORT}"

    First, install some essentials (reboot after upgrade if you like).
    echo "Acquire::http::Proxy \"${PROXY}\";" \
    > /etc/apt/apt.conf.d/90proxy
    apt-get update && apt-get upgrade
    apt-get install git openssh-server rsync socat screen vim

    Configure global proxies

    For any http:// or https:// repositories we can just set a shell environment variable. We’ll set this in /etc/environment so that all future shells have it automatically.

    cat >> /etc/environment << EOF
    export http_proxy="${PROXY}"
    export https_proxy="${PROXY}"
    export HTTP_PROXY="${PROXY}"
    export HTTPS_PROXY="${PROXY}"
    export ftp_proxy="${PROXY}"
    export FTP_PROXY="${PROXY}"
    export no_proxy=localhost
    export NO_PROXY=localhost
    EOF

    Source this to set the proxy variables in your current shell.
    source /etc/environment

    Tell sudo to keep these environment variables
    echo 'Defaults env_keep = "http_proxy https_proxy ftp_proxy \
    no_proxy HTTP_PROXY HTTPS_PROXY FTP_PROXY NO_PROXY"' \
    > /etc/sudoers.d/01_proxy

    Configure Git

    For any git:// repositories we need to make a script that uses socat (you could use netcat) and tell Git to use this as the proxy.

    cat > /usr/local/bin/git-proxy.sh << EOF
    #!/bin/bash
    # \$1 = hostname, \$2 = port
    exec socat STDIO PROXY:${PROXY_HOST}:\${1}:\${2},proxyport=${PROXY_PORT}
    EOF

    Make it executable.
    chmod a+x /usr/local/bin/git-proxy.sh

    Tell Git to proxy connections through this script.
    git config --global core.gitProxy /usr/local/bin/git-proxy.sh

    Clone OpenStack Ansible

    OK, let’s clone the OpenStack Ansible repository! We’re living on the edge and so will build from the tip of the master branch.
    git clone git://git.openstack.org/openstack/openstack-ansible \
    /opt/openstack-ansible
    cd /opt/openstack-ansible/

    If you would prefer to build from a specific release, such as the latest stable, feel free to now check out the appropriate tag. For example, at the time of writing this is tag 13.3.1. You can get a list of tags by running the git tag command.

    # Only run this if you want to build the 13.3.1 release
    git checkout -b tag-13.3.1 13.3.1

    Or if you prefer, you can checkout the tip of the stable branch which prepares for the upcoming stable minor release.

    # Only run this if you want to build the latest stable code
    git checkout -b stable/matika origin/stable/mitaka

    Prepare log location

    If something goes wrong, it’s handy to be able to have the log available.

    export ANSIBLE_LOG_PATH=/root/ansible-log

    Bootstrap Ansible

    Now we can kick off the ansible bootstrap. This prepares the system with all of the Ansible roles that make up an OpenStack environment.
    ./scripts/bootstrap-ansible.sh

    Upon success, you should see:

    System is bootstrapped and ready for use.

    Bootstrap OpenStack Ansible All In One

    Now let’s bootstrap the all in one system. This configures the host with appropriate disks and network configuration, etc ready to run the OpenStack environment in containers.
    ./scripts/bootstrap-aio.sh

    Run the Ansible playbooks

    The final task is to run the playbooks, which sets up all of the OpenStack components on the host and containers. Before we proceed, however, this requires some additional configuration for the proxy.

    The user_variables.yml file under the root filesystem at /etc/openstack_deploy/user_variables.yml is where we configure environment variables for OSA to export and set some other options (again, note the leading / before etc – do not modify the template file at /opt/openstack-ansible/etc/openstack_deploy by mistake).

    cat >> /etc/openstack_deploy/user_variables.yml << EOF
    #
    ## Proxy settings
    proxy_env_url: "\"${PROXY}\""
    no_proxy_env: "\"localhost,127.0.0.1,{{ internal_lb_vip_address }},{{ external_lb_vip_address }},{% for host in groups['all_containers'] %}{{ hostvars[host]['container_address'] }}{% if not loop.last %},{% endif %}{% endfor %}\""
    global_environment_variables:
      HTTP_PROXY: "{{ proxy_env_url }}"
      HTTPS_PROXY: "{{ proxy_env_url }}"
      NO_PROXY: "{{ no_proxy_env }}"
      http_proxy: "{{ proxy_env_url }}"
      https_proxy: "{{ proxy_env_url }}"
      no_proxy: "{{ no_proxy_env }}"
    EOF

    Secondly, if you’re running the latest stable, 13.3.x, you will need to make a small change to pip package list for the keystone (authentication component) container. Currently it pulls in httplib2 version 0.8, however this does not appear to respect the NO_PROXY variable and so keystone provisioning fails. Version 0.9 seems to fix this problem.

    sed -i 's/state: present/state: latest/' \
    /etc/ansible/roles/os_keystone/tasks/keystone_install.yml

    Now run the playbooks!

    Note: This will take a long time, perhaps a few hours, so run it in a screen or tmux session.

    screen
    time ./scripts/run-playbooks.sh

    Verify containers

    Once the playbooks complete, you should be able to list your running containers and see their status (there will be a couple of dozen).
    lxc-ls -f

    Log into OpenStack

    Now that the system is complete, we can start using OpenStack!

    You should be able to use your web browser to log into Horizon, the OpenStack Dashboard, at your AIO hosts’s IP address.

    If you’re not sure what IP that is, you can find out by looking at which address port 443 is running on.

    netstat -ltnp |grep 443

    The admin user’s password is available in the user_secrets.yml file on the AIO host.
    grep keystone_auth_admin_password \
    /etc/openstack_deploy/user_secrets.yml

    osa-aio

    A successful login should reveal the admin dashboard.

    osa-aio-admin

    Enjoy your OpenStack Ansible All-in-one!

    Windows 3.11 nostalgia

    Because OS/2 didn’t go so well… let’s try something I’m a lot more familiar with. To be honest, the last time I in earnest used Windows on the desktop was around 3.11, so I kind of know it back to front (fun fact: I’ve read the entire Windows 3.0 manual).

    It turns out that once you have MS-DOS installed in qemu, installing Windows 3.11 is trivial. I didn’t even change any settings for Qemu, I just basically specced everything up to be very minimal (50MB RAM, 512mb disk).

    win31-setupwin31-disk4win31-installedWindows 3.11 was not a fun time as soon as you had to do anything… nightmares of drivers, CONFIG.SYS and AUTOEXEC.BAT plague my mind. But hey, it’s damn fast on a modern processor.

    Swift + Xena would make the perfect digital preservation solution

    Those of you might not know, but for some years I worked at the National Archives of Australia working on, at the time, their leading digital preservation platform. It was awesome, opensource, and they paid me to hack on it.
    The most important parts of the platform was Xena and Digital Preservation Recorder (DPR). Xena was, and hopefully still is amazing. It takes in a file, guesses the format. If it’s a closed proprietary format and it had the right xena plugin it would convert it to an open standard and optionally turned it into a .xena file ready to be ingested into the digital repository for long term storage.

    We did this knowing that proprietary formats change so quickly and if you want to store a file format long term (20, 40, 100 years) you won’t be able to open it. An open format on the other hand, even if there is no software that can read it any more is open, so you can get your data back.

    Once a file had passed through Xena, we’d use DPR to ingest it into the archive. Once in the archive, we had other opensource daemons we wrote which ensured we didn’t lose things to bitrot, we’d keep things duplicated and separated. It was a lot of work, and the size of the space required kept growing.

    Anyway, now I’m an OpenStack Swift core developer, and wow, I wish Swift was around back then, because it’s exactly what is required for the DPR side. It duplicates, infinitely scales, it checks checksums, quarantines and corrects. Keeps everything replicated and separated and does it all automatically. Swift is also highly customise-able. You can create your own middleware and insert it in the proxy pipeline or in any of the storage node’s pipelines, and do what ever you need it to do. Add metadata, do something to the object on ingest, or whenever the object is read, updating some other system.. really you can do what ever you want. Maybe even wrap Xena into some middleware.

    Going one step further, IBM have been working on a thing called storlets which uses swift and docker to do some work on objects and is now in the OpenStack namespace. Currently storlets are written in Java, and so is Xena.. so this might also be a perfect fit.

    Anyway, I got talking with Chris Smart, a mate who also used to work in the same team at NAA, so it got my mind thinking about all this and so I thought I’d place my rambling thoughts somewhere in case other archives or libraries are interested in digital preservation and needs some ideas.. best part, the software is open source and also free!

    Happy preserving.

    August 07, 2016

    SM2000 – Part 8 – Gippstech 2016 Presentation

    Justin, VK7TW, has published a video of my SM2000 presentation at Gippstech, which was held in July 2016.

    Brady O’Brien, KC9TPA, visited me in June. Together we brought the SM2000 up to the point where it is decoding FreeDV 2400A waveforms at 10.7MHz IF, which we demonstrate in this video. I’m currently busy with another project but will get back to the SM2000 (and other FreeDV projects) later this year.

    Thanks Justin and Brady!

    FreeDV and this video was also mentioned on this interesting Reddit post/debate from Gary KN4AQ on VHF/UHF Digital Voice – a peek into the future

    OS/2 Warp Nostalgia

    Thanks to the joys of abandonware websites, you can play with some interesting things from the 1990s and before. One of those things is OS/2 Warp. Now, I had a go at OS/2 sometime in the 1990s after being warned by a friend that it was “pretty much impossible” to get networking going. My experience of OS/2 then was not revolutionary… It was, well, something else on a PC that wasn’t that exciting and didn’t really add a huge amount over Windows.

    Now, I’m nowhere near insane enough to try this on my actual computer, and I’ve managed to not accumulate any ancient PCs….

    Luckily, qemu helps with an emulator! If you don’t set your CPU to Pentium (or possibly something one or two generations newer) then things don’t go well. Neither does a disk that by today’s standards would be considered beyond tiny. Also, if you dare to try to use an unpartitioned hard disk – OH MY are you in trouble.

    Also, try to boot off “Disk 1” and you get this:
    os2-wrong-floppyPossibly the most friendly error message ever! But, once you get going (by booting the Installation floppy)… you get to see this:

    Screenshot from 2016-08-07 19-12-19and indeed, you are doing the time warp of Operating Systems right here. After a bit of fun, you end up in FDISK:

    os2-installos2-1gb-too-muchWhy I can’t create a partition… WHO KNOWS. But, I tried again with a 750MB disk that already had a partition on it and…. FAIL. I think this one was due to partition type, so I tried again with partition type of 6 – plain FAT16, and not W95 FAT16 (LBA). Some memory is coming back to me of larger drives and LBA and nightmares…

    But that worked!

    warp4-installingos2-checkingThen, the OS/2 WARP boot screen… which seems to stick around for a long time…..

    os2-install-2and maybe I could get networking….

    os2-networkLadies and Gentlemen, the wonders of having to select DHCP:

    os2-dhcpIt still asked me for some config, but I gleefully ignored it (because that must be safe, right!?) and then I needed to select a network adapter! Due to a poor choice on my part, I started with a rtl8139, which is conspicuously absent from this fine list of Token Ring adapters:

    os2-tokenringand then, more installing……

    os2-more-installingbefore finally rebooting into….

    os2-failand that, is where I realized there was beer in the fridge and that was going to be a lot more fun.

    August 04, 2016

    Remembering Seymour Papert

    papert-01

    Today we’re remembering Seamour Papert, as we’ve received news that he died a few days ago (31st July 2016) at the age of 88.  Throughout his life, Papert did so much for computing and education, he even worked with the famous Jean Piaget who helped Papert further develop his views on children and learning.

    For us at OpenSTEM, Papert is also special because in the late 1960s (yep that far back) he invented the Logo programming language, used to control drawing “turtles”.  The Mirobot drawing turtle we use in our Robotics Program is a modern descendant of those early (then costly) adventures.

    I sadly never met him, but what a wonderful person he was.

    For more information, see the media release at MIT’s Media Lab (which he co-founded) or search for his name online.

     

    Supercomputers: Current Status and Future Trends

    The somewhat nebulous term "supercomputer" has a long history. Although first coined in the 1920s to refer to IBMs tabulators, in electronic computing the most important initial contribution was the CDC6600 in the 1960s, due to its advanced performance over competitors. Over time major technological advancements included vector processing, cluster architecture, massive processors counts, GPGPU technologies, multidimensional torus architectures for interconnect.

    read more

    August 01, 2016

    Putting Prometheus node_exporter behind apache proxy

    I’ve been playing with Prometheus monitoring lately. It is fairly new software that is getting popular. Prometheus works using a pull architecture. A central server connects to each thing you want to monitor every few seconds and grabs stats from it.

    In the simplest case you run the node_exporter on each machine which gathers about 600-800 (!) metrics such as load, disk space and interface stats. This exporter listens on port 9100 and effectively works as an http server that responds to “GET /metrics HTTP/1.1” and spits several hundred lines of:

    node_forks 7916
    node_intr 3.8090539e+07
    node_load1 0.47
    node_load15 0.21
    node_load5 0.31
    node_memory_Active 6.23935488e+08

    Other exporters listen on different ports and export stats for apache or mysql while more complicated ones will act as proxies for outgoing tests (via snmp, icmp, http). The full list of them is on the Prometheus website.

    So my problem was that I wanted to check my virtual machine that is on Linode. The machine only has a public IP and I didn’t want to:

    1. Allow random people to check my servers stats
    2. Have to setup some sort of VPN.

    So I decided that the best way was to just use put a user/password on the exporter.

    However the node_exporter does not  implement authentication itself since the authors wanted the avoid maintaining lots of security code. So I decided to put it behind a reverse proxy using apache mod_proxy.

    Step 1 – Install node_exporter

    Node_exporter is a single binary that I started via an upstart script. As part of the upstart script I told it to listen on localhost port 19100 instead of port 9100 on all interfaces

    # cat /etc/init/prometheus_node_exporter.conf
    description "Prometheus Node Exporter"
    
    start on startup
    
    chdir /home/prometheus/
    
    script
    /home/prometheus/node_exporter -web.listen-address 127.0.0.1:19100
    end script
    
    

    Once I start the exporter a simple “curl 127.0.0.1:19100/metrics” makes sure it is working and returning data.

    Step 2 – Add Apache proxy entry

    First make sure apache is listening on port 9100 . On Ubuntu edit the /etc/apache2/ports.conf file and add the line:

    Listen 9100

    Next create a simple apache proxy without authentication (don’t forget to enable mod_proxy too):

    # more /etc/apache2/sites-available/prometheus.conf 
    <VirtualHost *:9100>
     ServerName prometheus
    
    CustomLog /var/log/apache2/prometheus_access.log combined
     ErrorLog /var/log/apache2/prometheus_error.log
    
    ProxyRequests Off
     <Proxy *>
    Allow from all
     </Proxy>
    
    ProxyErrorOverride On
     ProxyPass / http://127.0.0.1:19100/
     ProxyPassReverse / http://127.0.0.1:19100/
    
    </VirtualHost>

    This simply takes requests on port 9100 and forwards them to localhost port 19100 . Now reload apache and test via curl to port 9100. You can also use netstat to see what is listening on which ports:

    Proto Recv-Q Send-Q Local Address   Foreign Address State  PID/Program name
    tcp   0      0      127.0.0.1:19100 0.0.0.0:*       LISTEN 8416/node_exporter
    tcp6  0      0      :::9100         :::*            LISTEN 8725/apache2

     

    Step 3 – Get Prometheus working

    I’ll assume at this point you have other servers working. What you need to do now is add the following entries for you server in you prometheus.yml file.

    First add basic_auth into your scape config for “node” and then add your servers, eg:

    - job_name: 'node'
    
      scrape_interval: 15s
    
      basic_auth: 
        username: prom
        password: mypassword
    
      static_configs:
        - targets: ['myserver.example.com:9100']
          labels: 
             group: 'servers'
             alias: 'myserver'

    Now restart Prometheus and make sure it is working. You should see the following lines in your apache logs plus stats for the server should start appearing:

    10.212.62.207 - - [31/Jul/2016:11:31:38 +0000] "GET /metrics HTTP/1.1" 200 11377 "-" "Go-http-client/1.1"
    10.212.62.207 - - [31/Jul/2016:11:31:53 +0000] "GET /metrics HTTP/1.1" 200 11398 "-" "Go-http-client/1.1"
    10.212.62.207 - - [31/Jul/2016:11:32:08 +0000] "GET /metrics HTTP/1.1" 200 11377 "-" "Go-http-client/1.1"

    Notice that connections are 15 seconds apart, get http code 200 and are 11k in size. The Prometheus server is using Authentication but apache doesn’t need it yet.

    Step 4 – Enable Authentication.

    Now create an apache password file:

    htpasswd -cb /home/prometheus/passwd prom mypassword

    and update your apache entry to the followign to enable authentication:

    # more /etc/apache2/sites-available/prometheus.conf
     <VirtualHost *:9100>
     ServerName prometheus
    
     CustomLog /var/log/apache2/prometheus_access.log combined
     ErrorLog /var/log/apache2/prometheus_error.log
    
     ProxyRequests Off
     <Proxy *>
     Order deny,allow
     Allow from all
     #
     AuthType Basic
     AuthName "Password Required"
     AuthBasicProvider file
     AuthUserFile "/home/prometheus/passwd"
     Require valid-user
     </Proxy>
    
     ProxyErrorOverride On
     ProxyPass / http://127.0.0.1:19100/
     ProxyPassReverse / http://127.0.0.1:19100/
     </VirtualHost>

    After you reload apache you should see the following:

    10.212.56.135 - prom [01/Aug/2016:04:42:08 +0000] "GET /metrics HTTP/1.1" 200 11394 "-" "Go-http-client/1.1"
    10.212.56.135 - prom [01/Aug/2016:04:42:23 +0000] "GET /metrics HTTP/1.1" 200 11392 "-" "Go-http-client/1.1"
    10.212.56.135 - prom [01/Aug/2016:04:42:38 +0000] "GET /metrics HTTP/1.1" 200 11391 "-" "Go-http-client/1.1"

    Note that the “prom” in field 3 indicates that we are logging in for each connection. If you try to connect to the port without authentication you will get:

    Unauthorized
    This server could not verify that you
    are authorized to access the document
    requested. Either you supplied the wrong
    credentials (e.g., bad password), or your
    browser doesn't understand how to supply
    the credentials required.

    That is pretty much it. Note that will need to add additional Virtualhost entries for more ports if you run other exporters on the server.

     

    FacebookGoogle+Share

    Playing with Shifter – NERSC’s tool to use Docker containers in HPC

    Early days yet, but playing with NERSC’s Shifter to let us use Docker containers safely on our test RHEL6 cluster is looking really interesting (given you can’t use Docker itself under RHEL6, and if you could the security concerns would cancel it out anyway).

    To use a pre-built Ubuntu Xenial image, for instance, you tell it to pull the image:

    [samuel@bruce ~]$ shifterimg pull ubuntu:16.04

    There’s a number of steps it goes through, first retrieving the container from the Docker Hub:

    2016-08-01T18:19:57 Pulling Image: docker:ubuntu:16.04, status: PULLING

    Then disarming the Docker container by removing any setuid/setgid bits, etc, and repacking as a Shifter image:

    2016-08-01T18:20:41 Pulling Image: docker:ubuntu:16.04, status: CONVERSION

    …and then it’s ready to go:

    2016-08-01T18:21:04 Pulling Image: docker:ubuntu:16.04, status: READY

    Using the image from the command line is pretty easy:

    [samuel@bruce ~]$ cat /etc/lsb-release
    LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
    
    [samuel@bruce ~]$ shifter --image=ubuntu:16.04 cat /etc/lsb-release
    DISTRIB_ID=Ubuntu
    DISTRIB_RELEASE=16.04
    DISTRIB_CODENAME=xenial
    DISTRIB_DESCRIPTION="Ubuntu 16.04 LTS"

    and the shifter runtime will copy in a site specified /etc/passwd, /etc/group and /etc/nsswitch.conf files so that you can do user/group lookups easily, as well as map in site specified filesystems, so your home directory is just where it would normally be on the cluster.

    [samuel@bruce ~]$ shifter --image=debian:wheezy bash --login
    samuel@bruce:~$ pwd
    /vlsci/VLSCI/samuel

    I’ve not yet got to the point of configuring the Slurm plugin so you can queue up a Slurm job that will execute inside a Docker container, but very promising so far!

    Correction: a misconception on my part – Shifter doesn’t put a Slurm batch job inside the container. It could, but there are good reasons why it’s better to leave that to the user (soon to be documented on the Shifter wiki page for Slurm integration).

    This item originally posted here:

    Playing with Shifter – NERSC’s tool to use Docker containers in HPC

    July 29, 2016

    Australia moves fast: North-West actually

    Australia on globeThis story is about the tectonic plate on which we reside.  Tectonic plates move, and so continents shift over time.  They generally go pretty slow though.

    What about Australia?  It appears that every year, we move 11 centimetres West and 7 centimetres North.  For a tectonic plate, that’s very fast.

    The last time scientists marked our location on the globe was in 1994, with the Geocentric Datum of Australia 1994 (GDA1994) – generally called GDA94 in geo-spatial tools (such as QGIS).  So that datum came into force 22 years ago.  Since then, we’ve moved an astonishing 1.5 metres!  You may not think much of this, but right now it actually means that if you use a GPS in Australia to get coordinates, and plot it onto a map that doesn’t correct for this, you’re currently going to be off by 1.5 metres.  Depending on what you’re measuring/marking, you’ll appreciate this can be very significant and cause problems.

    Bear in mind that, within Australia, GDA94 is not wrong as such, as its coordinates are relative to points within Australia. However, the positioning of Australia in relation to the rest of the globe is now outdated.  Positioning technologies have also improved.  So there’s a new datum planned for Australia, GDA2020.  By the time it comes into force, we’ll have shifted by 1.8 metres relative to GDA94.

    We can have some fun with all this:

    • If you stand and stretch both your arms out, the tips of your fingers are about 1.5 metres apart – of course this depends a bit on the length of your arms, but it’ll give you a rough idea.  Now imagine a pipe or cable in the ground at a particular GPS position,  move 1.5 metres.  You could clean miss that pipe or cable… oops!  Unless your GPS is configured to use a datum that gets updated, such as WGS84.  However, if you had the pipe or cable plotted on a map that’s in GDA94, it becomes messy again.
    • If you use a tool such as Google Earth, where is Australia actually?  That is, will a point be plotted accurately, or be 1.5 metres out, or somewhere in between?
      Well, that would depend on when the most recent broad scale photos were taken, and what corrections the Google Earth team possibly applies during processing of its data (for example, Google Earth uses a different datum – WGS 84 for its calculations).
      Interesting question, isn’t it…
    • Now for a little science/maths challenge.  The Northern most tip of Australia, Cape York, is just 150km South of Papua New Guinea (PNG).  Presuming our plate maintains its present course and speed, roughly how many years until the visible bits (above sea level) of Australia and PNG collide?  Post your answer with working/reasoning in a comment to this post!  Think about this carefully and do your research.  Good luck!

    July 28, 2016

    Neuroscience in PSYOPS, World Order, and More

    One of the funny things that I've heard is that people from one side believe that people from another side are somehow 'brainwashed' into believing what they do. As we saw in out last post there is a lot of manipulation and social engineering going on if you think about it, http://dtbnguyen.blogspot.com/2016/07/social-engineeringmanipulation-rigging.html We're going to examine just exactly why

    License quibbles (aka Hiro & linux pt 2)

    Since my last post regarding the conversion of media from Channel 9’s Catch Up service, I have been in discussion with the company behind this technology, Hiro-Media. My concerns were primarily around their use of the open source xvid media codec and whilst I am not a contributor to xvid (and hence do not have any ownership under copyright), I believe it is still my right under the GPL to request a copy of the source code.

    First off I want to thank Hiro-Media for their prompt and polite responses. It is clear that they take the issue of license violations very seriously. Granted, it would be somewhat hypocritical for a company specialising in DRM to not take copyright violations within their own company seriously, but it would not be the first time.

    I initially asserted that, due to Hiro’s use (and presumed modification) of xvid code, that this software was considered a derivative and therefore bound in its entirety by the GPL. Hiro-Media denied this stating they use xvid in its original, unmodified state and hence Hiro is simply a user of rather than a derivative of xvid. This is a reasonable statement albeit one that is difficult to verify. I want to stress at this point that in my playing with the Hiro software I have NOT in anyway reverse engineered it nor have I attempted to decompile their binaries in any way.

    In the end, the following points were revealed:

    • The Mac version of Hiro uses a (claimed) unmodified version of the Perian Quicktime component
    • The Windows version of Hiro currently on Channel 9’s website IS indeed modified, what Hiro-Media terms an ‘accidental internal QA’ version. They state that they have sent a new version to Channel 9 that corrects this. The xvid code they are using can be found at http://www.koepi.info/xvid.html
    • Neither version has included a GPL preamble within their EULA as required. Again, I am assured this is to be corrected ASAP.

    I want to reiterate that Hiro-Media have been very cooperative about this and appear to have genuine concern. I am impressed by the Hiro system itself and whilst I am still not a fan of DRM in general, this is by far the best compromise I have seen to date. They just didn’t have a linux version.

    This brings me to my final, slightly more negative point. On my last correspondence with Hiro-Media, they concluded with the following:

    Finally, please note our deepest concerns as to any attempt to misuse our system, including the content incorporated into it, as seems to be evidenced in your website. Prima facia, such behavior consists a gross and fundamental breach of our license (which you have already reviewed). Any such misuse may cause our company, as well as any of our partners, vast damages.

    I do not wish to label this a threat (though I admit to feeling somewhat threatened by it), but I do want to clear up a few things about what I have done. The statement alleges I have violated Hiro’s license (pot? kettle? black?) however this is something I vehemently disagree with. I have read the license very careful (Obviously as I went looking for the GPL) and the only relevant part is:

    You agree that you will not modify, adapt or translate, or disassemble, decompile, reverse engineer or otherwise attempt to discover the source code of the Software.

    Now I admit to being completely guilty of a single part of this, I have attempted to discover the source code. BUT (and this is a really big BUT), I have attempted this by emailing Hiro-Media and asking them for it, NOT by decompiling (or in any other way inspecting) the software. In my opinion, the inclusion of that specific part in their license also goes against the GPL as such restrictions are strictly forbidden by it.
    But back to the point, I have not modified, translated, disassembled, decompiled or reverse engineered the Hiro software. Additionally, I do not believe I have adapted it either. It is still doing exactly the same thing as it was originally, that is taking an incoming video stream, modifying it and decoding it. Importantly, I do not modify any files in any way. What I have altered is how Quicktime uses the data returned by Hiro. All my solution does is (using official OSX/Quicktime APIs) divert the output to a file rather than to the screen. In essence I have not done anything different to the ‘Save As’ option found in Quicktime Pro, however not owning Quicktime Pro, I merely found another way of doing this.

    So that’s my conclusion. I will reply to Hiro-Media with a link to this post asking whether they still take issue with what I have done and take things from there.
    To the guys from Hiro if you are reading this, I didn’t do any of this to start trouble. All I wanted was a way to play these files on my linux HTPC, with or without ads. Thankyou.

    Channel 9, Catch Up, Hiro and linux

    About 6 months ago, Channel 9 launched their ‘Catch Up’ service. Basically this is their way of fighting piracy and allowing people to download Australian made TV shows to watch on their PC. Now, of course, no ‘old media’ service would possibly do this without the wonders of DRM. Channel 9 though, are taking a slightly different approach.

    Instead of the normal style of DRM that prevents you copying the file, Channel 9 employs technology from a company called Hiro. Essentially you install the Hiro player, download the file and watch it. The player will insert unskippable ads throughout the video, supposedly even targeted at your demographic. Now this is actually a fairly neat system, Channel 9 actually encourage you to share the video files over bittorrent etc! The problem, as I’m sure you can guess, is that there’s no player for linux.

    So, just to skip to the punchline, yes it IS possible to get these files working on free software (completely legally & without the watermark)! If you just want to know how to do it, jump to the end as I’m going to explain a bit of background first.

    Hiro

    The Hiro technology is interesting in that it isn’t simply some custom player. The files you download from Channel 9 are actually xvid encoded, albeit a bastard in-bred cousin of what xvid should be. If you simply download the file and play it with vlc or mplayer, it will run, however you will get a nasty watermark over the top of the video and it will almost certainly crash about 30s in when it hits the first advertising blob. There is also some trickiness going on with the audio as, even if you can get the video to keep playing, the audio will jump back to the beginning at this point. Of course, the watermark isn’t just something that’s placed over the top in post-processing like a subtitle, its in the video data itself. To remove it you actually need to filter the video to modify the area covered by the watermark to darken/lighten the pixels affected. Sounds crazy and a tremendous amount of work right? Well thankfully its already been done, by Hiro themselves.

    When you install Hiro, you don’t actually install a media player, you install either a DirectShow filter or a Quicktime component depending on your platform. This has the advantage that you can use reasonably standard software to play the files. Its still not much help for linux though.

    Before I get onto how to create a ‘normal’ xvid file, I just want to mention something I think should be a concern for free software advocates. As you might know, xvid is an open codec, both for encoding and decoding. Due to the limitations of Quicktime and Windows Media Player, Hiro needs to include an xvid decoder as part of their filter. I’m sure its no surprise to anyone though that they have failed to release any code for this filter, despite it being based off a GPL’d work. IA(definitely)NAL, but I suspect there’s probably some dodginess going on here.

    Using Catchup with free software

    Very basically, the trick to getting the video working is that it needs to be passed through the filter provided by Hiro. I tried a number of methods to get the files converted for use with mplayer or vlc and in the end, unfortunately, I found that I needed to be using either Windows or OSX to get it done. Smarter minds than mine might be able to get the DirectShow filter (HiroTransform.ax) working with mplayer in a similar manner to how CoreAVC on linux works, but I had no luck.

    But, if you have access to OSX, here’s how to do it:

    1. Download and install the Hiro software for Mac. You don’t need to register or anything, in fact, you can delete the application the moment you finish the install. All you need is the Quicktime component it added.
    2. Grab any file from the Catch Up Service (http://video.ninemsn.com.au/catchuptv). I’ve tested this with Underbelly, but all videos should work.
    3. Install ffmpegx (http://ffmpegx.com)
    4. Grab the following little script: CleanCatch.sh
    5. Run:
      chmod +x CleanCatch.sh
      ./CleanCatch.sh <filename>
    6. Voila. Output will be a file called ‘<filename>.clean.MP4′ and should be playable in both VLC and mplayer

    Distribution

    So, I’m the first to admit that the above is a right royal pain to do, particularly the whole requiring OSX part. To save everyone the hassle though, I believe its possible to simply distribute the modified file. Now again, IANAL, but I’ve gone over the Channel 9 website with a fine tooth comb and can see nothing that forbids me from distributing this newly encoded file. I agreed to no EULA when I downloaded the original video and their site even has the following on it:

    You can share the episode with your friends and watch it as many times as you like – online or offline – with no limitations

    That whole ‘no limitations’ part is the bit I like. Not only have Channel 9 given me permission to distribute the file, they’ve given it to me unrestricted. I’ve not broken any locks and in fact have really only used the software provided by Channel 9 and a standard transcoding package.

    This being the case, I am considering releasing modified versions of Channel 9’s files over bittorrent. I’d love to hear people’s opinions about this before doing so though in case they know more than I (not a hard thing) about such matters.

    The rallyduino lives

    [Update: The code or the rallyduino can be found at: http://code.google.com/p/rallyduino/]

    Amidst a million other things (Today is T-1 days until our little bubs technical due date! No news yet though) I finally managed to get the rallyduino into the car over the weekend. It actually went in a couple of weeks ago, but had to come out again after a few problems were found.

    So, the good news, it all works! I wrote an automagic calibration routine that does all the clever number crunching and comes up with the calibration number, so after a quick drive down the road, everything was up and running. My back of the envelope calculations for what the calibration number for our car would be also turned out pretty accurate, only about 4mm per revolution out. The unit, once calibrated, is at least as accurate as the commercial devices and displays considerably more information. Its also more flexible as it can switch between modes dynamically and has full control from the remote handset. All in all I was pretty happy, even the instantaneous speed function worked first time.

    To give a little background, the box uses a hall effect sensor mounted next to a brake caliper that pulses each time a wheel stud rotates past. This is a fairly common method for rally computers to use and to make life simpler, the rallyduino is pin compatible with another common commercial product, the VDO Minicockpit. As we already had a Minicockpit in the car, all we did was ‘double adapt’ the existing plug and pop in the rallyduino. This means there’s 2 computers running off the 1 sensor, in turn making it much simpler (assuming 1 of them is known to be accurate) to determine if the other is right.

    After taking the car for a bit of a drive though, a few problems became apparent. The explain them, a pic is required:
    8856285

    The 4 devices in the pic are:

    1. Wayfinder electronic compass
    2. Terratrip (The black box)
    3. Rallyduino (Big silver box with the blue screen)
    4. VDO Minicockpit (Sitting on top of the rallyduino)

    The major problem should be immediately obvious. The screen is completely unsuitable. Its both too small and has poor readability in daylight. I’m currently looking at alternatives and it seems like the simplest thing to do is get a 2×20 screen the same physical size as the existing 4×20. This, however, means that there would only be room for a single distance tracker rather than the 2 currently in place. The changeover is fairly trivial as the code, thankfully, is nice and flexible and the screen dimensions can be configured at compile time (From 2×16 to 4×20). Daylight readable screens are also fairly easily obtainable (http://www.crystalfontz.com is the ultimate small screen resource) so its just a matter of ordering one. In the long term I’d like to look at simply using a larger 4×20 screen but, as you can see, real estate on the dash is fairly tight.

    Speaking of screens, I also found the most amazing little LCD controller from web4robot.com. It has both serial and I2C interfaces, a keypad decoder (with inbuilt debounce) and someone has gone to all the hard work of work of writing a common arduino interface library for it and other I2C LCD controllers (http://www.wentztech.com/radio/arduino/projects.html) . If you’re looking for such a LCD controller and you are working with an arduino, I cannot recommend these units enough. They’re available on eBay for about $20 Au delivered too. As much as I loved the unit from Phil Anderson, it simply doesn’t have the same featureset as this one, nor is it as robust.

    So that’s where things are at. Apologies for the brain dump nature of this post, I just needed to get everything down while I could remember it all.

    1 + 1 = 3

    No updates to this blog in a while I’m afraid, things have just been far too busy to have had anything interesting (read geeky) to write about. That said, I am indulging and making this post purely to show off.

    On Monday 25th May at 8:37am, after a rather long day/night, Mel and I welcomed Spencer Bailey Stewart into the world. There were a few little issues throughout the night (Particularly towards the end) and he had some small hurdles to get over in his first hour, but since then he has gone from strength to strength and both he and Mum and now doing amazingly well.
    Obligatory Pic:

    Spence

    Spencer Bailey Stewart

    He’s a very placid little man and would quite happily sleep through an earthquake, much to our delight. And yes, that is a little penguin he’s holding on to in that pic

    So that’s all really. I am very conscious of not becoming a complete baby bore so unless something actually worth writing about happens, this will hopefully be my only baby post for the sake of a baby post.

    Boxee iView back online

    Just a quick post to say the ABC iView Boxee app has been updated to version 0.7 and should now be fully functional once again. I apologise to anyone who has been using the app for how long this update has taken and I wish I could say I’ve been off solving world hunger or something, but in reality I’ve just been flat out with work and family. I’ve also got a few other projects on the go that have been keeping me busy. These may or may not ever reach a stage of public consumption, but if they do it’ll be cool stuff.

    For anyone using Boxee, you may need to remove the app from My Apps and wait for Boxee to refresh its repository index, but eventually version 0.7 should pop up. Its a few rough in places so I hope to do another cleanup within the next few weeks, but at least everything is functional again.

    Going in to business

    Lately my toying around with media centers has created some opportunities for commercial work in this area, which has been a pleasant change. As a result of this I’ve formed Noisy Media, a company specialising in the development of media centre apps for systems such as Boxee as well as the creation of customised media applications using XBMC (and similar) as a base.

    Whilst I don’t expect this venture to ever be huge, I can see the market growing hugely in the future as products such as Google TV (Something I will be looking at VERY closely) and the Boxee Box are released and begin bringing streaming Video on Demand to the loungeroom. Given this is something I have a true passion for, the ability to turn work in this area into something profitable is very appealing, and exciting.

    Here’s to video on demand!

    ASX RSS down

    Just a quick note to advise that the ASX RSS feed at http://noisymime.org/asx is currently not functional due to a change in the source data format.  I am in the process of performing a rewrite on this now and should have it back up and running (Better and more robust than ever) within the next few days.

    Apologies for the delay in getting things back up and running.

    Cortina Fuel Injection – Part 1 The Electronics

    Besides being your run of the mill computer geek, I’ve always been a bit of a car geek as well. This often solicits down-the-nose looks from others who associate such people with V8 Supercar lovin’ petrolheads, which has always surprised me little because the most fun parts of working on a car are all just testing physics theories anyway. With that in mind, I’ll do this writeup from the point of view of the reader being a non-car, but scientifically minded person. First a bit of background…

    Background

    For the last 3 years or so my dad and I have been working a project to fuel inject our race car. The car itself is a 1968 Mk2 Cortina and retains the original 40 year old 1600 OHV engine. This engine is originally carbureted, meaning that it has a device that uses the vacuum created by the engine to mix fuel and air. This mixture is crucial to the running of an engine as the ratio of fuel to air dramatically alters power, response and economy. Carburetors were used for this function for a long time and whilst they achieve the basics very well, they are, at best, a compromise for most engines. To overcome these limitations, car manufacturers started moving to fuel injection in the 80’s, which allowed precise control of the amount of fuel added through the use of electronic signals. Initially these systems were horrible however, being driven by analog or very basic digital computers that did not have the power or inputs needed to accurately perform this function. These evolved to something useful throughout the 90’s and by the 00’s cars were having full sequential system (more on this later) that could deliver both good performance and excellent economy. It was our plan to fit something like the late 90’s type systems (ohh how did this change by the end though) to the Cortina with the aims of improving the power and drivability of the old engine. IN this post I’m going to run through the various components needed from the electrical side to make this all happen, as well as a little background on each. Our starting point was this:

    The System

    To have a computer control when are how much fuel to inject, it requires a number of inputs:

    • A crank sensor. This is the most important thing and tells the computer where in the 4-stroke cycle (HIGHLY recommended reading if you don’t know about the 4 strokes and engine goes through) the engine is and therefore WHEN to inject the fuel. Typically this is some form of toothed wheel that is on the end of the crankshaft with a VR or Hall effect sensor that pulses each time a tooth goes past it. The more teeth the wheel has, the more precisely the computer knows where the engine is (Assuming it can keep up with all the pulses). By itself the toothed wheel is not enough however as the computer needs a reference point to say when the cycle is beginning. This is typically done by either a 2nd sensor that only pulses once every 4 strokes, or by using what’s know as a missing tooth wheel, which is the approach we have taken. This works by having a wheel that would ordinarily have, say, 36 teeth, but has then had one of them removed. This creates an obvious gap in the series of pulses which the computer can use as a reference once it is told where in the cycle the missing tooth appears. The photos below show the standard Cortina crankshaft end and the wheel we made to fit onto the end

      Standard Crankshaft pulley

      36-1 sensor wheel added. Pulley is behind the wheel

      To read the teeth, we initially fitted a VR sensor, which sat about 0.75mm from the teeth, however due to issues with that item, we ended up replacing it with a Hall Effect unit.

    • Some way of knowing how much air is being pulled into the engine so that it knows HOW MUCH fuel to inject. In earlier fuel injection systems this was done with a Manifold Air Flow (MAF) sensor, a device which heated a wire that was in the path of the incoming air. By measuring the drop in temperature of the wire, the amount of air flowing over it could be determined (Although guessed is probably a better word as most of these systems tended to be fairly inaccurate). More recently systems (From the late 90’s onwards) have used Manifold Absolute Pressure (MAP) sensors to determine the amount of air coming in. Computationally these are more complex as there are a lot more variables that need to be known by the computer, but they tend to be much more accurate for the purposes of fuel injection. Nearly all aftermarket computers now use MAP and given how easy it is the setup (just a single vacuum hose going from the manifold to the ECU) this is the approach we took.

    The above are the bare minimum inputs required for a computer to control the injection, however typically more sensors are needed in order to make the system operate smoothly. We used:

    • Temperature sensors: As the density of air changes with temperature, the ECU needs to know how hot or cold the incoming air is. It also needs to know the temperature of the water in the coolant system to know whether it is running too hot or cold so it can make adjustments as needed.
    • Throttle position sensor: The ratio of fuel to air is primarily controlled by the MAf or MAP sensor described above, however as changes in these sensors are not instantaneous, the ECU needs to know when the accelerator is pressed so it can add more fuel for the car to be able to accelerate quickly. These sensors are typically just a variable resistor fitted to the accelerator.
    • Camshaft sensor: I’ll avoid getting too technical here, but injection can essentially work in 2 modes, batched or sequential. In the 4 strokes an engine goes through, the crankshaft will rotate through 720 degrees (ie 2 turns). With just a crank sensor, the ECU can only know where the shaft is in the 0-360 degree range. To overcome this, most fuel injection systems up to about the year 2000 ran in ‘batched’ mode, meaning that the fuel injectors would fire in pairs, twice (or more) per 720 degrees. This is fine and cars can run very smoothly in this mode, however it means that after being injected, some fuel mixture sits in the intake manifold before being sucked into the chamber. During this time, the mixture starts to condense back into a liquid which does not burn as efficiently, leading to higher emissions and fuel consumption. To improve the situation, car manufacturers starting moving to sequential injection, meaning that the fuel is only ever injected at the time it can go straight into the combustion chamber. To do this, the ECU needs to know where in 720 degrees the engine is rather than just in 360 degrees. As the camshaft in a car runs at half the crankshaft speed, all you need to do this is place a similar sensor on this that produces 1 pulse every revolution (The equivalent of 1 pulse every 2 turns of the crank). In our case, we had decided that to remove the distributor (which is driven off the crank) and converted it to provide this pulse. I’ll provide a picture of this shortly, but it uses a single tooth that passes through a ‘vane’ type hall effect sensor, so that the signal goes high when the tooth enters the sensor and low when it leaves.
    • Oxygen sensor (O2) – In order to give some feedback to the ECU about how the engine is actually running, most cars these days run a sensor in the exhaust system to determine how much of the fuel going in is actually being burned. Up until very recently, virtually all of these sensors were what is known as narrowband, which in short means that they can determine whether the fuel/air mix is too lean (Not enough fuel) or too rich (Too much fuel), but not actually by how much. The upshot of this is that you can only ever know EXACTLY what the fuel/air mixture is when it switches from one state to the other. To overcome this problem, there is a different version of the sensor, known as wideband, that (within a certain range) can determine exactly how rich or lean the mixture is. If you ever feel like giving yourself a headache, take a read through http://www.megamanual.com/PWC/LSU4.htm which is the theory behind these sensors. They are complicated! Thankfully despite all the complication, they are fairly easy to use and allow much easier and quicker tuning once the ECU is up and running.

    So with all of the above, pretty much the complete electronics system is covered. Of course, this doesn’t even start to cover off the wiring, fusing, relaying etc that has to go into making all of it work in the terribly noisy environment of a car, but that’s all the boring stuff

    ECU

    Finally the part tying everything together is the ECU (Engine Control Unit) itself. There are many different types of programmable ECUs available and they vary significantly in both features and price, ranging from about $400 to well over $10,000. Unsurprisingly there’s been a lot of interest in this area from enthusiasts looking to make their own and despite there having been a few of these to actually make it to market, the most successfully has almost certainly been Megasquirt. when we started with this project we had planned on using the 2nd generation Megasquirt which, whilst not having some of the capabilities of the top end systems, provided some great bang for bang. As we went along though, it became apparent that the Megasquirt 3 would be coming out at about the right time for us and so I decided to go with one of them instead. I happened to fluke one of the first 100 units to be produced and so we had it in our hands fairly quickly.

    Let me just say that this is an AMAZING little box. From what I can see it has virtually all the capabilities of the (considerably) more expensive commercial units including first class tuning software (Multi platform, Win, OSX, linux) and a very active developer community. Combined with the Extra I/O board (MS3X) the box can do full sequential injection and ignition (With direct coil driving of ‘smart’ coils), launch control, traction control, ‘auto’ tuning in software, generic I/O of basically any function you can think of (including PID closed loop control), full logging via onboard SD slot and has a built in USB adaptor to boot!

    Megasquirt 3 box and unassembled components

    In the next post I’ll go through the hardware and setup we used to make all this happen. I’ll also run through the ignition system that we switched over to ECU control.

    Command line Skype

    Despite risking the occasional dirty look from a certain type of linux/FOSS supporter, I quite happily run the (currently) non-free Skype client on my HTPC. I have a webcam sitting on top of the TV and works flawlessly for holding video chats with family and friends.

    The problem I initially faced however, was that my HTPC is 100% controlled by a keyboard only. Unlike Windows, the linux version of Skype has no built in shortcut keys (user defined or otherwise) for basic tasks such as answering and terminating calls. This makes it virtually impossible to use out of the box. On the upside though, the client does have an API and some wonderful person out there has created a python interface layer for it, aptly named, Skype4Py.

    A little while ago when I still had free time on weekends, I sat down and quickly hacked together a python script for answering and terminating calls, as well as maximising video chats from the command line. I then setup a few global shortcut keys within xfce to run the script with the appropriate arguments.

    I won’t go into the nitty gritty of the script as it really is a hack in some places (Particularly the video maximising), but I thought I’d post it up in case it is of use to anyone else. I’ve called it mythSkype, simply because the primary function of the machine I run it on is MythTV, but it has no dependencies on Myth at all.

    The depedencies are:

    • Python – Tested with 2.6, though 2.5 should work
    • Skype4Py –  any version
    • xdotool – Only required for video maximising

    To get the video maximising to work you’ll need to edit the file and set the screen_width and screen_height variables to match your resolution.
    Make sure you have Skype running, then simply execute one of the following:

    • ./mythSkype -a (Answer any ringing calls)
    • ./mythSkype -e (End active calls)
    • ./mythSkype -m (Maximise the current video)

    The first time you run the script, you will get a prompt from Skype asking if you are happy with Skype4Py having access to the application. Obviously you must give your ascent or nothing will work.

    Its nothing fancy, but I hope its of use to others out there.

    Download: mythSkype

    ABC iView on Boxee

    A few months ago I switched from using the standard mythfrontend to Boxee, a web enhanced version of the popular XBMC project. Now Boxee has a LOT of potential and the upcoming Boxee Box looks very promising, but its fantastic web capabilities are let down here in Australia as things such as Hulu and Netflix streaming are not available here.

    What we do have though is a national broadcaster with a reasonably good online facility. The ABCs iView has been around for some time and has a really great selection of current programs available on it. Sounds like the perfect candidate for a Boxee app to me.

    So with the help of Andy Botting and using Jeremy’s Vissers Python iView download app for initial guidance, I put together a Boxee app for watching iView programs fullscreen. For those wishing to try it out, just add the following repository within Boxee:
    http://noisymime.org/boxee

    Its mostly feature complete although there are a few things that still need to be added. If you have any suggestions or find a bug either leave a comment or put in a ticket at http://code.google.com/p/xbmc-boxee-iview/issues/list (A Google code site by Andy that I am storing this project at)

    So that’s the short version of the story. Along the way however there has been a few hiccups and I want to say something about what the ABC (and more recently the BBC) have done to ‘protect’ their content.

    The ABC have enabled a function called SWF Verification on their RTMP stream. This is something that Adobe offer on top of their RTMP products despite the fact that they omitted it from the public RTMP spec. That wouldn’t be so bad, except that they are now going after open source products that implement this, threatening them with cease and desists. Going slightly technical for a minute, SWF Verification is NOT a real protection system. It does not encrypt the content being sent nor does it actually prevent people copying it. The system works by requesting a ‘ping’ every 60-90 seconds. If the player can’t provide the correct response (Which is made up of things such as the date and time and the phrase “Genuine Adobe Flash Player 001″) then the server stops the streaming. Hardly high tech stuff.

    In my opinion the ABC has made a huge mistake in enabling this as it achieves nothing in stopping piracy or restricting the service to a certain platform and serves only to annoy and frustrate the audience. There is a patch available at http://code.google.com/p/xbmc-boxee-iview that allows Boxee to read these streams directly, however until such time as this is included in Boxee/XBMC mainline (Watch this space: http://trac.xbmc.org/ticket/8971) or the ABC come to their senses as disable this anti-feature, this Boxee app will use the flash interface instead (boo!)

    So that’s it. Hope the app is useful to people and, as stated above, if there are any problems, please let me know.

    [EDIT]
    I should’ve mentioned this originally, Andy and I did actually contact the iView team at the ABC regarding SWF verification. They responded with the following:

    Thanks for contacting us re-Boxee. We agree that it’s a great platform and ultimately appropriate for iView iteration. Currently we’re working out our priorities this year for our small iView team, in terms of extended content offerings, potential platforms and general enhancements to the site.

    Just some background on our security settings. We have content agreements with various content owners (individuals, production companies, US TV networks etc) a number require additional security, such as SWF hashing. Our content owners also consider non-ABC rendering of that content as not in the spirit of those agreements.

    We appreciate the effort you have put into the plug-in and your general interest in all things iView. So once we are on our way with our development schedule for “out of the browser” iView, we’ll hopefully be in a position to share our priorities a little more. We would like to keep in touch with you this year and if you have any questions or comments my direct email is ********@abc.net.au.

    [STATUS]
    The app is currently in a WORKING state. If you are experiencing any problems, please send me a copy of your Boxee log file and I will investigate the issue.

    July 27, 2016

    LUV Main August 2016 Meeting: M.2 / CRCs

    Aug 2 2016 18:30
    Aug 2 2016 20:30
    Aug 2 2016 18:30
    Aug 2 2016 20:30
    Location: 

    6th Floor, 200 Victoria St. Carlton VIC 3053

    Speakers:

    • Russell Coker, M.2
    • Rodney Brown, CRCs

    200 Victoria St. Carlton VIC 3053

    Late arrivals, please call (0490) 049 589 for access to the venue.

    Before and/or after each meeting those who are interested are welcome to join other members for dinner. We are open to suggestions for a good place to eat near our venue. Maria's on Peel Street in North Melbourne is currently the most popular place to eat after meetings.

    LUV would like to acknowledge Red Hat and Infoxchange for their help in obtaining the meeting venues.

    Linux Users of Victoria Inc. is an incorporated association, registration number A0040056C.

    August 2, 2016 - 18:30

    read more

    LUV Beginners August Meeting: File Sharing in Linux

    Aug 20 2016 12:30
    Aug 20 2016 16:30
    Aug 20 2016 12:30
    Aug 20 2016 16:30
    Location: 

    Infoxchange, 33 Elizabeth St. Richmond

    This hands-on presentation and tutorial with Wen Lin will introduce the various types of file sharing in Linux - from the more traditional NFS & Samba to the newer cloud-based ones like Dropbox, Google Drive and OwnCloud. The primary audience of the talk will be for beginners (newbies), but hopefully some of you who are familiar with Linux will get something out of it as well.

    The meeting will be held at Infoxchange, 33 Elizabeth St. Richmond 3121 (enter via the garage on Jonas St.)

    Late arrivals, please call (0490) 049 589 for access to the venue.

    LUV would like to acknowledge Infoxchange for the venue.

    Linux Users of Victoria Inc., is an incorporated association, registration number A0040056C.

    August 20, 2016 - 12:30

    read more

    Get off my lawn: separating Docker workloads using cgroups

    On my team, we do two different things in our Continuous Integration setup: build/functional tests, and performance tests. Build tests simply test whether a project builds, and, if the project provides a functional test suite, that the tests pass. We do a lot of MySQL/MariaDB testing this way. The other type of testing we do is performance tests: we build a project and then run a set of benchmarks against it. Python is a good example here.

    Build tests want as much grunt as possible. Performance tests, on the other hand, want a stable, isolated environment. Initially, we set up Jenkins so that performance and build tests never ran at the same time. Builds would get the entire machine, and performance tests would never have to share with anyone.

    This, while simple and effective, has some downsides. In POWER land, our machines are quite beefy. For example, one of the boxes I use - an S822L - has 4 sockets, each with 4 cores. At SMT-8 (an 8 way split of each core) that gives us 4 x 4 x 8 = 128 threads. It seems wasteful to lock this entire machine - all 128 threads - just so as to isolate a single-threaded test.1

    So, can we partition our machine so that we can be running two different sorts of processes in a sufficiently isolated way?

    What counts as 'sufficiently isolated'? Well, my performance tests are CPU bound, so I want CPU isolation. I also want memory, and in particular memory bandwith to be isolated. I don't particularly care about IO isolation as my tests aren't IO heavy. Lastly, I have a couple of tests that are very multithreaded, so I'd like to have enough of a machine for those test results to be interesting.

    For CPU isolation we have CPU affinity. We can also do something similar with memory. On a POWER8 system, memory is connected to individual P8s, not to some central point. This is a 'Non-Uniform Memory Architecture' (NUMA) setup: the directly attached memory will be very fast for a processor to access, and memory attached to other processors will be slower to access. An accessible guide (with very helpful diagrams!) is the relevant RedBook (PDF), chapter 2.

    We could achieve the isolation we want by dividing up CPUs and NUMA nodes between the competing workloads. Fortunately, all of the hardware NUMA information is plumbed nicely into Linux. Each P8 socket gets a corresponding NUMA node. lscpu will tell you what CPUs correspond to which NUMA nodes (although what it calls a CPU we would call a hardware thread). If you install numactl, you can use numactl -H to get even more details.

    In our case, the relevant lscpu output is thus:

    NUMA node0 CPU(s):     0-31
    NUMA node1 CPU(s):     96-127
    NUMA node16 CPU(s):    32-63
    NUMA node17 CPU(s):    64-95
    

    Now all we have to do is find some way to tell Linux to restrict a group of processes to a particular NUMA node and the corresponding CPUs. How? Enter control groups, or cgroups for short. Processes can be put into a cgroup, and then a cgroup controller can control the resouces allocated to the cgroup. Cgroups are hierarchical, and there are controllers for a number of different ways you could control a group of processes. Most helpfully for us, there's one called cpuset, which can control CPU affinity, and restrict memory allocation to a NUMA node.

    We then just have to get the processes into the relevant cgroup. Fortunately, Docker is incredibly helpful for this!2 Docker containers are put in the docker cgroup. Each container gets it's own cgroup under the docker cgroup, and fortunately Docker deals well with the somewhat broken state of cpuset inheritance.3 So it suffices to create a cpuset cgroup for docker, and allocate some resources to it, and Docker will do the rest. Here we'll allocate the last 3 sockets and NUMA nodes to Docker containers:

    cgcreate -g cpuset:docker
    echo 32-127 > /sys/fs/cgroup/cpuset/docker/cpuset.cpus
    echo 1,16-17 > /sys/fs/cgroup/cpuset/docker/cpuset.mems
    echo 1 > /sys/fs/cgroup/cpuset/docker/cpuset.mem_hardwall
    

    mem_hardwall prevents memory allocations under docker from spilling over into the one remaining NUMA node.

    So, does this work? I created a container with sysbench and then ran the following:

    root@0d3f339d4181:/# sysbench --test=cpu --num-threads=128 --max-requests=10000000 run
    

    Now I've asked for 128 threads, but the cgroup only has CPUs/hwthreads 32-127 allocated. So If I run htop, I shouldn't see any load on CPUs 0-31. What do I actually see?

    htop screenshot, showing load only on CPUs 32-127

    It works! Now, we create a cgroup for performance tests using the first socket and NUMA node:

    cgcreate -g cpuset:perf-cgroup
    echo 0-31 > /sys/fs/cgroup/cpuset/perf-cgroup/cpuset.cpus
    echo 0 > /sys/fs/cgroup/cpuset/perf-cgroup/cpuset.mems
    echo 1 > /sys/fs/cgroup/cpuset/perf-cgroup/cpuset.mem_hardwall
    

    Docker conveniently lets us put new containers under a different cgroup, which means we can simply do:

    dja@p88 ~> docker run -it --rm --cgroup-parent=/perf-cgroup/ ppc64le/ubuntu bash
    root@b037049f94de:/# # ... install sysbench
    root@b037049f94de:/# sysbench --test=cpu --num-threads=128 --max-requests=10000000 run
    

    And the result?

    htop screenshot, showing load only on CPUs 0-31

    It works! My benchmark results also suggest this is sufficient isolation, and the rest of the team is happy to have more build resources to play with.

    There are some boring loose ends to tie up: if a build job does anything outside of docker (like clone a git repo), that doesn't come under the docker cgroup, and we have to interact with systemd. Because systemd doesn't know about cpuset, this is quite fiddly. We also want this in a systemd unit so it runs on start up, and we want some code to tear it down. But I'll spare you the gory details.

    In summary, cgroups are surprisingly powerful and simple to work with, especially in conjunction with Docker and NUMA on Power!


    1. It gets worse! Before the performance test starts, all the running build jobs must drain. If we have 8 Jenkins executors running on the box, and a performance test job is the next in the queue, we have to wait for 8 running jobs to clear. If they all started at different times and have different runtimes, we will inevitably spend a fair chunk of time with the machine at less than full utilisation while we're waiting. 

    2. At least, on Ubuntu 16.04. I haven't tested if this is true anywhere else. 

    3. I hear this is getting better. It is also why systemd hasn't done cpuset inheritance yet. 

    Personal submission to the Productivity Commission Review on Public Sector Data

    My name is Pia Waugh and this is my personal submission to the Productivity Commission Review on Public Sector Data. It does not reflect the priorities or agenda of my employers past, present or future, though it does draw on my expertise and experience in driving the open data agenda and running data portals in the ACT and Commonwealth Governments from 2011 till 2015. I was invited by the Productivity Commission to do a submission and thought I could provide some useful ideas for consideration. I note I have been on maternity leave since January 2016 and am not employed by the Department of Prime Minister and Cabinet or working on data.gov.au at the time of writing this submission. This submission is also influenced by my work and collaboration with other Government jurisdictions across Australia, overseas and various organisations in the private and community sectors. I’m more than happy to discuss these ideas or others if useful to the Productivity Commission.

    I would like to thank all those program and policy managers, civic hackers, experts, advocates, data publishers, data users, public servants and vendors whom I have had the pleasure to work with and have contributed to my understanding of this space. I’d also like to say a very special thank you to the Australian Government Chief Technology Officer, John Sheridan, who gave me the freedom to do what was needed with data.gov.au, and to Allan Barger who was my right hand man in rebooting the agenda in 2013, supporting agencies and helping establish a culture of data publishing and sharing across the public sector. I think we achieved a lot in only a few years with a very small but highly skilled team. A big thank you also to Alex Sadleir and Steven De Costa who were great to work with and made it easy to have an agile and responsive approach to building the foundation for an important piece of data infrastructure for the Australian Government.

    Finally, this is a collection of some of my ideas and feedback for use by the Productivity Commission however, it doesn’t include everything I could possibly have to say on this topic because, frankly, we have a small baby who is taking most of my time at the moment. Please feel free to add your comments, criticisms or other ideas to the comments below! It is all licensed as Creative Commons 4.0 By Attribution, so I hope it is useful to others working in this space.

    The Importance of Vision

    Without a vision, we stumble blindly in the darkness. Without a vision, the work and behaviours of people and organisations are inevitably driven by other competing and often short term priorities. In the case of large and complex organisms like the Australian Public Service, if there is no cohesive vision, no clear goal to aim for, then each individual department is going to do things their own way, driven by their own priorities, budgets, Ministerial whims and you end up with what we largely have today: a cacophony of increasingly divergent approaches driven by tribalism that make collaboration, interoperability, common systems and data reuse impossible (or prohibitively expensive).

    If however, you can establish a common vision, then even a strongly decentralised system can converge on the goal. If we can establish a common vision for public data, then the implementation of data programs and policies across the APS should become naturally more consistent and common in practice, with people naturally motivated to collaborate, to share expertise, and to reuse systems, standards and approaches in pursuit of the same goal.

    My vision for public data is two-pronged and a bit of a paradigm shift: data by design and gov as an API! “Data by design” is about taking a data driven approach to the business of government and “gov as an API” is about changing the way we use, consume, publish and share data to properly enable a data driven public service and a broader network of innovation. The implementation of these ideas would create mashable government that could span departments, jurisdictions and international boundaries. In a heavily globalised world, no government is in isolation and it is only by making government data, content and services API enabled and reusable/interfacable, that we, collectively, can start to build the kind of analysis, products and services that meet the necessarily cross jurisdictional needs of all Australians, of all people.

    More specifically, my vision is a data driven approach to the entire business of government that supports:

    • evidence based and iterative policy making and implementation;

    • transparent, accountable and responsible Government;

    • an open competitive marketplace built on mashable government data, content and services; and

    • a more efficient, effective and responsive public service.

    What this requires is not so simple, but is utterly achievable if we could embed a more holistic whole of government approach in the work of individual departments, and then identify and fill the gaps through a central capacity that is responsible for driving a whole of government approach. Too often we see the data agenda oversimplified into what outcomes are desired (data visualisations, dashboards, analysis, etc) however, it is only in establishing multipurpose data infrastructure which can be reused for many different purposes that we will enable the kind of insights, innovation, efficiencies and effectiveness that all the latest reports on realising the value of data allude to. Without actual data, all the reports, policies, mission statements, programs and governance committees are essentially wasting time. But to get better government data, we need to build capacity and motivation in the public sector. We need to build a data driven culture in government. We also need to grow consumer confidence because a) demand helps drive supply, and b) if data users outside the public sector don’t trust that they can find, use and rely upon at least some government data, then we won’t ever see serious reuse of government data by the private sector, researchers, non-profits, citizens or the broader community.

    Below is a quick breakdown of each of these priorities, followed by specific recommendations for each:

    data infrastructure that supports multiple types of reuse. Ideally all data infrastructure developed by all government entities should be built in a modular, API enabled way to support data reuse beyond the original purpose to enable greater sharing, analysis, aggregation (where required) and publishing. It is often hard for agencies to know what common infrastructure already exists and it is easy for gaps to emerge, so another part of this is to map the data infrastructure requirements for all government data purposes, identify where solutions exist and any gaps. Where whole of government approaches are identified, common data infrastructure should be made available for whole of government use, to reduce the barrier to publishing and sharing data for departments. Too often, large expensive data projects are implemented in individual agencies as single purpose analytics solutions that don’t make the underlying data accessible for any other purpose. If such projects separated the data infrastructure from the analytics solutions, then the data infrastructure could be built to support myriad reuse including multiple analytics solutions, aggregation, sharing and publishing. If government data infrastructure was built like any other national infrastructure, it should enable a competitive marketplace of analysis, products and service delivery both domestically and globally. A useful analogy to consider is the example of roads. Roads are not typically built just from one address to another and are certainly not made to only support certain types of vehicles. It would be extremely inefficient if everyone built their own custom roads and then had to build custom vehicles for each type of road. It is more efficient to build common roads to a minimum technical standard that any type of vehicle can use to support both immediate transport needs, but also unknown transport needs into the future. Similarly we need to build multipurpose data infrastructure to support many types of uses.

    greater publisher capacity and motivation to share and publish data. Currently the range of publishing capacity across the APS is extremely broad, from agencies that do nothing to agencies that are prolific publishers. This is driven primarily by different cultures and responsibilities of agencies and if we are to improve the use of data, we need to improve the supply of data across the entire public sector. This means education and support for agencies to help them understand the value to their BAU work. The time and money saved by publishing data, opportunities to improve data quality, the innovation opportunities and the ability to improve decision making are all great motivations once understood, but generally the data agenda is only pitched in political terms that have little to no meaning to data publishers. Otherwise there is no natural motivation to publish or share data, and the strongest policy or regulation in the world does not create sustainable change or effective outcomes if you cannot establish a motivation to comply. Whilst ever publishing data is seen as merely a compliance issue, it will be unlikely for agencies to invest the time and skills to publish data well, that is, to publish the sort of data that consumers want to use.

    greater consumer confidence to improve the value realised from government data. Supply is nothing without demand and currently there is a relatively small (but growing) demand for government data, largely because people won’t use what they don’t trust. In the current landscape is difficult to find data and even if one can find it, it is often not machine readable or not freely available, is out of date and generally hard to use. There is not a high level of consumer confidence in what is provided by government so many people don’t even bother to look. If they do look, they find myriad data sources of ranging quality and inevitably waste many hours trying to get an outcome. There is a reasonable demand for data for research and the research community tends to jump through hoops – albeit reluctantly and at great cost – to gain access to government data. However, the private and civic sectors are yet to seriously engage apart form a few interesting outliers. We need to make finding and using useful data easy, and start to build consumer confidence or we will never even scratch the surface of the billions of dollars of untapped potential predicted by various studies. The data infrastructure section is obviously an important part of building consumer confidence as it should make it easier for consumers to find and have confidence in what they need, but it also requires improving the data culture across the APS, better outreach and communications, better education for public servants and citizens on how to engage in the agenda, and targeted programs to improve the publishing of data already in demand. What we don’t need is yet another “tell us what data you want” because people want to see progress.

    a data driven culture that embeds in all public servants an understanding of the role of data in the every day work of the public service, from program management, policy development, regulation and even basic reporting. It is important to take data from being seen as a specialist niche delegated only to highly specialised teams and put data front and centre as part of the responsibilities of all public servants – especially management – in their BAU activities. Developing this culture requires education, data driven requirements for new programs and policies, some basic skills development but mostly the proliferation of an awareness of what data is, why it is important, and how to engage appropriate data skills in the BAU work to ensure a data driven approach. Only with data can a truly evidence driven approach to policy be taken, and only with data can a meaningful iterative approach be taken over time.

    Finally, obviously the approach above requires an appropriately skilled team to drive policy, coordination and implementation of the agenda in collaboration with the broader APS. This team should reside in a central agenda to have whole of government imprimatur, and needs a mix of policy, commercial, engagement and technical data skills. The experience of data programs around the world shows that when you split policy and implementation, you inevitably get both a policy team lacking in the expertise to drive meaningful policy and an implementation team paralysed by policy indecision and an unclear mandate. This space is changing so rapidly that policy and implementation need to be agile and mutually reinforcing with a strong focus on getting things done.

    As we examine the interesting opportunities presented by new developments such as blockchain and big data, we need to seriously understand the shift in paradigm from scarcity to surplus, from centralised to distributed systems, and from pre-planned to iterative approaches, if we are to create an effective public service for the 21st century.

    There is already a lot of good work happening, so the recommendations in this submission are meant to improve and augment the landscape, not replicate. I will leave areas of specialisation to the specialists, and have tried to make recommendations that are supportive of a holistic approach to developing a data-driven public service in Australia.

    Current Landscape

    There has been progress in recent years towards a more data driven public sector however, these initiatives tend to be done by individual teams in isolation from the broader public service. Although we have seen some excellent exemplars of big data and open data, and some good work to clarify and communicate the intent of a data driven public service through policy and reviews, most projects have simply expanded upon the status quo thinking of government as a series of heavily fortified castles that take the extraordinary effort of letting in outsiders (including other departments) only under strictly controlled conditions and with great reluctance and cost. There is very little sharing at the implementation level (though an increasing amount of sharing of ideas and experience) and very rarely are new initiatives consulted across the APS for a whole of government perspective. Very rarely are actual data and infrastructure experts encouraged or supported to work directly together across agency or jurisdiction lines, which is a great pity. Although we have seen the idea of the value of data start to be realised and prioritised, we still see the implementation of data projects largely delegated to small, overworked and highly specialised internal teams that are largely not in the habit of collaborating externally and thus there is a lot of reinvention and diversity in what is done.

    If we are to realise the real benefits of data in government and the broader economy, we need to challenge some of the status quo thinking and approaches towards data. We need to consider government (and the data it collects) as a platform for others to build upon rather than the delivery mechanism for all things to all people. We also need to better map what is needed for a data-driven public service rather than falling victim to the attractive (and common, and cheap) notion of simply identifying existing programs of work and claiming them to be sufficient to meet the goals of the agenda.

    Globally this is still a fairly new space. Certain data specialisations have matured in government (eg. census/statistics, some spatial, some science data) but there is still a lack of a cohesive approach to data in any one agency. Even specialist data agencies tend to not look beyond the specialised data to have a holistic data driven approach to everything. In this way, it is critical to develop a holistic approach to data at all levels of the public service to embed the principles of data driven decision making in everything we do. Catalogues are not enough. Specialist data projects are not enough. Publishing data isn’t enough. Reporting number of datasets quickly becomes meaningless. We need to measure our success in this space by how well data is helping the public service to make better decisions, build better services, develop and iterate responsive and evidence based policy agendas, measure progress and understand the environment in which we operate.

    Ideally, government agencies need to adopt a dramatic shift in thinking to assume in the first instance that the best results will be discovered through collaboration, through sharing, through helping people help themselves. There also needs in the APS to be a shift away from thinking that a policy, framework, governance structure or other artificial constructs are sufficient outcomes. Such mechanisms can be useful, but they can also be a distraction from getting anything tangible done. Such mechanisms often add layers of complexity and cost to what they purport to achieve. Ultimately, it is only what is actually implemented that will drive an outcome and I strongly believe an outcomes driven approach must be applied to the public data agenda for it to achieve its potential.

    References

    In recent years there has been a lot of progress. Below is a quick list to ensure they are known and built upon for the future. It is also useful to recognise the good work of the government agencies to date.

    • Public Data Toolkit – the data.gov.au team have pulled together a large repository of information, guidance and reports over the past 3 years on our open data toolkit at http://toolkit.data.gov.au. There are also some useful contributions from the Department of Communications Spatial Policy Branch. The Toolkit has links to various guidance from different authoritative agencies across the APS as well as general information about data management and publishing which would be useful to this review.

    • The Productivity Commission is already aware of the Legislative and other Barriers Workshop I ran at PM&C before going on maternity leave, and I commend the outcomes of that session to the Review.

    • The Financial Sector Inquiry (the “Murray Inquiry”) has some excellent recommendations regarding the use of data-drive approaches to streamline the work and reporting of the public sector which, if implemented, would generate cost and time savings as well as the useful side effect of putting in place data driven practices and approaches which can be further leveraged for other purposes.

    • Gov 2.0 Report and the Ahead of the Game Report – these are hard to find copies of online now, but have some good recommendations and ideas about a more data centric and evidence based public sector and I commend them both to the Review. I’m happy to provide copies if required.

    • There are many notable APS agency efforts which I recommend the Productivity Commission engage with, if they haven’t already. Below are a few I have come across to date, and it is far from an exhaustive list:

      • PM&C (Public Data Management Report/Implementation & Public Data Policy Statement)

      • Finance (running and rebooting data.gov.au, budget publishing, data integration in GovCMS)

      • ABS (multi agency arrangement, ABS.Stat)

      • DHS (analytics skills program, data infrastructure and analysis work)

      • Immigration (analytics and data publishing)

      • Social Services (benefits of data publishing)

      • Treasury (Budget work)

      • ANDS (catalogue work and upskilling in research sector)

      • NDI (super computer functionality for science)

      • ATO (smarter data program, automated and publications data publishing, service analytics, analytics, dev lab, innovationspace)

      • Industry (Lighthouse data integration and analysis, energy ratings data and app)

      • CrimTRAC and AUSTRAC (data collection, consolidation, analysis, sharing)

    • Other jurisdictions in Australia have done excellent work as well and you can see a list (hopefully up to date) of portals and policies on the Public Data Toolkit. I recommend the Productivity Commission engage with the various data teams for their experiences and expertise in this matter. There are outstanding efforts in all the State and Territory Governments involved as well as many Local Councils with instructive success stories, excellent approaches to policy, implementation and agency engagement/skills and private sector engagement projects.

    Som current risks/issues

    There are a number of issues and risks that exist in pursuing the current approach to data in the APS. Below are some considerations to take into account with any new policies or agendas to be developed.

    • There is significant duplication of infrastructure and investment from building bespoke analytics solutions rather than reusable data infrastructure that could support multiple analytics solutions. Agencies build multiple bespoke analytics projects without making the underpinning data available for other purposes resulting in duplicated efforts and under-utilised data across government.

    • Too much focus on pretty user interfaces without enough significant investment or focus on data delivery.

    • Discovery versus reuse – too many example of catalogues linking to dead data. Without the data, discovery is less than useful.

    • Limitations of tech in agencies by ICT Department – often the ICT Department in an agency is reticent to expand the standard operating environment beyond the status quo, creating an issue of limitation of tools and new technologies.

    • Copyright and legislation – particularly old interpretations of each and other excuses to not share.

    • Blockers to agencies publishing data (skills, resources, time, legislation, tech, competing priorities e.g. assumed to be only specialists that can do data).

    • Often activities in the public sector are designed to maintain the status quo (budgets, responsibilities, staff count) and there is very little motivation to do things more efficiently or effectively. We need to establish these motivations for any chance to be sustainable.

    • Public perceptions about the roles and responsibilities of government change over time and it is important to stay engaged when governments want to try something new that the public might be uncertain about. There has been a lot of media attention about how data is used by government with concerns aired about privacy. Australians are concerned about what Government plans to do with their data. Broadly the Government needs to understand and engage with the public about what data it holds and how it is used. There needs to be trust built to both improve the benefits from data and to ensure citizen privacy and rights are protected. Where government wants to use data in new ways, it needs to prosecute the case with the public and ensure there are appropriate limitations to use in place to avoid misuse of the data. Generally, where Australians can directly view the benefit of their data being used and where appropriate limitations are in place, they will probably react positively. For example, tax submission are easier now that their data auto-fills from their employers and health providers when completing Online Tax. People appreciate the concept of having to only update their details once with government.

    Benefits

    I agree with the benefits identified by the Productivity Commission discussion paper however I would add the following:

    • Publishing government data, if published well, enables a competitive marketplace of service and product delivery, the ability to better leverage public and academic analysis for government use and more broadly, taps into the natural motivation of the entire community to innovate, solve problems and improve life.

    • Establishing authoritative data – often government is the authoritative source of information it naturally collects as part of the function of government. When this data is not then publicly available (through anonymised APIs if necessary) then people will use whatever data they can get access to, reducing the authority of the data collected by Government

    • A data-drive approach to collecting, sharing and publishing data enables true iterative approaches to policy and services. Without data, any changes to policy are difficult to justify and impossible to track the impact, so data provides a means to support change and to identify what is working quickly. Such feedback loops enable iterative improvements to policies and programs that can respond to the changing financial and social environment the operate in.

    • Publishing information in a data driven way can dramatically streamline reporting, government processes and decision making, freeing up resources that can be used for more high value purposes.

    Public Sector Data Principles

    The Public Data Statement provides a good basis of principles for this agenda. Below are some principles I think are useful to highlight with a brief explanation of each.

    Principles:

    • build for the future - legacy systems will always be harder to deal with so agencies need to draw a line in the sand and ensure new systems are designed with data principles, future reuse and this policy agenda in mind. Otherwise we will continue to build legacy systems into the future. Meanwhile, just because a legacy system doesn’t natively support APIs or improved access doesn’t mean you can’t affordably build middleware solutions to extract, transform, share and publish data in an automated way.

    • data first - wherever data is used to achieve an outcome, publish the data along with the outcome. This will improve public confidence in government outcomes and will also enable greater reuse of government data. For example, where graphs or analysis are published also publish the data. Where a mobile app is using data, publish the data API. Where a dashboard is set up, also provide access to the underpinning data.

    • use existing data, from the source where possible - this may involve engaging with or even paying for data from private sector or NGOs, negotiating with other jurisdictions or simply working with other government entities to gain access.

    • build reusable data infrastructure first - wherever data is part of a solution, the data should be accessible through APIs so that other outcomes and uses can be realised, even if the APIs are only used for internal access in the first instance.

    • data driven decision making to support iterative and responsive policy and implementations approaches – all decisions should be evidence based, all projects, policies and programs should have useful data indicators identified to measure and monitor the initiative and enable iterative changes backed by evidence.

    • consume your own data and APIs - agencies should consider how they can better use their own data assets and build access models for their own use that can be used publicly where possible. In consuming their own data and APIs, there is a better chance the data and APIs will be designed and maintained to support reliable reuse. This could raw or aggregate data APIs for analytics, dashboards, mobile apps, websites, publications, data visualisations or any other purpose.

    • developer empathy - if government agencies start to prioritise the needs of data users when publishing data, there is a far greater likelihood the data will be published in a way developers can use. For instance, no developer likes to use PDFs, so why would an agency publish data in a PDF (hint: there is no valid reason. PDF does not make your data more secure!).

    • standardise where beneficial but don’t allow the perfect to delay the good - often the focus on data jumps straight to standards and then multi year/decade standards initiatives are stood up which creates huge delays to accessing actual data. If data is machine readable, it can often be used and mapped to some degree which is useful, more useful than having access to nothing.

    • automate, automate, automate! – where human effort is required, tasks will always be inefficient and prone to error. Data collection, sharing and publishing should be automated where possible. For example, when data is regularly requested, agencies should automate the publishing of data and updates which both reduces the work for the agency and improves the quality for data users.

    • common platforms - where possible agencies should use existing common platforms to share and publish data. Where they need to develop new infrastructure, efforts should be made to identify where new platforms might be useful in a whole of government or multi agency context and built to be shared. This will support greater reuse of infrastructure as well as data.

    • a little less conversation a little more action – the public service needs to shift from talking about data to doing more in this space. Pilot projects, experimentation, collaboration between implementation teams and practitioners, and generally a greater focus on getting things done.

    Recommendations for the Public Data agenda

    Strategic

    1. Strong Recommendation: Develop a holistic vision and strategy for a data-driven APS. This could perhaps be part of a broader digital or ICT strategy, but there needs to be a clear goal that all government entities are aiming towards. Otherwise each agency will continue to do whatever they think makes sense just for them with no convergence in approach and no motivation to work together.

    2. Strong Recommendation: Develop and publish work program and roadmap with meaningful measures of progress and success regularly reported publicly on a public data agenda dashboard. NSW Government already have a public roadmap and dashboard to report progress on their open data agenda.

    Whole of government data infrastructure

    1. Strong Recommendation: Grow the data.gov.au technical team to at least 5 people to grow the whole of government catalogue and cloud based data hosting infrastructure, to grow functionality in response to data publisher and data user requirements, to provide free technical support and training to agencies, and to regularly engage with data users to grow public confidence in government data. The data.gov.au experience demonstrated that even just a small motivated technical team could greatly assist agencies to start on their data publishing journey to move beyond policy hypothesising into practical implementation. This is not something that can be efficiently or effectively outsourced in my experience.

    • I note that in the latest report from PM&C, Data61 have been engaged to improve the infrastructure (which looks quite interesting) however, there still needs to be an internal technical capability to work collaboratively with Data61, to support agencies, to ensure what is delivered by contractors meets the technical needs of government, to understand and continually improve the technical needs and landscape of the APS, to contribute meaningfully to programs and initiatives by other agencies, and to ensure the policies and programs of the Public Data Branch are informed by technical realities.

    1. Recommendation: Establish/extend a data infrastructure governance/oversight group with representatives from all major data infrastructure provider agencies including the central public data team to improve alignment of agendas and approaches for a more holistic whole of government approach to all major data infrastructure projects. The group would assess new data functional requirements identified over time, would identify how to best collectively meet the changing data needs of the public sector and would ensure that major data projects apply appropriate principles and policies to enable a data driven public service. This work would also need to be aligned with the work of the Data Champions Network.

    2. Recommendation: Map out, publish and keep up to date the data infrastructure landscape to assist agencies in finding and using common platforms.

    3. Recommendation: Identify on an ongoing basis publisher needs and provide whole of government solutions where required to support data sharing and publishing (eg – data.gov.au, ABS infrastructure, NationalMap, analytics tools, github and code for automation, whole of gov arrangements for common tools where they provide cost benefits).

    4. Recommendation: Create a requirement for New Policy Proposals that any major data initiatives (particularly analytics projects) also make the data available via accessible APIs to support other uses or publishing of the data.

    5. Recommendation: Establish (or build upon existing efforts) an experimental data playground or series of playgrounds for agencies to freely experiment with data, develop skills, trial new tools and approaches to data management, sharing, publishing, analysis and reuse. There are already some sandbox environments available and these could be mapped and updated over time for agencies to easily find and engage with such initiatives.

    Grow consumer confidence

    1. Strong Recommendation: Build automated data quality indicators into data.gov.au. Public quality indicators provide an easy way to identify quality data, thus reducing the time and effort required by data users to find something useful. This could also support a quality search interface, for instance data users could limit searches to “high quality government data” or choose granular options such as “select data updated this year”. See my earlier blog (from PM&C) draft of basic technical quality indicators which would be implemented quickly, giving data users a basic indication of how usable and useful data is in a consistent automated way. Additional quality indicators including domain specific quality indicators could be implemented in a second or subsequent iteration of the framework.

    2. Strong Recommendation: Establish regular public communications and engagement to improve relations with data users, improve perception of agenda and progress and identify areas of data provision to prioritise. Monthly blogging of progress, public access to the agenda roadmap and reporting on progress would all be useful. Silence is generally assumed to mean stagnation, so it is imperative for this agenda to have a strong public profile, which in part relies upon people increasingly using government data.

    3. Strong Recommendation: Establish a reasonable funding pool for agencies to apply for when establishing new data infrastructure, when trying to make existing legacy systems more data friendly, and for responding to public data requests in a timely fashion. Agencies should also be able to apply for specialist resource sharing from the central and other agencies for such projects. This will create the capacity to respond to public needs faster and develop skills across the APS.

    4. Strong Recommendation: The Australian Government undertake an intensive study to understand the concerns Australians hold relating to the use of their data and develop a new social pact with the public regarding the use and limitations of data.

    5. Recommendation: establish a 1-2 year project to support Finance in implementing the data driven recommendations from the Murray Inquiry with 2-3 dedicated technical resources working with relevant agency teams. This will result in regulatory streamlining, improved reporting and analysis across the APS, reduced cost and effort in the regular reporting requirements of government entities and greater reuse of the data generated by government reporting.

    6. Recommendation: Establish short program to focus on publishing and reporting progress on some useful high value datasets, applying the Public Data Policy Statement requirements for data publishing. The list of high value datasets could be drawn from the Data Barometer, the Murray Inquiry, existing requests from data.gov.au, and work from PM&C. The effort of determining the MOST high value data to publish has potentially got in the way of actual publishing, so it would be better to use existing analysis and prioritise some data sets but more importantly to establish data by default approach across govt for the kinds of serendipitous use of data for truly innovation outcomes.

    7. Recommendation: Citizen driven privacy – give citizens the option to share data for benefits and simplified services, and a way to access data about themselves.

    Grow publisher capacity and motivation

    1. Strong Recommendation: Document the benefits for agencies to share data and create better guidance for agencies. There has been a lot of work since the reboot of data.gov.au to educate agencies on the value of publishing data. The value of specialised data sharing and analytics projects is often evident driving those kinds of projects, but traditionally there hasn’t been a lot of natural motivations for agencies to publish data, which had the unfortunate result of low levels of data publishing. There is a lot of anecdotal evidence that agencies have saved time and money by publishing data publicly, which have in turn driven greater engagement and improvements in data publishing by agencies. If these examples were better documented (now that there are more resources) and if agencies were given more support in developing holistic public data strategies, we would likely see more data published by agencies.

    2. Strong Recommendation: Implement an Agency League Table to show agency performance on publishing or otherwise making government data publicly available. I believe such a league table needs to be carefully designed to include measures that will drive better behaviours in this space. I have previously mapped out a draft league table which ranks agency performance by quantity (number of data resources, weighted by type), quality (see previous note on quality metrics), efficiency (the time and/or money saved in publishing data) and value (a weighted measure of usage and reuse case studies) and would be happy to work with others in re-designing the best approach if useful.

    3. Recommendation: Establish regular internal hackfests with tools for agencies to experiment with new approaches to data collection, sharing, publishing and analysis – build on ATO lab, cloud tools, ATO research week, etc.

    4. Recommendation: Require data reporting component for New Policy Proposals and new tech projects wherein meaningful data and metrics are identified that will provide intelligence on the progress of the initiative throughout the entire process, not just at the end of the project.

    5. Recommendation: Add data principles and API driven and automated data provision to the digital service standard and APSC training.

    6. Recommendation: Require public APIs for all government data, appropriately aggregated where required, leveraging common infrastructure where possible.

    7. Recommendation: Establish a “policy difference engine” – a policy dashboard that tracks the top 10 or 20 policy objectives for the government of the day which includes meaningful metrics for each policy objective over time. This will enable the discovery of trends, the identification of whether policies are meeting their objectives, and supports an evidence based iterative approach to the policies because the difference made by any tweaks to the policy agenda will be evident.

    8. Recommendation: all publicly funded research data to be published publicly, and discoverable on central research data hub with free hosting available for research institutions. There has been a lot of work by ANDS and various research institutions to improve discovery of research data, but a large proportion is still only available behind a paywall or with an education logon. A central repository would reduce the barrier for research organisations to publicly publish their data.

    9. Recommendation: Require that major ICT and data initiatives consider cloud environments for the provision, hosting or analysis of data.

    10. Recommendation: Identify and then extend or provide commonly required spatial web services to support agencies in spatially enabling data. Currently individual agencies have to run their own spatial services but it would be much more efficient to have common spatial web services that all agencies could leverage.

    Build data drive culture across APS

    1. Strong Recommendation: Embed data approaches are considered in all major government investments. For example, if data sensors were built into major infrastructure projects it would create more intelligence about how the infrastructure is used over time. If all major investments included data reporting then perhaps it would be easier to keep projects on time and budget.

    2. Recommendation: Establish a whole of government data skills program, not just for specialist skills, but to embed in the entire APS and understanding of data-driven approaches for the public service. This would ideally include mandatory data training for management (in the same way OH&S and procurement are mandatory training). At C is a draft approach that could be taken.

    3. Recommendation: Requirement that all government contracts have create new data make that data available to the contracting gov entity under Creative Commons By Attribution only licence so that government funded data is able to published publicly according to government policy. I have seen cases of contracts leaving ownership with companies and then the data not being reusable by government.

    4. Recommendation: Real data driven indicators required for all new policies, signed off by data champions group, with data for KPIs publicly available on data.gov.au for public access and to feed policy dashboards. Gov entities must identify existing data to feed KPIs where possible from gov, private sector, community and only propose new data collection where new data is clearly required.

    • Note: it was good to see a new requirement to include evidence based on data analytics for new policy proposals and to consult with the Data Champions about how data can support new proposals in the recently launched implementation report on the Public Data Management Report. However, I believe it needs to go further and require data driven indicators be identified up front and reported against throughout as per the recommendation above. Evidence to support a proposal does not necessarily provide the ongoing evidence to ensure implementation of the proposal is successful or has the intended effect, especially in a rapidly changing environment.

    1. Recommendation: Establish relationships with private sector to identify aggregate data points already used in private sector that could be leveraged by public sector rather. This would be more efficient and accurate then new data collection.

    2. Recommendation: Establish or extend a cross agency senior management data champions group with specific responsibilities to oversee the data agenda, sign off on data indicators for NPPs as realistic, provide advice to Government and Finance on data infrastructure proposals across the APS.

    3. Recommendation: Investigate the possibilities for improving or building data sharing environments for better sharing data between agencies.

    4. Recommendation: Take a distributed and federated approach to linking unit record data. Secure API access to sensitive data would avoid creating a honey pot.

    5. Recommendation: Establish data awards as part of annual ICT Awards to include: most innovative analytics, most useful data infrastructure, best data publisher, best data driven policy.

    6. Recommendation: Extend the whole of government service analytics capability started at the DTO and provide access to all agencies to tap into a whole of government view of how users interact with government services and websites. This function and intelligence, if developed as per the original vision, would provide critical evidence of user needs as well as the impact of changes and useful automated service performance metrics.

    7. Recommendation: Support data driven publishing including an XML platform for annual reports and budgets, a requirement for data underpinning all graphs and datavis in gov publications to be published on data.gov.au.

    8. Recommendation: develop a whole of government approach to unit record aggregation of sensitive data to get consistency of approach and aggregation.

    Implementation recommendations

    1. Move the Public Data Branch to an implementation agency – Currently the Public Data Branch sits in the Department of Prime Minister and Cabinet. Considering this Department is a policy entity, the questions arises as to whether it is the right place in the longer term for an agenda which requires a strong implementation capability and focus. Public data infrastructure needs to be run like other whole of government infrastructure and would be better served as part of a broader online services delivery team. Possible options would include one of the shared services hubs, a data specialist agency with a whole of government mandate, or the office of the CTO (Finance) which runs a number of other whole of government services.

    Downloadable copy

    July 26, 2016

    Gather-ing some thoughts on societal challenges

    On the weekend I went to the GatherNZ event in Auckland, an interesting unconference. I knew there were going to be some pretty awesome people hanging out which gave a chance for me to catch up with and introduce the family to some friends, hear some interesting ideas, and road test some ideas I’ve been having about where we are all heading in the future. I ran a session I called “Choose your own adventure, please” and it was packed! Below is a bit of a write up of what was discussed as there was a lot of interest in how to keep the conversation going. I confess, I didn’t expect so much interest as to be asked where the conversation could be continued, but this is a good start I think. I was particularly chuffed when a few attendee said the session blew their minds :)

    I’m going to be blogging a fair bit over the coming months on this topic in any case as it relates to a book I’m in the process of researching and writing, but more on that next week!

    Choose your own adventure, please

    We are at a significant tipping point in history. The world and the very foundations our society were built on have changed, but we are still largely stuck in the past in how we think and plan for the future. If we don’t make some active decisions about how we live, think and prioritise, then we will find ourselves subconsciously reinforcing the status quo at every turn and not in a position to genuinely create a better future for all. I challenge everyone to consider how they think and to actively choose their own adventure, rather than just doing what was done before.

    How has the world changed? Well many point to the changes in technology and science, and the impact these have had on our quality of life. I think the more interesting changes are in how power and perspectives has changed, which created the environment for scientific and technological progress in the first instance, but also created the ability for many many more individuals to shape the world around them. We have seen traditional paradigms of scarcity, centralisation and closed systems be outflanked and outdated by modern shifts to surplus, distribution and open systems. When you were born a peasant and died one, what power did you have to affect your destiny? Nowadays individuals are more powerful than ever in our collective history, with the traditionally centralised powers of publishing, property, communications, monitoring and even enforcement now distributed internationally to anyone with access to a computer and the internet, which is over a third of the world’s population and growing. I blogged about this idea more here. Of course, these shifts are proving challenging for traditional institutions and structures to keep up, but individuals are simply routing around these dinosaurs, putting such organisations in the uncomfortable position of either adapting or rendering themselves irrelevant.

    Choices, choices, choices

    We discussed a number of specific premises or frameworks that underpinned the development of much of the world we know today, but are now out of touch with the changing world we live in. It was a fascinating discussion, so thank you to everyone who came and contributed and although I think we only scratched the surface, I think it gave a lot of people food for thought :)

    • Open vs closed – open systems (open knowledge, data, government, source, science) are outperforming closed ones in almost everything from science, technology, business models, security models, government and political systems, human knowledge and social models. Open systems enable rapid feedback loops that support greater iteration and improvements in response to the world, and open systems create a natural motivation for the players involved to perform well and gain the benefits of a broader knowledge, experience and feedback base. Open systems also support a competitive collaborative environment, where organisations can collaborate on the common, but compete on their specialisation. We discussed how security by obscurity was getting better understood as a largely false premise and yet, there are still so many projects, decisions, policies or other initiatives where closed is the assumed position, in contrast to the general trend towards openness across the board.
    • Central to distributed – many people and organisations still act like kings in castles, protecting their stuff from the masses and only collaborating with walls and moats in place to keep out the riff raff. The problem is that everything is becoming more distributed, and the smartest people will never all be in the one castle, so if you want the best outcomes, be it for a policy, product, scientific discovery, service or anything else, you need to consider what is out there and how you can be a part of a broader ecosystem. Building on the shoulders of giants and being a shoulder for others to build upon. Otherwise you will always be slower than those who know how to be a node in the network. Although deeply hierarchical systems still exist, individuals are learning how to route around the hierarchy (which is only an imaginary construct in any case). There will always be specialists and the need for central controls over certain things however, if whatever you do is done in isolation, it will only be effective in isolation. Everything and everyone is more and more interconnected so we need to behave more in this way to gain the benefits, and to ensure what we do is relevant to those we do it for. By tapping into the masses, we can also tap into much greater capacity and feedback loops to ensure how we iterate is responsive to the environment we operate in. Examples of the shift included media, democracy, citizen movements, ideology, security, citizen science, gov as an API, transnational movements and the likely impact of blockchain technologies on the financial sector.
    • Scarcity to surplus – the shift from scarcity to surplus is particularly interesting because so much of our laws, governance structures, business models, trade agreements and rules for living are based around antiquated ideas of scarcity and property. We now apply the idea of ownership to everything and I shared a story of a museum claiming ownership on human remains taken from Australia. How can you own that and then refuse to repatriate the remains to that community? Copyright was developed when the ability to copy something was costly and hard. Given digital property (including a lot of “IP”) is so easily replicated with low/zero cost, it has wrought havoc with how we think about IP and yet we have continued to duplicate this antiquated thinking in a time of increasing surplus. This is a problem because new technologies could genuinely create surplus in physical properties, especially with the developments in nano-technologies and 3D printing, but if we bind up these technologies to only replicate the status quo, we will never realise the potential to solve major problems of scarcity, like hunger or poverty.
    • Nationalism and tribalism – because of global communications, more people feel connected with their communities of interest, which can span geopolitical, language, disability and other traditional barriers to forming groups. This will also have an impact on loyalties because people will have an increasingly complex relationship with the world around them. Citizens can and will increasingly jurisdiction shop for a nation that supports their lifestyle and ideological choices, the same way that multinational corporates have jurisdiction shopped for low tax, low regulation environments for some time. On a more micro level, individuals engage in us vs them behaviours all the time, and it gets in the way of working together.
    • Human augmentation and (dis)ability – what it means to look and be human will start to change as more human augmentation starts to become mainstream. Not just cosmetic augmentations, but functional. The body hacking movement has been playing with human abilities and has discovered that the human brain can literally adapt to and start to interpret foreign neurological inputs, which opens up the path to nor just augmenting existing human abilities, but expanding and inventing new human abilities. If we consider the olympics have pretty much found the limit of natural human sporting achievement and have become arguably a bit boring, perhaps we could lift the limitations on the para-olympics and start to see rocket powered 100m sprints, or cyborg Judo competitions. As we start to explore what we can do with ourselves physically, neurologically and chemically, it will challenge a lot of views on what it means to be human. By why should we limit ourselves?
    • Outsourcing personal responsibility – with advances in technology, many have become lazy about how far their personal responsibility extends. We outsource small tasks, then larger ones, then strategy, then decision making, and we end up having no personal responsibility for major things in our world. Projects can fail, decisions become automated, ethics get buried in code, but individuals can keep their noses clean. We need to stop trying to avoid risk to the point where we don’t do anything and we need to ensure responsibility for human decisions are not automated beyond human responsibility.
    • Unconscious bias of privileged views, including digital colonialism – the need to be really aware of our assumptions and try to not simply reinvent the status quo or reinforce “structural white supremacy” as it was put by the contributor. Powerful words worth pondering! Explicit inclusion was put forward as something to prioritise.
    • Work – how we think about work! If we are moving into a more automated landscape, perhaps how we think about work will fundamentally change which would have enormous ramifications for the social and financial environment. Check out Tim Dunlop’s writing on this :)
    • Facts to sensationalism – the flow of information and communications are now so rapid that people, media and organisations are motivated to ever more sensationalism rather than considered opinions or facts. Definitely a shift worth considering!

    Other feedback from the room included:

    • The importance of considering ethics, values and privilege in making decisions.
    • The ability to route around hierarchy, but the inevitable push back of established powers on the new world.
    • The idea that we go in cycles of power from centralised to distributed and back again. I confess, this idea is new to me and I’ll be pondering on it more.

    Any feedback, thinking or ideas welcome in the comments below :) It was a fun session.

    July 23, 2016

    Gather Conference 2016 – Afternoon

    The Gathering

    Chloe Swarbrick

    • Whose responsibility is it to disrupt the system?
    • Maybe try and engage with the system we have for a start before writing it off.
    • You disrupt the system yourself or you hold the system accountable

    Nick McFarlane

    • He wrote a book
    • Rock Stars are dicks to work with

    So you want to Start a Business

    • Hosted by Reuben and Justin (the accountant)
    • Things you need to know in your first year of business
    • How serious is the business, what sort of structure
      • If you are serious, you have to do things properly
      • Have you got paying customers yet
      • Could just be an idea or a hobby
    • Sole Trader vs Incorporated company vs Trust vs Partnership
    • Incorperated
      • Directors and Shareholders needed to be decided on
      • Can take just half an hour
    • when to get a GST number?
      • If over $60k turnover a year
      • If you have lots of stuff you plan to claim back.
    • Have an accounting System from Day 1 – Xero Pretty good
    • Get an advisor or mentor that is not emotionally invested in your company
    • If partnership then split up responsibilities so you can hold each other accountable for specific items
    • If you are using Xero then your accountant should be using Xero directly not copying it into a different system.
    • Remuneration
      • Should have a shareholders agreement
      • PAYE possibility from drawings or put 30% aside
      • Even if only a small hobby company you will need to declare income to IRD especially non-trivial level.
    • What Level to start at Xero?
      • Probably from the start if the business is intended to be serious
      • A bit of pain to switch over later
    • Don’t forget about ACC
    • Remember you are due provisional tax once you get over the the $2500 for the previous year.
    • Home Office expense claim – claim percentage of home rent, power etc
    • Get in professionals to help

    Diversity in Tech

    • Diversity is important
      • Why is it important?
      • Does it mean the same for everyone
    • Have people with different “ways of thinking” then we will have a diverse views then wider and better solutions
    • example “Polish engineer could analysis a Polish specific character input error”
    • example “Controlling a robot in Samoan”, robots are not just in english
    • Stereotypes for some groups to specific jobs, eg “Indians in tech support”
    • Example: All hires went though University of Auckland so had done the same courses etc
    • How do you fix it when people innocently hire everyone from the same background? How do you break the pattern? No be the first different-hire represent everybody in that group?
    • I didn’t want to be a trail-blazer
    • Wow’ed out at “Women in tech” event, first time saw “majority of people are like me” in a bar.
    • “If he is a white male and I’m going to hire him on the team that is already full of white men he better be exception”
    • Worried about implication that “diversity” vs “Meritocracy” and that diverse candidates are not as good
    • Usual over-representation of white-males in the discussion even in topics like this.
    • Notion that somebody was only hired to represent diversity is very harmful especially for that person
    • If you are hiring for a tech position then 90% of your candidates will be white-males, try place your diversity in getting more diverse group applying for the jobs not tilt in the actual hiring.
    • Even in maker spaces where anyone is welcome, there are a lot fewer women. Blames mens mags having things unfinished, women’s mags everything is perfect so women don’t want to show off something that is unfinished.
    • Need to make the workforce diverse now to match the younger people coming into it
    • Need to cover “power income” people who are not exposed to tech
    • Even a small number are role models for the future for the young people today
    • Also need to address the problem of women dropping out of tech in the 30s and 40s. We can’t push girls into an “environment filled with acid”
    • Example taking out “cocky arrogant males” from classes into “advanced stream” and the remaining class saw women graduating and staying in at a much higher rate.

    Podcasting

    • Paul Spain from Podcast New Zealand organising
    • Easiest to listen to when doing manual stuff or in car or bus
    • Need to avoid overload of commercials, eg interview people from the company about the topic of interest rather than about their product
    • Big firms putting money into podcasting
    • In the US 21% of the market are listening every single month. In NZ perhaps more like 5% since not a lot of awareness or local content
    • Some radios shows are re-cutting and publishing them
    • Not a good directory of NZ podcasts
    • Advise people use proper equipment if possible if more than a once-off. Bad sound quality is very noticeable.
    • One person: 5 part series on immigration and immigrants in NZ
    • Making the charts is a big exposure
    • Apples “new and noteworthy” list
    • Domination by traditional personalities and existing broadcasters at present. But that only helps traction within New Zealand

     

     

    FacebookGoogle+Share