Planet Linux Australia
Celebrating Australians & Kiwis in the Linux and Free/Open-Source community...

May 27, 2016

Oryx and Crake




ISBN: 9780385721677
LibraryThing
I bought this book ages ago, on the recommendation of a friend (I don't remember who), but I only just got around to reading it. Its a hard book to read in places -- its not hopeful, or particularly fun, and its confronting in places -- especially the plot that revolves around child exploitation. There's very little to like about the future society that Atwood posits here, but perhaps that's the point.

Despite not being a happy fun story, the book made me think about things like genetic engineering in a way I didn't before and I think that's what Atwood was seeking to achieve. So I'd have to describe the book as a success.

Tags for this post: book margaret_atwood apocalypse genetic_engineering
Related posts: The Exterminator's Want Ad; Cyteen: The Vindication; East of the Sun, West of the Moon; The White Dragon; Runner; Cyteen: The Betrayal
Comment Recommend a book

Using OpenVPN on iOS and OSX

I have written instructions on how to connect to your own OpenVPN server using Network Manager as well as Android.

Here is how to do it on iOS and OSX assuming you have followed my instructions for the server setup.

Generate new keys

From the easy-rsa directory you created while generating the server keys, create a new keypair for your phone:

./build-key iphone   # "iphone" as Name, no password

and for your laptop:

./build-key osx      # "osx" as Name, no password

Using OpenVPN Connect on iOS

The app you need to install from the App Store is OpenVPN Connect.

Once it's installed, connect your phone to your computer and transfer the following files using iTunes:

  • ca.crt
  • iphone.crt
  • iphone.key
  • iphone.ovpn
  • ta.key

You should then be able to select it after launching the app. See the official FAQ if you run into any problems.

iphone.ovpn is a configuration file that you need to supply since the OpenVPN Connect app doesn't have a configuration interface. You can use this script to generate it or write it from scratch using this template.

On Linux, you can also create a configuration file using Network Manager 1.2, use the following command:

nmcli connection export hafnarfjordur > iphone.ovpn

though that didn't quite work in my experience.

Here is the config I successfully used to connect to my server:

client
remote hafnarfjordur.fmarier.org 1194
ca ca.crt
cert iphone.crt
key iphone.key
cipher AES-256-CBC
auth SHA384
comp-lzo yes
proto udp
tls-remote server
remote-cert-tls server
ns-cert-type server
tls-auth ta.key 1

Using Viscosity on Mac OSX

One of the possible OpenVPN clients you can use on OSX is Viscosity.

Here are the settings you'll need to change when setting up a new VPN connection:

  • General
    • Remote server: hafnarfjordur.fmarier.org
  • Authentication
    • Type: SSL/TLS client
    • CA: ca.crt
    • Cert: osx.crt
    • Key: osx.key
    • Tls-Auth: ta.key
    • direction: 1
  • Options
    • peer certificate: require server nsCertType
    • compression: turn LZO on
  • Networking
    • send all traffic on VPN
  • Advanced
    • add the following extra OpenVPN configuration commands:

      cipher AES-256-CBC
      auth SHA384
      

Adventures in Container Sharding – SQLite performance problem and the pivot point.

Hey world it’s been a while, turns out I’m not much of a blogger. But I know how useful for myself it is to do write-ups occasionally so I can actually find them later.. having said that. In my last post I mentioned I was an OpenStack Developer.. and it’s still true. I spend my time hacking and working on Openstack Swift the awesome OpenSource object storage cluster.

One thing I’ve been trying to tackle recently is container sharding in Swift, I will not go into full details as there is a Swift Spec that is relatively recent, and I’ve also gave a high level talk on it at LCA in Geelong.

The tl;dr being, Swift accounts and containers (or the metadata layer of Swift) are SQLite databases that get treated like objects themselves and replicated throughout the cluster. Which works amazingly well. Until you add millions and millions of objects to a container. And what I’m talking about here is container level object metadata, not the objects themselves. When this happens, SQLite being a file starts to have latency and locking issues, as one would expect. The solution to this is shard up these container databases throughout the cluster, which is what I’ve been working on.

At the last OpenStack summit in Austin, the awesome people at SwiftStack, whom I work quite closely with in the community gave me a container database they generated that has 700,000,000 objects in it (metadata again). This SQLite file is about 105G so not small. Plugging this into a small cluster I have to test my sharding implementation has been interesting to say the least.

When sharding a container down, we have a simple idea, split it in half. That is to say find someplace in the object table to pivot on. We can then keep pivoting giving us a list of ranges (which can be treated as a binary tree). The problem is finding the pivot point. In all my testing up til now I had what I thought was the perfect and simple way:

SELECT name
FROM object
WHERE deleted=0 ORDER BY name LIMIT 1 OFFSET (
SELECT object_count / 2
FROM policy_stat);

This did amazingly well in all my tests.. but I obviously never got big enough. This simple SQL statement would do plenty well if sharding in Swift was turned on from day dot, but the sharding plans for Swift is for once it’s solved in this POC to add to Swift as a beta which can be turned ‘on’ at a container by container basis when you want. After it graduates from beta but is still a switch. To finally once we are confident in it’s ability to have it on permanently. In the latter case container’s would never get big enough to worry about.. However, in the earlier stages a user would only turn it on when the container is _very_ slow.

Using the pivot SQL statement on the large container I was given ground to a halt, I’m sure it would have come back to be eventually, but I got tired of waiting after what seemed for ages.. there has to be a better way.

Turns out the OFFSET statement in SQLite, even when hitting an index still does a scan to find the offset.. This is slow when you get to a very large table size. Turns out under the hood, an Index is stored as a double-linked list, which I’m sure probably has optimisations but anyway I was struggling to think of a good way to find a good enough middle value that didn’t involve some table scanning. You can see from the SQL statement, we know have many objects we have in the container, but the problem is because swift is eventually consistent we need to temporally store objects that have been deleted. So randomly picking an index doesn’t help, and it wont necessarily be in name order.

So on really large containers OFFSET needs to be thrown out the window. Turns out the sharding implementation can deal with shrinking the number of shards, merging smaller ranges together, not just growing/splitting. This means we don’t actually need to be exact, we also don’t actually need to split on an existing object, just a name that would be somewhere in the middle and so long as it’s cutting down the large container then it’s good enough. So what can we do?

Turns out there is an optimisation in SQLite, because an index is a double-linked list and ordered by it’s index, it’s really quick if all we want to do is go to the first or last element. So that’s what I’ve done:

SELECT min(name) as name FROM object WHERE deleted = 0;
SELECT max(name) as name FROM object WHERE deleted = 0;

These two statements are blindingly fast due to the fact that we already have a compound index on name and delete (for cleaning up). Note however they have to be run as 2 separate commands, combine the two into one and you loose your optimisation and you’ll have to scan all elements. Having the min and max name is a good start, and even when dealing with already sharded containers, they are just smaller ranges so this still works. The question is now what?

In the perfect work we have an even distribution of objects between the min and max names, so we just need to find a middle name between the two to pivot on. Turns out even in a not evenly distributed container we will still be shrinking the container, even at worst case only be a few objects. But these will be cleaned up later (merged into a neighbour range by the implementation). And so long as the container gets smaller, eventually it’ll shrink small enough to be usable.

Next step is finding the middle value, to do this I just wrote some python:

from itertools import izip_longest
import sys

lower = unicode(sys.argv[1])
upper = unicode(sys.argv[2])

def middle_str(str1, str2):
    result = []
    for l, u in izip_longest(map(ord, str1), map(ord, str2), fillvalue=0):
        result.append((l + u) // 2)
    return u''.join(map(unichr, result))

if __name__ == "__main__":
    print(middle_str(lower, upper))

What does it do. Calling middle_str(min, max) will grab the unicode versions of the strings, turn them into there interger values, find the middle and turn them back into a word. After matching the prefix that is. So:

$ python middle_str.py 'aaaaaaaaa' 'zzzzzzzzz'
mmmmmmmmm

$ python middle_str.py 'aaaaaaaaa' 'aazzzzzzz'
aammmmmmm

$ python middle_str.py 'DFasjiojsaoi' 'ZZsdkmfi084f'
OPjkjkjiQLQg

I am now plugging this into my implementation and lets tackle this large container again.

May 26, 2016

More Social and Economic Systems, Music Stuff, and More

- modern economics just gibberish. St Louis Federal Reserve Database. Unemployment data distorted. One hour at soup kitchen (even volunteering) is 'employed'? International rule not Western centric. This rule (and others) developed in last 20 years. US still in sustained slump? Shadow stats. Neo-liberalism assumes that private sector will allocate more effectively? Using margin to control and

May 23, 2016

Third party testing with Turbo-Hipster

Why is this hipster voting on my code?!

Soon you are going to see a new robot barista leaving comments on Nova code reviews. He is obsessed with espresso, that band you haven’t heard of yet, and easing the life of OpenStack operators.

Doing a large OpenStack deployment has always been hard when it came to database migrations. Running a migration requires downtime, and when you have giant datasets that downtime could be hours. To help catch these issues Turbo-Hipster (http://josh.people.rcbops.com/2013/09/building-a-zuul-worker/) will now run your patchset’s migrations against copies of real databases. This will give you valuable feedback on the success of the patch, and how long it might take to migrate.

Depending on the results, Turbo-Hipster will add a review to your patchset that looks something like this:

Example turbo-hipster post

What should I do if Turbo-Hipster fails?

That depends on why it has failed. Here are some scenarios and steps you can take for different errors:

FAILURE – Did not find the end of a migration after a start

  • If you look at the log you should find that a migration began but never finished. Hopefully there’ll be a traceroute for you to follow through to get some hints about why it failed.

WARNING – Migration %s took too long

  • In this case your migration took a long time to run against one of our test datasets. You should reconsider what operations your migration is performing and see if there are any optimisations you can make, or if each step is really necessary. If there is no way to speed up your migration you can email us at rcbau@rcbops.com for an exception.

FAILURE – Final schema version does not match expectation

  • Somewhere along the line the migrations stopped and did not reach the expected version. The datasets start at previous releases and have to upgrade all the way through. If you see this inspect the log for traceroutes or other hints about the failure.

FAILURE – Could not setup seed database. FAILURE – Could not find seed database.

  • These two are internal errors. If you see either of these, contact us at rcbau@rcbops.com to let us know so we can fix and rerun the tests for you.

FAILURE – Could not import required module.

  • This error probably shouldn’t happen as Jenkins should catch it in the unit tests before Turbo-Hipster launches. If you see this, please contact us at rcbau@rcbops.com and let us know.

If you receive an error that you think is a false positive, leave a comment on the review with the sole contents of recheck migrations.

If you see any false positives or have any questions or problems please contact us on rcbau@rcbops.com

git.openstack.org adventures

Over the past few months I started to notice occasional issues when cloning repositories (particularly nova) from git.openstack.org.

It would fail with something like

git clone -vvv git://git.openstack.org/openstack/nova .
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

The problem would occur sporadically during our 3rd party CI runs causing them to fail. Initially these went somewhat ignored as rechecks on the jobs would succeed and the world would be shiny again. However, as they became more prominent the issue needed to be addressed.

When a patch merges in gerrit it is replicated out to 5 different cgit backends (git0[1-5].openstack.org). These are then balanced by two HAProxy frontends which are on a simple DNS round-robin.

                          +-------------------+
                          | git.openstack.org |
                          |    (DNS Lookup)   |
                          +--+-------------+--+
                             |             |
                    +--------+             +--------+
                    |           A records           |
+-------------------v----+                    +-----v------------------+
| git-fe01.openstack.org |                    | git-fe02.openstack.org |
|   (HAProxy frontend)   |                    |   (HAProxy frontend)   |
+-----------+------------+                    +------------+-----------+
            |                                              |
            +-----+                                    +---+
                  |                                    |
            +-----v------------------------------------v-----+
            |    +---------------------+  (source algorithm) |
            |    | git01.openstack.org |                     |
            |    |   +---------------------+                 |
            |    +---| git02.openstack.org |                 |
            |        |   +---------------------+             |
            |        +---| git03.openstack.org |             |
            |            |   +---------------------+         |
            |            +---| git04.openstack.org |         |
            |                |   +---------------------+     |
            |                +---| git05.openstack.org |     |
            |                    |  (HAProxy backend)  |     |
            |                    +---------------------+     |
            +------------------------------------------------+

Reproducing the problem was difficult. At first I was unable to reproduce locally, or even on an isolated turbo-hipster run. Since the problem appeared to be specific to our 3rd party tests (little evidence of it in 1st party runs) I started by adding extra debugging output to git.

We were originally cloning repositories via the git:// protocol. The debugging information was unfortunately limited and provided no useful diagnosis. Switching to https allowed for more CURL output (when using GIT_CURL_VERBVOSE=1 and GIT_TRACE=1) but this in itself just created noise. It actually took me a few days to remember that the servers are running arbitrary code anyway (a side effect of testing) and therefore cloning from the potentially insecure http protocol didn’t provide any further risk.

Over http we got a little more information, but still nothing that was conclusive at this point:

git clone -vvv http://git.openstack.org/openstack/nova .

error: RPC failed; result=18, HTTP code = 200
fatal: The remote end hung up unexpectedly
fatal: protocol error: bad pack header

After a bit it became more apparent that the problems would occur mostly during high (patch) traffic times. That is, when a lot of tests need to be queued. This lead me to think that either the network turbo-hipster was on was flaky when doing multiple git clones in parallel or the git servers were flaky. The lack of similar upstream failures lead me to initially think it was the former. In order to reproduce I decided to use Ansible to do multiple clones of repositories and see if that would uncover the problem. If needed I would have then extended this to orchestrating other parts of turbo-hipster in case the problem was systemic of something else.

Firstly I need to clone from a bunch of different servers at once to simulate the network failures more closely (rather than doing multiple clones on the one machine or from the one IP in containers for example). To simplify this I decided to learn some Ansible to launch a bunch of nodes on Rackspace (instead of doing it by hand).

Using the pyrax module I put together a crude playbook to launch a bunch of servers. There is likely much neater and better ways of doing this, but it suited my needs. The playbook takes care of placing appropriate sshkeys so I could continue to use them later.

    ---
    - name: Create VMs
      hosts: localhost
      vars:
        ssh_known_hosts_command: "ssh-keyscan -H -T 10"
        ssh_known_hosts_file: "/root/.ssh/known_hosts"
      tasks:
        - name: Provision a set of instances
          local_action:
            module: rax
            name: "josh-testing-ansible"
            flavor: "4"
            image: "Ubuntu 12.04 LTS (Precise Pangolin) (PVHVM)"
            region: "DFW"
            count: "15"
            group: "raxhosts"
            wait: yes
          register: raxcreate

        - name: Add the instances we created (by public IP) to the group 'raxhosts'
          local_action:
            module: add_host
            hostname: "{{ item.name }}"
            ansible_ssh_host: "{{ item.rax_accessipv4 }}"
            ansible_ssh_pass: "{{ item.rax_adminpass }}"
            groupname: raxhosts
          with_items: raxcreate.success
          when: raxcreate.action == 'create'

        - name: Sleep to give time for the instances to start ssh
          #there is almost certainly a better way of doing this
          pause: seconds=30

        - name: Scan the host key
          shell: "{{ ssh_known_hosts_command}} {{ item.rax_accessipv4 }} >> {{ ssh_known_hosts_file }}"
          with_items: raxcreate.success
          when: raxcreate.action == 'create'

    - name: Set up sshkeys
      hosts: raxhosts
      tasks:
       - name: Push root's pubkey
         authorized_key: user=root key="{{ lookup('file', '/root/.ssh/id_rsa.pub') }}"

From here I can use Ansible to work on those servers using the rax inventory. This allows me to address any nodes within my tenant and then log into them with the seeded sshkey.

The next step of course was to run tests. Firstly I just wanted to reproduce the issue, so in order to do that it would crudely set up an environment where it can simply clone nova multiple times.

    ---
    - name: Prepare servers for git testing
      hosts: josh-testing-ansible*
      serial: "100%"
      tasks:
        - name: Install git
          apt: name=git state=present update_cache=yes
        - name: remove nova if it is already cloned
          shell: 'rm -rf nova'

    - name: Clone nova and monitor tcpdump
      hosts: josh-testing-ansible*
      serial: "100%"
      tasks:
        - name: Clone nova
          shell: "git clone http://git.openstack.org/openstack/nova"

By default Ansible runs with 5 folked processes. Meaning that Ansible would work on 5 servers at a time. We want to exercise git heavily (in the same way turbo-hipster does) so we use the –forks param to run the clone on all the servers at once. The plan was to keep launching servers until the error reared its head from the load.

To my surprise this happened with very few nodes (less than 15, but I left that as my minimum testing). To confirm I also ran the tests after launching further nodes to see it fail at 50 and 100 concurrent clones. It turned out that the more I cloned the higher the failure rate percentage was.

Now that I had the problem reproducing, it was time to do some debugging. I modified the playbook to capture tcpdump information during the clone. Initially git was cloning over IPv6 so I turned that off on the nodes to force IPv4 (just in case it was a v6 issue, but the problem did present itself on both networks). I also locked git.openstack.org to one IP rather than randomly hitting both front ends.

    ---
    - name: Prepare servers for git testing
      hosts: josh-testing-ansible*
      serial: "100%"
      tasks:
        - name: Install git
          apt: name=git state=present update_cache=yes
        - name: remove nova if it is already cloned
          shell: 'rm -rf nova'

    - name: Clone nova and monitor tcpdump
      hosts: josh-testing-ansible*
      serial: "100%"
      vars:
        cap_file: tcpdump_{{ ansible_hostname }}_{{ ansible_date_time['epoch'] }}.cap
      tasks:
        - name: Disable ipv6 1/3
          sysctl: name="net.ipv6.conf.all.disable_ipv6" value=1 sysctl_set=yes
        - name: Disable ipv6 2/3
          sysctl: name="net.ipv6.conf.default.disable_ipv6" value=1 sysctl_set=yes
        - name: Disable ipv6 3/3
          sysctl: name="net.ipv6.conf.lo.disable_ipv6" value=1 sysctl_set=yes
        - name: Restart networking
          service: name=networking state=restarted
        - name: Lock git.o.o to one host
          lineinfile: dest=/etc/hosts line='23.253.252.15 git.openstack.org' state=present
        - name: start tcpdump
          command: "/usr/sbin/tcpdump -i eth0 -nnvvS -w /tmp/{{ cap_file }}"
          async: 6000000
          poll: 0 
        - name: Clone nova
          shell: "git clone http://git.openstack.org/openstack/nova"
          #shell: "git clone http://github.com/openstack/nova"
          ignore_errors: yes
        - name: kill tcpdump
          command: "/usr/bin/pkill tcpdump"
        - name: compress capture file
          command: "gzip {{ cap_file }} chdir=/tmp"
        - name: grab captured file
          fetch: src=/tmp/{{ cap_file }}.gz dest=/var/www/ flat=yes

This gave us a bunch of compressed capture files that I was then able to seek the help of my colleagues to debug (a particular thanks to Angus Lees). The results from an early run can be seen here: http://119.9.51.216/old/run1/

Gus determined that the problem was due to a RST packet coming from the source at roughly 60 seconds. This indicated it was likely we were hitting a timeout at the server or a firewall during the git-upload-pack of the clone.

The solution turned out to be rather straight forward. The git-upload-pack had simply grown too large and would timeout depending on the load on the servers. There was a timeout in apache as well as the HAProxy config for both frontend and backend responsiveness. The relative patches can be found at https://review.openstack.org/#/c/192490/ and https://review.openstack.org/#/c/192649/

While upping the timeout avoids the problem, certain projects are clearly pushing the infrastructure to its limits. As such a few changes were made by the infrastructure team (in particular James Blair) to improve git.openstack.org’s responsiveness.

Firstly git.openstack.org is now a higher performance (30GB) instance. This is a large step up from the previous (8GB) instances that were used as the frontend previously. Moving to one frontend additionally meant the HAProxy algorithm could be changed to leastconn to help balance connections better (https://review.openstack.org/#/c/193838/).

                          +--------------------+
                          | git.openstack.org  |
                          | (HAProxy frontend) |
                          +----------+---------+
                                     |
                                     |
            +------------------------v------------------------+
            |  +---------------------+  (leastconn algorithm) |
            |  | git01.openstack.org |                        |
            |  |   +---------------------+                    |
            |  +---| git02.openstack.org |                    |
            |      |   +---------------------+                |
            |      +---| git03.openstack.org |                |
            |          |   +---------------------+            |
            |          +---| git04.openstack.org |            |
            |              |   +---------------------+        |
            |              +---| git05.openstack.org |        |
            |                  |  (HAProxy backend)  |        |
            |                  +---------------------+        |
            +-------------------------------------------------+

All that was left was to see if things had improved. I rerun the test across 15, 30 and then 45 servers. These were all able to clone nova reliably where they had previously been failing. I then upped it to 100 servers where the cloning began to fail again.

Post-fix logs for those interested:
http://119.9.51.216/run15/
http://119.9.51.216/run30/
http://119.9.51.216/run45/
http://119.9.51.216/run100/
http://119.9.51.216/run15per100/

At this point, however, I’m basically performing a Distributed Denial of Service attack against git. As such, while the servers aren’t immune to a DDoS the problem appears to be fixed.

New Blog

Welcome to my new blog.

You can find my old one here: http://josh.opentechnologysolutions.com/blog/joshua-hesketh

I intend on back-porting those posts into this one in due course. For now though I’m going to start posting about my adventures in openstack!Wordpress

Introducing turbo-hipster for testing nova db migrations

Zuul is the continuous integration utility used by OpenStack to gate patchsets against tests. It takes care of communicating with gerrit (the code review system) and the test workers – usually Jenkins. You can read more about how the systems tie together on the OpenStack Project Infrastructure page.

The nice thing is that zuul doesn’t require you to use Jenkins. Anybody can provide a worker to zuul using the gearman protocol (which is a simple job server). Enter turbo-hipster*.

“Turbo-hipster is a CI worker with pluggable tasks initially designed to test OpenStack’s database migrations against copies of real databases.”

This will hopefully catch scenarios where changes to the database schema may not work due to outliers in real datasets and also help find where a migration may take an unreasonable amount of time against a large database.

In zuuls layout configuration we are able to specify which jobs should be ran against which projects in which pipelines. For example, for nova we want to run tests when a patchset is created, but we don’t need to run tests against it (necessarily) once it is merged etc. So in zuul we specify a new gate (aka job) to test nova against real databases.

turbo-hipster then listens for jobs created on that gate using the gearman protocol. Once it receives a patchset from zuul it creates a virtual environment and tests the upgrades. It then compiles and sends back the results.

At the moment turbo-hipster is still under heavy development but I hope to have it reporting results back to gerrit patchsets soon as part of zuuls report summary. For the moment I have a separate zuul instance running to test new nova patches and email the results back to me. Here is an example result report:

<code>Build succeeded.

- http://thw01.rcbops.com/logviewer/?q=/results/47/47162/9/check/gate-real-db-upgrade_nova_mysql/c4bc35c/index.html : SUCCESS in 13m 31s
</code>

Turbo Hipster Meme

*The name was randomly generated and does not necessarily contain meaning.

LinuxCon Europe

linuxcon-europe-2011After travelling very close to literally the other side of the world[0] I’m in Edinburgh for LinuxCon EU recovering from jetlag and getting ready to attend. I’m very much looking forward to my first LinuxCon, meeting new people and learning lots :-).

If you’re around and would like to catch up drop me a comment here. Otherwise I’ll see you at the conference!

[0] http://goo.gl/maps/JeJO2

OpenStack infrastructure swift logs and performance

Turns out I’m not very good at blogging very often. However I thought I would put what I’ve been working on for the last few days here out of interest.

For a while the OpenStack Infrastructure team have wanted to move away from storing logs on disk to something more cloudy – namely, swift. I’ve been working on this on and off for a while and we’re nearly there.

For the last few weeks the openstack-infra/project-config repository has been uploading its CI test logs to swift as well as storing them on disk. This has given us the opportunity to compare the last few weeks of data and see what kind of effects we can expect as we move assets into an object storage.

  • I should add a disclaimer/warning, before you read, that my methods here will likely make statisticians cringe horribly. For the moment though I’m just getting an indication for how things compare.

The set up

Fetching files from an object storage is nothing particularly new or special (CDN’s have been doing it for ages). However, for our usage we want to serve logs with os-loganalyze giving the opportunity to hyperlink to timestamp anchors or filter by log severity.

First though we need to get the logs into swift somehow. This is done by having the job upload its own logs. Rather than using (or writing) a Jenkins publisher we use a bash script to grab the jobs own console log (pulled from the Jenkins web ui) and then upload it to swift using credentials supplied to the job as environment variables (see my zuul-swift contributions).

This does, however, mean part of the logs are missing. For example the fetching and upload processes write to Jenkins’ console log but because it has already been fetched these entries are missing. Therefore this wants to be the very last thing you do in a job. I did see somebody do something similar where they keep the download process running in a fork so that they can fetch the full log but we’ll look at that another time.

When a request comes into logs.openstack.org, a request is handled like so:

  1. apache vhost matches the server
  2. if the request ends in .txt.gz, console.html or console.html.gz rewrite the url to prepend /htmlify/
  3. if the requested filename is a file or folder on disk, serve it up with apache as per normal
  4. otherwise rewrite the requested file to prepend /htmlify/ anyway

os-loganalyze is set up as an WSGIScriptAlias at /htmlify/. This means all files that aren’t on disk are sent to os-loganalyze (or if the file is on disk but matches a file we want to mark up it is also sent to os-loganalyze). os-loganalyze then does the following:

  1. Checks the requested file path is legitimate (or throws a 400 error)
  2. Checks if the file is on disk
  3. Checks if the file is stored in swift
  4. If the file is found markup (such as anchors) are optionally added and the request is served
    1. When serving from swift the file is fetched via the swiftclient by os-loganlayze in chunks and streamed to the user on the fly. Obviously fetching from swift will have larger network consequences.
  5. If no file is found, 404 is returned

If the file exists both on disk and in swift then step #2 can be skipped by passing ?source=swift as a parameter (thus only attempting to serve from swift). In our case the files exist both on disk and in swift since we want to compare the performance so this feature is necessary.

So now that we have the logs uploaded into swift and stored on disk we can get into some more interesting comparisons.

Testing performance process

My first attempt at this was simply to fetch the files from disk and then from swift and compare the results. A crude little python script did this for me: http://paste.openstack.org/show/122630/

The script fetches a copy of the log from disk and then from swift (both through os-loganalyze and therefore marked-up) and times the results. It does this in two scenarios:

  1. Repeatably fetching the same file over again (to get a good average)
  2. Fetching a list of recent logs from gerrit (using the gerrit api) and timing those

I then ran this in two environments.

  1. On my local network the other side of the world to the logserver
  2. On 5 parallel servers in the same DC as the logserver

Running on my home computer likely introduced a lot of errors due to my limited bandwidth, noisy network and large network latency. To help eliminate these errors I also tested it on 5 performance servers in the Rackspace cloud next to the log server itself. In this case I used ansible to orchestrate the test nodes thus running the benchmarks in parallel. I did this since in real world use there will often be many parallel requests at once affecting performance.

The following metrics are measured for both disk and swift:

  1. request sent – time taken to send the http request from my test computer
  2. response – time taken for a response from the server to arrive at the test computer
  3. transfer – time taken to transfer the file
  4. size – filesize of the requested file

The total time can be found by adding the first 3 metrics together.

 

Results

Home computer, sequential requests of one file

 

The complementary colours are the same metric and the darker line represents swift’s performance (over the lighter disk performance line). The vertical lines over the plots are the error bars while the fetched filesize is the column graph down the bottom. Note that the transfer and file size metrics use the right axis for scale while the rest use the left.

As you would expect the requests for both disk and swift files are more or less comparable. We see a more noticable difference on the responses though with swift being slower. This is because disk is checked first, and if the file isn’t found on disk then a connection is sent to swift to check there. Clearly this is going to be slower.

The transfer times are erratic and varied. We can’t draw much from these, so lets keep analyzing deeper.

The total time from request to transfer can be seen by adding the times together. I didn’t do this as when requesting files of different sizes (in the next scenario) there is nothing worth comparing (as the file sizes are different). Arguably we could compare them anyway as the log sizes for identical jobs are similar but I didn’t think it was interesting.

The file sizes are there for interest sake but as expected they never change in this case.

You might notice that the end of the graph is much noisier. That is because I’ve applied some rudimentary data filtering.

request sent (ms) – disk request sent (ms) – swift response (ms) – disk response (ms) – swift transfer (ms) – disk transfer (ms) – swift size (KB) – disk size (KB) – swift
Standard Deviation 54.89516183 43.71917948 56.74750291 194.7547117 849.8545127 838.9172066 7.121600095 7.311125275
Mean 283.9594368 282.5074598 373.7328851 531.8043908 5091.536092 5122.686897 1219.804598 1220.735632

 

I know it’s argued as poor practice to remove outliers using twice the standard deviation, but I did it anyway to see how it would look. I only did one pass at this even though I calculated new standard deviations.

 

request sent (ms) – disk request sent (ms) – swift response (ms) – disk response (ms) – swift transfer (ms) – disk transfer (ms) – swift size (KB) – disk size (KB) – swift
Standard Deviation 13.88664039 14.84054789 44.0860569 115.5299781 541.3912899 515.4364601 7.038111654 6.98399691
Mean 274.9291111 276.2813889 364.6289583 503.9393472 5008.439028 5013.627083 1220.013889 1220.888889

 

I then moved the outliers to the end of the results list instead of removing them completely and used the newly calculated standard deviation (ie without the outliers) as the error margin.

Then to get a better indication of what are average times I plotted the histograms of each of these metrics.

Here we can see a similar request time.
 

Here it is quite clear that swift is slower at actually responding.
 

Interestingly both disk and swift sources have a similar total transfer time. This is perhaps an indication of my network limitation in downloading the files.

 

Home computer, sequential requests of recent logs

Next from my home computer I fetched a bunch of files in sequence from recent job runs.

 

 

Again I calculated the standard deviation and average to move the outliers to the end and get smaller error margins.

request sent (ms) – disk request sent (ms) – swift response (ms) – disk response (ms) – swift transfer (ms) – disk transfer (ms) – swift size (KB) – disk size (KB) – swift
Standard Deviation 54.89516183 43.71917948 194.7547117 56.74750291 849.8545127 838.9172066 7.121600095 7.311125275
Mean 283.9594368 282.5074598 531.8043908 373.7328851 5091.536092 5122.686897 1219.804598 1220.735632
Second pass without outliers
Standard Deviation 13.88664039 14.84054789 115.5299781 44.0860569 541.3912899 515.4364601 7.038111654 6.98399691
Mean 274.9291111 276.2813889 503.9393472 364.6289583 5008.439028 5013.627083 1220.013889 1220.888889

 

What we are probably seeing here with the large number of slower requests is network congestion in my house. Since the script requests disk, swift, disk, swift, disk.. and so on this evens it out causing a latency in both sources as seen.
 

Swift is very much slower here.

 

Although comparable in transfer times. Again this is likely due to my network limitation.
 

The size histograms don’t really add much here.
 

Rackspace Cloud, parallel requests of same log

Now to reduce latency and other network effects I tested fetching the same log over again in 5 parallel streams. Granted, it may have been interesting to see a machine close to the log server do a bunch of sequential requests for the one file (with little other noise) but I didn’t do it at the time unfortunately. Also we need to keep in mind that others may be access the log server and therefore any request in both my testing and normal use is going to have competing load.
 

I collected a much larger amount of data here making it harder to visualise through all the noise and error margins etc. (Sadly I couldn’t find a way of linking to a larger google spreadsheet graph). The histograms below give a much better picture of what is going on. However out of interest I created a rolling average graph. This graph won’t mean much in reality but hopefully will show which is faster on average (disk or swift).
 

You can see now that we’re closer to the server that swift is noticeably slower. This is confirmed by the averages:

 

  request sent (ms) – disk request sent (ms) – swift response (ms) – disk response (ms) – swift transfer (ms) – disk transfer (ms) – swift size (KB) – disk size (KB) – swift
Standard Deviation 32.42528982 9.749368282 245.3197219 781.8807534 1082.253253 2737.059103 0 0
Mean 4.87337544 4.05191168 39.51898688 245.0792916 1553.098063 4167.07851 1226 1232
Second pass without outliers
Standard Deviation 1.375875503 0.8390193564 28.38377158 191.4744331 878.6703183 2132.654898 0 0
Mean 3.487575109 3.418433003 7.550682037 96.65978872 1389.405618 3660.501404 1226 1232

 

Even once outliers are removed we’re still seeing a large latency from swift’s response.

The standard deviation in the requests now have gotten very small. We’ve clearly made a difference moving closer to the logserver.

 

Very nice and close.
 

Here we can see that for roughly half the requests the response time was the same for swift as for the disk. It’s the other half of the requests bringing things down.
 

The transfer for swift is consistently slower.

 

Rackspace Cloud, parallel requests of recent logs

Finally I ran just over a thousand requests in 5 parallel streams from computers near the logserver for recent logs.

 

Again the graph is too crowded to see what is happening so I took a rolling average.

 

 

request sent (ms) – disk request sent (ms) – swift response (ms) – disk response (ms) – swift transfer (ms) – disk transfer (ms) – swift size (KB) – disk size (KB) – swift
Standard Deviation 0.7227904332 0.8900549012 434.8600827 909.095546 1913.9587 2132.992773 6.341238774 7.659678352
Mean 3.515711867 3.56191383 145.5941102 189.947818 2427.776165 2875.289455 1219.940039 1221.384913
Second pass without outliers
Standard Deviation 0.4798803247 0.4966553679 109.6540634 171.1102999 1348.939342 1440.2851 6.137625464 7.565931993
Mean 3.379718381 3.405770445 70.31323922 86.16522485 2016.900047 2426.312363 1220.318912 1221.881335

 

The averages here are much more reasonable than when we continually tried to request the same file. Perhaps we’re hitting limitations with swifts serving abilities.

 

I’m not sure why we have sinc function here. A network expert may be able to tell you more. As far as I know this isn’t important to our analysis other than the fact that both disk and swift match.
 

Here we can now see swift keeping a lot closer to disk results than when we only requested the one file in parallel. Swift is still, unsurprisingly, slower overall.
 

Swift still loses out on transfers but again does a much better job of keeping up.
 

Error sources

I haven’t accounted for any of the following swift intricacies (in terms of caches etc) for:

  • Fetching random objects
  • Fetching the same object over and over
  • Fetching in parallel multiple different objects
  • Fetching the same object in parallel

I also haven’t done anything to account for things like file system caching, network profiling, noisy neighbours etc etc.

os-loganalyze tries to keep authenticated with swift, however

  • This can timeout (causes delays while reconnecting, possibly accounting for some spikes?)
  • This isn’t thread safe (are we hitting those edge cases?)

We could possibly explore getting longer authentication tokens or having os-loganalyze pull from an unauthenticated CDN to add the markup and then serve. I haven’t explored those here though.

os-loganalyze also handles all of the requests not just from my testing but also from anybody looking at OpenStack CI logs. In addition to this it also needs to deflate the gzip stream if required. As such there is potentially a large unknown (to me) load on the log server.

In other words, there are plenty of sources of errors. However I just wanted to get a feel for the general responsiveness compared to fetching from disk. Both sources had noise in their results so it should be expected in the real world when downloading logs that it’ll never be consistent.

Conclusions

As you would expect the request times are pretty much the same for both disk and swift (as mentioned earlier) especially when sitting next to the log server.

The response times vary but looking at the averages and the histograms these are rarely large. Even in the case where requesting the same file over and over in parallel caused responses to go slow these were only in the magnitude of 100ms.

The response time is the important one as it indicates how soon a download will start for the user. The total time to stream the contents of the whole log is seemingly less important if the user is able to start reading the file.

One thing that wasn’t tested was streaming of different file sizes. All of the files were roughly the same size (being logs of the same job). For example, what if the asset was a few gigabytes in size, would swift have any significant differences there? In general swift was slower to stream the file but only by a few hundred milliseconds for a megabyte. It’s hard to say (without further testing) if this would be noticeable on large files where there are many other factors contributing to the variance.

Whether or not these latencies are an issue is relative to how the user is using/consuming the logs. For example, if they are just looking at the logs in their web browser on occasion they probably aren’t going to notice a large difference. However if the logs are being fetched and scraped by a bot then it may see a decrease in performance.

Overall I’ll leave deciding on whether or not these latencies are acceptable as an exercise for the reader.

Potato Point

I went to Potato Point with the Scouts for a weekend wide game. Very nice location, apart from the ticks!

                                       

See more thumbnails

Tags for this post: blog pictures 20160523 photo coast scouts bushwalk
Related posts: Exploring the Jagungal; Scout activity: orienteering at Mount Stranger

Comment

May 22, 2016

PyCon Australia 2016: Registration Opens!

We are delighted to announce that online registration is now open for PyCon Australia 2016. The seventh PyCon Australia is being held in Melbourne, Victoria from August 12th – 16th at the Melbourne Convention and Exhibition Centre, will draw hundreds of Python developers, enthusiasts and students from Australasia and afar.

Starting today, early bird offers are up for grabs. To take advantage of these discounted ticket rates, be among the first 90 to register. Early bird registration starts from $60 for full-time students, $190 for enthusiasts and $495 for professionals. Offers this good won’t last long, so register right away.

We strongly encourage attendees to organise their accommodation as early as possible, as demand for cheaper rooms is very strong during the AFL season.

PyCon Australia has endeavoured to keep tickets as affordable as possible. Financial assistance is also available: for information about eligibility, head to our financial assistance page and apply. We are able to make such offers thanks to our Sponsors and Contributors.

To begin the registration process, and find out more about each level of ticket, visit our registration information page.

Important Dates to Help You Plan

  • 22 May: Registration opens - ‘Early bird’ prices for the first 90 tickets
  • 17 June: Last day to apply for financial assistance
  • 26 June: Last day to purchase conference dinner tickets
  • 9 July: Last day to order conference t-shirts
  • 12 August: PyCon Australia 2016 begins!

About PyCon Australia

PyCon Australia is the national conference for the Python programming community. The seventh PyCon Australia will be held on August 12-16 2016 in Melbourne, bringing together professional, student and enthusiast developers with a love for programming in Python. PyCon Australia informs the country’s developers with presentations by experts and core developers of Python, as well as the libraries and frameworks that they rely on.

To find out more about PyCon Australia 2016, visit our website at pycon-au.org, follow us at @pyconau or e-mail us at contact@pycon-au.org.

PyCon Australia is presented by Linux Australia (www.linux.org.au) and acknowledges the support of our Platinum Sponsors, DevDemand.co and IRESS; and our Gold sponsors, Google Australia and Optiver. For full details of our sponsors, see our website.

Django and PostgreSQL composite types

PostgreSQL has this nifty feature called composite types that you can use to create your own types from the built-in PostgreSQL types. It’s a bit like hstore, only structured, which makes it great for structured data that you might reuse multiple times in a model, like addresses.

Unfortunately to date, they were pretty much a pain to use in Django. There were some older implementations for versions of Django before 1.7, but they tended to do things like create surprise new objects in the namespace, not be migrateable, and require connection to the DB at any time (i.e. during your build).

Anyway, after reading a bunch of their implementations and then the Django source code I wrote django-postgres-composite-types.

Install with:

pip install django-postgres-composite-types

Then you can define a composite type declaratively:

from django.db import models
from postgres_composite_type import CompositeType


class Address(CompositeType):
    """An address."""

    address_1 = models.CharField(max_length=255)
    address_2 = models.CharField(max_length=255)

    suburb = models.CharField(max_length=50)
    state = models.CharField(max_length=50)

    postcode = models.CharField(max_length=10)
    country = models.CharField(max_length=50)

    class Meta:
        db_type = 'x_address'  # Required

And use it in a model:

class Person(models.Model):
    """A person."""

    address = Address.Field()

The field should provide all of the things you need, including formfield etc and you can even inherit this field to extend it in your own way:

class AddressField(Address.Field):
    def __init__(self, in_australia=True, **kwargs):
        self.in_australia = in_australia

        super().__init__(**kwargs)

Finally to set up the DB there is a migration operation that will create the type that you can add:

import address
from django.db import migrations


class Migration(migrations.Migration):

    operations = [
        # Registers the type
        address.Address.Operation(),
        migrations.AddField(
            model_name='person',
            name='address',
            field=address.Address.Field(blank=True, null=True),
        ),
    ]

It’s not smart enough to add it itself (can you do that?). Nor would it be smart enough to write the operations to alter a type. That would be a pretty cool trick. But it’s useful functionality all the same, especially when the alternative is creating lots of 1:1 models that are hard to work with and hard to garbage collect.

It’s still pretty early days, so the APIs are subject to change. PRs accepted of course.

May 21, 2016

Restoring gitstats

gitstats tool has stopped working on our project after upgrade to Ubuntu 16.04. Finally I have got time to have a look. There were two issues with it:

  1. we do not need to use process wait as process communicate waits until process termination and the last process in the pipeline do not finish until all processes before it in the pipeline terminate, plus process wait may deadlock on pipes with huge output, see notice at https://docs.python.org/2/library/subprocess.html
  2. On Ubuntu 16.04 grep has started to give "Binary file (standard input) matches" notice into the pipe which breaks parsing.

I have made a pull request which fixes this issue: https://github.com/hoxu/gitstats/pull/65
Also you can clone fixed version from my account: https://github.com/Maxime2/gitstats

May 20, 2016

Heatsink for RPi3

I ordered a passive heatsink for system-on-chip of the Raspberry Pi 3 model B. Since it fits well I'll share the details:

Order

  • Fischer Elektronik ICK S 14 X 14 X 10 heatsink (Element 14 catalogue 1850054, AUD3.70).

  • Fischer Elektronik WLFT 404 23X23 thermally conductive foil, adhesive (Element 14 catalogue 1211707, AUD2.42 ).

Install

To install you need these parts: two lint-free isopropyl alcohol swabs; and these tools: sharp craft knife, a anti-static wrist strap.

Prepare the heatsink: Swab the base of the heatsink. Wait for it to dry. Remove the firm clear plastic from the thermal foil, taking care not to get fingerprints in the centre of the exposed sticky side. Put the foil on the bench, sticky side up. Plonk the heatsink base onto the sticky side, rolling slightly to avoid air bubbles and then pressing hard. Trim around the edges of the heatsink with the craft knife.

Prepare the Raspberry Pi 3 system-on-chip: Unlug everything from the RPi3, turn off the power, wait a bit, plug the USB power lead back in but don't reapply power (this gives us a ground reference). If the RPi3 is in a case, just remove the lid. Attach wrist strap and clamp to ethernet port surround or some other convenient ground. Swab the largest of the chips on the board, ensuring no lint remains.

Attach heat sink: Remove the plastic protection from the thermal foil, exposing the other sticky side. Do not touch the sticky side. With care place the heatsink squarely and snuggly on the chip. Press down firmly with finger of grounded hand for a few seconds. Don't press too hard: we're just ensuring the glue binds.

Is it worth it?

This little passive heatsink won't stop the RPi3 from throttling under sustained full load, despite this being one of the more effective passive heatsinks on the market. You'll need a fan blowing air across the heatsink to prevent that happening, and you might well need a heatsink on the RAM too.

But the days of CPUs being able to run at full rate continuously are numbered. Throttling the CPU performance under load is common in phones and tablets, and is not rare in laptops.

What the heatsink allows is for a delay to the moment of throttling. So a peaky load can have more chance of not causing throttling. Since we're only talking AUD7.12 in parts a passive heatsink is worth it if you are going to use the RPi3 for serious purposes.

Of course the heatsink is also a more effective radiator. When running cpuburn-a53 the CPU core temperature stabilises at 80C with a CPU clock of 700MHz (out of 1200MHz). It's plain that 80C is the target core temperature for this version of the RPi3's firmware. That's some 400MHz higher than without the heatsink. But if your task needs sustained raw CPU performance then you are much better off with even the cheapest of desktops, let alone a server.

[mtb/events] UTA100 - The big dance through the blue mountains again

Back at Ultra Trail Australia running through the Blue Mountains wilderness

I am still fascinated by seeing how I can improve in this event, after running in pairs twice and now solo twice I signed up to come back this year still seeing how much time I can lop off my lap of the course. Though I continually claim I am not a runner with my mountain biking and adventure racing background I have been getting out on foot a lot since I got into doing this event. With an arbitrary number I apply to the time around this course before I may admit I am a runner of 12 hours I as coming back to see how close to this goal I would get.

My first year solo in 2014 I was positive I would finish just now sure how fast, thinking on the day I may take around 15 hours I managed 13:44 which at the time had me happy and a little surprised. In 2015 I had a few things interrupt my lead up and not everything felt great so though I hoped to go under 13 hours I was not sure, managing 13:15 was not what I wanted but I got around the loop again anyway.

In 2016 I continued to not have a training program and simply work toward goals by judging effort in my head and race schedule leading up to the event. However most running science seems to suggest the more you can run without getting injured the better. So on January 1st 2016 I kicked off a running streak to see how long it would last. I managed to run every day in 2016 until Wednesday before UTA100, so 132 days in a row with a minimum distance of 5km. This included the days before and after efforts such as the razorback ultra in Victoria and the Six Foot Track marathon in the Blue Mountains.

I never really managed to get much speed work into my prep again this year however had definitely upped my volume doing between 70 and 125km every week of the year with most of it on trails with some good altitude gain at times. I also remained un injured and able to run every day which was great, even with the odd fall or problem I could work around and keep moving through I was feeling good before the event. Due to my tendency to waste time at the check points on course I also had my sister here to support me this year so I would be able to run into CP 3, 4 and 5. Grab new bottles, have food shoved at me and head on out.

All was looking fairly good and I was sure I could go under 13 hours this year the question remained how far under I could get. Then Wednesday night before the race I got home feeling awful and shivering and needed to crawl into bed early and get sleep, waking up Thursday I felt worse if possible and was worried it was all over I had gotten sick and nothing would help. I left work at 2pm that day and headed home to sleep the rest of the day. Fortunately by the time I woke on Friday morning I no longer felt so awful, and actually felt I may be able to run the next day. I had stopped my running streak on Wednesday, no real need to continue it and feeling so bad for two days definitely had to stop.

I arrived Friday afternoon, spent money with Graham and Hanny in their store for some stuff I needed from Find Your Feet and headed to the briefing. The welcome to country form David King was once again a highlight of the runners briefing it is a fantastic part of the race every year and really heart felt, genuine and funny. Met my sister Jane at our accommodation and discussed the race day and estimated times while eating dinner. Fortunately I finally felt ready to run again by the time I went to sleep Friday night. I had a few runs the week before with what I call Happy Legs where you feel awesome running and light and happy on your feet. Though I hoped for that on Saturday I knew I just had to get out on the track and keep moving well.

I was in wave 1 and starting at 6:20am, had a chat with my mate Tom Reeve on the start line and then we got moving, taking it easy on the 5km bitumen loop I had a chat with Phil Whitten who was worried after stomach issues in six foot caused him problems he may have issues today too (in the end he did alas), still it was nice to be moving and cruising along the out and back before the steps. In wave 1 it was nice and open and even the descent down Furber steps was pretty open. Ran through toward the golden stairs feeling OK, never awesome but not like it was going to be a horrible day out.

I got onto the fire road out Narrow Neck and realised I was probably a few beats higher than I probably should be HR wise however decided to stay with it and ensure I not push too hard on the hills climbs along here. With the start out and back slightly extended this year it was good to pass through CP1 in the same time as last year so on course for slightly faster, however I would not have a proper idea of time ad how I was going until Dunphys camp. On the climb from Cedar gap I noticed some people around me seemed to be pushing harder than I thought they should however that had nothing to do with me so I kept moving and hoping I survived. On the descent down to the camp I had my left adductor cramp a bit which seems to happen here every year so I have to manage it and keep going.

At Dunphys CP I had a chat to Myf happy to actually see her or Matt this year (I missed seeing them here last year) and got moving aware I would need to take it easy on iron pot to keep the cramps at bay. I got onto Iron Pot and loved being able to say thanks to David King and his colleagues welcoming us to country with Didgeridoo and clap sticks up there, the short out and back made it easier this year and then I took it really easy on the loose ski slope sort of descent down due to cramps being close to the surface. Continued taking it easy chatting with other runners as we went back past the outgoing track on our right and then we dropped down to the bottom of the valley to start heading up Megalong Rd.

Looking at my watch I was probably behind time to do sub 12 hours already at this point but would have a much better idea once I got to Six Foot CP in a little while. I took it easy climbing the rd at strong power walk and then managed a comfortable 4 to 5 minute pace along the road into the CP. I got out of CP3 just before the 5 hour mark, this was confirming I was unlikely to go under 12 hours, I expected I needed to be gone from here in 4h40m to manage sub 12 knowing how I was feeling. I grabbed some risotto and baked potatoes with salt from Jane to see if I could eat these for some variety rather than sweet crap while climbing to Katoomba. On the way into the CP I passed Etienne who had an injury so asked her to see if he needed help when he got in (though that made it harder for her to get to e in time at Katoomba, fortunately Etienne had his parents there to help him out when he had to withdraw there)

Trying to eat the solid food was difficult and slowing me down so I gave up by the time I hit the single track just before the stairs. I had a chat with a blonde woman (it may have been Daniela Burton) and it was her first 100 so I told her not to get discouraged how long the next leg (CP4 to CP5) takes and to keep focusing on moving forward. I also had a chat with Ben Grimshaw a few times on the way up Nellies as I was passed by him while trying to eat solid food and then caught him again on the stairs once I started pushing up there reasonably fast once more. We cruised through the single track at the top passing a few runners and got into CP4 pretty much together.

I had to refill my water bladder here as well as get two new bottles, still with Jane's help I got out of here fast and left by 6 hours 30 minutes on the race clock. Though behind Ben now as he was quicker in the CP. Now I was happy to hit my race goal of feeling pretty good at Katoomba and still being keen to run which is always the way I think you need to feel at this point as the next leg is the the crux of the race, the half marathon of stairs is really a tough mental and physical barrier to get through.

I headed along to echo point through some crowds on the walk way near the cliff edge and it was nice to have some of the tourists cheering us on, a few other runners were near by and we got through nicely. On the descent down the giant stair case I seemed to pass a few people pretty comfortably and then on to Dardanelle's pass and it was nice running through there for a while. Of course getting down to Leura forest we got to see some 50km runners coming the other way (a few asked me where I was going worried they had made a wrong turn, when I said I was a 100km runner they realised all was cool told me well done and kept going).

I caught Ben again on the way up the stairs from Leura forest and we were near each other a bit for a while then however I seemed to pull ahead on stairs a bit so over the next while I got away from him (he caught me later in the race anyway though). Last year I had a diabetic low blood sugar incident in this leg, somewhere just before the wentworth falls lookout carpark I think. So I was paying more attention through the day on constant calorie intake with lots of clif shot blocks and gu gels. I kept moving well enough through this whole leg so that turned out well. I Said hi to Graham (Hammond) who was cheering runners on at the Fairmont resort water station and ran on for a few more stairs.

Running in to CP 5 on king tableland road I still felt alright and managed to eat another three cubes of shot block there. I had run out of plain water (bladder) again so had not had a salt tablet for a little while. This year I had decided to run with more salt consumption and had bought hammer enduralyte salt tablets, I was downing 1 or 2 of them every time I ate all day which I think may have helped, though I still had cramps around Dunphys that happens every year and I knew I had run a bit hard early anyway (hoping to hit splits needed for sub 12). However even though it was a hot day and many people struggled more in the heat than other years I seemed to deal with it well. However I had discovered I struggled to down the tablets with electrolyte drink from my bottles (high 5 tablets, usually berry flavour) so I needed plain water from the camelback for them.

I got more food from Jane at CP5, re lubed myself a bit refilled the bladder and got moving. I also grabbed a second head torch, though I was carrying one already I liked the beam pattern more on the one I grabbed here, though with full water, bottles and the extra torch I felt pretty heavy running out of CP 5. Still just 3 hours to go now I expected. I got out of there at 9h25m on the race clock which was good, thus if I could have a good run through here I may be able to get in under 12h20m (2h50m run would be nice for this leg at this point). I got moving on the approach to the kedumba descent joking with a few others around me it was time to smash the quads and say good bye to them as they were no longer needed after this really. (only one short sort of descent to Leura creek) I was asked if we needed quads on the stairs, my response was they were a glute fest and allowed use of arms due to the railing so who needs quads after Kedumba.

However as I got on to the descent and passed under the overhang I noticed my legs were a bit off and I could not open up well, I thought about it and realised I was probably low on sugar and needed to eat, eating at this sort of downhill pace was a bit hard (especially as some food was making me feel like throwing up (gels)). I thought I would try to hang on until the bottom as I could walk up out of Jamisons creek eating. However I needed to slow to a walk just after passing the Mt Solitary turn off and down a gel. Then a few minutes later trying to run still did not work so I had to stop and walk and eat for a while again rather than descending at full speed. Doing all of that I was passed by a few people (I think the woman who came 5th, the guy I joked about not needing Quads with and a few others).

Oh well I should have eaten more while stopped at the CP or on the flat at the top, oops, lost time (in the results comparing with people I ran similar splits all day to I may have lost as much as 15 minutes here with this issue). Once I got onto the climb out of Jamisons creek I ate some more and focused on holding a reasonably strong hike, the people who passed me were long gone and I could not motivate myself to push hard to see if I would catch them or not. I was passing a number of 50km runners by this point (I think the sweep must have been at CP5 when I went through). They were fun to cheer on and chat with as I caught and passed them, getting down to Leura creek was good as it was still day light and I could get moving up there to the last aid and onto the finish before I thought about lights.

Ben caught me a gain here saying he had really pushed hard on the kedumba descent and he was looking good so sat a little ahead of me up to the aid station. I refilled my bottles and kept going chatting with 50 km runners as I passed them. I got to the poo farm a bit quicker than I expected (going on feeling as I was not looking at my watch much) however it was good to finally be up on Federal pass not long after that and this is where I decided to focus on moving fast. The last two years I crawled along here and I think I lost a lot of time, I know last year I had mentally given up so was crawling, the year before I think I was just a bit stuffed by then. This time I focused on running whenever it was not a steep up and on getting over to the stairs as quickly as possible.

It was still fun cheering on the 50km runners and chatting with them as I passed, I even saw some women in awesome pink outfits I had seen here a few weeks earlier while training so it was good to cheer them on, when I asked them about it they said it was them and they recognised me (it's pinky they exclaimed) as I passed. I got to the base of the stairs at 12:14 so knew I had to work hard to finish in under 12:30 but it was time to get that done if possible. On the climb up the stairs it felt like I was getting stuck behind 50km runners on many of the narrow sections of stairs however it probably was not much time slowing up the pace (one occasion a race doctor was walking up the stairs with a runner just to help them get to the finish). I managed to get across the finish line in 12:29:51 (57th overall) which was a good result all things considered.

Thanks go to Jane for coming up from Sydney and supporting me all day, Tom, Al and AROC for keeping the fun happening for all the runners, Dave and co for some excellent course markings, all the other AROC people and volunteers. David, Julie, Alex and others for company on lots of the training the last few months. I have a few ideas for what I need to work on next to faster on this course, however am thinking I may have a year off UTA100 to go do something else. The Hubert race in South Australia at the start of may looks like it could be awesome (running in the wilpena pound area through the Flinders ranges) and it will probably be good to develop my base and speed a bit more over time before my next attempt to see if I can become a runner (crack 12 hours on this course).

UTA100 really is the pinnacle of trail running in Australia with the level of competition, course fun quality, vibe on ocurse and the welcome to country, the event history and everything else so I hightly recommend it to anyone keen to challenge themselves. Even if so far this year the event that has really grabbed my attention the most is probably the Razorback Ultra, it is a very different day out to UTA100 so it is all good fun to get outdoors and enjoy the Australian wilderness.

May 19, 2016

Introducing: Linkify for Chrome

In WordPress 4.2, a fun little feature was quietly snuck into Core, I’m always delighted to see people’s reactions when they discover it.

But there’s still a problem – WordPress is only ~26% of the internet, how can you get the same feature on the other 74%? Well, that problem has now been rectified. Introducing, Linkify for Chrome:

Thank you to Davide for creating Linkify’s excellent icon!

Linkify is a Chrome extension to automatically turn a pasted URL into a link, just like you’re used to in WordPress. It also supports Trac and Markdown-style links, so you can paste links on your favourite bug trackers, too.

Speaking of bug trackers, if there are any other link formats you’d like to see, post a ticket over on the Linkify GitHub repo!

Oh, and speaking of Chrome extensions, you might be like me, and find the word “emojis” to be extraordinarily awkward. If so, I have another little extension, just for you.

May 18, 2016

Fuzzing Firmware – afl-fuzz + skiboot

In what is likely to be a series on how firmware makes some normal tools harder to use, first I’m going to look at american fuzzy lop – a tool for fuzz testing that if you’re not using then you most certainly have bugs it’ll find for you.

I first got interested in afl-fuzz during Erik de Castro Lopo’s excellent linux.conf.au 2016 in Geelong earlier this year: “Fuzz all the things!“. In a previous life, the Random Query Generator managed to find a heck of a lot of bugs in MySQL (and Drizzle). For randgen info, see Philip Stoev’s talk on it from way back in 2009, a recent (2014) blog post on how Tokutek uses it and some notes on how it was being used at Oracle from 2013. Basically, the randgen was a specialized fuzzer that (given a grammar) would randomly generate SQL queries, and then (if the server didn’t crash), compare the result to some other database server (e.g. your previous version).

The afl-fuzz fuzzer takes a different approach – it’s a much more generic fuzzer rather than a targeted tool. Also, while tools such as the random query generator are extremely powerful and find specialized bugs, they’re hard to get started with. A huge benefit of afl-fuzz is that it’s really, really simple to get started with.

Basically, if you have a binary that takes input on stdin or as a (relatively small) file, afl-fuzz will just work and find bugs for you – read the Quick Start Guide and you’ll be finding bugs in no time!

For firmware of course, we’re a little different than a simple command line program as, well, we aren’t one! Luckily though, we have unit tests. These are just standard binaries that include a bunch of firmware code and get run in user space as part of “make check”. Also, just like unit tests for any project, people do send me patches that break tests (which I reject).

Some of these tests act on data we get from a place – maybe reading other parts of firmware off PNOR or interacting with data structures we get from other bits of firmware. For testing this code, it can be relatively easy to (for the test), read these off disk.

For skiboot, there’s a data structure we get from the service processor on FSP machines called HDAT. Basically, it’s just like the device tree, but different. Because yet another binary format is always a good idea (yes, that is laced with a heavy dose of sarcasm). One of the steps in early boot is to parse the HDAT data structure and convert it to a device tree. Luckily, we structured our code so that creating a unit test that can run in userspace was relatively easy, we just needed to dump this data structure out from a running machine. You can see the test case here. Basically, hdat_to_dt is a binary that reads the HDAT structure out of a pair of files and prints out a device tree. One of the regression tests we have is that we always produce the same output from the same input.

So… throwing that into AFL yielded a couple of pretty simple bugs, especially around aborting out on invalid data (it’s better to exit the process with failure rather than hit an assert). Nothing too interesting here on my simple input file, but it does mean that our parsing code exits “gracefully” on invalid data.

Another utility we have is actually a userspace utility for accessing the gard records in the flash. A GARD record is a record of a piece of hardware that has been deconfigured due to a fault (or a suspected fault). Usually this utility operates on PNOR flash through /dev/mtd – but really what it’s doing is talking to the libflash library, that we also use inside skiboot (and on OpenBMC) to read/write from flash directly, via /dev/mtd or just from a file. The good news? I haven’t been able to crash this utility yet!

So I modified the pflash utility to read from a file to attempt to fuzz the partition reading code we have for the partitioning format that’s on PNOR. So far, no crashes – although to even get it going I did have to fix a bug in the file handling code in pflash, so that’s already a win!

But crashing bugs aren’t the only type of bugs – afl-fuzz has exposed several cases where we act on uninitialized data. How? Well, we run some test cases under valgrind! This is the joy of user space unit tests for firmware – valgrind becomes a tool that you can run! Unfortunately, these bugs have been sitting in my “todo” pile (which is, of course, incredibly long).

Where to next? Fuzzing the firmware calls themselves would be nice – although that’s going to require a targeted tool that knows about what to pass each of the calls. Another round of afl-fuzz running would also be good, I’ve fixed a bunch of the simple things and having a better set of starting input files would be great (and likely expose more bugs).

May 17, 2016

More PSYOPS, Social Systems, and More

- I think that most people would agree that the best social systems revolve around the idea that we have fair and just laws. If the size of the security apparatus exceeds a certain point (which seems to be happening in a lot of places) are we certain that we have the correct laws and societal laws in place? If they can't convince through standard argumentation then the policy is probably not

May 14, 2016

London roadshow wrap-up, see you in Paris next week

Just a few days ago, I presented at the MariaDB Roadshow in London, and I had a lot of fun. While I had canned slides, I did know the topic intimately well, so it was good to get further in-depth. In addition, we had these MasterMind sessions, basically the place to get one-on-one time with Anders/Luisa/or me, I noticed that pretty much everyone said they were buying services afterwards (which more or less must mean the event was rather successful from that standpoint!).

In addition to that, I was happy to see that from attendee feedback, I did have the highest averages – thank you!

So here’s to repeating this in Paris next week — Gestion des données pour les applications vitales – MariaDB Roadshow Paris. I look forward to seeing you there, and I know we are repeating the MasterMind sessions. To fine-tune it, try to bring as much information as you possibly can so our time can be extremely effective.

The Politics of Resentment

I’ve been reading The Archdruid Report regularly for a long time now, because unlike me, John Michael Greer posts every week and always writes something interesting. Given that we’ve got a federal election coming up in Australia and that I’ve mentioned one of JMG’s notions on the current state of politics to several people over the last few months, I though I’d provide a TL;DR here:

If you want, you can split people in the US into four classes, based on how they get most of their income:

  1. The investment class (income derived from returns on investment)
  2. The salary class (who receive a monthly salary)
  3. The wage class (who receive an hourly wage)
  4. The welfare class (who receive welfare payments)

According to JMG, over the last fifty years or so, three of these classes of people have remained roughly where they are; the investment class still receives returns on investment (modulo a recession or two), the salary class still draws a reasonable salary, and life still sucks for people on welfare. But the wage class, to be blunt, has been systematically fucked over this time period. There’s a lot of people there, and it’s this disenfranchised group who sees someone outside the political establishment status quo (Trump) as someone they can get behind. Whether or not Trump is elected in the US, there’s still going to be a whole lot of people out there pissed off with the current state of things, and it’s going to be really interesting to see how this plays out.

You should probably go read the full post, because I doubt I’ve done it justice here, but I don’t think it’s unreasonable to imagine the same (or a similar) thesis might be valid for Australia, so my question is: what, if anything, does this mean for our 2016 federal election?

Xen CPU Use per Domain again

8 years ago I wrote a script to summarise Xen CPU use per domain [1]. Since then changes to Xen required changes to the script. I have new versions for Debian/Wheezy (Xen 4.1) and Debian/Jessie (Xen 4.4).

Here’s a new script for Debian/Wheezy:

#!/usr/bin/perl
use strict;

open(LIST, "xm list --long|") or die "Can't get list";

my $name = "Dom0";
my $uptime = 0.0;
my $cpu_time = 0.0;
my $total_percent = 0.0;
my $cur_time = time();

open(UPTIME, "</proc/uptime") or die "Can't open /proc/uptime";
my @arr = split(/ /, <UPTIME>);
$uptime = $arr[0];
close(UPTIME);

my %all_cpu;

while(<LIST>)
{
  chomp;
  if($_ =~ /^\)/)
  {
    my $cpu = $cpu_time / $uptime * 100.0;
    if($name =~ /Domain-0/)
    {
      printf("%s uses %.2f%% of one CPU\n", $name, $cpu);
    }
    else
    {
      $all_cpu{$name} = $cpu;
    }
    $total_percent += $cpu;
    next;
  }
  $_ =~ s/\).*$//;
  if($_ =~ /start_time /)
  {
    $_ =~ s/^.*start_time //;
    $uptime = $cur_time – $_;
    next;
  }
  if($_ =~ /cpu_time /)
  {
    $_ =~ s/^.*cpu_time //;
    $cpu_time = $_;
    next;
  }
  if($_ =~ /\(name /)
  {
    $_ =~ s/^.*name //;
    $name = $_;
    next;
  }
}
close(LIST);

sub hashValueDescendingNum {
  $all_cpu{$b} <=> $all_cpu{$a};
}

my $key;

foreach $key (sort hashValueDescendingNum (keys(%all_cpu)))
{
  printf("%s uses %.2f%% of one CPU\n", $key, $all_cpu{$key});
}

printf("Overall CPU use approximates %.1f%% of one CPU\n", $total_percent);

Here’s the script for Debian/Jessie:

#!/usr/bin/perl

use strict;

open(UPTIME, "xl uptime|") or die "Can't get uptime";
open(LIST, "xl list|") or die "Can't get list";

my %all_uptimes;

while(<UPTIME>)
{
  chomp $_;

  next if($_ =~ /^Name/);
  $_ =~ s/ +/ /g;

  my @split1 = split(/ /, $_);
  my $dom = $split1[0];
  my $uptime = 0;
  my $time_ind = 2;
  if($split1[3] eq "days,")
  {
    $uptime = $split1[2] * 24 * 3600;
    $time_ind = 4;
  }
  my @split2 = split(/:/, $split1[$time_ind]);
  $uptime += $split2[0] * 3600 + $split2[1] * 60 + $split2[2];
  $all_uptimes{$dom} = $uptime;
}
close(UPTIME);

my $total_percent = 0;

while(<LIST>)
{
  chomp $_;

  my $dom = $_;
  $dom =~ s/ .*$//;

  if ( $_ =~ /(\d+)\.[0-9]$/ )
  {
    my $percent = $1 / $all_uptimes{$dom} * 100.0;
    $total_percent += $percent;
    printf("%s uses %.2f%% of one CPU\n", $dom, $percent);
  }
  else
  {
    next;
  }
}

printf("Overall CPU use approximates  %.1f%% of one CPU\n", $total_percent);

May 13, 2016

Signal Return Orientated Programming attacks

When a process is interrupted, the kernel suspends it and stores its state in a sigframe which is placed on the stack. The kernel then calls the appropriate signal handler code and after a sigreturn system call, reads the sigframe off the stack, restores state and resumes the process. However, by crafting a fake sigframe, we can trick the kernel into executing something else.

My friend Rashmica, an intern at OzLabs, has written an interesting blog post about this for some work she’s doing with the POWER architecture in Linux.

SROP Mitigation

What is SROP?

Sigreturn Oriented Programming - a general technique that can be used as an exploit, or as a backdoor to exploit another vulnerability.

Okay, but what is it?

Yeah... Let me take you through some relevant background info, where I skimp on the details and give you the general picture.

In Linux, software interrupts are called signals. More about signals here! Generally a signal will convey some information from the kernel and so most signals will have a specific signal handler (some code that deals with the signal) setup.

Signals are asynchronous - ie they can be sent to a process/program at anytime. When a signal arrives for a process, the kernel suspends the process. The kernel then saves the 'context' of the process - all the general purpose registers (GPRs), the stack pointer, the next-instruction pointer etc - into a structure called a 'sigframe'. The sigframe is stored on the stack, and then the kernel runs the signal handler. At the very end of the signal handler, it calls a special system call called 'sigreturn' - indicating to the kernel that the signal has been dealt with. The kernel then grabs the sigframe from the stack, restores the process's context and resumes the execution of the process.

This is the rough mental picture you should have:

Double Format

Okay... but you still haven't explained what SROP is..?

Well, if you insist...

The above process was designed so that the kernel does not need to keep track of what signals it has delivered. The kernel assumes that the sigframe it takes off the stack was legitimately put there by the kernel because of a signal. This is where we can trick the kernel!

If we can construct a fake sigframe, put it on the stack, and call sigreturn, the kernel will assume that the sigframe is one it put there before and will load the contents of the fake context into the CPU's registers and 'resume' execution from where the fake sigframe tells it to. And that is what SROP is!

Well that sounds cool, show me!

Firstly we have to set up a (valid) sigframe:

By valid sigframe, I mean a sigframe that the kernel will not reject. Luckily most architectures only examine a few parts of the sigframe to determine the validity of it. Unluckily, you will have to dive into the source code to find out which parts of the sigframe you need to set up for your architecture. Have a look in the function which deals with the syscall sigreturn (probably something like sys_sigreturn() ).

For a real time signal on a little endian powerpc 64bit machine, the sigframe looks something like this:

struct rt_sigframe {
        struct ucontext uc;
        unsigned long _unused[2];
        unsigned int tramp[TRAMP_SIZE];
        struct siginfo __user *pinfo;
        void __user *puc;
        struct siginfo info;
        unsigned long user_cookie;
        /* New 64 bit little-endian ABI allows redzone of 512 bytes below sp */
        char abigap[USER_REDZONE_SIZE];
} __attribute__ ((aligned (16)));

The most important part of the sigframe is the context or ucontext as this contains all the register values that will be written into the CPU's registers when the kernel loads in the sigframe. To minimise potential issues we can copy valid values from the current GPRs into our fake ucontext:

register unsigned long r1 asm("r1");
register unsigned long r13 asm("r13");
struct ucontext ctx = { 0 };

/* We need a system thread id so copy the one from this process */
ctx.uc_mcontext.gp_regs[PT_R13] = r13;

/*  Set the context's stack pointer to where the current stack pointer is pointing */
ctx.uc_mcontext.gp_regs[PT_R1] = r1;

We also need to tell the kernel where to resume execution from. As this is just a test to see if we can successfully get the kernel to resume execution from a fake sigframe we will just point it to a function that prints out some text.

/* Set the next instruction pointer (NIP) to the code that we want executed */
ctx.uc_mcontext.gp_regs[PT_NIP] = (unsigned long) test_function;

For some reason the sys_rt_sigreturn() on little endian powerpc 64bit checks the endianess bit of the ucontext's MSR register, so we need to set that:

/* Set MSR bit if LE */
ctx.uc_mcontext.gp_regs[PT_MSR] = 0x01;

Fun fact: not doing this or setting it to 0 results in the CPU switching from little endian to big endian! For a powerpc machine sys_rt_sigreturn() only examines ucontext, so we do not need to set up a full sigframe.

Secondly we have to put it on the stack:

/* Set current stack pointer to our fake context */
r1 = (unsigned long) &ctx;

Thirdly, we call sigreturn:

/* Syscall - NR_rt_sigreturn */
asm("li 0, 172\n");
asm("sc\n");

When the kernel receives the sigreturn call, it looks at the userspace stack pointer for the ucontext and loads this in. As we have put valid values in the ucontext, the kernel assumes that this is a valid sigframe that it set up earlier and loads the contents of the ucontext in the CPU's registers "and resumes" execution of the process from the address we pointed the NIP to.

Obviously, you need something worth executing at this address, but sadly that next part is not in my job description. This is a nice gateway into the kernel though and would pair nicely with another kernel vulnerability. If you are interested in some more in depth examples, have a read of this paper.

So how can we mitigate this?

Well, I'm glad you asked. We need some way of distinguishing between sigframes that were put there legitimately by the kernel and 'fake' sigframes. The current idea that is being thrown around is cookies, and you can see the x86 discussion here.

The proposed solution is to give every sighand struct a randomly generated value. When the kernel constructs a sigframe for a process, it stores a 'cookie' with the sigframe. The cookie is a hash of the cookie's location and the random value stored in the sighand struct for the process. When the kernel receives a sigreturn, it hashes the location where the cookie should be with the randomly generated number in sighand struct - if this matches the cookie, the cookie is zeroed, the sigframe is valid and the kernel will restore this context. If the cookies do not match, the sigframe is not restored.

Potential issues:

  • Multithreading: Originally the random number was suggested to be stored in the task struct. However, this would break multi-threaded applications as every thread has its own task struct. As the sighand struct is shared by threads, this should not adversely affect multithreaded applications.
  • Cookie location: At first I put the cookie on top of the sigframe. However some code in userspace assumed that all the space between the signal handler and the sigframe was essentially up for grabs and would zero the cookie before I could read the cookie value. Putting the cookie below the sigframe was also a no-go due to the ABI-gap (a gap below the stack pointer that signal code cannot touch) being a part of the sigframe. Putting the cookie inside the sigframe, just above the ABI gap has been fine with all the tests I have run so far!
  • Movement of sigframe: If you move the sigframe on the stack, the cookie value will no longer be valid... I don't think that this is something that you should be doing, and have not yet come across a scenario that does this.

For a more in-depth explanation of SROP, click here.

Tell Me About Petitboot

A Google search for 'Petitboot' brings up results from a number of places, some describing its use on POWER servers, others talking about how to use it on the PS3, in varying levels of detail. I tend to get a lot of general questions about Petitboot and its behaviour, and have had a few requests for a broad "Welcome to Petitboot" blog, suggesting that existing docs deal with more specific topics.. or that people just aren't reading them :)

So today we're going to take a bit of a crash course in the what, why, and possibly how of Petitboot. I won't delve too much into technical details, and this will be focussed on Petitboot in POWER land since that's where I spend most of my time. Here we go!

What

Aside from a whole lot of firmware and kernel logs flying past, the first thing you'll see when booting a POWER serverIn OPAL mode at least... is Petitboot's main menu:

Main Menu

Petitboot is the first interact-able component a user will see. The word 'BIOS' gets thrown around a lot when discussing this area, but that is wrong, and the people using that word are wrong.

When the OPAL firmware layer Skiboot has finished its own set up, it loads a certain binary (stored on the BMC) into memory and jumps into it. This could hypothetically be anything, but for any POWER server right now it is 'Skiroot'. Skiroot is a full Linux kernel and userspace, which runs Petitboot. People often say Petitboot when they mean Skiroot - technically Petitboot is the server and UI processes that happen to run within Skiroot, and Skiroot is the full kernel and rootfs package. This is more obvious when you look at the op-build project - Petitboot is a package built as part of the kernel and rootfs created by Buildroot.

Petitboot is made of two major parts - the UI processes (one for each available console), and the 'discover' server. The discover server updates the UI processes, manages and scans available disks and network devices, and performs the actual booting of host operating systems. The UI, running in ncurses, displays these options, allows the user to edit boot options and system configuration, and tells the server which boot option to kexec.

Why

The 'why' delves into some of the major architectural differences between a POWER machine and your average x86 machine which, as always, could spread over several blog posts and/or a textbook.

POWER processors don't boot themselves, instead the attached Baseboard Management Controller (BMC) does a lot of low-level poking that gets the primary processor into a state where it is ready to execute instructions. PowerVM systems would then jump directly into the PHYP hypervisor - any subsequent OS, be it AIX or Linux, would then run as a 'partition' under this hypervisor.

What we all really want though is to run Linux directly on the hardware, which meant a new boot process would have to be thought up while still maintaining compatibility with PowerVM so systems could be booted in either mode. Thus became OPAL, and its implementation Skiboot. Skipping over so much detail, the system ends up booting into Skiboot which acts as our firmware layer. Skiboot isn't interactive and doesn't really care about things like disks, so it loads another binary into memory and executes it - Skiroot!

Skiroot exists as an alternative to writing a whole new bootloader just for POWER in OPAL mode, or going through the effort to port an existing bootloader to understand the specifics of POWER. Why do all that when Linux already exists and already knows how to handle disks, network interfaces, and a thousand other things? Not to mention that when Linux gains support for fancy new devices so do we, and adding new features of our own is as simple as writing your average Linux program.

Skiroot itself (not including Skiboot) is roughly comparable to UEFI, or at least much more so than legacy BIOS implementations. But whereas UEFI tends to be a monolithic blob of fairly platform-specific code (in practice), Skiroot is simply a small Linux environment that anyone could put together with Buildroot.

A much better insight into the development and motivation behind Skiroot and Petitboot is available in Jeremy's LCA2013 talk

Back to Petitboot

Petitboot is the part of the 'bootloader' that did need to be written, because users probably wouldn't be too thrilled if they had to manually mount disks and kexec their kernels every time they booted the machine. The Petitboot server process mounts any available disk devices and scans them for available operating systems. That's not to say that it scans the entire disk, because otherwise you could be waiting for quite some time, but rather it looks in a list of common locations for bootloader configuration files. This is handy because it means the operating system doesn't need to have any knowledge of Petitboot - it just uses its usual install scripts and Petitboot reads them to know what is available. At the same time Petitboot makes PXE requests on configured network interfaces so we can netboot, and allows these various sources to be given relative priorities for auto-boot, plus a number of other ways to specially configure booting behaviour.

A particularly neat feature of existing in a Linux environment is the ability to easily recover from boot problems; whereas on another system you might need to use a Live CD to fix a misconfiguration or recover a broken filesystem, in Skiroot you can just drop to the shell and fix the issue right there.

In summary, Petitboot/Skiroot is a small but capable Linux environment that every OPAL POWER machine boots into, gathering up all the various local and remote boot possibilities, and presenting them to you in a state-of-the-art ncurses interface. Petitboot updates all the time, and if you come across a feature that you think Petitboot is missing, patches are very welcome at petitboot@lists.ozlabs.org (or hassle me on IRC)!

SROP Mitigation

What is SROP?

Sigreturn Oriented Programming - a general technique that can be used as an exploit, or as a backdoor to exploit another vulnerability.

Okay, but what is it?

Yeah... Let me take you through some relevant background info, where I skimp on the details and give you the general picture.

In Linux, software interrupts are called signals. More about signals here! Generally a signal will convey some information from the kernel and so most signals will have a specific signal handler (some code that deals with the signal) setup.

Signals are asynchronous - ie they can be sent to a process/program at anytime. When a signal arrives for a process, the kernel suspends the process. The kernel then saves the 'context' of the process - all the general purpose registers (GPRs), the stack pointer, the next-instruction pointer etc - into a structure called a 'sigframe'. The sigframe is stored on the stack, and then the kernel runs the signal handler. At the very end of the signal handler, it calls a special system call called 'sigreturn' - indicating to the kernel that the signal has been dealt with. The kernel then grabs the sigframe from the stack, restores the process's context and resumes the execution of the process.

This is the rough mental picture you should have:

Double Format

Okay... but you still haven't explained what SROP is..?

Well, if you insist...

The above process was designed so that the kernel does not need to keep track of what signals it has delivered. The kernel assumes that the sigframe it takes off the stack was legitimately put there by the kernel because of a signal. This is where we can trick the kernel!

If we can construct a fake sigframe, put it on the stack, and call sigreturn, the kernel will assume that the sigframe is one it put there before and will load the contents of the fake context into the CPU's registers and 'resume' execution from where the fake sigframe tells it to. And that is what SROP is!

Well that sounds cool, show me!

Firstly we have to set up a (valid) sigframe:

By valid sigframe, I mean a sigframe that the kernel will not reject. Luckily most architectures only examine a few parts of the sigframe to determine the validity of it. Unluckily, you will have to dive into the source code to find out which parts of the sigframe you need to set up for your architecture. Have a look in the function which deals with the syscall sigreturn (probably something like sys_sigreturn() ).

For a real time signal on a little endian powerpc 64bit machine, the sigframe looks something like this:

struct rt_sigframe {
        struct ucontext uc;
        unsigned long _unused[2];
        unsigned int tramp[TRAMP_SIZE];
        struct siginfo __user *pinfo;
        void __user *puc;
        struct siginfo info;
        unsigned long user_cookie;
        /* New 64 bit little-endian ABI allows redzone of 512 bytes below sp */
        char abigap[USER_REDZONE_SIZE];
} __attribute__ ((aligned (16)));

The most important part of the sigframe is the context or ucontext as this contains all the register values that will be written into the CPU's registers when the kernel loads in the sigframe. To minimise potential issues we can copy valid values from the current GPRs into our fake ucontext:

register unsigned long r1 asm("r1");
register unsigned long r13 asm("r13");
struct ucontext ctx = { 0 };

/* We need a system thread id so copy the one from this process */
ctx.uc_mcontext.gp_regs[PT_R13] = r13;

/*  Set the context's stack pointer to where the current stack pointer is pointing */
ctx.uc_mcontext.gp_regs[PT_R1] = r1;

We also need to tell the kernel where to resume execution from. As this is just a test to see if we can successfully get the kernel to resume execution from a fake sigframe we will just point it to a function that prints out some text.

/* Set the next instruction pointer (NIP) to the code that we want executed */
ctx.uc_mcontext.gp_regs[PT_NIP] = (unsigned long) test_function;

For some reason the sys_rt_sigreturn() on little endian powerpc 64bit checks the endianess bit of the ucontext's MSR register, so we need to set that:

/* Set MSR bit if LE */
ctx.uc_mcontext.gp_regs[PT_MSR] = 0x01;

Fun fact: not doing this or setting it to 0 results in the CPU switching from little endian to big endian! For a powerpc machine sys_rt_sigreturn() only examines ucontext, so we do not need to set up a full sigframe.

Secondly we have to put it on the stack:

/* Set current stack pointer to our fake context */
r1 = (unsigned long) &ctx;

Thirdly, we call sigreturn:

/* Syscall - NR_rt_sigreturn */
asm("li 0, 172\n");
asm("sc\n");

When the kernel receives the sigreturn call, it looks at the userspace stack pointer for the ucontext and loads this in. As we have put valid values in the ucontext, the kernel assumes that this is a valid sigframe that it set up earlier and loads the contents of the ucontext in the CPU's registers "and resumes" execution of the process from the address we pointed the NIP to.

Obviously, you need something worth executing at this address, but sadly that next part is not in my job description. This is a nice gateway into the kernel though and would pair nicely with another kernel vulnerability. If you are interested in some more in depth examples, have a read of this paper.

So how can we mitigate this?

Well, I'm glad you asked. We need some way of distinguishing between sigframes that were put there legitimately by the kernel and 'fake' sigframes. The current idea that is being thrown around is cookies, and you can see the x86 discussion here.

The proposed solution is to give every sighand struct a randomly generated value. When the kernel constructs a sigframe for a process, it stores a 'cookie' with the sigframe. The cookie is a hash of the cookie's location and the random value stored in the sighand struct for the process. When the kernel receives a sigreturn, it hashes the location where the cookie should be with the randomly generated number in sighand struct - if this matches the cookie, the cookie is zeroed, the sigframe is valid and the kernel will restore this context. If the cookies do not match, the sigframe is not restored.

Potential issues:

Multithreading: Originally the random number was suggested to be stored in the task struct. However, this would break multi-threaded applications as every thread has its own task struct. As the sighand struct is shared by threads, this should not adversely affect multithreaded applications.
Cookie location: At first I put the cookie on top of the sigframe. However some code in userspace assumed that all the space between the signal handler and the sigframe  was essentially up for grabs and would zero the cookie before I could read the cookie value. Putting the cookie below the sigframe was also a no-go due to the ABI-gap (a gap below the stack pointer that signal code cannot touch) being a part of the sigframe. Putting the cookie inside the sigframe, just above the ABI gap has been fine with all the tests I have run so far!
Movement of sigframe: If you move the sigframe on the stack, the cookie value will no longer be valid... I don't think that this is something that you should be doing, and have not yet come across a scenario that does this.

For a more in-depth explanation of SROP, click here.

May 12, 2016

Doubles in hex and why Kernel addresses ~= -2

It started off a regular Wednesday morning when I hear from my desk a colleague muttering about doubles and their hex representation. "But that doesn't look right", "How do I read this as a float", and "redacted you're the engineer, you do it". My interest piqued, I headed over to his desk to enquire about the great un-solvable mystery of the double and its hex representation. The number which would consume me for the rest of the morning: 0xc00000001568fba0.

That's a Perfectly Valid hex Number!

I hear you say. And you're right, if we were to treat this as a long it would simply be 13835058055641365408 (or -4611686018068186208 if we assume a signed value). But we happen to know that this particular piece of data which we have printed is supposed to represent a double (-2 to be precise). "Well print it as a double" I hear from the back, and once again we should all know that this can be achieved rather easily by using the %f/%e/%g specifiers in our print statement. The only problem is that in kernel land (where we use printk) we are limited to printing fixed point numbers, hence why our only easy option was to print our double in it's raw hex format.

This is the point where we all think back to that university course where number representations were covered in depth, and terms like 'mantissa' and 'exponent' surface in our minds. Of course as we rack our brains we realise there's no way that we're going to remember exactly how a double is represented and bring up the IEEE 754 Wikipedia page.

What is a Double?

Taking a step back for a second, a double (or a double-precision floating-point) is a number format used to represent floating-point numbers (those with a decimal component). They are made up of a sign bit, an exponent and a fraction (or mantissa):

Double Format

Where the number they represent is defined by:

Double Formula

So this means that a 1 in the MSB (sign bit) represents a negative number, and we have some decimal component (the fraction) which we multiply by some power of 2 (as determined by the exponent) to get our value.

Alright, so what's 0xc00000001568fba0?

The reason we're all here to be begin with, so what's 0xc00000001568fba0 if we treat it as a double? We can first split it into the three components:

0xc00000001568fba0:

Sign bit: 1 -> Negative
Exponent: 0x400 -> 2(1024 - 1023)
Fraction: 0x1568fba0 -> 1.something

And then use the formula above to get our number:

(-1)1 x 1.something x 2(1024 - 1023)

But there's a much easier way! Just write ourselves a little program in userspace (where we are capable of printing floats) and we can save ourselves most of the trouble.

#include <stdio.h>

void main(void)
{
    long val = 0xc00000001568fba0;

    printf("val: %lf\n", *((double *) &val));
}

So all we're doing is taking our hex value and storing it in a long (val), then getting a pointer to val, casting it to a double pointer, and dereferencing it and printing it as a float. Drum Roll And the answer is?

"val: -2.000000"

"Wait a minute, that doesn't quite sound right". You're right, it does seem a bit strange that this is exactly -2. Well it may be that we are not printing enough decimal places to see the full result, so update our print statement to:

printf("val: %.64lf\n", *((double *) &val));

And now we get:

"val: -2.0000001595175973534423974342644214630126953125000000"

Much better... But still where did this number come from and why wasn't it the -2 that we were expecting?

Kernel Pointers

At this point suspicions had been raised that what was being printed by my colleague was not what he expected and that this was in fact a Kernel pointer. How do you know? Lets take a step back for a second...

In the PowerPC architecture, the address space which can be seen by an application is known as the effective address space. We can take this and translate it into a virtual address which when mapped through the HPT (hash page table) gives us a real address (or the hardware memory address).

The effective address space is divided into 5 regions:

Effective Address Table

As you may notice, Kernel addresses begin with 0xc. This has the advantage that we can map a virtual address without the need for a table by simply masking the top nibble.

Thus it would be reasonable to assume that our value (0xc00000001568fba0) was indeed a pointer to a Kernel address (and further code investigation confirmed this).

But What is -2 as a Double in hex?

Well lets modify the above program and find out:

include <stdio.h>

void main(void)
{
        double val = -2;

        printf("val: 0x%lx\n", *((long *) &val));
}

Result?

"val: 0xc000000000000000"

Now that sounds much better. Lets take a closer look:

0xc000000000000000:

Sign Bit: 1 -> Negative
Exponent: 0x400 -> 2(1024 - 1023)
Fraction: 0x0 -> Zero

So if you remember from above, we have:

(-1)1 x 1.0 x 2(1024 - 1023) = -2

What about -1? -3?

-1:

0xbff0000000000000:

Sign Bit: 1 -> Negative
Exponent: 0x3ff -> 2(1023 - 1023)
Fraction: 0x0 -> Zero

(-1)1 x 1.0 x 2(1023 - 1023) = -1

-3:

0xc008000000000000:

Sign Bit: 1 -> Negative
Exponent: 0x400 -> 2(1024 - 1023)
Fraction: 0x8000000000000 -> 0.5

(-1)1 x 1.5 x 2(1024 - 1023) = -3

So What Have We Learnt?

Firstly, make sure that what you're printing is what you think you're printing.

Secondly, if it looks like a Kernel pointer then you're probably not printing what you think you're printing.

Thirdly, all Kernel pointers ~= -2 if you treat them as a double.

And Finally, with my morning gone, I can say for certain that if we treat it as a double, 0xc00000001568fba0 = -2.0000001595175973534423974342644214630126953125.

TRIM on LVM on LUKS on SSD, revisited

A few years ago I wrote about enabling trim on an SSD that was running with LVM on top of LUKS. Since then things have changed slightly, a few times.

With Fedora 24 you no longer need to edit the /etc/crypttab file and rebuild your initramfs. Now systemd supports a kernel boot argument rd.luks.options=discard which is the only thing you should need to do to enable trim on your LUKS device.

Edit /etc/default/grub and add the rd.luks.options=discard argument to the end of GRUB_CMDLINE_LINUX, e.g.:
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-de023401-ccec-4455-832bf-e5ac477743dc rd.luks.uuid=luks-a6d344739a-ad221-4345-6608-e45f16a8645e rhgb quiet rd.luks.options=discard"
GRUB_DISABLE_RECOVERY="true"

Next, rebuild your grub config file:
sudo grub2-mkconfig -o /boot/grub2/grub.cfg

If you’re using LVM, the setting is the same as the previous post. Edit the /etc/lvm/lvm.conf file and enabled the issue_discards option:
issue_discards = 1

If using LVM you will need to rebuild your initramfs so that the updated lvm.conf is in there.
sudo dracut -f

Reboot and try fstrim:
sudo fstrim -v /

Now also thanks to systemd, you can just enable the fstrim timer (cron) to do this automatically:
sudo systemctl enable fstrim.timer

May 11, 2016

LUV Main June 2016 Meeting: Talks TBA

Jun 7 2016 18:30
Jun 7 2016 20:30
Jun 7 2016 18:30
Jun 7 2016 20:30
Location: 

6th Floor, 200 Victoria St. Carlton VIC 3053

200 Victoria St. Carlton VIC 3053

Late arrivals, please call (0490) 049 589 for access to the venue.

Before and/or after each meeting those who are interested are welcome to join other members for dinner. We are open to suggestions for a good place to eat near our venue. Maria's on Peel Street in North Melbourne is currently the most popular place to eat after meetings.

LUV would like to acknowledge Red Hat and Infoxchange for their help in obtaining the meeting venues.

Linux Users of Victoria Inc. is an incorporated association, registration number A0040056C.

June 7, 2016 - 18:30

LUV Beginners May Meeting: Apache Cassandra Workshop

May 21 2016 12:30
May 21 2016 16:30
May 21 2016 12:30
May 21 2016 16:30
Location: 

Infoxchange, 33 Elizabeth St. Richmond

This hands-on workshop will provide participants with an introduction to the Cassandra distributed "NoSQL" database management system, including deployment, keyspace and table manipulation, replication, creating multiple datacenters and creating users.

Participants will:

  • Install a Cassandra server
  • Create a keyspace
  • Create a table and insert data
  • Replicate data across a three-node cluster
  • Replicate data across a two-datacenter cluster
  • Set up Cassandra and JMX authentication

The meeting will be held at Infoxchange, 33 Elizabeth St. Richmond 3121 (enter via the garage on Jonas St.)

Late arrivals, please call (0490) 049 589 for access to the venue.

LUV would like to acknowledge Infoxchange for the venue.

Linux Users of Victoria Inc., is an incorporated association, registration number A0040056C.

May 21, 2016 - 12:30

read more

Towards GStreamer 1.0 talk

 GStreamer logo

I gave my talk titled “Towards GStreamer 1.0” at the Gran Canaria Desktop Summit on Sunday. The slides are available here

My intention with the talk was to present some of the history and development of the GStreamer project as a means to look at where we might go next. I talked briefly about the origins of the project, its growth, and some of my personal highlights from the work we’ve done in the last year. To prepare the talk, I extracted some simple statistics from our commit history. In those, it’s easy to see both the general growth of the project, in terms of development energy/speed, as well as the increase in the number of contributors. It’s also possible to see the large hike in productivity that switching to Git in January has provided us.

The second part of the talk was discussing some of the pros and cons around considering whether to embark on a new major GStreamer release cycle leading up to a 1.0 release. We’ve successfully maintained the 0.10 GStreamer release series with backwards-compatible ABI and API (with some minor glitches) for 3.5 years now, and been very successful at adding features and improving the framework while doing so.

After 3.5 years of stable development, it’s clear to me that when we made GStreamer 0.10, it really ought to have been 1.0. Nevertheless, there are some parts of GStreamer 0.10 that we’re collectively not entirely happy with and would like to fix, but can’t without breaking backwards compatibility – so I think that even if we had made 0.10 at that point, I’d want to be doing 1.2 by now.

Some examples of things that are hard to do in 0.10:

  • Replace ugly or hard to use API
  • ABI mistakes such as structure members that should be private having been accidentally exposed in some release.
  • Running out of padding members in public structures, preventing further expansion
  • Deprecated API (and associated dead code paths) we’d like to remove

There are also some enhancements that fall into a more marginal category, in that they are technically possible to achieve in incremental steps during the 0.10 cycle, but are made more difficult by the need to preserve backwards compatibility. These include things like adding per-buffer metadata to buffers (for extensible timestamping/timecode information, pan & scan regions and others), variable strides in video buffers and creating/using more base classes for common element types.

In the cons category are considerations like the obvious migration pain that breaking ABI will cause our applications, and the opportunity cost of starting a new development cycle. The migration cost is mitigated somewhat by the ability to have parallel installations of GStreamer. GStreamer 0.10 applications will be able to coexist with GStreamer 1.0 applications.

The opportunity cost is a bit harder to ignore. When making the 0.9 development series, we found that the existing 0.8 branch became essentially unmaintained for 1.5 years, which is a phenomenon we’d all like to avoid with a new release series. I think that’s possible to achieve this time around, because I expect a much smaller scope of change between 0.10 and 1.0. Apart from the few exceptions above, GStreamer 0.10 has turned out really well, and has become a great framework being used in all sorts of exciting ways that doesn’t need large changes.

Weighing up the pros and cons, it’s my opinion that it’s worth making GStreamer 1.0. With that in mind, I made the following proposal at the end of my talk:

  • We should create a shared Git playground and invite people to use it for experimental API/ABI branches
  • Merge from the 0.10 master regularly into the playground regularly, and rebase/fix experimental branches
  • Keep developing most things in 0.10, relying on the regular merges to get them into the playground
  • After accumulating enough interesting features, pull the experimental branches together as a 0.11 branch and make some released
  • Target GStreamer 1.0 to come out in time for GNOME 3.0 in March 2010

This approach wasn’t really possible the last time around when everything was stored in CVS – it’s having a fast revision control system with easy merging and branch management that will allow it.

GStreamer Summit

On Thursday, we’re having a GStreamer summit in one of the rooms at the university. We’ll be discussing my proposal above, as well as talking about some of the problems people have with 0.10, and what they’d like to see in 1.0. If we can, I’d like to draw up a list of features and changes that define GStreamer 1.0 that we can start working towards.

Please come along if you’d like to help us push GStreamer forward to the next level. You’ll need to turn up at the university GCDS venue and then figure out on your own which room we’re in. We’ve been told there is one organised, but not where – so we’ll all be in the same boat.

The summit starts at 11am.

Proof of life – A New Adventure!

Hi world! It’s been several years since I used this blog, and there’s been a lot of things happen to us since then. I don’t even live on the same continent as I did.

More on that in a future post. Today, I have an announcement to make – a new Open Source company! Together with fellow GStreamer hackers Tim-Philipp Müller and Sebastian Dröge, I have founded a new company: Centricular Ltd.

From 2007 until July, I was working at Oracle on Sun Ray thin client firmware. Oracle shut down the project in July, and my job along with it – opening up this excellent opportunity to try something I’ve wanted for a while and start a business, while getting back to Free Software full time.

Our website has more information about the Open Source technologies and services we plan to offer. This list is not complete and we will try to broaden it over time, so if you have anything interesting that is not listed there but you think we can help with, get in touch

As Centricular’s first official contribution to the software pool, here’s my Raspberry Pi Camera GStreamer module. It wraps code from Raspivid to allow direct capture from the official camera module and hardware encoding to H.264 in a GStreamer pipeline – without the shell pipe and fdsrc hack people have been using to date. Take a look at the README for more information.

Raspberry Pi Camera GStreamer element

Sebastian, Tim and I will be at the GStreamer Conference in Edinburgh next week.

New York trip, DVD stuff

We’re leaving tomorrow afternoon for 11 days holiday in New York and Washington D.C. While we’re there, I’m hoping to catch up with Luis and Krissa and Thom May. It’s our first trip to either city, so we’re really excited – there’s a lot of fun, unique stuff to do in both places and we’re looking forward to trying to do all of it in our short visit.

On the GStreamer front, I just pushed a bunch of commits I’ve been working on for the past few weeks upstream into Totem, gst-plugins-base and gst-plugins-bad. Between them they fix a few DVD issues like multiangle support and playback in playbin2. The biggest visible feature though is the API that allowed me to (finally!) hook up the DVD menu items in Totem’s UI. Now the various ‘DVD menu’, ‘Title Menu’ etc menu items work, as well as switching angles in multiangle titles, and it provides the nice little ‘cursor switches to a hand when over a clickable button’ behaviour.

I actually had it all ready yesterday, but people told me April 1 was the wrong day to announce any big improvements in totem-gstreamer DVD support :-)

DVD playback in GStreamer 1.0

Some time in 2012, the GStreamer team was busy working toward the GStreamer 1.0 major release. Along the way, I did my part and ported the DVD playback components from 0.10. DVD is a pretty complex playback scenario (let’s not talk about Blu-ray)

I gave a talk about it at the GStreamer conference way back in 2010 – video here. Apart from the content of that talk, the thing I liked most was that I used Totem as my presentation tool :)

With all the nice changes that GStreamer 1.0 brought, DVD playback worked better than ever. I was able to delete a bunch of hacks and workarounds from the 0.10 days. There have been some bugs, but mostly minor things. Recently though, I became aware of a whole class of DVDs that didn’t work for a very silly reason. The symptom was that particular discs would error out at the start with a cryptic “The stream is in the wrong format” message.

It turns out that these are DVDs that begin with a piece of video that has no sound.

Sometimes, that’s implemented on a disc as a video track with accompanying silence, but in the case that was broken the DVDs have no audio track for that initial section at all. For a normal file, GStreamer would handle that by not creating any audio decoder chain or audio sink output element and just decode and play video. For DVD though, there are very few discs that are entirely without audio – so we’re going to need the audio decoder chain sooner or later. There’s no point creating and destroying when the audio track appears and disappears.

Accordingly, we create an audio output pad, and GStreamer plugs in a suitable audio output sink, and then nothing happens because the pipeline can’t get to the Playing state – the pipeline is stuck in the Paused state. Before a pipeline can start playing, it has to progress through Ready and Paused and then to Playing state. The key to getting from Paused to Playing is that each output element (video sink and audio sink) in our case, has to receive some data and be ready to output it. A process called Pre-roll. Pre-rolling the pipeline avoids stuttering at the start, because otherwise the decoders would have to race to try and deliver something in time for it to get on screen.

With no audio track, there’s no actual audio packets to deliver, and the audio sink can’t Pre-roll. The solution in GStreamer 1.0 is a GAP event, sent to indicate that there is a space in the data, and elements should do whatever they need to to skip or fill it. In the audio sink’s case it should handle it by considering itself Pre-rolled and allowing the pipeline to go to Playing, starting the ring buffer and the audio clock – from which the rest of the pipeline will be timed.

Everything up to that point was working OK – the sink received the GAP event… and then errored out. It expects to be told what format the audio samples it’s receiving are so it knows how to fill in the gap… when there’s no audio track and no audio data, it was never being told.

In the end, the fix was to make the dummy place-holder audio decoder choose an audio sample format if it gets a GAP event and hasn’t received any data yet – any format, it doesn’t really matter as long as it’s reasonable. It’ll be discarded and a new format selected and propagated when some audio data really is encountered later in playback.

That fix is #c24a12 – later fixed up a bit by thiagoss to add the ‘sensible’ part to format selection. The initial commit liked to choose a samplerate of 1Hz :)

If you have any further bugs in your GStreamer DVD playback, please let us know!

Network clock examples

Way back in 2006, Andy Wingo wrote some small scripts for GStreamer 0.10 to demonstrate what was (back then) a fairly new feature in GStreamer – the ability to share a clock across the network and use it to synchronise playback of content across different machines.

Since GStreamer 1.x has been out for over 2 years, and we get a lot of questions about how to use the network clock functionality, it’s a good time for an update. I’ve ported the simple examples for API changes and to use the gobject-introspection based Python bindings and put them up on my server.

To give it a try, fetch play-master.py and play-slave.py onto 2 or more computers with GStreamer 1 installed. You need a media file accessible via some URI to all machines, so they have something to play.

Then, on one machine run play-master.py, passing a URI for it to play and a port to publish the clock on:

./play-master.py http://server/path/to/file 8554

The script will print out a command line like so:

Start slave as: python ./play-slave.py http://server/path/to/file [IP] 8554 1071152650838999

On another machine(s), run the printed command, substituting the IP address of the machine running the master script.

After a moment or two, the slaved machine should start playing the file in synch with the master:

Network Synchronised Playback

If they’re not in sync, check that you have the port you chose open for UDP traffic so the clock synchronisation packets can be transferred.

This basic technique is the core of my Aurena home media player system, which builds on top of the network clock mechanism to provide file serving and a simple shuffle playlist.

For anyone still interested in GStreamer 0.10 – Andy’s old scripts can be found on his server: play-master.py and play-slave.py

2014 GStreamer Conference

I’ve been home from Europe over a week, after heading to Germany for the annual GStreamer conference and Linuxcon Europe.

We had a really great turnout for the GStreamer conference this year

GstConf2k14

as well as an amazing schedule of talks. All the talks were recorded by Ubicast, who got all the videos edited and uploaded in record time. The whole conference is available for viewing at http://gstconf.ubicast.tv/channels/#gstreamer-conference-2014

I gave one of the last talks of the schedule – about my current work adding support for describing and handling stereoscopic (3D) video. That support should land upstream sometime in the next month or two, so more on that in a bit.

elephant

There were too many great talks to mention them individually, but I was excited by 3 strong themes across the talks:

  • WebRTC/HTML5/Web Streaming support
  • Improving performance and reducing resource usage
  • Building better development and debugging tools

I’m looking forward to us collectively making progress on all those things and more in the upcoming year.

OSSbarcamp 2 – GNOME 3.0 talk

I gave a talk at the second Dublin OSSbarcamp yesterday. My goal was to provide some insight into the goals for GNOME 3.0 for people who didn’t attend GCDS.

Actually, the credit for the entire talk goes to Vincent and friends, who gave the GNOME 3.0 overview during the GUADEC opening at GCDS and to Owen for his GNOME Shell talk. I stole content from their slides shamelessly.

The slides are available in ODP form, or as a PDF

New gst-rpicamsrc features

I’ve pushed some new changes to my Raspberry Pi camera GStreamer wrapper, at https://github.com/thaytan/gst-rpicamsrc/

These bring the GStreamer element up to date with new features added to raspivid since I first started the project, such as adding text annotations to the video, support for the 2nd camera on the compute module, intra-refresh and others.

Where possible, you can now dynamically update any of the properties – where the firmware supports it. So you can implement digital zoom by adjusting the region-of-interest (roi) properties on the fly, or update the annotation or change video effects and colour balance, for example.

The timestamps produced are now based on the internal STC of the Raspberry Pi, so the audio video sync is tighter. Although it was never terrible, it’s now more correct and slightly less jittery.

The one major feature I haven’t enabled as yet is stereoscopic handling. Stereoscopic capture requires 2 cameras attached to a Raspberry Pi Compute Module, so at the moment I have no way to test it works.

I’m also working on GStreamer stereoscopic handling in general (which is coming along). I look forward to releasing some of that code soon.

 

Mysterious Parcel

I received a package in the mail today!
Mysterious Package

Everything arrived all nicely packaged up in a hobby box and ready for assembly.
Opening the box

Lots of really interesting goodies in the box!
Out on the table

After a little while, I’ve got the first part together.First part assembled

The rest will have to wait for another day. In the meantime, have fun guessing what it is, and enjoy this picture of a cake I baked on the weekend:
Strawberry Sponge Cake

See you later!

A glimpse of audio nirvana

This is post is basically a love letter to the Pulseaudio and Gnome Bluetooth developers.

I upgraded my laptop to Ubuntu Karmic recently, which brought with it the ability to use my Bluetooth A2DP headphones natively. Getting them running is now as simple as using the Bluetooth icon in the panel to pair the laptop with the headphones, and then selecting them in the Sound Preferences applet, on the Output tab.

As soon as the headphones are connected, they show up as a new audio device. Selecting it instantly (and seamlessly) migrates my sounds and music from the laptop sound device onto the headphones. The Play/Pause, Next Track and Previous Track buttons all generate media key keypresses – so Rhythmbox and Totem behave like they’re supposed to. It’s lovely.

If that we’re all, it would already be pretty sweet in my opinion, but wait – there’s more!

A few days after starting to use my bluetooth headphones, my wife and I took a trip to Barcelona (from Dublin, where we live for the next few weeks… more on that later). When we got to the airport, the first thing we learned was that our flight had been delayed by 3 hours. Since I occasionally hack on multimedia related things, I typically have a few DVDs
with me for testing. In this case, I had Vicky Christina Barcelona on hand, and we hadn’t watched it yet – a perfect choice for 2 people on their way to Barcelona.

Problem! There are four sets of ears wanting to listen to the DVD, and only 2 audio channels produced. I could choose to send the sound to either the in built sound device, and listen on the earbuds my wife had, or I could send it to my bluetooth headphones, but not both.

Pulseaudio to the rescue! With a bit of command-line fu (no GUI for this, but that’s totally do-able), I created a virtual audio device, using Pulseaudio’s “combine” module. Like the name suggests, it combines multiple other audio devices into a single one. It can do more complex combinations (such as sending some channels hither and others thither), but I just needed a straight mirroring of the devices. In a terminal, I ran:

pactl load-module module-combine sink_name=shared_play adjust_time=3 slaves=”alsa_output.pci-0000_00_1b.0.analog-stereo,bluez_sink.00_15_0F_72_70_E1″

Hey presto! Now there’s a third audio device available in the Sound Preferences to send the sound to, and it comes out both the wired ear buds and my bluetooth headphones (with a very slight sync offset, but close enough for my purposes).

Also, for those interested – the names of the 2 audio devices in my pactl command line came from the output of ‘pactl list’.

This kind of seamless migration of running audio streams really isn’t possible to do without something like Pulseaudio that can manage stream routing on the fly. I’m well aware that Pulseaudio integration into the distributions has been a bumpy ride for lots of people, but I think the end goal justifies the painful process of fixing all the sound drivers. I hope you do too!

edit
Lennart points out that the extra paprefs application has a “Add virtual output device for simultaneous output on all local sound cards” check-box that does the same thing as loading the combine module manual, but also handles hot-plugging of devices as they appear and disappear.

Planet Linux Australia... rebooted

Recently Linux Australia needed to move its infrastructure to a different place, and so we took the opportunity to build a fresh new instance of the Planet Linux Australia blog aggregator.

It made me realise how crusty the old site had become, how many things I had planned to do which I had left undone, and how I hadn't applied simple concepts such as Infrastructure as Code which have become accepted best-practices in the time since I originally set this up.

Of course things have changed in this time.  People blog less now, so I've also taken the opportunity to remove what appear to be dead blogs from the aggregator.   If you have a blog of interest to the Linux Australia community, you can ask to be added via emailing planet at linux dot org dot au. All you need is a valid Atom or RSS feed.

The other thing that is that the blog aggregator software we use hasn't seen an update since 2011. It started out as PlanetPlanet, then moved on to Venus, and so I've taken a fork to hopefully improve this some more when I find my round tuit. Fortunately I don't still need to run it under python 2.4 which is getting a little long in the tooth.

Finally, the config for Planet Linux Australia is up on github.  Just like the venus code itself, pull requests welcome.  Share and Enjoy :-)

Simple Squid access log reporting.

Squid is one of the biggest and most used proxies on the interwebs. And generating reports from the access logs is already a done deal, there are many commercial and OSS apps that support the squid log format. But I found my self in a situation where I wanted stats but didn’t want to install a web server on my proxy or use syslog to push my logs to a centralised server which was running such software, and also wasn’t in a position to go buy one of those off the shelf amazing wiz bang Squid reporting and graphing tools.

As a Linux geek I surfed the web to see what others have done. I came across a list provided by the Squid website. Following a couple of links, I came across a awk script called ‘proxy_stats.gawk’ written by Richard Huveneers.

I downloaded it and tried it out… unfortunately it didn’t work, looking at the code.. which he nicely commented showed that he had it set up for access logs  from version 1.* of squid. Now the squid access log format from squid 2.6+ hasn’t changed too much from version 1.1. all they have really done is add a “content type” entry at the end of each line.

So as a good Linux geek does, he upgrades the script, my changes include:

  • Support for squid 2.6+
  • Removed the use a deprecated switches that now isn’t supported in the sort command.
  • Now that there is a an actual content type “column” lets use it to improve the ‘Object type report”.
  • Add a users section, as this was an important report I required which was missing.
  • And in a further hacked version, an auto generated size of the first “name” column.

Now with the explanation out of the way, let me show you it!

For those who are new to awk, this is how I’ve been running it:

zcat <access log file> | awk -f proxy_stats.gawk > <report-filename>

NOTE: I’ve been using it for some historical analysis, so I’m running it on old rotated files, which are compressed thus the zcat.

You can pass more then one file at a time and it order doesn’t matter, as each line of an access log contains the date in epoch time:

zcat `find /var/log/squid/ -name "access.log*"` |awk -f proxy_stats.gawk

The script produces an ascii report (See end of blog entry for example), which could be generated and emailed via cron. If you want it to look nice in any email client using html the I suggest wrapping it in <pre> tags.:

<html>
<head><title>Report Title</title></head>
Report title<body>
<pre>
... Report goes here ...
</pre>
</body>
</html>

For those experienced Linux sys admins out there using cron + ‘find -mtime’ would be a very simple way of having an automated daily, weekly or even monthly report.
But like I said earlier I was working on historic data, hundreds of files in a single report, hundreds because for business reasons we have been rotating the squid logs every hour… so I did what I do best, write a quick bash script to find all the files I needed to cat into the report:

#!/bin/bash

ACCESS_LOG_DIR="/var/log/squid/access.log*"
MONTH="$1"

function getFirstLine() {
	if [ -n  "`echo $1 |grep "gz$"`" ]
	then
		zcat $1 |head -n 1
	else
		head -n 1 $1 
	fi
}

function getLastLine() {
	if [ -n  "`echo $1 |grep "gz$"`" ]
	then
		zcat $1 |tail -n 1
	else
		tail -n 1 $1 
	fi
}

for log in `ls $ACCESS_LOG_DIR`
do
	firstLine="`getFirstLine $log`"
	epochStr="`echo $firstLine |awk '{print $1}'`"
	month=`date -d @$epochStr +%m`
	
	if [ "$month" -eq "$MONTH" ]
	then
		echo $log
		continue
	fi

	
	#Check the last line
	lastLine="`getLastLine $log`"
	epochStr="`echo $lastLine |awk '{print $1}'`"
        month=`date -d @$epochStr +%m`

        if [ "$month" -eq "$MONTH" ]
        then
                echo $log
        fi
	
done

So there you go, thanks to the work of Richard Huveneers there is a script that I think generates a pretty good acsii report, which can be automated or integrated easily into any Linux/Unix work flow.

If you interested in getting hold of the most up to date version of the script you can get it from my sysadmin github repo here.

As promised earlier here is an example report:

Parsed lines  : 32960
Bad lines     : 0

First request : Mon 30 Jan 2012 12:06:43 EST
Last request  : Thu 09 Feb 2012 09:05:01 EST
Number of days: 9.9

Top 10 sites by xfers           reqs   %all %xfers   %hit         MB   %all   %hit     kB/xf      kB/s
------------------------- ------------------------------- ------------------------ -------------------
213.174.155.216                   20   0.1% 100.0%   0.0%        0.0   0.0%   0.0%       1.7       2.5
30.media.tumblr.com                1   0.0% 100.0%   0.0%        0.0   0.0%   0.0%      48.3      77.4
28.media.tumblr.com                1   0.0% 100.0%   0.0%        0.1   0.0%   0.0%      87.1       1.4
26.media.tumblr.com                1   0.0%   0.0%      -        0.0   0.0%      -         -         -
25.media.tumblr.com                2   0.0% 100.0%   0.0%        0.1   0.0%   0.0%      49.2      47.0
24.media.tumblr.com                1   0.0% 100.0%   0.0%        0.1   0.0%   0.0%     106.4     181.0
10.1.10.217                      198   0.6% 100.0%   0.0%       16.9   0.9%   0.0%      87.2    3332.8
3.s3.envato.com                   11   0.0% 100.0%   0.0%        0.1   0.0%   0.0%       7.6      18.3
2.s3.envato.com                   15   0.0% 100.0%   0.0%        0.1   0.0%   0.0%       7.5      27.1
2.media.dorkly.cvcdn.com           8   0.0% 100.0%  25.0%        3.2   0.2%   0.3%     414.1     120.5

Top 10 sites by MB              reqs   %all %xfers   %hit         MB   %all   %hit     kB/xf      kB/s
------------------------- ------------------------------- ------------------------ -------------------
zulu.tweetmeme.com                 2   0.0% 100.0% 100.0%        0.0   0.0% 100.0%       3.1     289.6
ubuntu.unix.com                    8   0.0% 100.0% 100.0%        0.1   0.0% 100.0%       7.5     320.0
static02.linkedin.com              1   0.0% 100.0% 100.0%        0.0   0.0% 100.0%      36.0     901.0
solaris.unix.com                   2   0.0% 100.0% 100.0%        0.0   0.0% 100.0%       3.8     223.6
platform.tumblr.com                2   0.0% 100.0% 100.0%        0.0   0.0% 100.0%       1.1     441.4
i.techrepublic.com.com             5   0.0%  60.0% 100.0%        0.0   0.0% 100.0%       6.8    2539.3
i4.zdnetstatic.com                 2   0.0% 100.0% 100.0%        0.0   0.0% 100.0%      15.3     886.4
i4.spstatic.com                    1   0.0% 100.0% 100.0%        0.0   0.0% 100.0%       4.7     520.2
i2.zdnetstatic.com                 2   0.0% 100.0% 100.0%        0.0   0.0% 100.0%       7.8    2920.9
i2.trstatic.com                    9   0.0% 100.0% 100.0%        0.0   0.0% 100.0%       1.5     794.5

Top 10 neighbor report          reqs   %all %xfers   %hit         MB   %all   %hit     kB/xf      kB/s
------------------------- ------------------------------- ------------------------ -------------------
www.viddler.com                    4   0.0% 100.0%   0.0%        0.0   0.0%      -       0.0       0.0
www.turktrust.com.tr              16   0.0% 100.0%   0.0%        0.0   0.0%      -       0.0       0.0
www.trendmicro.com                 5   0.0% 100.0%   0.0%        0.0   0.0%      -       0.0       0.0
www.reddit.com                     2   0.0% 100.0%   0.0%        0.0   0.0%      -       0.0       0.0
www.linkedin.com                   2   0.0% 100.0%   0.0%        0.0   0.0%      -       0.0       0.0
www.google-analytics.com           2   0.0% 100.0%   0.0%        0.0   0.0%      -       0.0       0.0
www.facebook.com                   2   0.0% 100.0%   0.0%        0.0   0.0%      -       0.0       0.0
www.dynamicdrive.com               1   0.0% 100.0%   0.0%        0.0   0.0%      -       0.0       0.0
www.benq.com.au                    1   0.0% 100.0%   0.0%        0.0   0.0%      -       0.0       0.0
wd-edge.sharethis.com              1   0.0% 100.0%   0.0%        0.0   0.0%      -       0.0       0.0

Local code                      reqs   %all %xfers   %hit         MB   %all   %hit     kB/xf      kB/s
------------------------- ------------------------------- ------------------------ -------------------
TCP_CLIENT_REFRESH_MISS         2160   6.6% 100.0%   0.0%        7.2   0.4%   0.0%       3.4      12.9
TCP_HIT                          256   0.8% 100.0%  83.2%       14.0   0.8% 100.0%      56.0    1289.3
TCP_IMS_HIT                      467   1.4% 100.0% 100.0%       16.9   0.9% 100.0%      37.2    1747.4
TCP_MEM_HIT                      426   1.3% 100.0% 100.0%       96.5   5.3% 100.0%     232.0    3680.9
TCP_MISS                       27745  84.2%  97.4%   0.0%     1561.7  85.7%   0.3%      59.2      18.2
TCP_REFRESH_FAIL                  16   0.0% 100.0%   0.0%        0.2   0.0%   0.0%      10.7       0.1
TCP_REFRESH_MODIFIED             477   1.4%  99.8%   0.0%       35.0   1.9%   0.0%      75.3    1399.4
TCP_REFRESH_UNMODIFIED          1413   4.3% 100.0%   0.0%       91.0   5.0%   0.0%      66.0     183.5

Status code                     reqs   %all %xfers   %hit         MB   %all   %hit     kB/xf      kB/s
------------------------- ------------------------------- ------------------------ -------------------
000                              620   1.9% 100.0%   0.0%        0.0   0.0%      -       0.0       0.0
200                            29409  89.2% 100.0%   2.9%     1709.7  93.8%   7.7%      59.5     137.1
204                              407   1.2% 100.0%   0.0%        0.2   0.0%   0.0%       0.4       1.4
206                              489   1.5% 100.0%   0.0%      112.1   6.1%   0.0%     234.7     193.0
301                               82   0.2% 100.0%   0.0%        0.1   0.0%   0.0%       0.7       1.5
302                              356   1.1% 100.0%   0.0%        0.3   0.0%   0.0%       0.8       2.7
303                                5   0.0% 100.0%   0.0%        0.0   0.0%   0.0%       0.7       1.5
304                              862   2.6% 100.0%  31.2%        0.4   0.0%  30.9%       0.4      34.2
400                                1   0.0%   0.0%      -        0.0   0.0%      -         -         -
401                                1   0.0%   0.0%      -        0.0   0.0%      -         -         -
403                               47   0.1%   0.0%      -        0.0   0.0%      -         -         -
404                              273   0.8%   0.0%      -        0.0   0.0%      -         -         -
500                                2   0.0%   0.0%      -        0.0   0.0%      -         -         -
502                               12   0.0%   0.0%      -        0.0   0.0%      -         -         -
503                               50   0.2%   0.0%      -        0.0   0.0%      -         -         -
504                              344   1.0%   0.0%      -        0.0   0.0%      -         -         -

Hierarchie code                 reqs   %all %xfers   %hit         MB   %all   %hit     kB/xf      kB/s
------------------------- ------------------------------- ------------------------ -------------------
DIRECT                         31843  96.6%  97.7%   0.0%     1691.0  92.8%   0.0%      55.7      44.3
NONE                            1117   3.4% 100.0% 100.0%      131.6   7.2% 100.0%     120.7    2488.2

Method report                   reqs   %all %xfers   %hit         MB   %all   %hit     kB/xf      kB/s
------------------------- ------------------------------- ------------------------ -------------------
CONNECT                         5485  16.6%  99.2%   0.0%      132.8   7.3%   0.0%      25.0       0.3
GET                            23190  70.4%  97.7%   4.9%     1686.3  92.5%   7.8%      76.2     183.2
HEAD                            2130   6.5%  93.7%   0.0%        0.7   0.0%   0.0%       0.3       1.1
POST                            2155   6.5%  99.4%   0.0%        2.9   0.2%   0.0%       1.4       2.0

Object type report              reqs   %all %xfers   %hit         MB   %all   %hit     kB/xf      kB/s
------------------------- ------------------------------- ------------------------ -------------------
*/*                                1   0.0% 100.0%   0.0%        0.0   0.0%   0.0%       1.6       3.2
application/cache-digest         396   1.2% 100.0%  50.0%       33.7   1.8%  50.0%      87.1    3655.1
application/gzip                   1   0.0% 100.0%   0.0%        0.1   0.0%   0.0%      61.0      30.8
application/javascript           227   0.7% 100.0%  12.3%        2.2   0.1%   7.7%       9.9      91.9
application/json                 409   1.2% 100.0%   0.0%        1.6   0.1%   0.0%       4.1       6.0
application/ocsp-response        105   0.3% 100.0%   0.0%        0.2   0.0%   0.0%       1.9       2.0
application/octet-stream         353   1.1% 100.0%   6.8%       81.4   4.5%   9.3%     236.1     406.9
application/pdf                    5   0.0% 100.0%   0.0%       13.5   0.7%   0.0%    2763.3      75.9
application/pkix-crl              96   0.3% 100.0%  13.5%        1.0   0.1%   1.7%      10.6       7.0
application/vnd.google.sa       1146   3.5% 100.0%   0.0%        1.3   0.1%   0.0%       1.1       2.4
application/vnd.google.sa       4733  14.4% 100.0%   0.0%       18.8   1.0%   0.0%       4.1      13.4
application/x-bzip2               19   0.1% 100.0%   0.0%       78.5   4.3%   0.0%    4232.9     225.5
application/x-gzip               316   1.0% 100.0%  59.8%      133.4   7.3%  59.3%     432.4    3398.1
application/x-javascript        1036   3.1% 100.0%   5.8%        9.8   0.5%   3.4%       9.7      52.1
application/xml                   46   0.1% 100.0%  34.8%        0.2   0.0%  35.1%       3.5     219.7
application/x-msdos-progr        187   0.6% 100.0%   0.0%       24.4   1.3%   0.0%     133.7     149.6
application/x-pkcs7-crl           83   0.3% 100.0%   7.2%        1.6   0.1%   0.4%      19.8      10.8
application/x-redhat-pack         13   0.0% 100.0%   0.0%       57.6   3.2%   0.0%    4540.7     156.7
application/x-rpm                507   1.5% 100.0%   6.3%      545.7  29.9%   1.5%    1102.2     842.8
application/x-sdlc                 1   0.0% 100.0%   0.0%        0.9   0.0%   0.0%     888.3     135.9
application/x-shockwave-f        109   0.3% 100.0%  11.9%        5.4   0.3%  44.5%      50.6     524.1
application/x-tar                  9   0.0% 100.0%   0.0%        1.5   0.1%   0.0%     165.3      36.4
application/x-www-form-ur         11   0.0% 100.0%   0.0%        0.1   0.0%   0.0%       9.9      15.4
application/x-xpinstall            2   0.0% 100.0%   0.0%        2.5   0.1%   0.0%    1300.6     174.7
application/zip                 1802   5.5% 100.0%   0.0%      104.0   5.7%   0.0%      59.1       2.5
Archive                           89   0.3% 100.0%   0.0%        0.0   0.0%      -       0.0       0.0
audio/mpeg                         2   0.0% 100.0%   0.0%        5.8   0.3%   0.0%    2958.2      49.3
binary/octet-stream                2   0.0% 100.0%   0.0%        0.0   0.0%   0.0%       5.5      14.7
font/ttf                           2   0.0% 100.0%   0.0%        0.0   0.0%   0.0%      15.5      12.5
font/woff                          1   0.0% 100.0% 100.0%        0.0   0.0% 100.0%      42.5    3539.6
Graphics                         126   0.4% 100.0%   0.0%        0.1   0.0%   0.0%       0.6       2.5
HTML                              14   0.0% 100.0%   0.0%        0.0   0.0%   0.0%       0.1       0.1
image/bmp                          1   0.0% 100.0%   0.0%        0.0   0.0%   0.0%       1.3       3.9
image/gif                       5095  15.5% 100.0%   2.4%       35.9   2.0%   0.7%       7.2       9.5
image/jpeg                      1984   6.0% 100.0%   4.3%       52.4   2.9%   0.6%      27.0      62.9
image/png                       1684   5.1% 100.0%  10.3%       28.6   1.6%   1.9%      17.4     122.2
image/vnd.microsoft.icon          10   0.0% 100.0%  30.0%        0.0   0.0%  12.8%       1.0       3.3
image/x-icon                      72   0.2% 100.0%  16.7%        0.2   0.0%   6.0%       3.2      15.0
multipart/bag                      6   0.0% 100.0%   0.0%        0.1   0.0%   0.0%      25.2      32.9
multipart/byteranges              93   0.3% 100.0%   0.0%       16.5   0.9%   0.0%     182.0     178.4
text/cache-manifest                1   0.0% 100.0%   0.0%        0.0   0.0%   0.0%       0.7       3.1
text/css                         470   1.4% 100.0%   7.9%        3.4   0.2%   5.8%       7.4      59.7
text/html                       2308   7.0%  70.7%   0.4%        9.6   0.5%   0.6%       6.0      14.7
text/javascript                 1243   3.8% 100.0%   2.7%       11.1   0.6%   5.2%       9.1      43.3
text/json                          1   0.0% 100.0%   0.0%        0.0   0.0%   0.0%       0.5       0.7
text/plain                      1445   4.4%  99.4%   1.5%       68.8   3.8%   5.5%      49.0      41.9
text/x-cross-domain-polic         24   0.1% 100.0%   0.0%        0.0   0.0%   0.0%       0.7       1.7
text/x-js                          2   0.0% 100.0%   0.0%        0.0   0.0%   0.0%      10.1       6.4
text/x-json                        9   0.0% 100.0%   0.0%        0.0   0.0%   0.0%       3.0       8.5
text/xml                         309   0.9% 100.0%  12.9%       12.9   0.7%  87.5%      42.8     672.3
unknown/unknown                 6230  18.9%  99.3%   0.0%      132.9   7.3%   0.0%      22.0       0.4
video/mp4                          5   0.0% 100.0%   0.0%        3.2   0.2%   0.0%     660.8      62.7
video/x-flv                      117   0.4% 100.0%   0.0%      321.6  17.6%   0.0%    2814.9     308.3
video/x-ms-asf                     2   0.0% 100.0%   0.0%        0.0   0.0%   0.0%       1.1       4.7

Ident (User) Report             reqs   %all %xfers   %hit         MB   %all   %hit     kB/xf      kB/s
------------------------- ------------------------------- ------------------------ -------------------
-                              32960 100.0%  97.8%   3.5%     1822.6 100.0%   7.2%      57.9     129.0

Weekly report                   reqs   %all %xfers   %hit         MB   %all   %hit     kB/xf      kB/s
------------------------- ------------------------------- ------------------------ -------------------
2012/01/26                     14963  45.4%  97.6%   3.6%      959.8  52.7%   1.8%      67.3     104.5
2012/02/02                     17997  54.6%  98.0%   3.4%      862.8  47.3%  13.2%      50.1     149.4

Total report                    reqs   %all %xfers   %hit         MB   %all   %hit     kB/xf      kB/s
------------------------- ------------------------------- ------------------------ -------------------
All requests                   32960 100.0%  97.8%   3.5%     1822.6 100.0%   7.2%      57.9     129.0

Produced by : Mollie's hacked access-flow 0.5
Running time: 2 seconds

Happy squid reporting!

Use xmllint and vim to format xml documents

If you want vim to nicely format an XML file (and a xena file in this example, 2nd line) then add this to your ~/.vimrc file:
" Format *.xml and *.xena files by sending them to xmllint
au FileType xml exe ":silent 1,$!xmllint --format --recover - 2>/dev/null"
au FileType xena exe ":silent 1,$!xmllint --format --recover - 2>/dev/null"

This uses the xmllint command to format the xml file.. useful on xml docs that aren’t formatted in the file.

I’m now an OpenStack developer.

Hello world,

It’s been a while since I have blogged on this site, I apologise for that. My previous position was a tad proprietary, so although I worked with Linux, what I was doing needs to be sanitised before I can post about it. I have a bunch of posts in the cooker from those days still awaiting sanitation. But I have some great news… I am now an Openstack developer.

It’s been a busy year, married moved over to the UK to work for an amazing company who needs no introduction, Rackspace. Over there I was working with Linux in a Support/DevOps style role, but am back in Oz now with a new team at Rackspace! The Rackspace Cloud Builders. In this role I’ll be getting my development hat on and developing for upstream Openstack again and am so excited about it.

Watch this space!!!

Matt

chkconfig-ify an exising init script.

If you are using a 3rd party application / package installer to install a service onto a system that using chkconfig to manage your run-levels, or writing your own which are incompatible with chkconfig. That is to say when trying to add them you get the following error:

# chkconfig <service> on
service <service> does not support chkconfig

Then it needs to be converted to support chkconfig. Don’t worry, it isn’t a rewrite, its just adding some meta-data to the init script.
Just edit the config and add the following lines just below the sha-bang (#!/bin/bash or #!/bin/sh).

# chkconfig: 2345 95 05
# description:
# processname:

NOTE: The numbers on the chkconfig line mean:

That on runlevels 2,3,4 and 5, this subsystem will be activated with priority 95 (one of the lasts), and deactivated with priority 05 (one of the firsts).

The above quote comes from this post where I found this solution, so I am passing it on.

For those playing along at home, chkconfig is the Redhat/Centos/Fedora way of managing your run-levels.

Centos 4 / RHEL 4 Bind 9.7.3-8 RPMs.

In case anyone out there in internet land happen to have a BIND DNS server still running RHEL 4 or Centos 4 and require a version that has been back ported from the Centos 6.2 source, one that has the CVE-2012-1667 fix. Then you can download the RPMs I build from here.

NOTE: I’ve only just built them, so haven’t tested them yet, but thought it’ll be better to share. Also they aren’t x86_64, if you need them, let me know and I’ll build some.

Debian 6 GNU/KFreeBSD Grub problems on VirtualBox

Debian 6 was released the other day, with this release they not only released a Linux kernel version but they now support a FreeBSD version as well!
So I decided to install it under VirtualBox and check it out…

The install process went smoothly until I got to the end when it was installing and setting up grub2. It installed ok on the MBR but got an error in the installer while trying to set it up. I jumped into the console to take a look around.

I started off trying to run the update-grub command which fails silently (checking $? shows the return code of 1). On closer inspection I noticed the command created an incomplete grub config named /boot/grub/grub.cfg.new

So all we need to do is finish off this config file. So jump back into the installer and select continue without boot loader, this will pop up a message about what you must set the root partition as when you do set up a boot loader, so take note of it.. mine was /dev/ad0s5.

OK, with that info we can finish off our config file. Firstly lets rename the incomplete one:
cp /boot/grub/grub.cfg.new /boot/grub/grub.cfg

Now my /boot/grub/grub.cfg ended like:
### BEGIN /etc/grub.d/10_kfreebsd ###
menuentry 'Debian GNU/kFreeBSD, with kFreeBSD 8.1-1-amd64' --class debian --class gnu-kfreebsd --class gnu --class os {
insmod part_msdos
insmod ext2


set root='(hd0,1)'
search --no-floppy --fs-uuid --set dac05f8a-2746-4feb-a29d-31baea1ce751
echo 'Loading kernel of FreeBSD 8.1-1-amd64 ...'
kfreebsd /kfreebsd-8.1-1-amd64.gz

So I needed to add the following to finish it off (note this I’ll repeat that last part):
### BEGIN /etc/grub.d/10_kfreebsd ###
menuentry 'Debian GNU/kFreeBSD, with kFreeBSD 8.1-1-amd64' --class debian --class gnu-kfreebsd --class gnu --class os {
insmod part_msdos
insmod ext2
insmod ufs2


set root='(hd0,1)'
search --no-floppy --fs-uuid --set dac05f8a-2746-4feb-a29d-31baea1ce751
echo 'Loading kernel of FreeBSD 8.1-1-amd64 ...'
kfreebsd /kfreebsd-8.1-1-amd64.gz
set kFreeBSD.vfs.root.mountfrom=ufs:/dev/ad0s5
set kFreeBSD.vfs.root.mountfrom.options=rw
}

Note: My root filesytem was UFS, thus the ‘ufs:/dev/ad0s5′ in the mountfrom option.

That’s it, you Debian GNU/kFreeBSD should now boot successfully :)

NTLM Authentication in Squid using Winbind.

Some old windows servers require authentication through the old NTLM protocol, luckily with the help from squid, samba and winbind we can do this under Linux.

Some URLs a much of this information was gathered from are:

  • http://wiki.squid-cache.org/ConfigExamples/Authenticate/NtlmCentOS5
  • http://wiki.squid-cache.org/ConfigExamples/Authenticate/Ntlm

HOW TO

In order to authenticate through winbind we will be using that and samba to connect to a windows domain, so you will need to have a domain and the details for it or all this will be for naught. I’ll use some fake credentials for this post.

Required Packages
Let’s install all the required packages:

yum install squid krb5-workstation samba-common ntp samba-winbind authconfig

NTP (Network Time Protocol)
Kerberos and windbind can be a little thingy about date and time, so its a good idea to use NTP for your network, I’ll assume your domain controller (DC) will be also your NTP server in which case lets set it up.

Comment out any lines that begin with server and create only one that points to your Active Directory PDC.

# vim /etc/ntp.conf
server pdc.test.lan

Now add it to the default runlevels and start it.

chkconfig ntpd on
/etc/init.d/ntpd start

Samba, Winbind and Kerberos
We will the use the authconfig package/command we installed earlier to configure Samba, Winbind and perform the join in one step, this makes things _SO_ much
easier!!!

NOTE: If you don’t have DNS set up then you will need to add the DC to your hosts file, and it is important to use the name the DC machine knows itself as in AD.


authconfig --enableshadow --enablemd5 --passalgo=md5 --krb5kdc=pdc.test.lan \
--krb5realm=TEST.LAN --smbservers=pdc.test.lan --smbworkgroup=TESTLAN \
--enablewinbind --enablewinbindauth --smbsecurity=ads --smbrealm=TEST.LAN \
--smbidmapuid="16777216-33554431" --smbidmapgid="16777216-33554431" --winbindseparator="+" \
--winbindtemplateshell="/bin/false" --enablewinbindusedefaultdomain --disablewinbindoffline \
--winbindjoin=administrator --disablewins --disablecache --enablelocauthorize --updateall

NOTE: Replace pdc.test.lan with that of your FQDN of your DC server, TESTLAN with your domain, TEST.LAN with the full name of the domain/realm, and make sure you set ‘–winbindjoin’ with a domain admin.

If that succeeds lets test it:

# wbinfo -u
# wbinfo -g

If you are able to enumerate your Active Directory Groups and Users, everything is working.

Next lets test that we can authenticate with winbind:

# wbinfo -a

E.G:

# wbinfo -a testuser
Enter testuser's password:
plaintext password authentication succeeded
Enter testuser's password:
challenge/response password authentication succeeded

Great, we have been added to the domain, so now we can setup squid for NTLM authentication.

SQUID Configuration
Squid comes with its own ntlm authentication binary (/usr/lib64/squid/ntlm_smb_lm_auth) which uses winbind, but as of Samba 3.x, samba bundle their own which is the recommended binary to use (according to the squid and samba projects). So the binary we use comes from the samba-winbind package we installed earlier:

/usr/bin/ntlm_auth

Add the following configuration elements to the squid.conf to enable NTLM authentication:

#NTLM
auth_param ntlm program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
auth_param ntlm children 5
auth_param ntlm keep_alive on

acl ntlm proxy_auth REQUIRED
http_access allow ntlm

NOTE: The above is allowing anyone access as long as they authenticate themselves via NTLM, you could use further acl’s to restrict this more.

The ntlm_auth binary has other switches that might be of use, such as restricting users by group membership:

auth_param ntlm program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp --require-membership-of=EXAMPLE+ADGROUP

Before we are complete there is one more thing we need to do, for squid to be allowed to use winbind, the squid user (which was created when the squid package was installed) needs to be a member of a wbpriv group:

gpasswd -a squid wbpriv

IMPORTANT!
NTLM authentication WILL FAIL if you have “cache_effective_group squid” set, if you do then remove it! As this overrides the effective group and squid then isn’t seen as part of the ‘wbpriv’ group which breaks authentication!!!
/IMPORTANT!

Add squid to the runlevels and start it:

# chkconfig squid on
# /etc/init.d/squid start

Trouble shooting
Make sure you open the port in iptables, if squid is listening on 3128 then:

# iptables -I INPUT 1 -p tcp --dport 3128 -j ACCEPT
# /etc/init.d/iptables save

NOTE: The ‘/etc/init.d/iptables save’ command saves the current running configuration so the new rule will be applied on reboot.

Happy squid-ing.

Posfix – Making sense of delays in mail

The maillog

The maillog is easy enough to follow, but when you understand what all the delay and delays numbers mean then this may help really understand what is going on!
A standard email entry in postfix looks like:

Jan 10 10:00:00 testmtr postfix/smtp[20123]: 34A1B160852B: to=, relay=mx1.example.lan[1.2.3.4]:25, delay=0.49, delays=0.2/0/0.04/0.25, dsn=2.0.0, status=sent

Pretty straight forward: date, email identifier in the mailq (34A1B160852B), recipient, which server the email is being sent to (relay). It is the delay and delays I’d like to talk about.

Delay and Delays
If we take a look at the example email from above:

Jan 10 10:00:00 testmtr postfix/smtp[20123]: 34A1B160852B: to=, relay=mx1.example.lan[1.2.3.4]:25, delay=0.49, delays=0.2/0/0.04/0.25, dsn=2.0.0, status=sent

The delay parameter (delay=0.49) is fairly self explanatory, it is the total amount of time this email (34A1B160852B) has been on this server. But what is the delays parameter all about?

delays=0.2/0/0.04/0.25

NOTE: Numbers smaller than 0.01 seconds are truncated to 0, to reduce the noise level in the logfile.

You might have guessed it is a break down of the total delay, but what do each number represent?

Well from the release notes we get:

delays=a/b/c/d:
a=time before queue manager, including message transmission;
b=time in queue manager;
c=connection setup time including DNS, HELO and TLS;
d=message transmission time.

There for looking at our example:

  • a (0.2): The time before getting to the queue manager, so the time it took to be transmitted onto the mail server and into postfix.
  • b (0): The time in queue manager, so this email didn’t hit the queues, so it was emailed straight away.
  • c (0.04): The time it took to set up a connection with the destination mail relay.
  • d (0.25): The time it took to transmit the email to the destination mail relay.

However if the email is deferred, then when the email is attempted to be sent again:

Jan 10 10:00:00 testmtr postfix/smtp[20123]: 34A1B160852B: to=, relay=mx1.example.lan[1.2.3.4]:25, delay=82, delays=0.25/0/0.5/81, dsn=4.4.2, status=deferred (lost connection with mx1.example.lan[1.2.3.4] while sending end of data -- message may be sent more than once)

Jan 10 testmtr postfix/smtp[20123]: 34A1B160852B: to=, relay=mx1.example.lan[1.2.3.4]:25, delay=1092, delays=1091/0.2/0.8/0.25, dsn=2.0.0, status=sent

This time the first entry shows how long it took before the destination mail relay took to time out and close the connection:

delays=0.25/0/0.5/81
Therefore: 81 seconds.

The email was deferred then about 15 minutes later (1009 seconds [delays – <total delay from last attempt> ]) another attempt is made.
This time the delay is a lot larger, as the total time this email has spent on the server is a lot longer.

delay=1092, delays=1091/0.2/0.8/0.25

What is interesting though is the value of ‘a’ is now 1091, which means when an email is resent the ‘a’ value in the breakdown also includes the amount of time this email has currently spend on the system (before this attempt).

So there you go, those delays values are rather interesting and can really help solve where bottlenecks lie on your system. In the above case we obviously had some problem communicating to the destination mail relay, but worked the second time, so isn’t a problem with our system… or so I’d like to think.

Identically partition disks.. the easy way!

Was just looking into a software RAID howto.. for no reason really, but kinda glad I did! When you set up software raid you want to make sure all disks are partitioned the same, right. so check this out:

3. Create partitions on /dev/sda identical to the partitions on /dev/sdb:

sfdisk -d /dev/sdb | sfdisk /dev/sda

That’s a much easier way 😉

This gem is thanks to: http://www.howtoforge.com/how-to-create-a-raid1-setup-on-an-existing-centos-redhat-6.0-system

Reverse proxy using squid + Redirection

Squid – Reverse Proxy

In computer networks, a reverse proxy is a type of proxy server that retrieves resources on behalf of a client from one or more servers. These resources are then returned to the client as though it originated from the reverse proxy itself. While a forward proxy is usually situated between the client application (such as a web browser) and the server(s) hosting the desired resources, a reverse proxy is usually situated closer to the server(s) and will only return a configured set of resources.

See: http://en.wikipedia.org/wiki/Reverse_proxy

Configuration

Squid should already be installed, if not then install it:

yum install squid

Then we edit squid config:


vim /etc/squid/squid.conf

Add we add the following to the top of the file:

http_port 80 vhost
https_port 443 cert=/etc/squid/localhost.crt key=/etc/squid/localhost.key vhost

cache_effective_user squid
cache_effective_group squid

cache_peer 1.2.3.4 parent 80 0 no-query originserver login=PASS name=site1-http
cache_peer 1.2.3.5 parent 443 0 no-query originserver login=PASS ssl sslflags=DONT_VERIFY_PEER name=site2-ssl
cache_peer_domain site1-http site1.example.lan
cache_peer_domain site2-ssl site2.anotherexample.lan

acl bad_requests urlpath_regex -i cmd.exe \/bin\/sh \/bin\/bash default\.ida?XXX insert update delete select
http_access deny bad_requests

Now I’ll walk us through the above configuration.

http_port 80 vhost
https_port 443 cert=/etc/squid/localhost.crt key=/etc/squid/localhost.key vhost

This sets the http and https ports squid is listening on. Note the cert options for https, we can get squid to use https up to the proxy and unencrytped link to the last hop if we want.. which is cool. If for some reason the server doesn’t support https.


cache_effective_user squid
cache_effective_group squid

Set the effective user and group for squid.. this may not be required, but doesn’t hurt.


cache_peer 1.2.3.4 parent 80 0 no-query originserver name=site1-http
cache_peer 1.2.3.5 parent 443 0 no-query originserver ssl sslflags=DONT_VERIFY_PEER name=site2-ssl
cache_peer_domain site1-http site1.example.lan
cache_peer_domain site2-ssl site2.anotherexample.lan

This is the magic, the first two lines, tell squid which peer to reverse proxy for and what port to use. Note if you use ssl the ‘sslflags=DONT_VERIFY_PEER’ is useful otherwise if your using a self signed cert you’ll have certificate errors.

IMPORTANT: If you want to allow http authentication (auth handled by the web server, such as htaccess) then you need to add ‘login=PASS’ otherwise squid will try and authenticate to squid rather than the http server.

The last two lines, reference the first two and tell squid the domains to listen to, so if someone connects to squid looking for that domain it knows where to go/cache.


acl bad_requests urlpath_regex -i cmd.exe \/bin\/sh \/bin\/bash default\.ida?XXX insert update delete select
http_access deny bad_requests

NOTE: The acl line has been cut over two lines, this should be on one. There should be the ACL line and the http_access line.
These lines set up some bad requests to which we deny access to, this is to help prevent SQL injection, and other hack attempts, etc.

That’s it, after a (re)start to squid you it will be reverse proxying the domains.

Redirect to SSL

We had a requirement to automatically redirect to https if someone came in on http. Squid allows redirecting through a variety of ways, you can write a redirect script at get squid to use it, but there is a simpler way, using all squid internals and acls.

Add the following to the entries added in the last section:


acl port80 myport 80
acl site1 dstdomain site1.example.lan
http_access deny port80 site1
deny_info https://site1.example.lan/ site1

acl site2 dstdomain site2.anotherexample.lan
http_access deny port80 site2
deny_info https://site2.anotherexample.lan/ site2

We create an acl for the squids port 80 and then one for the domain we want to redirect. We then use “http_access deny” to cause squid to deny access to that domain coming in on port 80 (http). This causes a deny which is caught by the deny_info which redirects it to https.

The order used of the acl’s in the http_access and the deny_info is important. Squid only remembers the last acl used by a http_access command and will look for a corresponding deny_info matched to that acl. So make sure the last acl matches the acl used in the deny_info statement!

NOTE: See http://www.squid-cache.org/Doc/config/deny_info/

Appendix

The following is the configuration all put together now.

Reverse proxy + redirection:

http_port 80 vhost
https_port 443 cert=/etc/squid/localhost.crt key=/etc/squid/localhost.key vhost

cache_effective_user squid
cache_effective_group squid

cache_peer 1.2.3.4 parent 80 0 no-query originserver login=PASS name=site1-http
cache_peer 1.2.3.5 parent 443 0 no-query originserver login=PASS ssl sslflags=DONT_VERIFY_PEER name=site2-ssl
cache_peer_domain site1-http site1.example.lan
cache_peer_domain site2-ssl site2.anotherexample.lan

acl bad_requests urlpath_regex -i cmd.exe \/bin\/sh \/bin\/bash default\.ida?XXX insert update delete select
http_access deny bad_requests

acl port80 myport 80
acl site1 dstdomain site1.example.lan
http_access deny port80 site1
deny_info https://site1.example.lan/ site1

acl site2 dstdomain site2.anotherexample.lan
http_access deny port80 site2
deny_info https://site2.anotherexample.lan/ site2

Follow up: The woeful state of communications in Australia’s capital city

In January 2011, I posted about my experiences in trying to get an internet connection provisioned at my new home

I am now posting from our Internode naked DSL connection. To be honest, this has been working for many months, I have been slack in posting this follow up!

The Telstra guy did come back and install the line. But only after we ordered a full phone line, dial tone and all, at around $30/month. Not to mention the $299 installation fee.

After that was installed, Internode activated the ADSL. Even that took multiple calls to get the technicians back to the exchange as things went wrong.

After that was all sorted out, it was then converted to a ‘naked ADSL’ service. Effectively cancelling the dial tone service.

The rampant stupidity of the Australian communications system is truly breathtaking. And expensive. What should have been a very simple thing to get going – a naked ADSL line – proved to be extremely difficult and expensive.

But now we have Internode naked ADSL and NodePhone. Finally.

(As an interesting side note, we retained our Melbourne based phone NodePhone (VoIP) number. When the Mitchell chemical fire occurred the other day and half of Canberra was on alert, we received a call on the VoIP number, as it is registered at this address. Both mine and my wife’s mobile phones are through Optus, also registered at this address and didn’t get an SMS or call. Either the emergency alerting system or Optus messed up there. I’d be guessing the latter.)

Unfortunately, we are so far away from the exchange that we only get around 500 KB a second (half a MB a second). Back in Melbourne, close to the exchange, I was getting 2.2 MB a second, so around four times faster).

But at least we have it 🙂

A call to “standardised user account requirements” arms

We need to have a standard for management of user accounts.

Given the number of high profile companies that have been cracked into lately, I have been going through the process of closing accounts for services I no longer use.

Many of these accounts were established when I was more trusting and included real data. However now, unless I am legally required to, I no longer use my real name or real data.

But I have been bitterly disappointed by the inability of some companies to shut down old accounts. For example, one service told me that “At this time, we do not directly delete user accounts…”. I also couldn’t change my username. Another service emailed my credentials in plain text.

To protect the privacy and security of all users, an enforceable standard needs to be established covering management of user accounts. It needs to be applied across the board to all systems connected to the internet. I know how ridiculous this sounds, and that many sites wouldn’t use it, but high profile services should be able to support something like this.

Included in the standard should be:

  • the ability to completely delete accounts (unless there’s some kind of legislative requirement to keep, and then they should only retain the data that is absolutely necessary)
  • the ability to change all details including usernames
  • a requirement to encrypt and salt the password (that covers the credentials in plain text issue noted above)
  • determine the minimum practicable data set that you need to maintain an account and only ask for that. If there’s no need to retain particular account details, don’t collect them. For example, I’ve never been contacted by phone by any of these companies so why was I forced to enter a phone number?

This is a short list from my frustrations today. Please comment to help me flesh this out with other things that should be done on a properly supported user account management system.

And please let me know of your experiences with companies that were unable to properly protect your privacy and security.

The woeful state of communications in Australia’s capital city

For those who may not know, I recently moved from Melbourne, Victoria to Canberra, Australian Capital Territory (ACT) and am now living in a house in the inner north-west. Of course, being a geek, I wanted to get the internet connected as soon as possible! After such a smooth transition I’d expected some problems and this is where they all cropped up.

In Melbourne I had an Internode ADSL connection and before I moved I called them up to relocate this service. This, of course, relied on getting an active Telstra line at the new house. I knew it would take a bit of time to relocate the service, so in the interim I bought a Telstra wi-fi internet device. This is actually a ZTE MF30 and supports up to 5 connections via wi-fi, so I can get both my iPhone and laptop on at the same time. Quite simply, this device is brilliant at what it does and I couldn’t be happier with it.

So, at the moment I’m online via the Telstra device, which is just as well really, as I soon encounter communication issue number 1: Optus.

It appears that Optus have a woeful network in Canberra. I have an iPhone 3GS, which I know can only use 850MHz and 2100MHz 3G networks. Optus uses 900MHz and 2100MHz for their 3G, so the iPhone will only work in Optus 2100MHz coverage. In Melbourne I never had a problem getting on the internet at good speeds.

When I looked at the Optus overage maps for ACT and click on “3G Single band” (the 2100MHz network coverage), it shows the inner north-west being well covered. It really isn’t. Both from home and at work in Belconnen, I can barely get two bars of GSM phone signal. The connectivity is so bad that I can barely make phone calls and send SMSs. Occasionally, I get the “Searching…” message which tells me that it has completely lost GSM connectivity. This never happened in Melbourne, where I had 4-5 bars of signal pretty much all the time.

The 3G connection drops in and out so often that I have to be standing in exactly the right location to be able to access the internet on my iPhone. Even this afternoon in Kingston in the inner south, I wasn’t able to get onto the internet and post to Twitter. I had to use the Telstra device, which hasn’t missed a beat in any location for network connectivity, to establish a connection. This really isn’t good enough for the middle of Canberra. I am seriously considering calling Optus, lodging a complaint and trying to get out of my 2 year contract (which has another 10 months to run), so I can switch over to Telstra. I never thought I’d say this, but I actually want to use a Telstra service!!!

Communications issue number 2: TransACT. From what I can find out TransACT have a cable TV network which also has telephone and internet capabilities. When this network was established about a decade ago, it was revolutionary and competitive. Today the network has been expanded to support ADSL connections, but there is no ability to get a naked service as all connections require an active phone service. Additionally, as a quick look at some of the internet connectivity plans show, after factoring in the required phone service, it is a costly service for below average download allowances.

When I moved into the house, the process of relocating the Internode ADSL service from Melbourne to Canberra triggered a visit from a Telstra technician. However, he wasn’t able to find a physical Telstra line into the house. Being an older suburb of Canberra, this house will have a Telstra cable. Or rather will have had as apparently it is not unknown for TransACT installers to cut the Telstra cables out as “You won’t need THAT anymore!”

So now I have to pay for a new cable to be installed from the house to the “Telstra network boundary” (presumably the street or nearest light pole where it can be connected to Telstra’s infrastructure). Then we have to pay again for a new Telstra connection at a cost of $299. Considering that if the Telstra cable had been left in place, the connection cost would be $55, this is turning into quite an expensive proposition just to get a naked DSL service.

All in all I am not impressed with the state of communications in Australia’s capital city, Canberra. All I can say is please, please, please bring on the National Broadband Network (NBN)!

 

 

In an ideal world … how to change my address

Recently I moved house.

I  hate moving. Not just for the having to pack everything into boxes at one end then then unpack everything at the destination (which for this move I didn’t have to do!), but mostly because I have to go through the pain that is changing my address.

It turns out that I interact with a lot of organisations, from finance institutions (banks, credit card companies, car insurance, house insurance, health insurance, etc), to official organisations (driver licencing, Medicare, electoral, organ donor register, etc), to community (Red Cross blood donor, 3RRRFM, etc) and mundane organisations (Costco, etc). And that’s just a fraction of them.

I was thinking that, rather than having to fill in what feels like a million forms and waste time that could be spent being a productive public servant or dad for my kid, why isn’t there a central contact details database that I update once? I’m sure that smarter minds than mine have considered this, but I think an opportunity exists for some organisation (government or private) to do this. In the day and age of ‘over-sharing’, are people still averse to putting their address, phone number and email details into a central database? Login security could be addressed using two-factor authentication, such as used by Google Authenticator, or sending a one-time code via SMS or email.

Many services, such as Twitter and Facebook, are set up to authorise other apps to access them. An example of this is when I used my Facebook account to sign up for Freecycle which operates as a Yahoo Group.  I ‘authorised’ Facebook to talk to Yahoo. I’ve also authorised Twicca on my Android smartphone to talk to my Twitter account.

In the same way, in this theoretical single contact details database, I could let the various companies and organisations that I interact with, access my updated contact details. Maybe they could poll this database once a week to look for updated details. I understand they’d have many different backend CRM systems so there may be some manipulation required, but nothing that’s too hard to fix with a bit of scripting.

I could also remove their access when I cease using their services. If I’m not longer banking with Bank A, then I revoke their access so they can’t find out how to contact me.

Does this sound sensible or silly? If sensible why hasn’t Google or someone done this already?

My new laptop!

In May 2010, I posted about what I thought were some pretty underwhelming specifications for laptops.

I have bitten the bullet and upgraded to a laptop with 1366×768 display resolution anyway.

But on a 13.3 inch screen. So it actually works pretty well.

It is a system worth about $2500 that I got for around $700. And no, it didn’t fall off the back of a truck! It fell off the back of the Dell Outlet Store.

Specs:

  • Dell Latitude E6320
  • Core i5-2520M
  • 4GB RAM (although as an ‘Enterprise’ system, it came with Windows 7 32-bit, so only 3.2GB is visible to Windows. Fixed that by dual-booting Ubuntu 64-bit)
  • 250GB HDD
  • Wi-fi
  • Bluetooth (which I personally think is next to useless)
  • Backlit keyboard (which I think is the BEST thing ever!)
  • 6 cell battery

It’s also mil-spec hardened (or something) which means that it’s almost child-proof!

It does 1080p video and with 4 cores (2 physical and 2 virtual ‘hyper-threading’) video editing works well. Really well.

I want to post up a full review at some stage, but it may not be soon.

Fun with JavaScript!

Hoping someone can help me with this JavaScript problem. I’m trying to pass an array to a getElementById with the purpose of making multiple cells in the table take the class. I can get it working with one array location but not with more than one. Please help!

<!DOCTYPE html>
<html>
<head>
<title>Title</title>
<script>
function changecolors(redsarray,yellowsarray,greensarray,graysarray)
{

var redsarray = new Array();
redsarray[0]=’r1_c1′;
//redsarray[1]=’r1_c2′;

var yellowsarray = new Array();
yellowsarray[0]=’r2_c1′;
//yellowsarray[1]=’r2_c2′;

var greensarray = new Array();
greensarray[0]=’r3_c1′;
//greensarray[1]=’r3_c2′;

var graysarray = new Array();
graysarray[0]=’r4_c1′;
//graysarray[1]=’r4_c2′;

document.getElementById(redsarray).className=’red’;
document.getElementById(yellowsarray).className=’yellow’;
document.getElementById(greensarray).className=’green’;
document.getElementById(graysarray).className=’gray’;
}
</script>
<style type=”text/css”>
.red {background-color:red;}
.yellow {background-color:yellow;}
.green {background-color:green;}
.gray {background-color:gray;]
</style>
</head>
<body>
Content …
<table border=”1″>
<tr>
<td id=”r1_c1″>
r1_c1
</td>
<td id=”r1_c2″>
r1_c2
</td>
</tr>
<tr>
<td id=”r2_c1″>
r2_c1
</td>
<td id=”r2_c2″>
r2_c2
</td>
</tr>
<tr>
<td id=”r3_c1″>
r3_c1
</td>
<td id=”r3_c2″>
r3_c2
</td>
</tr>
<tr>
<td id=”r4_c1″>
r4_c1
</td>
<td id=”r4_c2″>
r4_c2
</td>
</table>

<button type=”button” onclick=”changecolors()”;>Button</button>

</body>

</html>

 

Back to WordPress!

I’ve given up on Blogger and returned to WordPress. I’ll update the look and feel from the defaults and try to update it a bit more often!

Stupidity with passwords

We all know and understand how important passwords are. We all know that we should be using strong passwords.

What’s a strong password? Something that uses:

  • lower case characters
  • UPPER CASE CHARACTERS
  • punctuation, such as !@#$%^&*()<>?”:{}+_
  • and should be 8 characters or longer

So, to put it mildly, it really annoys me when I come across services that don’t allow me to use strong passwords. If I possibly could, I’d boycott these services, but sometimes that’s just not possible.

For example, my internet banking is limited to a password of between 6-8 characters. WTF?! This is hardly a secure password policy!

Another financial service I use is limited to 15 characters and doesn’t allow most of the punctuation set. Why? Is it too difficult to extend your database validation rules to cover all of the character set?

Ironically, I didn’t have a problem with Posterous, Facebook or Twitter (and others) in using properly secure passwords. So, these free services give me a decent level of security, but Australian financial services companies can’t. It’s stupidity in the extreme.

Idea from BarCamp Canberra #barcampcbr

Yesterday I went to the second half of BarCamp Canberra 2012 (I was busy in the morning and couldn’t make it).

As per usual for a BarCamp there were many great ideas being discussed. Someone (Craig?) suggested that we all go home and write blog posts about our own great ideas.  So here goes …

My ideas is this: to build a website to facilitate the transfer of mobile phone credit from people who have a surplus to people who need it.

My wife and I are currently using Telstra pre-paid and every so often when it gets near the expiry date, if there’s any unused credit we transfer some (or all) of that to the other account. Telstra call this ‘CreditMe2U’ and my understanding is that it can be used on any post- or pre-paid accounts. There’s a few limitations, such a maximum of $10 per day and some limit per month.

I see the site facilitating someone posting up that they need, say $5 credit. Anyone should be able to do this for any reason. The request could be as little as just a phone number and an amount.

Someone else, who has surplus credit, would transfer them some credit from their account, and then mark that the transaction has happened. This ensures that the requester doesn’t get flooded with credit transfers and multiple people who have surplus credit don’t end up  helping just one person. The requester would also not be able to make another request for 24 hours (based on phone number).

I would be reluctant to require people to register for accounts, as I think that would kill it entirely. It should be able to be truly anonymous. I would also be really keen to see that the site is not indexed in any way (robots.txt, archive.org exclusions, etc), so that numbers can’t be linked with requests.

I’m not sure if carriers other than Telstra have this option, but it’s worth investigating.

While there would be obvious ways to ‘game’ this system, and it’s not a fully thought through idea, it could become so with some feedback. So, what do you all think?

Three Ubuntu 11.10 annoyances

A while back I posted up a few of the issues I was having with Ubuntu 10.04 “Lucid Lynx”.

I’m now using the latest version (for the next few weeks), Ubuntu 11.10 “Oneric Ocelot”. And while it works well on my new laptop, it suffers from three pretty annoying issues.

  1. IPv6 and Network Manager. I am experiencing regular wireless drop outs when I enable IPv6 on my router. When I disable IPv6 on Network Manager it is perfectly stable again.
  2. For most USB keys, write speeds are really slow. And I mean excruciatingly slow. USB HDDs seems to be OK. The issue seems to be in the way that Ubuntu deals with caching.
  3. Sandy Bridge power draining. This is a well known and documented issue, with fixes that have been issued (to be incorporated into the 3.3.x kernel). They are not being integrated into the current version of Ubuntu (which uses the 3.0.x kernel), but are being backported into the next version, 12.04 (which will use the 3.2.x kernel).

These things are quite frustrating, and while I am pretty confident that the power issues will be resolved, I really hope that the other problems are addressed for the next version which is due 26 April 2012. From those bug reports and blog posts, it looks like they will be, which is heartening.

Eric Allman to be keynote speaker at lca2011

The second confirmed keynote speaker for lca2011 is the original author of Sendmail, co-founder and Chief Scientist of Sendmail, Inc., and co-author of Sendmail, published by O’Reilly and Associates.  lca2011 gives a hearty welcome to Eric Allman. We know that his work has had a profound effect on how  people across the globe have communicated and there is no doubt that this is going to be another great keynote not to be missed.  For the media release please see http://lca2011.linux.org.au/media/news/56

What about lca2011 registrations?

Wow – what a week!  I often forget  just how much work is involved in organising a conference and how important it is  to have a dedicated and committed organising team to make things happen.  As the lead organiser for linux.conf.au 2011 I am very lucky to have such a great team that will work such long hours in their own time to ensure that the conference will be a success.  Over the last few days, weeks and months there has been a lot of work going on behind the scenes to make lca2011 a conference to remember.  We had intended to open registrations a couple of weeks ago, however due to some added functionality and some really cool changes to the conference we have had to change a few things (which were not trivial) this year which has taken a little more preparation time.  Having said that though, we really are only a few days away from making the registrations live.  I know there are many people waiting on this and wanting to plan all sorts of things in preparation for the conference.

I know that without the team I am working with, I really would be lost and want to acknowledge all the hard work and long hours they are putting into the conference.  I hope to release a number of media statements over the next week or two which I know will excite quite a few people.  More updates very very soon and to all those students out there in the community, I think you will also be very happy with what we have done in terms of costs for the conference.

Geoff Huston to be keynote speaker at lca2011

The third confirmed keynote speaker for lca2011 is the Chief Scientist at the Asia Pacific Network Information Centre (APNIC), the Regional Internet Registry serving the Asia Pacific region. He is an author and researcher who has been closely involved with the development of the Internet for many years, particularly in Australia, where he is known for the initial build of the Internet within the Australian academic and research sector with the Australian Academic and Research Network (AARNet).  lca2011 gives a hearty welcome to Geoff Huston.

Please see the full media release at http://lca2011.linux.org.au/media/news/58

Dr Vint Cerf to be keynote speaker at lca2011

Some great news – Dr Vint Cerf is to be a keynote speaker at lca2011 – http://linux.conf.au.  Widely known as one of the “Fathers of the Internet,” Cerf is the co-designer of the TCP/IP protocols and the architecture of the Internet.  For more information please see the oficial announcement on the lca2011 web site – http://lca2011.linux.org.au/media/news/70

linux.conf.au 2011 was a real blast!

linux.conf.au 2011 is now over  and its time to catch up on a few hundred hours of sleep (after a little packing up and cleaning).  It is amazing how much work still has to be done after the conference has finished, but I am hopeful (perhaps skeptical) that I will have most of it out of the way before I officially go back to work in a weeks time.

The conference was awesome,  given the many challenges the team had to endure before, during and after the event.  lca2011 will truly be a conference the   whole team (and many others) will remember for a long time to come.  While we were unable to deliver everything we had hoped for (time and financial constraints due to floods), the majority of the feedback has been extremely positive.  There were a few interesting requests during the conference that added extra challenges (such as tethering a helium balloon 100m above the university oval) and the rocketry miniconf near the NOC kept us fairly high on paint fumes for many of the days they were constructing their rockets.

Reflecting back on the event – we had a venue (eventually), we had great speakers and delegates, supportive sponsors, a working network with Internet prior to the conference (even with a little IPv6 thrown in here and there) and great food.  The PDNS at the Maritime museum saw fine weather and a relaxed environment (even with some food left over from the catering) with a few fireworks thrown in across the river.  Interestingly, we also managed to have many video’s uploaded to http://linuxconfau.blip.tv/ during the conference which was a credit to our AV team and the many hours of planning and training that they undertook.  Once encoding is finished all the video’s will be available at the Linux Australia Mirror (as per usual).  The speakers dinner was a real drumming event where everyone was able to get in to the rhythm and hit drums as hard as they could – an event that received continual praise from many speakers during the week.  The penguin dinner was relaxed and perhaps a little slow to kick off the fund raising activities – however a total of $23 239 was raised for the Qld Flood Appeals and another $5000 for the Glenleighden School at Fig Tree Pocket in Brisbane ($28 239 in total).

Open Day seemed to attract a good crowd of people during the morning while the final day of conference activities was the launch of the Rockets built during the Rocketry miniconf (thanks to Bdale, Keith, AJ and Clinton for their efforts here).  A few of the pics from the rocket launch day can be found at http://fishtank.kangawallafox.com/index.php/linux-conf-au/rockets-lca2011

lca2011 rocket launch day

lca2011 rocket launch day

Rockets, Rockets

Rockets, Rockets and Rockets

preparing for the flight

preparing for the flight

2 more sleeps to an awesome lca2011

For most people they will know that there is only 2 more sleeps till an awesome linux.conf.au (lca2011) in Brisbane, at the Qld. University of Technology – Kelvin Grove Campus.

As most people will know from identi.ca, twitter, email and website updates – the lead up to the conference has been fraught with many challenges, but the team has been working hard to ensure that they can provide the very best linux.conf.au experience that they can for all attendee’s while also trying to help friends and family affected by the recent Brisbane floods.  As a consequence of the floods we have also had to move the entire conference venue location as well as deal with the logistics of this and changing many of the events and plans we have been making since we found out about our successful bid 13  mths ago.  Essentially – we have had to change most of our plans (as well as many kool things we had wanted to do) and then achieve it all in about 10 days.

Most of the team has had very little sleep and some of them even seem to be showing signs of being slightly stressed (uncontrollable twitching or twittering, changed vocabulary, poor dress sense and an innate desire to play on loop “Death is not the End” by Nick cave while drinking from enhanced frozen coke. So, if you see one of these creatures floating around the venue, please be kind and smile at them.  If you find something that is not working or not up to the standard you expect – then please go to the registration desk and let us know.  We want this to be a great experience for everyone.

Now – the very special experience we are giving to everyone delegate at this years conference is the opportunity to actually use the switches that Jared and myself rescued from the Brisbane floods while helping clean up. Some of them did need a little tlc but we are confident that this will just enhance the experience for all

…really looking forward to seeing everyone at the conference on Monday morning – especially when we give out the first computer in the welcome talk.  Here are a few pics!

clean boys with cisco switches for lca2011

clean boys with cisco switches for lca2011

Saturday night switch config for Urbanest and other area's

Saturday night switch config for Urbanest and other area's

AP config for urbanest

AP config for urbanest

Early Bird registrations nearly sold out for linux.conf.au

With just over 24hrs to go till registrations close for lca2011 we only have a few early bird registrations left to sell which means it is very likely that they could close before midnight on the 8th November. If you are keen to grab one of these discounted registrations then please make sure you visit the registration page as soon as possible and register and pay (you must pay to get the early bird rate).  For those of you also looking for the volunteer form please also see the volunteer page.

lca2011 – prizes and more

A long time ago the lca2011 team decided to have a small competition to capture the ideas of what people wanted at the next linux.conf.au.  We received a lot of responses to this including some very very special requests for things like stickers and cup cakes.  However, while we all wanted to have a conference full of stickers and pretty cup cakes and take on board as many other suggestions as feasibly possible, there was one suggestion that the team decided should be taken on board – that was the inclusion of a poster type session.  Alec Clews having made this suggestion through the competition website has received a complimentary conference registration including penguin dinner.  Congratulations from the team Alec!

There is only 26 days to go till lca2011 officially kicks off!  Final decisions on numbers for  many of the different events will be made shortly so if you are intend to be a part of an awesome 2011 linux.conf.au please ensure that you have registered as soon as possible – time is now running out and there is a lot planned for lca2011 delegates.  A lot of  the plans for lca2011 are now falling into place and the team will be even busier in the next couple of weeks leading up to the conference. We have also received some really cool Loongson lemote mini computers which will be given away during the conference – a few more details on these shortly. Most of the goodies for the schwag bag have arrived and are awaiting being packed while we are also testing the  Internet connections to the main accommodation venue at Urbanest (straight across the river from the conference venue at South Brisbane).

lca2011 registrations are open

yippee! lca2011 registrations are now open – here is a direct link to the prices http://lca2011.linux.org.au/register/prices.  Please note that there have been a few changes to things this year, though we have tried to keep prices as low as possible for a the 5 day conference.  There is also a new miniconf this year which I am sure will excite a number of people in the community – this is the Rocket miniconf which will include a live launch of the rockets built during the miniconf.  There is an additional fee attached to this miniconf and places are limited to 24 so if this is something that interests you I would suggest registering as soon as possible so that you do not miss out.  More announcements soon – for now I need a lot of sleep :-)

linux.conf.au 2011 early bird closes in 10 days

The early bird tickets for linux.conf.au 2011 are selling quickly with over half of them sold.  It really looks as though they will sell out before the 8/11/2010 so if you are keen to take advantage of these discounted prices I would suggest that you act on this as soon as possible.  The conference is on target to be another great lca with an awesome line up of keynote speakers, and other conference speakers and activities.  The Rocket miniconf is also proving to be very popular  with numbers growing quickly.  There will be a launch of the rockets on the Sunday after the main conference finishes (30/01/2011) so this should also be a heap of fun – especially from some of the ideas I am hearing about what people want to build.

What’s the difference between 0xffffffff and 0xffffffffu?

In C++, what is the difference between 0xffffffff and 0xffffffffu?

This one’s pretty easy to answer with this information from the C++ standard:

The type of an integer literal is the first of the corresponding list in Table 6 in which its value can be represented.

0xffffffff is a hexadecimal constant, it’s too big to be represented in a (signed) int, so — by the terms of the standard — the type of 0xffffffff is unsigned int.

Furthermore, each of these hexadecimal literals will have a different type:

0x7fffffff   // int
0xffffffff   // unsigned int
0x1ffffffff  // long int (or long long int)
0x1ffffffffu // unsigned long int (or unsigned long long int)

But to answer the original question, there is no difference between 0xffffffff and 0xffffffffu apart from this:

Another C++11 ‘countof’

Note: There’s an update here.

Read “Better array ‘countof’ implementation with C++ 11” for context. Specifically, it presents Listing 5 as an implementation of countof() using C++11 constexpr:

  • template<typename T, std::size_t N>
    constexpr std::size_t countof(T const (&)[N]) noexcept
    {
      return N;
    }

But this falls short. Just a little.

There are arguments that could be passed to a naive sizeof(a)/sizeof(a[0]) macro that will cause the above to fail to compile.

Consider:

struct S
{
  int a[4];
};

void f(S* s)
{
  constexpr size_t s_a_count = countof(s->a); 
  int b[s_a_count]; 
  // do things...
}

This does not compile. s is not constant, and countof() is a constexpr function whose result is needed at compile time, and so expects a constexpr-friendly argument. Even though it is not used.

Errors from this kind of thing can look like this from clang-3.7.0:

error: constexpr variable 's_a_count' must be initialized by a 
       constant expression
note:  read of non-constexpr variable 's' is not allowed in a 
       constant expression

or this from Visual Studio 2015 Update 1:

error: C2131: expression did not evaluate to a constant

(Aside: At the time of writing, the error C2131 seems to be undocumented for VS2015. But Visual Studio 6.0 had an error with the same number)

Here’s a C++11 version of countof() that will give the correct result for countof(s->a) above:

#include <type_traits>

template<typename Tin>
constexpr std::size_t countof()
{
  using T = typename std::remove_reference<Tin>::type;
  static_assert(std::is_array<T>::value, 
                "countof() requires an array argument");
  static_assert(std::extent<T>::value > 0,  // [0]
                "zero- or unknown-size array");
  return std::extent<T>::value;
}

#define countof(a) countof<decltype(a)>()

Some of the details:

Adding a countof() macro allows use of decltype() in the caller’s context, which provides the type of the member array of a non-const object at compile time.

std::remove_reference is needed to get the array type from the result of decltype(). Without it, std::is_array and std::extent produce false and zero, respectively.

The first static assert ensures that countof() is being called on an actual array. The upside over failed template instantiation or specialization is that you can write your own human-readable, slightly more context aware error message (better than mine).

The second static assert validates that the array size is known, and is greater than zero. Without it, countof<int[]>() will return zero (which will be wrong) without error. And zero-sized arrays will also result in zero — in practice they rarely actually contain zero elements. This isn’t a function for finding the size of those arrays.

And then std::extent<T>::value produces the actual count of the elements of the array.


Addendum:

If replacing an existing sizeof-based macro with a constexpr countof() alternate, Visual Studio 2015 Update 1 will trigger warnings in certain cases where there previously were no warnings.

warning C4267: conversion from 'size_t' to 'int', possible loss of data

It is unfortunate to have to add explicit casts when the safety of such operations is able to be determined by the compiler. I have optimistically submitted this as an issue at connect.microsoft.com.

[0] Typo fix thanks to this commentor

The Growth of Modern C++ Support

 

Completing what I started here, I’ve charted the numbers from Christophe’s data for C++11, C++11 Concurrency, C++14 and C++17.

The data is taken entirely from the linked pdf with one exception: N3664 is a clarification that permits optimization, not a requirement for compliance. Compilers that do not perform this optimization are no less compliant with C++14. I’ve recomputed the percentages for all compiler versions to take this into account.

In addition to the references from the previous post, the approval date of C++14 was taken from http://en.wikipedia.org/wiki/C++14

C++14 and volatile implicity

[Update 2016-03-07: It appears that this was a bug in VS2015, and has been fixed in Update 2 RC]

In the process of upgrading Visual Studio 2012 to Visual Studio 2015, I encountered some brand new link errors that looked something like this:

error LNK2001: unresolved external symbol 
"public: __cdecl FooData::FooData(struct FooData& const &)"

It’s not a new error in VS2015 — VS2012 can certainly produce it. I mean “new” in the sense that there were no problems linking this code when using the older compiler.

The struct in question looks vaguely like this:

struct FooData
{
  int m_Bar;
  volatile int m_Baz;
};

The problem is m_Baz. In C++14, the language was changed to say that structs are not trivially constructible if they have non-static volatile members. And that, I think, is why there’s no default copy constructor being generated. I can’t quote chapter and verse to back up that assertion, though.

[Update: Actually… maybe not? I’m beginning to wonder if VS2015 is doing the wrong thing here.]

But the fix is simple: add a copy constructor. And then, when the program fails to compile, declare a default constructor (because, of course, adding a copy constructor causes the implicit default constructor to be marked as deleted).

I found that developing an understanding of exactly what was happening and why to be the more difficult problem. Initially because the the compiler gave no indication that there was a problem at all, and willingly generated calls to a copy constructor that couldn’t possibly exist. Deeper than that, I’m still trying to piece together my own understanding of exactly why (and how) this change was made to the standard.

Aside: Over-engineered Min() [C++, variadic templates, constexpr, fold left]

Q: Given a function constexpr int Min(int a, int b), construct a function constexpr int Min(Args... args) that returns the minimum of all the provided args. Fail to justify your over-engineering.

A: Rename Min(int, int) as MinImpl(int, int) or stick it in a namespace. Overloading the function is not only unnecessary, it gets in the way of the implementation.

constexpr int MinImpl(int a, int b)
{
  return a < b ? a : b;
}

Implement a constexpr fold left function. If we can use it for Min(), we should be able to do the same for Max(), and other similar functions. Should we be able to find any (#prematuregeneralization).

template<typename ArgA, typename ArgB, typename Func>
constexpr auto foldl(Func func, ArgA a, ArgB b)
{
  return func(a, b);
}

template<typename ArgA, typename ArgB,
typename Func, typename ...Args>
constexpr auto foldl(Func func, ArgA a, ArgB b, Args... args)
{
  return foldl(func, func(a, b), args...);
}

Combine the two.

template<typename ...Args>
constexpr auto Min(Args... args)
{
  return foldl(MinImpl, args...);
}

Add the bare minimum amount of testing for a constexpr function: slap a static_assert() on it.

static_assert(Min(6, 4, 5, 3, 9) == 3), "Nope");

I did so with Visual Studio 2015 Update 2. It did not object.

Addendum: Some discussion with @nlguillemot and @DrPizza led to this attempt to do something similar with a C++17/C++1z fold-expression:

#include <limits.h>

constexpr int MinImpl1(int a, int b) { return a < b ? a : b; }

constexpr void MinImpl2(int* m, int a, int b) { *m = a < b ? a : b; }

template<typename ...Args>
constexpr int Min(Args... args)
{
  int m = INT_MAX;
  // a binary expression in an operand of a fold-expression 
  // is not allowed, so this won't compile:
  //((m = MinImpl1(m, args), ...);

  // But this does:
  (MinImpl2(/*out*/&m, m, args), ...);
  return m;
}

int main()
{
  static_assert(Min(3,4,5) == 3, "nope");
}

This compiles with a gcc-6 pre-release snapshot.

Update: Here’s a further updated version, based on a refinement by @dotstdy.

floats, bits, and constant expressions

Can you access the bits that represent an IEEE754 single precision float in a C++14 constant expression (constexpr)?

(Why would you want to do that? Maybe you want to run a fast inverse square root at compile time. Or maybe you want to do something that is actually useful. I wanted to know if it could be done.)

For context: this article is based on experiences using gcc-5.3.0 and clang-3.7.1 with -std=c++14 -march=native on a Sandy Bridge Intel i7. Where I reference sections from the C++ standard, I’m referring to the November 2014 draft.

Before going further, I’ll quote 5.20.6 from the standard:

Since this International Standard imposes no restrictions on the accuracy of floating-point operations, it is unspecified whether the evaluation of a floating-point expression during translation yields the same result as the evaluation of the same expression (or the same operations on the same values) during program execution.88

88) Nonetheless, implementations are encouraged to provide consistent results, irrespective of whether the evaluation was performed during translation and/or during program execution.

In this post, I document things that worked (and didn’t work) for me. You may have a different experience.

Methods of conversion that won’t work

(Error text from g++-5.3.0)

You can’t access the bits of a float via a typecast pointer [which is undefined behavior, and covered by 5.20.2.5]:

  constexpr uint32_t bits_cast(float f)
  {
    return *(uint32_t*)&f; // [2]
  }
error:
  accessing value of 'f' through a 'uint32_t {aka unsigned int}' glvalue
  in a constant expression

You can’t convert it via a reinterpret cast [5.20.2.13]

constexpr uint32_t bits_reinterpret_cast(float f) 
{ 
  const unsigned char* cf = reinterpret_cast<const unsigned char*>(&f); 
  // endianness notwithstanding
  return (cf[3] << 24) | (cf[2] << 16) | (cf[1] << 8) | cf[0]; 
} 
error:
  '*(cf + 3u)' is not a constant expression

(gcc reports an error with the memory access, but does not object to the reinterpret_cast. clang produces a specific error for the cast.)

You can’t convert it through a union [gcc, for example, permits this for non-constant expressions, but the standard forbids it in 5.20.2.8]:

constexpr uint32_t bits_union(float f) 
{ 
  union Convert { 
    uint32_t u;
    float f;
    constexpr Convert(float f_) : f(f_) {}
  };
  return Convert(f).u;
}
error:
  accessing 'bits_union(float)::Convert::u' member instead of 
  initialized 'bits_union(float)::Convert::f' member 
  in constant expression

You can’t use memcpy() [5.20.2.2]:

constexpr uint32_t bits_memcpy(float f) 
{
  uint32_t u = 0;
  memcpy(&u, &f, sizeof f);
  return u;
}
error:
  'memcpy(((void*)(&u)), ((const void*)(&f)), 4ul)' 
  is not a constant expression

And you can’t define a constexpr memcpy()-like function that is capable of the task [5.20.2.11]:

constexpr void* memcpy(void* dest, const void* src, size_t n)
{
  char* d = (char*)dest;
  const char* s = (const char*)src;
  while(n-- > 0)
    *d++ = *s++;
  return dest;
}

constexpr uint32_t bits_memcpy(float f)
{
  uint32_t u = 0;
  memcpy(&u, &f, sizeof f);
  return u;
}
error:
  accessing value of 'u' through a 'char' glvalue
  in a constant expression

So what can you do?

Floating point operations in constant expressions

For constexpr float f = 2.0f, g = 2.0f the following operations are available [as they are not ruled out by anything I can see in 5.20]:

  • Comparison of floating point values e.g.
    static_assert(f == g, "not equal");
  • Floating point arithmetic operations e.g.
    static_assert(f * 2.0f == 4.0f, "arithmetic failed");
  • Casts from float to integral value, often with well-defined semantics e.g.
    constexpr int i = (int)2.0f; static_assert(i == 2, "conversion failed");

So I wrote a function (uint32_t bits(float)) that will return the binary representation of an IEEE754 single precision float. The full function is at the end of this post. I’ll go through the various steps required to produce (my best approximation of) the desired result.

1. Zero

When bits() is passed the value zero, we want this behavior:

static_assert(bits(0.0f) == 0x00000000);

And we can have it:

  if (f == 0.0f)
    return 0;

Nothing difficult about that.

2. Negative zero

In IEEE754 land, negative zero is a thing. Ideally, we’d like this behavior:

static_assert(bits(-0.0f) == 0x80000000)

But the check for zero also matches negative zero. Negative zero is not something that the C++ standard has anything to say about, given that IEEE754 is an implementation choice [3.9.1.8: “The value representation of floating-point types is implementation defined”]. My compilers treat negative zero the same as zero for all comparisons and arithmetic operations. As such, bits() returns the wrong value when considering negative zero, returning 0x00000000 rather than the desired 0x80000000.

I did look into other methods for detecting negative zero in C++, without finding something that would work in a constant expression. I have seen divide by zero used as a way to detect negative zero (resulting in ±infinity, depending on the sign of the zero), but that doesn’t compile in a constant expression:

constexpr float r = 1.0f / -0.0f;
error: '(1.0e+0f / -0.0f)' is not a constant expression

and divide by zero is explicitly named as undefined behavior in 5.6.4, and so by 5.20.2.5 is unusable in a constant expression.

Result: negative zero becomes positive zero.

3. Infinity

We want this:

static_assert(bits(INFINITY) == 0x7f800000);

And this:

  else if (f == INFINITY)
    return 0x7f800000;

works as expected.

4. Negative Infinity

Same idea, different sign:

static_assert(bits(-INFINITY) == 0xff800000);
  else if (f == -INFINITY)
    return 0xff800000;

Also works.

5. NaNs

There’s no way to generate arbitrary NaN constants in a constant expression that I can see (not least because casting bits to floats isn’t possible in a constant expression, either), so it seems impossible to get this right in general.

In practice, maybe this is good enough:

static_assert(bits(NAN) == 0x7fc00000);

NaN values can be anywhere in the range of 0x7f800001 -- 0x7fffffff and 0xff800001 -- 0xffffffff. I have no idea as to the specific values that are seen in practice, nor what they mean. 0x7fc00000 shows up in /usr/include/bits/nan.h on the system I’m using to write this, so — right or wrong — I’ve chosen that as the reference value.

It is possible to detect a NaN value in a constant expression, but not its payload. (At least that I’ve been able to find). So there’s this:

  else if (f != f) // NaN
    return 0x7fc00000; // This is my NaN...

Which means that of the 2*(223-1) possible NaNs, one will be handled correctly (in this case, 0x7fc00000). For the other 16,777,213 values, the wrong value will be returned (in this case, 0x7fc00000).

So… partial success? NaNs are correctly detected, but the bits for only one NaN value will be returned correctly.

(On the other hand, the probability that it will ever matter could be stored as a denormalized float)

6. Normalized Values

// pseudo-code
static_assert(bits({  0x1p-126f, ...,  0x1.ffff7p127})
                == { 0x00800000, ...,  0x7f7fffff});
static_assert(bits({ -0x1p-126f, ..., -0x1.ffff7p127})
                == { 0x80800000, ...,  0xff7fffff});

[That 0x1pnnnf format happens to be a convenient way to represent exact values that can be stored as binary floating point numbers]

It is possible to detect and correctly construct bits for every normalized value. It does requires a little care to avoid truncation and undefined behavior. I wrote a few different implementations — the one that I describe here requires relatively little code, and doesn’t perform terribly [0].

The first step is to find and clear the sign bit. This simplifies subsequent steps.

 bool sign = f < 0.0f; 
  float abs_f = sign ? -f : f;

Now we have abs_f — it’s positive, non-zero, non-infinite, and not a NaN.

What happens when a float is cast to an integral type?

uint64_t i = (uint64_t)f;

The value of f will be stored in i, according to the following rules:

  • The value will be rounded towards zero which, for positive values, means truncation of any fractional part.
  • If the value in f is too large to be represented as a uint64_t (i.e. f > 264-1) the result is undefined.

If truncation takes place, data is lost. If the number is too large, the result is (probably) meaningless.

For our conversion function, if we can scale abs_f into a range where it is not larger than (264-1), and it has no fractional part, we have access to an exact representation of the bits that make up the float. We just need to keep track of the amount of scaling being done.

Single precision IEEE 754 floating point numbers have, at most, (23+1) bits of precision (23 in the significand, 1 implicit). This means that we can scale down large numbers and scale up small numbers into the required range.

Multiplying by powers of two change only the exponent of the float, and leave the significand unmodified. As such, we can arbitrarily scale a float by a power of two and — so long as we don’t over- or under-flow the float — we will not lose any of the bits in the significand.

For the sake of simplicity (believe it or not [1]), my approach is to scale abs_f in steps of 241 so that (abs_f ≥ 287) like so:

  int exponent = 254; 

  while(abs_f < 0x1p87f) 
  { 
    abs_f *= 0x1p41f; 
    exponent -= 41; 
  }

If abs_f ≥ 287, the least significant bit of abs_f, if set, is 2(87-23)==264.

Next, abs_f is scaled back down by 264 (which adds no fractional part as the least significant bit is 264) and converted to an unsigned 64 bit integer.

  uint64_t a = (uint64_t)(abs_f * 0x1p-64f);

All of the bits of abs_f are now present in a, without overflow or truncation. All that is needed now is to determine where they are:

  int lz = count_leading_zeroes(a);

adjust the exponent accordingly:

  exponent -= lz;

and construct the result:

  uint32_t significand = (a << (lz + 1)) >> (64 - 23); // [3]
  return (sign << 31) | (exponent << 23) | significand;

With this, we have correct results for every normalized float.

7. Denormalized Values

// pseudo-code
static_assert(bits({  0x1.0p-149f, ...,  0x1.ffff7p-127f})
                == {  0x00000001,  ...,  0x007fffff});
static_assert(bits({ -0x1.0p-149f, ..., -0x1.ffff7p-127f})
                == {  0x80000001,  ...,  0x807fffff});

The final detail is denormalized values. Handling of normalized values as presented so far fails because denormals will have additional leading zeroes. They are fairly easy to account for:

   if (exponent <= 0) 
  { 
    exponent = 0; 
    lz = 8 - 1; 
  }

To attempt to demystify that lz = 8 - 1 a little: there are 8 leading bits that aren’t part of the significand of a denormalized single precision float after the repeated 2-41 scaling that has taken place. There is also no leading 1 bit that is present in all normalized numbers (which is accounted for in the calculation of significand above as (lz + 1)). So the leading zero count (lz) is set to account for the 8 bits of offset to the start of the denormalized significand, minus the one that the subsequent calculation assumes it needs to skip over.

And that’s it. All the possible values of a float are accounted for.

(Side note: If you’re compiling with -ffast-math, passing denormalized numbers to bits() will return invalid results. That’s -ffast-math for you. With gcc or clang, you could add an #ifdef __FAST_MATH__ around the test for negative exponent.)

Conclusion

You can indeed obtain the bit representation of a floating point number at compile time. Mostly. Negative zero is wrong, NaNs are detected but otherwise not accurately converted.

Enjoy your compile-time bit-twiddling!


The whole deal:

// Based on code from 
// https://graphics.stanford.edu/~seander/bithacks.html
constexpr int count_leading_zeroes(uint64_t v) 
{ 
  constexpr char bit_position[64] = {  
     0,  1,  2,  7,  3, 13,  8, 19,  4, 25, 14, 28,  9, 34, 20, 40, 
     5, 17, 26, 38, 15, 46, 29, 48, 10, 31, 35, 54, 21, 50, 41, 57, 
    63,  6, 12, 18, 24, 27, 33, 39, 16, 37, 45, 47, 30, 53, 49, 56, 
    62, 11, 23, 32, 36, 44, 52, 55, 61, 22, 43, 51, 60, 42, 59, 58 }; 
   
  v |= v >> 1; // first round down to one less than a power of 2  
  v |= v >> 2; 
  v |= v >> 4; 
  v |= v >> 8; 
  v |= v >> 16; 
  v |= v >> 32; 
  v = (v >> 1) + 1; 
   
  return 63 - bit_position[(v * 0x0218a392cd3d5dbf)>>58]; // [3]
}
 
constexpr uint32_t bits(float f) 
{ 
  if (f == 0.0f) 
    return 0; // also matches -0.0f and gives wrong result 
  else if (f == INFINITY) 
    return 0x7f800000; 
  else if (f == -INFINITY) 
    return 0xff800000; 
  else if (f != f) // NaN 
    return 0x7fc00000; // This is my NaN...
 
  bool sign = f < 0.0f; 
  float abs_f = sign ? -f : f; 
 
  int exponent = 254; 
 
  while(abs_f < 0x1p87f) 
  { 
    abs_f *= 0x1p41f; 
    exponent -= 41; 
  } 
 
  uint64_t a = (uint64_t)(abs_f * 0x1p-64f); 
  int lz = count_leading_zeroes(a);
  exponent -= lz;
 
  if (exponent <= 0) 
  { 
    exponent = 0; 
    lz = 8 - 1;
  } 
 
  uint32_t significand = (a << (lz + 1)) >> (64 - 23); // [3]
  return (sign << 31) | (exponent << 23) | significand; 
}

[0] Why does runtime performance matter? Because that’s how I tested the conversion function while implementing it. I was applying Bruce Dawson’s advice for testing floats and the quicker I found out that I’d broken the conversion the better. For the implementation described in this post, it takes about 97 seconds to test all four billion float values on my laptop — half that time if I wasn’t testing negative numbers (which are unlikely to cause problems due to the way I handle the sign bit). The implementation I’ve described in this post is not the fastest solution to the problem, but it is relatively compact, and well behaved in the face of -ffast-math.

Admission buried in a footnote: I have not validated correct behavior of this code for every floating point number in actual compile-time constant expressions. Compile-time evaluation of four billion invocations of bits() takes more time than I’ve been willing to invest so far.

[1] It is conceptually simpler to multiply abs_f by two (or one half) until the result is exactly positioned so that no leading zero count is required after the cast — at least, that was what I did in my first attempt. The approach described here was found to be significantly faster. I have no doubt that better-performing constant-expression-friendly approaches exist.

[2] Update 2016-03-28: Thanks to satbyy for pointing out the missing ampersand — it was lost sometime after copying the code into the article.

[3] Update 2016-03-28: Thanks to louiswins for pointing out additional code errors.

The Growth of C++11 Support

Update: This chart has been updated and I’ve added charts for C++11 Concurrency, C++14, and C++17 here.
 

A few days ago, Christophe Riccio tweeted a link to a pdf that shows the level of support for “Modern C++” standards in four C++ compilers: Visual C++, GCC, Clang, and ICC.

One of the things I wanted to see was not just how support had advanced between versions of each compiler, but how compilers had changed relative to one another over time. I extracted the numbers for C++11 from Christophe’s document, found the release dates for each compiler, and created a chart that puts it all together.

It’s interesting to see how far behind Clang starts in comparison to the others, and that it ends up in a close dance with GCC on the way to full C++11 support. It also highlights how disappointing VC++ has been in terms of language feature advancement — particularly when VS2010 was ahead of Clang and ICC for C++11 features.

Creating the chart also served as an opportunity to play around with data visualization using Bokeh. As such, you can click on the chart above and you’ll see a version that you can zoom, pan, and resize (which is only a small part of what Bokeh offers). I intend to write about my experiences with Bokeh at a later date.

 

Release dates for each compiler were taken from the following pages:

The date used to mark the approval of the C++11 standard is taken from http://en.wikipedia.org/wiki/C++11

Another another C++11 ‘countof’

My earlier post received this comment which is a pretty neat little improvement over the one from g-truc.net.

Here it is, with one further tweak:

template<typename T, std::size_t N>
constexpr 
std::integral_constant<std::size_t, N> countof(T const (&)[N]) noexcept
{
  return {};
}

#define COUNTOF(...) decltype(countof(__VA_ARGS__))::value

The change I’ve made to pfultz2’s version is to use ::value rather than {} after decltype in the macro.

This makes the type of the result std::size_t not std::integral_constant, so it can be used in va_arg settings without triggering compiler or static analysis warnings.

It also has the advantage of not triggering extra warnings in VS2015U1 (this issue).

Standards vs Compilers: Warning C4146

warning C4146: unary minus operator applied to unsigned type, result still unsigned

I saw this warning recently.

“Aha!” I thought. “A common source of errors, able to strike down the unsuspecting programmer. Thank you crafters of Visual C++ compiler warnings, tirelessly laboring to uncover wrong assumptions and naively written code.”

“What?” I exclaimed. “Of course the result is still unsigned. That’s how the language is designed, and that’s what I wanted!”

Nevertheless, I read the documentation for the warning to see if there was anything I could glean from it — particularly to see if I could find sufficient reason to not just #pragma disable it.

This is what you can find in the documentation:

Unsigned types can hold only non-negative values, so unary minus (negation) does not usually make sense when applied to an unsigned type. Both the operand and the result are non-negative.

Negation of an unsigned value may not make sense if you don’t know what it means — it is well defined. Regardless, this is a level 2 warning. It is designed to catch common mistakes and misunderstandings and notify the programmer to have them look more closely. It may be an entirely reasonable thing to warn about.

The documentation continues with some rationale:

Practically, this occurs when the programmer is trying to express the minimum integer value, which is -2147483648. This value cannot be written as -2147483648 because the expression is processed in two stages:

  1. The number 2147483648 is evaluated. Because it is greater than the maximum integer value of 2147483647, the type of 2147483648 is not int, but unsigned int.
  2. Unary minus is applied to the value, with an unsigned result, which also happens to be 2147483648.

The first point is wrong. Wrong for a standards-conformant C++ implementation, anyway. The second would be accurate if the first was accurate (because 232 – 231 == 231)

Here’s what the most recent draft of the C++ standard says about the integer literal types:

The type of an integer literal is the first of the corresponding list in Table 6 in which its value can be represented.

2147483648 is a decimal constant with no suffix. When using VC++ with it’s 32 bit long int type, the first of the corresponding list in which its value can be represented is the 64 bit long long int. An unsigned type is never an option.

Unary minus should then be applied to long long int 2147483648, which should result in long long int -2147483648. There’s nothing unsigned in this process

Use of the result should behave in an unsurprising way, too — long long int -2147483648 can be assigned to a variable of type int and nothing unexpected will happen. The type can be converted without affecting the value.

According to the standard, the rationale is flawed, and the warning seems pointless to me.

In theory, there’s no difference between theory and practise

So I tried compiling the example program from the documentation to see what would happen.

// C4146.cpp
// compile with: /W2
#include <stdio.h>

void check(int i)
{
  if (i > -2147483648) // C4146
    printf_s("%d is greater than the most negative int\n", i);
}

int main()
{
  check(-100);
  check(1);
}

The documentation predicts the following outcome:

The expected second line, 1 is greater than the most negative int, is not printed because ((unsigned int)1) > 2147483648 is false.

If I build the program with gcc 4.9.2, both lines print.

If I build the program with Visual C++ 2012, or even 2015 Preview, only one line is printed (as was predicted).

So there is legitimacy to this warning — this is an area that Visual C++ is not compliant with the standard.

Maybe it’s because the standard has changed? I looked at the earliest version of the text available in the cplusplus github repo dating from late 2011, and that has the same rules as quoted above.

I went back further and found copies of the standard from 2003 and 1998, both of which state:

The type of an integer literal depends on its form, value, and suffix. If it is decimal and has no suffix, it has the first of these types in which its value can be represented: int, long int; if the value cannot be represented as a long int, the behavior is undefined.

So it’s a detail that was previously undefined, which means that the compiler is permitted to do whatever it wants. In this case, we’ll get a warning, but only if the programmer has asked for it using option /W2.

The documentation is accurate, and Visual C++ hasn’t kept up with changes in the standard. This shouldn’t be surprising.

Update: long long int was added to the standard as part of C++11. It appears that VC++ has had long long support since at least Visual Studio .NET 2003

So what?

This investigation arose from my reading of Visual C++ documentation in the context of what I knew of a recent draft of the C++ standard. It turns out that these two things are less connected than I had assumed. Unsurprisingly, the Visual C++ documentation describes Visual C++, not the standard.

While it would be nice if deviations from the standard were clearly marked in the documentation, and even nicer if the Visual C++ compiler was consistent with the ISO standard, the reality is that they are not and it is not.

One should always pay close attention to context, which happens to apply as much when reading about the C++ language as it does when writing C++ code.

What is -1u?

In C++, what exactly is -1u?

It doesn’t seem like it should be difficult to answer — it’s only three characters: , 1, and u. And, knowing a little bit about C++, it seems like that’ll be (-1) negative one with that u making ((-1)u) an unsigned int. Right?

To be more specific, on an architecture where int is a 32 bit type, and negative numbers are represented using two’s complement (i.e. just about all of them), negative one has the binary value 11111111111111111111111111111111. And converting that to unsigned int should … still be those same thirty two ones. Shouldn’t it?

I can test that hypothesis! Here’s a program that will answer the question once and for all:

#include <stdio.h>
#include <type_traits>

int main()
{
 static_assert(std::is_unsigned<decltype(-1u)>::value, 
               "actually not unsigned");
 printf("-1u is %zu bytes, with the value %#08x\n ", 
        sizeof -1u, -1u);
}

Compile and run it like this:

g++ -std=c++11 minus_one_u.cpp -o minus_one_u && minus_one_u

If I do that, I see the following output:

-1u is 4 bytes, with the value 0xffffffff

I’m using -std=c++11 to be able to use std::is_unsigned, decltype and static_assert which combine to assure me that (-1u) is actually unsigned as the program wouldn’t have compiled if that wasn’t the case. And the output shows the result I had hoped for: it’s a four byte value, containing 0xffffffff (which is the same as that string of thirty two ones I was looking for).

I have now proven that -1u means “convert -1 to an unsigned int.” Yay me!

Not so much.

It just so happened that I was reading about integer literals in a recent draft of the ISO C++ standard. Here’s the part of the standard that describes the format of decimal integer literals:

2.14.2 Integer literals
1 An integer literal is a sequence of digits that has no period or exponent part, with optional separating single quotes that are ignored when determining its value. An integer literal may have a prefix that specifies its base and a suffix that specifies its type. The lexically first digit of the sequence of digits is the most significant. A decimal integer literal (base ten) begins with a digit other than 0 and consists of a sequence of decimal digits.

Can you see where it describes negative integer literals?

I can’t see where it describes negative integer literals.

Oh.

I though -1u was ((-1)u). I was wrong. Integer literals do not work that way.

Obviously -1u didn’t just stop producing an unsigned int with the value 0xffffffff (the program proved it!!1), but the reason it has that value is not the reason I thought.

So, what is -1u?

The standard says that 1u is an integer literal. So now I need to work out exactly what that  is doing. What does it mean to negate 1u? Back to the standard I go.

5.3.1 Unary operators
8 The operand of the unary – operator shall have arithmetic or unscoped enumeration type and the result is the negation of its operand. Integral promotion is performed on integral or enumeration operands. The negative of an unsigned quantity is computed by subtracting its value from 2n, where n is the number of bits in the promoted operand. The type of the result is the type of the promoted operand.

I feel like I’m getting closer to some real answers.

So there’s a numerical operation to apply to this thing. But first, this:

Integral promotion is performed on integral or enumeration operands.

Believe me when I tell you that this section changes nothing and you should skip it.

I have an integral operand (1u), so integral promotion must be performed. Here is the part of the standard that deals with that:

4.5 Integral promotions
1 A prvalue of an integer type other than bool, char16_t, char32_t, or wchar_t whose integer conversion rank (4.13) is less than the rank of int can be converted to a prvalue of type int if int can represent all the values of the source type; otherwise, the source prvalue can be converted to a prvalue of type unsigned int.

I’m going to cut a corner here: integer literals are prvalues, but I couldn’t find a place in the standard that explicitly declares this to be the case. It does seem pretty clear from 3.10 that they can’t be anything else. This page gives a good rundown on C++ value categories, and does state that integer literals are prvalues, so let’s go with that.

If 1u is a prvalue, and its type is unsigned int, I can collapse the standard text a little:

4.5 Integral promotions (prvalue edition)
A value of an integer type whose integer conversion rank (4.13) is less than the rank of int …

and I’m going to stop right there. Conversion rank what now? To 4.13!

4.13 Integer conversion rank
1 Every integer type has an integer conversion rank defined as follows:

Then a list of ten different rules, including this one:

— The rank of any unsigned integer type shall equal the rank of the corresponding signed integer type.

Without knowing more about conversion ranks, this rule gives me enough information to determine what 4.5 means for unsigned int values: unsigned int has the same rank as int. So I can rewrite 4.5 one more time like this:

4.5 Integral promotions (unsigned int edition)
1 [This space intentionally left blank]

Integral promotion of an unsigned int value doesn’t change a thing.

Where was I?

Now I can rewrite 5.3.1 with the knowledge that 1u requires no integral promotion:

5.3.1 Unary operators (unsigned int operand edition)
8 The [result of] the unary – operator … is the negation of its operand. The negative of an unsigned quantity is computed by subtracting its value from 2n, where n is the number of bits in the promoted operand. The type of the result is the type of the operand.

And, at long last, I get to do the negating. For an unsigned value that means:

[subtract] its value from 2n, where n is the number of bits in the promoted operand.

My unsigned int has 32 bits, so that would be 232 – 1. Which in hexadecimal looks something like this:

  0x100000000
- 0x000000001
  0x0ffffffff

But that leading zero I’ve left on the result goes away because

The type of the result is the type of the (promoted) operand.

And I am now certain that I know how -1u becomes an unsigned int with the value 0xffffffff. In fact, it’s not even dependent on having a platform that uses two’s complement  — nothing in the conversion relies on that.

But… when could this possibly ever matter?

For -1u? I don’t see this ever causing actual problems. There are situations that arise from the way that C++ integer literals are defined that can cause surprises (i.e. bugs) for the unsuspecting programmer.

There is a particular case described in the documentation for Visual C++ compiler warning C4146, but I think the rationale for that warning is wrong (or, at least, imprecise), but not because of something I’ve covered in this article. As I’ve already written far too many words about these three characters, I’ll keep that discussion for some time in the future.

The EV posts are comming !

EV i mievJuly 2013 it arrived

In July we purchased a 2012 I-miev ex-demo (Dec 2012 rego) from booths motor group Gosford for $25000 it’s now November so in 3-4 months we have done ~5000km. it’s had its 1500km service and the next service is at 15000km or 1 year.

Driving a EV is like driving a classic car, you rarely see them on the road and there is a nice community around them.

Like I said this is post number one more are coming !

NSW Solar Feed in Tariff

If you where lucky enough to install your solar panels before 30 June 2012 then you have nothing to worry about your nsw government guaranteed to get 60c for those pre 28/10/2010 in the scheme or 20c for those later. I didn’t own my house before that date so I missed out so we’ll mainly be looking at VFIT voluntary feed in tariff.

So to safe you the research I’ve put it in a table below using Energy Made Easy Website to look up power companies avaliable in NSW.

Company VFIT c/kwh Greenpower Website
Click Energy 10 No Click Energy
AGL 8 Yes AGL
Diamond Energy 8 No** Diamond Energy
Power Direct 7.7 Yes Power Direct
EnergyAustralia 7.7* Yes Energy Australia
Lumo 6.6 Yes Lumo Energy
Orgin 6 Yes Orgin energy
Red Energy 5* Yes Red Energy

*can’t confirm rate on company website need to ring for quote.
**Doesn’t offer certified green energy but does own green energy generators and no fossil fuel generators.

I export 3199.9 kWh/annually so at 8c it’s $255.99 and at the top rate of 10c/kWh it’s $319.99 so 10c give me an extra $64 per year. but that’s only a saving if the cost rate is the same.

There are other things to take into account when choosing a provide through, like the rates and discount for example click charge 27.39 c/kwh plus 7% discount 25.47 c/kwh which beat the discounted AGL rate of 26.44 c/kwh. Diamond energy with discount is ~26.56 c/kwh. daily service charge will also have to be factored in but it has less of an affect on the bill size. with click offering 78.10 c/day AGL 74.877 c/day and Diamond Energy 78.10 c/day.

I’m currently with AGL but I will be doing some further number crunching to work out if I can get enough savings out of click energy to justify me contract break fee that AGL will charge me if I leave.
I also dispprove on AGLs submission to the RET review so I’m not so willing to give them any more money.

Click energy doesn’t provide green power, this is a slight sticking point as purchasing green energy means not that my power comes from green sources but my money does. But that can be purchased seperatly to your energy bill and more directly to green providers so that might be an option.

New Electricity Retailer

So after crunching some more numbers and reading the green peace green energy guide I decided to change electricity retailers. Based of my need for a high VFIT (see previous post )  it was a choice between AGL (current provider), Click Energy and Diamond Energy.

Power Saving CalculationsPower Saving Calculations

Ok so the savings it’s not completely fair on AGL $55 of that $70 saving is 100% green energy which I’m not longer buying.  As click doesn’t offer it on their solar plan. but i can buy green energy from the a environmental trust for 4.2c/kWh and it’s a tax deduction.

Click saved me the most money has no contracts over AGLs 3 year killer and Diamonds 1 year one, it was also rated by green peace as middle range green. I’ve decided to move to click energy I’ll officially switch at my next meter read.

What about Gas well it’s going to switched later when click supports it. from twitter today:

If I’ve convinced you to switch and you want to get $50 click has a mates rates referral program  drop me a message and we’ll go from there.

EVSE for Sun Valley Toursit Park

So you might of seen a couple posts about Sun Valley Tourist Park, that is because we visit there a lot to visit grandma and grandpa (wife’s parents) .  So we decided because its outside of our return range we have to charge there to get home if we take the I-MIEV. but with the Electric Vehicle Supply Equipment (EVSE) that comes with the car limits the charge rate to 10amps max. So we convinced the park to install a 32amp EVSE.  This allow us to charge at the I-MIEV full rate of 13amps so 30% faster.

Aeroviroment RS at Sun ValleyAeroviroment EVSE-RS at Sun Valley

If you want to know more about the EVSE it’s an Aeroviroment EVSE RS.  It should work fine with the Holden volt, Mitsubishi Outlander PHEV, I-MIEV 2012 or later (may not work with 2010 models) and the Nissan LEAF.

If you are in the central coast and want somewhere to charge you can find the details on how to contact the park on plugshare. It’s available for public use depending on how busy the park is and the driver paying a nominal fee, and the driver phones ahead, during office hours.

 

Charging Infrastructure

A lot of people ask where do you charge. The answer is nearly for all EV drivers is at home. Some times the next question is do you need special equipment to which the answer is a powerpoint. (more specifically a 15amp powerpoint for the provide cable with the LEAF or I-miev). When out and about we have the following options.

 Commercial Infrastructure

There are two providers of commercial charging infrastructure in Australia, both are American Chargepoint and Blink, there was a 3rd Better place but unfortunately that company went broke.  Blink is yet to setup an Australian office so they are a bit harder to contact.

Chargepoint have a office in every state of Australia and have around 167 charge stations in the country. The chargepoint model is a low risk for them, it requires the person or business that wants a charging station to pay for the capital costs of supply and install of the charger.  It’s then up to the charging station owner if they want to charge the EV driver and chargepoint through the use of their RFID tags issued to drivers then take care of the payment system and charger driver accordingly, at present all the ones in Australia are free to use.

Chargepoint Network Blink Network

As you can see the blink network is much smaller, with only 5 sites and 7 chargers.  blink doesn’t let the site owner choose the price but instead charge $1USD per hour. I’m still waiting to hear from blink sales on if they have plans to expand in Australia.

Community Infrastructure

What if you don’t want to charge people or just provide a simple power point.  Well for these site there is a great site that EV drivers and Charging spot owners can used to share information.  http://www.recargo.com/search (you might have to pan to Australia) Allows you to sign up and add charge points to there map tell other EV drivers that a charging spot works by checking in (think foursquare and facebook), and upload pictures to help people find charge location.  This has probably been the most useful tool so far when it comes to charging infrastructure, I highly recommend all EV drivers install the app on the iphone or android.

Encouraging Infrastructure

Currently there still isn’t enough charging infrastructure, not because the current EV drivers need it but mainly because it puts people off buying a EV.

In a small effort to make it easier for business to understand what’s required to provide a service to EV drivers I prepared a primer

Electric Vehicle Charging Solutions for Businesses

Hopefully other EV drivers can use this when negotiating with companies about added a charging station to there site.

Kickstarter Field Hockey Game

For years in high school I had my hockey mad team mates telling me how great a field hockey game would be…

Looks like someone finally listerned there is currently a kick starter campaign to get one made http://www.kickstarter.com/projects/urbanwarfarestudios/the-field-hockey-game-pc-mac-linux

Love hockey but thing gaming is a waste of time ….. well think of this as a promotion of the sport well worth your time. I’ve backed it !

Betterplace, the first to fall.

Around May 2013 betterplace pulled out of Australian operations in an attempt to keep it’s parent company a float. This plan wasn’t overly great as the parent company still went bankrupt. http://www.smh.com.au/business/carbon-economy/how-better-place-got-lost-20130527-2n7gp.html this SMH article goes into a bit more depth, on why how and what’s to blame.

The real unfortunate thing is they put in quite a bit of infrastructure, with their own money and through government support through smart grid smarty city funding and now it’s being removed or unusable.

Below is pictures from at the Morisset Level 3 Charger, before and after removal.

November 2013 December 2013

The good news for the Level 3 fast chargers is that schneider electric have taken possession of these units back (they originally provided them) and at least they will be resold and not placed on the scrap heap.

It also appears it’s not just the level 3 but the level 2 after noticing a local shopping centre was adversing EV charging on there facebook page I emailed them to sadly only get this reply

Good morning Gabriel

Regarding your query about our Electric Car Charges, we did have charging areas however these have recently been removed from our Centre.

Kind regards

Michelle Thomas
Centre Manager
Commercial Property
Retail | Office | Industrial
Stockland Jesmond & Wallsend
28 Blue Gum Road, Jesmond NSW 2299
T +61 2 4955 9249 M +61 438 266 707
F +61 2 4955 8014 E michelle.thomas@stockland.com.au
www.stockland.com.au
BeGreen… and consider the environment before printing this email

So while I’m working madly to try and increase the amount of EV chargers in my area they are being removed at a faster rate that I can convince people to put them in.

Recharging NSW

So those who have been following this blog know that I’ve been a keen enthusiast for EVs attempting to grow and expand the amount of EVs in Australia and the related charging network.

Some of my frustration on how slowly it has been growing has turned into why don’t I do something about it.

So I have. I’m now a director of a new company Recharging NSW Pty Ltd. The main aim is to encourage and support EV uptake in Australia.  By increase both cars on the road and public charging.

So there isn’t much I can share at present everything is still in the planning phases. but stay tuned.

 

 

 

EV friendly accomondation

LEAF Meetup

On the 27 of October met up with some LEAF drivers from Australian LEAF Owners Forum at Sun Valley Tourist Park see the above picture for a line up of their Leafs. It was a good meet up most of them had not met before and basically the whole meet up was based around interest in the cars.  so naturally the talk of the next meet up came up and I suggested next years 2014 hunter EV festival it’s actually down for a weekend this day 1 at the race track day 2 the expo at the foreshore.

So in an attempt to help out I sent this email out to the accommodation around Newcastle

Hello
In the lead up the the 2014 hunter Electric Vehicle festival weekend I’m canvassing Newcastle accommodation for places that are Electric Vehicle friendly so i can provide a list to out of town EV drivers.

So the first question is would you like to be added to the list ?

If you are interested I would like to know how many power points you have within close proximity to a car park?  and what type they are ? (most will be 240 Volt 10 amp)

The last piece of information I need is if there is a surcharge to the accommodation cost for EV charging and how much will it be?

Thank you for your time

What I forgot the put in the email was the dates August 16th and 17th… and the approximate cost of power which I estimated at 15kwh (~6 hours of time) so at $0.25 per KWh $3.75.  So far only had one reply which was positive but you can keep and eye on my progress by checking the spreadsheet on google drive

 

 

Solar 1 year on

this time last year we had the solar installed well by this time it was well on the roof … this date last year we had the meter replaced and the solar turned on.

The statistics:

  • kWh Imported 4295.3
  • kWh Exported 3199.9
  • kWh generated according to the inverter 4936
  • Fit Collected at 8c kWh $255.99
  • kWh not purchased or sold 1736.1

I’m currently on a flat rate of 29.084 c/kWh (note my rate has gone up since solar was installed) minus 10% discount  plus 5.5  c /kWh for green energy = 31.94 c /kWh. So the amount not purchased or sold is what I saved by not buying 1736.1*31.94c = $554.51 + the money I got from Fit $255.99 so in a year it’s saved me ~$810.5.

Has the solar been a good investment no…the capital cost was around $8.5k so with the amount saved it’ll take 10 years to pay back. The main reason for this is that we export way too much and the fit is so low if we got paid what it costs us it would of saved us ~$1500 a year and only 5.5 years to pay back.  Do I care if it was a solid investment not really.

If we look at how green it is if we take imported – exported (1095 kWh)  that’s how much power I’ve used from other generators which for my area is black coal, but that has been offset by my green power money purchasing green power from wind and biogass. so does my house run emissions free when it comes to electricity according to an accountant yes, because every kWh of power I’ve used has been purchased from a green source  but maybe not according to an engineer.

Other interesting notes on the power bill:

average kWh used per day including solar from April to may 2013 before purchasing the EV was 12.1 kwh

average kWh used per day including solar from July 2013 to April 2014 post purchasing the EV is 16.5 kwh

so the effect of owning an EV on our power bills is about 4.4 kwh per day $1.40 increased cost. Note: this would also include seasonal cost extras like summertime air con and winter time heating so I won’t have a clear picture until we 1 year of EV ownership.

Linux on XDA…

I ended up starting to work a bit on getting Linux onto the XDA (It’s a very nice PDA phone, for those who don’t know). After spending about two weeks of fooling around, and getting help from people from #handhelds.org (joshua, pb, anpaza, and many others), I’ve finally got it booting. Currently with a very basic initrd image root, with a busybox shell at the serial console. Also trying to get the screen to work properly.

Some of my progress can be seen at the xda-linux photo section.

WINE 1 : Windows 0

Got this old but very popular game I wanted to play. It’s a Windows game, but not only that, the version I have is the simplified chinese version of it (I can read it, but not as good as traditional).

I was “lucky” and got someone else’s Windows XP box to try and see if it runs, and it doesn’t. It starts up and says it needs the version of the game only works on a simplified chinese version of Windows. Then I thought, I’ve seen similar problems with other programs. Let me try fiddling with those locale and default language and encoding settings in the control panel, not to mention that I need to reboot Windows almost after every change of the settings. However, none of the changes helped. The game still won’t run. And because you could ask, no, we don’t have a simplified version of Windows.

I know for the fact that the game actually runs under WINE pretty well (good thing being an old game I guess), so I gave it a go. Installation worked. Then when I run it, yup, got the same simplified chinese version error. But there’s something different. The texts of the error dialog box are all garbage. I imagine they’re simplified chinese, but somehow not using the right font or encoding. I decided to fix this problem first, which I knew it would be easy cos I have had same problems with other chinese games before. All you need is to set LANG, in this case, to zh_CN.

And guess what, that not only solves the chinese encoding issue, that also convinces the game to think I’m on simplified chinese version of Windows. The game worked beautifully. And need not to say, setting an environment variable doesn’t require rebooting… unlike some retarded operating system.

WINE wins!

Linux, ASUS W7J, and the ACPI brightness keys

Was meant to write up a page about Linux on my ASUS W7J, but I never got the time yet…

Anyway, here’s a possible quick fix for those with an ASUS laptop (W7J at least) and a non-working brightness acpi key (in my case, it’s Fn + F5 and Fn + F6), even though /sys/class/backlight/asus-laptop/brightness works perfectly.

I poked around the disassembled DSDT code and did a bit of googling. After a few reboots and testing, I figured out passing acpi_osi="!Windows 2006" to the kernel fixes the issue.

Apparently with the (current) Linux ACPI driver, the OS interface (OSI) code will respond to the ACPI as quite a few different versions of M$ windows (see drivers/acpi/utilities/uteval.c for the entire list). This list can be modified by the kernel parameter acpi_osi (see Documentation/kernel-parameters.txt for full details).

So my guess is the DSDT thinks I’m running “Windows 2006″ (aka Vista), and so it is doing (or not doing) something with these two particular ACPI keys. Telling the Linux ACPI driver not to pretend to be Vista got rid of the problem.

This quick workaround also means that I didn’t have to patch my own DSDT and build a kernel with it.

[Updated: 5th November 2007] I was testing 2.6.23.x on the W7J, and I noticed both brightness keys Fn + F5 and Fn + F6 work without acpi_osi. I looked into it and I’ve found out that the ACPI video driver makes the keys work. And for some reasons with older kernels (2.6.22.x) the video module is not loaded automatically. So you can probably ignore what I said above months ago :)

Pikkoro – Episode 2 – Loopback rootfs with Debian

My Linux live CF is going well. I have been using it on my wife’s laptop.

Next step I was trying to get suspending working. ACPI sleep state S3 (Suspend to RAM) works pretty well as expected. However when the laptop comes back from the suspend, there seems to be a delay, or timeout, or some issues with USB. As this rootfs is on USB, I’m getting read errors, and from that point on, the rootfs has gone to a funny state, and things stops working.

The easiest way to solve this problem for the moment, to me, is to totally avoid the problem. I need to get Linux running off the harddisk on this laptop. And at the same time I do not want to install, or repartition the existing windows installation. So I decided to go for loopback rootfs.

The loopback rootfs on Linux is not anything new. I briefly did a google search and there’s this loopback root filesystem HOWTO written back in 1999. Though, I want to get this done in a maintainable way. Being a Debian user, hence, I want to do this in a Debian way.

For this Linux live system, I’ve been a good Debian user and keep everything done in the Debian way, so that it’s easier for me to manage, recreate, and update/upgrade the system. Kernel initrd image is handled by initramfs-tools. So rather than putting together an initrd image manually to do all the tricks, I quickly looked at how initramfs works, and wrote a loop script, which is pretty much the same as the existing local script, but with one extra step. And I’ve also added a few more kernel parameters to make things easier. Here it is:

# Local loopback filesystem mounting

# Parameter: Where to mount the filesystem
mountroot ()
{
        for x in $(cat /proc/cmdline); do
                case $x in
                rootfstype=*)
                        rootfstype=${x#rootfstype=}
                        ;;
                loopfile=*)
                        loopfile=${x#loopfile=}
                        ;;
                loopfstype=*)
                        loopfstype=${x#loopfstype=}
                        ;;
                esac
        done

        [ "$quiet" != "y" ] && log_begin_msg "Running /scripts/local-top"
        run_scripts /scripts/local-top
        [ "$quiet" != "y" ] && log_end_msg

        # If the root device hasn't shown up yet, give it a little while
        # to deal with removable devices
        if [ ! -e "${ROOT}" ]; then
                log_begin_msg "Waiting for root file system..."
                if [ -x /sbin/usplash_write ]; then
                        /sbin/usplash_write "TIMEOUT 180" || true
                fi

                slumber=1800
                while [ ${slumber} -gt 0 -a ! -e "${ROOT}" ]; do
                        /bin/sleep 0.1
                        slumber=$(( ${slumber} - 1 ))
                done

                if [ ${slumber} -gt 0 ]; then
                        log_end_msg 0
                else
                        log_end_msg 1 || true
               fi
               if [ -x /sbin/usplash_write ]; then
                       /sbin/usplash_write "TIMEOUT 15" || true
               fi
        fi

        # We've given up, but we'll let the user fix matters if they can
        while [ ! -e "${ROOT}" ]; do
                panic "ALERT!  ${ROOT} does not exist.  Dropping to a shell!"
        done

        eval $(fstype < ${ROOT})

        [ "$quiet" != "y" ] && log_begin_msg "Running /scripts/local-premount"
        run_scripts /scripts/local-premount
        [ "$quiet" != "y" ] && log_end_msg

        # FIXME This has no error checking
        modprobe -q ${FSTYPE}

        # FIXME This has no error checking
        # Mount root
        mkdir /preroot

        if [ "${rootfstype}" ]; then
            opts="-t ${rootfstype}"
        fi
        # mount the device containing the file image as rw
        mount -w ${ROOTFLAGS} ${opts} ${ROOT} /preroot

        [ "$quiet" != "y" ] && log_begin_msg "Running /scripts/log-bottom"
        run_scripts /scripts/local-bottom
        [ "$quiet" != "y" ] && log_end_msg

        if [ "${loopfstype}" ]; then
            opts="-t ${loopfstype}"
        fi
        # mount the file image loopback
        mount -o loop ${opts} /preroot/${loopfile} ${rootmnt}

        if [ -e "${rootmnt}/preroot" ]; then
            # bind mount the real device so that it is accessible later
            mount -o bind /preroot ${rootmnt}/preroot
        fi
}

With this file in /usr/share/initramfs-tools/scripts/ or in my case, I simply made a copy of what’s in /usr/share/initramfs-tools/ and /etc/mkinitramfs/ and run mkinitramfs -d <confdir> -o initrd.img-pikkoro.loop. Anything I’ve done is adding modules I needed in the “modules” file. Mine looks like this:

usbcore
ehci-hcd
ohci-hcd
uhci-hcd
usb-storage
sd_mod
loop
ext2
ext3
vfat
jbd
nls_iso8859_1
nls_cp437

I’ve created a 4GB ext3 filesystem in a file, stored on the windows fat partition. In my case, the loopback file is on the windows partition /dev/hda5, with the path /pigeon/pikkoro.img. So my kernel parameters are:

root=/dev/hda5
boot=loop
rootfstype=vfat
loopfile=/pigeon/pikkoro.img
loopfstype=ext3

(shown in separate lines here for each parameter to make it clear)

And that’s it. Now I can boot this laptop via USB, then run Linux with the rootfs on the filesystem file, mounted as a loopback. And more more importantly, suspend-2-ram and resume is now working well.

RAAF airshow 2006

On 21st October I went to the first day of the Royal Australian Air Force (RAAF) airshow at their base in Richmond, NSW. It was roughly a two-hour train trip from home (one way).

I got there sometime just before 11 I think. Weather wasn’t very good and quite windy from time to time, but it was very comfortable as I was definitely not sweating, a lot better than one of those 37 degree weekends (which was the week before). It was very crowded, with a lot of families with kids and everything.

So I was pretty much standing and walking around for over 6 hours, taking photos, watching flying demos and aircraft displays. It’s absolutely good fun if you love planes. For me maybe it was too much of a warbirds and fighters show, but still very enjoyable. There were aerobatics, formation flying, attacks demo, helicopter joy riding (which I didn’t do, it wasn’t free :P), also some R/C aircraft and gliders displays. F-111, Hornets, Hercules, Boeing 707, Vampire, Tiger moth, and many many more…

I took almost 300 photos in total, though most of them look pretty dark and dull due to the weather. And to satisfy my panorama need, I’ve taken a 360° panorama view at the base at the end of the day.

Note to self:

  • Need better camera for this kind of show, a DSLR with like a 30x optical zoom lens would be great :)
  • Hang around at the end of the day when almost everyone are leaving, perfect time for taking even more photos wherever and however you like, without the extra random people.

My RAAF airshow 2006 album is here.

Click for more RAAF airshow 2006 photos Click for more RAAF airshow 2006 photos Click for more RAAF airshow 2006 photos Click for more RAAF airshow 2006 photos Click for more RAAF airshow 2006 photos
On a side note, there was this very odd Linux “sighting” at the show. If someone could explain it a bit that would be great ;)

Should probably start planning for the next airshow soon…

vimpress again

It was about 8 months ago when I started using vimpress. And I’ve actually modified it a bit since then. So far I’ve made two tiny changes.

The first is checking for vim python support before doing anything. I did this because in Debian vim (vim.basic) isn’t compiled with python support, and I prefer vim.basic over vim.gtk or vim.gnome as my default vim (and as for the reason, see my previous vimpress post.

The second is adding a :BlogSave command. Basically sending and saving an entry without publishing it, i.e. saving it as a draft, which I do a lot myself.

I’ve kept it in my git repo now at git://pigeond.net/vimpress.git

Gitweb at http://pigeond.net/git/?p=vimpress.git

FlightGear + Wiimote

So, we bought a Wii. Amazingly my wife managed to walk-in and bought apparently the second last Wii in that shop, on the first day of the release in Australia. We also got an extra Wiimote, Zelda the game, and two Wiimote grip protectors.

With thanks to WMD, you could use the Wiimote under Linux fairly easily. For those who don’t know Wiimote talks to the Wii console via bluetooth. So it works with any computer with bluetooth as well.

People have been using the Wiimote on the computers in all sorts of different ways. As a guitar (Wiitar), as a drum machine, for games like Counter Strike, a virtual light sabre, etc. As for me, using the Wiimote with a flightsim, like FlightGear, would naturally be the first thing I wanna try.

The Wiimote has three accelerometers for the three axes. I’ve modified slightly the WMD script to send the X and Z force to the uinput device as the absolute coordinates. jscal then takes care of the calibration of the two axes. Then I changed the commandMap in the script to send BTN_0 to BTN_9 from all the buttons on the Wiimote.

On the FlightGear side all I need is to write a joystick definition XML for the Wiimote. I’ve also made the ‘B’ button to be the modifier. For example I have arrow Up and Down for throttle, Left and Right for the rudder. Then if I hold down ‘B’, Up Down Left Right become panning the viewport. ‘1′ and ‘2′ for changing views, ‘+’ and ‘-’ for zooming in and out, ‘A’ for brakes and gears. All set. Now I can fasten my seatbelt^H^H^H^H^H^H^H^Hwrist strap, sit back and ready for take off.

Updated 5th Jan 2007: Here is a patch against WMD SVN repository. Like I said it’s pretty hacky at the moment. And you definitely have to use jscal to calibrate it before it will behave sanely. Also you probably want to make changes to the commandMap in wmd/Config.py to map more buttons.

And for those FlightGearers out there, here is the Wiimote joystick XML config I’ve made.

Here’s a video of FlightGear + Wiimote in action:

LCA 2007 Day 1

Hot day, cool people; nice LCA bag, bad wireless. My first day at LCA 2007.

It was pretty exciting to meet for the first time with a few people who I have known online for quite a while. George (or Gorilla) from the FlightGear gang, Leeds I know from #handhelds.org, and Endy from the ScummVM team showing off the latest version that has AGI support (for some of the old Sierra games like Space Quest I). Looking forward to his lightning talk at the Gaming Miniconf tomorrow.

Apparently Jeff ended up demonstrating the Wii in the Gnome miniconf (which I have missed). I reckon he should have demo using the Wiimote on Linux, which is pretty easy to setup. Or maybe that should be in the Gaming miniconf… :)

On the other hand, I was having weird trouble with the wireless on my laptop. It’s a Intel Centrino PRO/Wireless 2200BG, and I was using the ipw2200 module with the corresponding firmware. For some reasons it wasn’t able to get an IP from DHCP most of the time. I managed to get an IP once, but it dropped out pretty much straight away. I tried setting the same IP I got statically but it was only partially working (DNS working, for example, but nothing else worked, which was expected).

After consulting Jamie and John a bit they reckoned my wireless somehow kept dropping packets. And to be honest I haven’t actually use the wireless much on this laptop. I have no idea whether it’s a hardware/radio/firmware/driver problem or what. I ended up using my CF wifi card and it worked beautifully.

Anyone in the conference is welcomed to fix it for me :)

Took fewer photos than I wanted to today, but anyway they can be found here. I’ll try to put up new photos there for each LCA day.

Fun with Wine hacking

So I got this PC version of the game Shin Sangoku Musou 3 (真三國無雙). And of course it for some proprietary operating system (we do not speak of its names :P). As a Linux person the first thing you try is to try running it under Wine / WineX / Cedega.

A little bit of surprise, it ran with Wine latest development version, at least it’s gone thru all the splash screen, characters selection, until it gets to the actual game, crashed.

After turning on some debugging and tracing, followed by consulting on #winehq, I’ve learnt that this game hits one of the famous Wine and DirectX issue, multi-threaded Direct3D.

Still I’m a bit adventurous as usual, and did a bit of code reading and hacking in Wine, trying a few different quick-and-dirty approaches to get threads to use the same GLXContext, not that I’m an OpenGL guru.

After a few tries, I gave up. However I still kind of wanna be able to play this game under Linux. So I rethink the problem from a different approach, and did a even more dirty hack, so that only the main thread can do any GL calls. Any other threads will simply returns before trying. Yes, VERY UGLY indeed. So ugly that let’s not even talk about it anymore ;)

And guess what, that does the trick. It gets into the actual game. So apparently it’s creating a new thread only for the pre-game loading screen. So now instead of the loading animation screen, I simply get a black blank screen, and the game continues after that.

And just for fun, I’ve even created a character with the top Taiwanese model’s name :P

More screenshots here.