January 5th, 2012

Fixing the life support

Somehow, this year feels different.

Maybe it’s because of the increase in public awareness of the problems with the world at the end of 2011. A greater understanding that governments and people don’t run the world any more, that we have given money too much power, that we are running full tilt at a cliff-edge; it feels like it’s adding up.

Charlie Brooker said recently that 2011 felt like a series finale; I’m not sure about that, it felt to me more like the buildup in the penultimate episode. 2012 seems like the actual last episode when the big events will happen (probably with a few cliff-hangers left over for 2013).

This impending feeling that big events are coming could be imaginary of course, but I hope not. We need big events now (and I don’t mean the obvious 2012 fake-apocalypse fantasy). We are racing so fast towards collapse that we have to do something now, or we will be a cautionary tale in the Cockroach Book Of Ancient Civilisations. We still have the ability to save ourselves (just); I hope that this year brings the will to engage with the problem.

We need to grow up, realise the full-spectrum danger we are in, and do something about it. We need to mature as a species, and fast. We need to get over our selfish approach and make a better society for the future, together.

I am a software engineer; that makes me next to useless in a collapsed society. My daughter quite literally needs a functioning, liberal, caring society to keep her alive. Moving to a farmhouse, becoming self-sufficient and letting everyone else collapse is no good for me. A functioning technological society is an essential life support system.

With all that in mind, I am going into this year with an objective to make best use of the skills I have to attack the big problems where I can, and if it doesn’t work out, to fail magnificently having done my best

I hope you’ll join me. We’ve got a hell of a job to do.

July 1st, 2011

Hohm and Powermeter

The last week has been an interesting one for online energy monitoring. Not content with deprecating Powermeter a few weeks ago, Google decided to properly kill it off. And then, as if to say ‘us too!’ (as they did with the initial launch), Microsoft killed Hohm.

Both these efforts were unsuccessful because the companies that put them out weren’t serious about them. There is a place for experimentation, but if you’re a massive company, even your small experiments seem larger than other people’s entire companies. By half-heartedly grabbing the attention around energy monitoring, these experiments crowded out the people who are building a business around this, like Currentcost, Enio, Pachube and many more.

The fact is that online energy monitoring is not dead - the demise of these services will simply provide more room for other companies to innovate without being stepped on. Neither Powermeter or Hohm were doing anything particularly innovative that hadn’t been done many times before anyway! Powermeter also suffered from being a silo of data, not a hub. You could put data in, but not get it out, which is a bit pointless. A truly useful service in this space needs to act as an aggregator, letting data flow through it so that other innovations can cluster around it, like the Social Meter project, for instance.

Innovation needs openness, and by smothering a potentially innovative space with their distinctly non-innovative services, MS and Google did nobody any favours. We are better off without them.

June 8th, 2011

Resolving a method_missing fight

Recently, I was writing tests for a piece of code, and came up against some strange problems. The code talks to an external web service using RestClient (1.6.1), so of course the web service is mocked for the tests, in this case using Mocha (0.9.8).

The problem came when I tried to test failure modes; for instance, what happens when the service comes back with a 401 due to bad credentials? RestClient raises an exception for this containing the response, so the mock has to do the same.

request.stubs(:put).raises(RestClient::Unauthorized.new(
stub(:code => 401, :body => nil)
)

So far so good. The exception is raised, and all is well. The problem comes when we try to access the response code later on in our error-handling code:

rescue RestClient::Exception => e
  puts e.response.code

Seems reasonable, right? The response object should be a Mocha::Mock that responds to #code and gives back 401. Unfortunately not. We get a ‘stack too deep’ error thrown inside a recursive method_missing call. Huh?

This really should work, so let’s take a look inside the gems. It turns out that RestClient mixes something special into the response object (in this case the mock) for compatibility. The gem used to return Net::HTTP::Response objects, not RestClient::Response objects, so in order to retain compatibility with old code, it mixes in the behaviour of the Net::HTTP class to responses contained inside exceptions. This is taken from restclient/exceptions.rb:

# Compatibility : make the Response act like a 
# Net::HTTPResponse when needed module ResponseForException def method_missing symbol, *args if net_http_res.respond_to?(symbol) warn "[warning] ..." net_http_res.send symbol, *args else super end end end class Exception < RuntimeError attr_accessor :message, :response def initialize response = nil, initial_response_code = nil @response = response @initial_response_code = initial_response_code # compatibility: this make the exception behave like
# a Net::HTTPResponse response.extend ResponseForException if response end end

As you can see, it does this by adding its own method_missing, which accesses the net_http_res function of the response. In our mock, that doesn’t exist, so of course method_missing is called ad infinitum. So, let’s add it:

request.stubs(:put).raises(RestClient::Unauthorized.new(
stub(:code => 401, :body => nil, :net_http_res => nil)
)

Better. Does it work now? Hm. No. It’s still digging a method_missing hole to the Earth’s core.

The real problem is that the method_missing that is mixed in by RestClient::Exception is masking the one in Mocha::Mock, and that one is what does the actual handling of the stub code that we pass in. So, even though we added net_http_res up there, the bit of code that handles it is never called, so we’re still stuffed.

Now, there’s not a lot we can do about that without rewriting the gems. If I was going to do that, I’d try to make sure that RestClient::Exception doesn’t mask existing method_missing functions, but instead aliases them and makes sure they are still called. Not sure how easy that would be, I’ve not tried it, but it would be worth a shot.

I would try to make some general rule here about making sure your code plays nice with other code, but to be honest, I doubt you can consider every case where someone might be subverting your code by giving it things like mock objects (which could implement the mocking in any number of ways) instead of real ones. I guess there will always be a gap somewhere.

The solution I hit upon in the end was to change the stub. Instead of using Mocha, which relies on method_missing to implement its stubbing, I changed to OpenStruct, which doesn’t, and is sufficient in this case.

request.stubs(:put).raises(RestClient::Unauthorized.new(
OpenStruct.new(:code => 401, :body => nil, :net_http_res => nil)
)

Yay, by using a stub that doesn’t require method_missing to operate, it all works nicely now! My tests are saved and I can get on with some real work… about time.

April 20th, 2011

Last year, I went to the Rewired State Carbon & Energy hack day (#rscarbon), which was a great event. I went along with an idea, and even better actually got to build it! We got a great team together, and despite initial technical teething problems, had a demo running in time for the show and tell at the end (by the skin of our teeth). I’ve been meaning to write it up for ages, but instead, I decided to make a video.

The code is, as ever, available on Github, and there are other writeups on Paul Tanner’s blog and the Rewired State site.

November 26th, 2010

Francis Maude on the LRRA

In 2006, I wrote to my MP about the Legislative and Regulator Reform Bill (now Act). The bill contained extremely dangerous powers for Ministers to change anything they liked, giving them the ability to get away with pretty much anything without a proper vote. I campaigned against it, with many others, through Save Parliament.

Anyway, my MP happens to be Francis Maude, who is now Minister for the Cabinet Office. It turns out that his government is now advancing a bill with similar powers, the Public Bodies Bill. Now, I admit I’ve not had chance to follow this in detail, so I don’t know if it’s as bad, or if it’s more limited. However, others who know more might be interested to know what Francis Maude had to say about those sort of powers four years ago.

“I am particularly concerned at the potential for Parliament to be bypassed by the order-making powers contained in Part 1 of the Billl. These powers are extremely constitutionally significant.”
“It is imperative that the circumstances in which these powers can be used are limited and clearly set out in the Bill.”

The full scans are available on my Flickr stream, for anyone who is interested.

Maybe the PBB is consistent with what he said 4 years ago, but maybe not. I’d be interested to know.

October 25th, 2010

Rewired State: A Social Energy Monitor

The Rewired State Carbon & Energy hack weekend is coming up this weekend, and I’ve got a proposal for a hack I’d like to build (by kind permission of AMEE, my employer). I’m after a team to help, so if you’re interested, please get in touch.

Energy monitors are useful, but lack a certain something to keep people coming back to them. The web excels at social and game mechanics, but is ‘out there’, not present in the normal home environment. The idea is to bring the two together by building a system that takes realtime energy data from a home and displays similarly realtime social context for that data.

The end product is a simple display that shows not only current energy usage, but how that usage compares to other users of the system, in real time.

How it works: each home (probably just one in the initial hack, with the rest mocked up) has a CurrentCost meter. This, via a script or bridge device, submits 5-minute energy usage readings into AMEE, where it’s stored and its carbon equivalent calculated. A website is then created to take that information from all registered devices and do some simple stats. It will calculate a realtime league table, position for each user, realtime average power usage, and so on. This, along with the user’s own data, is then displayed on a screen in the home - I have an O2 Joggler touchscreen which would do the trick, either through a browser or via a native OpenPeak app.

The panel continuously shows the home user their energy usage, but also what their usage is compared to other users right now. It might display chart position, whether usage is going up or down, and other simple stats perhaps aggregated over time. This will bring some essential social context into the home energy monitoring space, and demonstrate the sort of innovation that can take place if smart metering systems are kept open.

Many parts of the system exist already, so I think this is buildable in a weekend at least as a prototype. Like I say, if you want to help me build it, get in touch in the comments or via Twitter (#rscarbon).

May 6th, 2010

What did you do in the election, Daddy?

One wondeful thing about the 2010 General Election is that so many great projects have happened online; Francis Irving covers them in a post on the OKFN blog.

I was personally lucky enough to find the time to use these services to build something. I couldn’t find election hustings in my area, so I decided to make a site to crowdsource and open up that information properly. Enter QuestionYourCandidates.

The site allows anyone to enter details of hustings in their area, basically, and link to them in a reliable way. It’s made from:

  • Rails 2.3.5 - this reminded me how excellent Rails is for rapid development
  • Constituency & candidate data from YourNextMP through their API.
  • Postcode search via the TheyWorkForYou API.

Both these APIs were incredibly easy to use, and because they use canonical names for both constituencies and candidates, it was easy to then link the site across to other services like Democracy Club and The Straight Choice.

The main dilemma while coding I had was whether to have user accounts. I decided against it in the end, just protecting the edit pages with a captcha. User accounts would have just added another layer of complexity, for really no extra functionality or security.

The whole thing took around 6 hours from start to deploy, and I’m pretty proud of it. Unfortunately I did it a bit late to really be useful for this election, but I will try to generalise it to is can continue to be used for future elections. And, as always, it’s open source. Code is on Github.

December 17th, 2009

In early September, Betavine organised a 24-hour hack event called EcoMo09, with an environmental theme. Unfortunately I couldn’t stay the whole time, but I went along as an AMEE representative to support other people using our API during the event. In the quiet periods, I managed to come up with my own hack, which used GreaseMonkey and AMEE to add carbon emissions to Google Maps.

You can even get the code and try it yourself. It’s very basic, and actually most of the work was dealing with Google Maps and making Greasemonkey behave properly - the AMEE bit, getting the carbon value, was dead easy.

November 10th, 2009

The Activist Geek Philosophy

So, first, let’s have a stab at identifying the community I’m talking about - this in itself is a pretty hard problem. I guess I’m talking about “activist geeks”, in a way. The sort of people who are involved in the work of MySociety, or Social Innovation Camp. The people who go along to events like geeKyoto, or perhaps Interesting. They are the people that go out and create innovative things that make the world better. I can’t put any names on a list here, though there are some obvious ones of course, but you will probably know if this is your sort of community or not.

So having made a (very) vague stab at who I’m talking about, what are the core philosophies that come out of that? One caveat: obviously this is my own personal viewpoint, and implies absolutely nothing on behalf of those examples linked above. I could be completely wrong on all of this (though I’d invite comment). I think I am what would traditionally be called “left wing”, so my please note that my views may be coloured by that.

Firstly, I think that while there are certain things that do unite the community, they often don’t fit along traditional political lines. You cannot say that this community is left-wing for instance; there are certainly members of all political parties involved. We’ll come back to some of the things that don’t unite us later on, but let’s find some common ground first.

Privacy
Even though many of us share vast amounts of information on various social networks, we have a strong belief in privacy of personal data. Perhaps this is better phrased as belief in control of personal data. I can delete my social network accounts at any time, and can decide who can see them, so I remain in control (as long as I trust the site). On the other side, I will not be able to remove my record from the ID card database or decide who accesses it, so I am not in control of that. This is part of a more fundamental belief in universal human rights.
Transparency
While personal information should be kept tightly controlled by its owner, information about (and generated by) government and public services should be fully transparent and open. The MP expenses scandal was an example of further erosion of trust in government because of an opaque system that was open to abuse. As they say (I think), sunlight is the best disinfectant. Government operations should be open to public scrutiny, and assets produced by that system (OS maps anyone?) should be freely available to the taxpayers who have paid for them.
Tolerance
I don’t think we give a monkeys where someone is from, what their background is, or about their religion, sexuality, or favourite colour. It’s just not relevant. The rest of you, get over it.
Anti-authoritarian
It always confuses me that a criticism of the left is that it wants to control people. Maybe that’s true, in which case I need to recalibrate my open personal compass a little, but my belief is that the community is very much about freedom of the individual. There is a strong anti-authoritarian streak - none of us want to end up in a police state.
Social Conscience
However, the above freedom of the individual does not lead to blind self-interest. I believe that the community is not one of self-interested people with no care for others, but of a group of empowered individuals who actively engage in and make society better.
Engagement
My final point is that the above lead to a community that is engaged with the world around them. We engage on issues of importance, such as my own personal focus, climate change, and they are interested in the way their society is governed. I believe that those qualities are fundamental for a functioning democracy, and we see that currently most of society does not measure up to that standard (a situation which we try to improve of course).

On the other hand, I seriously doubt we are all united on things like economics (I love that picture), the role of corporations, how to deal with climate change, foreign policy, and so on.

I’m sure there is more that can be said, but I will let it play out in the comments (if anyone wants to add to this). My final question is - if there is a central theme emerging, does that mean we should do something with it in these days when so many of the above viewpoints are rather missing from general society?

Discuss.

October 28th, 2009

IBM DeveloperWorks articles

Over the summer, I was asked to write an article for the IBM DeveloperWorks site about energy monitoring and so on. Eventually, that turned into a series of articles centered around AMEE and how you can track energy consumption and calculate carbon emissions with it.

So, now that they’re all published, here’s a bit of a linkdump to the various articles and other things that came out of them.

Introduction to AMEE
This is a technical introduction to the AMEE platform, how it works, and what you can do with it. It covers things like signing up, creating profiles, storing data, and getting results. It was also adapted into a screencast for the AMEE developer site.
Calculate your computer’s carbon footprint using AMEE
This one is an introduction to the wide variety of ways in which you can track energy data using AMEE, using the monitoring of computer energy use as a case study. It also shows some code examples for the AMEE ruby gem.
Monitor home energy with AMEE
This is my favourite of the three - a long tutorial, with lots of code, which shows a complete implementation of an online energy-monitoring system, using a CurrentCost display, AMEE, and a little Rails code. This uses a lot of the open source software I’ve been writing over the last year, which regular readers will have seen in previous posts, but it’s really nice to tie it all together and tell the world how it works!
James Smith on embedding environmental intelligence
Scott Laningham was nice enough to interview me for the DeveloperWorks podcast when the first article went out, so if you want to hear me blathering on about it in person, you can do so!