I have a theory. Its a variant of Conway's Law. It is:
developers write applications that work best in the network
infrastructure of their organisation
MS apps are pretty good on a laptop, provided the wifi is
high-bandwidth and highly available. Because the engineers get to
spend a lot of time in meetings in a campus with excellent wifi.
Sun products (like Java) cant handle roaming laptops as the
engineers don't work on the move enough. Open source apps are
fantastic with complex network conditions, though a lot of the
consumer centric apps can't handle proxies, because the developers
at home don't have them.
And nor, clearly, do the google team. Because Chrome, fast and
lovely that this, hands off proxy setup to the OS, which is
generally hopeless at the matter. Whereas with Mozilla and
switchproxy, all is well. So my laptop will be sticking with
mozilla. And my desktop, 64-bit unbuntu, will be sticking for a
while longer.
That doesn't mean that I don't think chrome is lovely; its a
browser designed to host heavy duty javascript client apps. I can't
help thinking of Netscape 3, with Java and and a Corba ORB built
in, but this one is at least driven by existing apps, not a
mistaken assumption about the future of distributed computing. It's
certainly far more appealing than the IE product chain, that seems
stuck with the problem of maintaining IE6 compatibility and deep OS
integration.
Its interesting because it shows how both open source and
externally-provisioned datacentres can be useful in computing
research; Azeez gets to extend Axis2 and related products, deploy
everything onto EC2, and write a nice paper on it. Well worth a
read.
From an academic perspective, I'd criticise it for not using or
citing Lamport's definition of liveness, which is essentially: A
live system is one that does useful work to external callers. If a
system is returning 200 on its happyaxis.jsp, it may still not be
able to talk to a database it needs, and hence isn't live.
If a system is successfully responding to local requests, but
unreachable from the outside, it isn't live. It thinks it
is, but the network has partitioned.
Once you adopt that view of liveness, then the only way to
determine if a system is really live is to call it from the
outside; in an ideal world you have monitor points or proxies on
all the main networks and pull them in to have a view of system
health.
They have a group system which tries to deal with the lack of
multicast by using well-known IP addresses to bootstrap the group;
that's a bit of a hack. You can eliminate those by using
XMPP+google talk (whose outage would probably indicate a network
partition) or a shared folder in S3, whose outage would stop your
EC2 app doing useful work anyway, so not impact the SLA at all.
Overall though, a nice paper. I think the claims to High
Availability are a bit ambitious; for that I'd like to see the
proofs. Scalability though, I think that's there.
This is half of my journey home, as videoed from a headcam, with
some of the slower bits (me pedalling) stripped out. Its to
reassure Stephane that there really is some good urban offroad
here, so not riding dutch-style commuter bikes offers some
entertainment. I am even looking forward to it getting dark in the
evenings so I can get the head torch out and do it at
nighttime.
That is provided it is a dry winter. A wet winter means a muddy
bike every night, which is a bit repetitive
For the curious, I'm heading south over Purdown; the tower at
the top of the hill is a telecomms mast; behind that is the remains
of a AAA gun emplacement from the last time that airplanes came to
deposit armaments on Filton. Then I descend down to a main road
(Muller Road), cross that and I'm St Werburgh's, near the indoor
climbing wall. After that, it will be uphill all the way home.
Someone once mentioned to me that it must be hard bringing up a
child in the uncertain post 9-11 world. I don't think it is. I
haven't had to explain Northern Ireland's Troubles, that
being the reason we never went as kids to the part of the country
where my dad grew up. Which is why all his family returned there:
to say farewell.
Today, Portadown, County Armargh is at peace, but it'a kind of
edgy peace.
The various factions haven't made friends, just agreed to stop
shooting or blowing each other up for a while.
You still need to know which taxi firm will handle your
particular part of the city, and when there's a march -one due the
end of the week- everyone from the other community makes themselves
scarce.
And they are still clearly two separate communities, Nationalist
and Unionist; Catholic and Protestant.
That is two groups that share the same belief in an untestable
supernatural hypothesis, which spent 30+ years killing each other
over which interpretation of the details, and which nation state
within the European Union they'd belong to.
However, it's still, disputed territory, as there is a recurrent
marching
issue that surfaces every summer.
But, now I've been, I've seen, I've done half the funeral work
though there's still a cremation in France to deal with, probably
in September now. I can see why my dad never brought us to
Portadown. Nor do I see any pressing need to bring my own son here.
A little bit of his heritage that's best treasured from a
distance.
The deployment project was in the building today, so he finally
got to see what a datacentre looked like. The main room is a bit
noisy, and if he did something like switch off the UK SAP
infrastructure I'd get into trouble, so we took him to where some
storage arrays live.
He liked the flashing lights and the raised floor, and also the
failed disk light on one disk of an array in a different rack. I
was tempted to show hot disk swapping, but he will only try the
same thing at home and I will end up very unhappy.
For those people with the "my
other computer is a datacenter" stickers, let's start getting
photos of the laptop with the datacentre, tag it
"myothercomputerisadatacenter". Here's Julio and myself in front of
a box full of flashing lights.
I did manage to get over to the Hadoop UK event in London
yesterday, which was sponsored by Skillsmatter, Last.fm and Yahoo!
and hosted at the Skillsmatter office near Farringdon tube station.
To get there we had to leave Bristol on the 7am train, which
involved getting up at half five and sharing a car to the Bristol
Parkway station, a station we didn't see again until half nine in
the evening -a fairly long day. But it was a day that involved free
beer, so it was a fun day.
My slides are
up for download, Fitz may be pleased to see his my other
computer is a datacentre stickers actually in a datacentre on
slide 2. I didn't demo any working code; I only got the tests
passing and a single-machine HDFS cluster up the day before on my
desktop, I wasn't going to try and get that replicated on a laptop
running a different OS on a Wifi LAN. What we have working, when
you patch SVN_HEAD Hadoop with HADOOP-3628
and our SVN_HEAD code is a version of Hadoop that has a standard
lifecycle for the main services, and which can be deployed and
pinged from SmartFrog. This lets us bring up clusters on local
machines, or, if you want to have fun, on dynamically allocated
machines by your CPU-time vendor of choice, though of course we
think everyone should choose HP.
All the other talks were interesting, with Doug Cutting giving
an update on the current status of Hadoop inside Yahoo!, looking
forward to the future, and Tom White talking about the future of
Hadoop on EC2. Its ironic, but S3 is precisely the wrong storage
form for Hadoop, as it hides where your data is. You don't want
that for an infrastructure that moves computation to the data. You
really want the storage vendor to offer Hadoop-time near the data,
with the CPUs chosen based on the data blocks you want to work
with.
There was a big Last.fm presence, which was fun -especially the
coverage of the "who deleted all the data" incident. It was also
interesting to see performance data stored in the HDFS cluster
being used to compare the merits of various performance tweaks of
the site, using A/B testing to compare each option against the
baseline, and determining performance from the logs. We also had
Miles Osborne from Edinburgh University talking about his use of
it, and how he's starting to seek CPU time for MSc/PhD students
running big MR jobs. Well, the NeSC grid facility has a few hundred
machines nearby, we just need to deploy Hadoop on top of an OGSA
grid. I could see how to do that, even if the grid management may
have issues. Remember, it is not the role of the management APIs to
dictate how your application talks to the other bits of the app.
The design decisions of the OGSA team may not match your needs. I
may have to get involved in this work.
Mark Butler's talk on Distributed Lucene got interest from those
people interested in doing such things. Right now there are three
prototypes out there; mark argued for having one containing the
best parts of each design...it's good to prototype, but better to
go into production.
The day (formally) closed with a panel, which was pretty
interesting -a good debate on whether and when to stop using the
relational database. Then we finished off the beer before heading
to a Belgian beer pub round the corner for more beer. Oh, and then
a sprint across London and a late train home. All in all, an
excellent day out!
I a phone call at work on wednesday from my sister: our father
died the previous night, down in the South of France, where he's
been living for 20-25 years.
Putting the loss-of-parent issues to one side, I now have to
deal with a cremation in france (date to be scheduled) and a
memorial event over in Portadown, Co. Armargh, where he was born.
Then comes the absolute nightmare of french inheritance rules,
where my sister and I appear to automatically get a large slice of
debt along with the houses, and a tax liability. I will have three
countries worth of paperwork to deal with (UK, US, Fr), and may
have to start avoiding France for tax reasons. It also means that
anything resembling a calendar for the next month is up in the air;
It may be Julio only speaking/demoing at the hadoop UK event.
