Johan is organising the next UK Hadoop meetup. I won't be there, but there will be at least one colleage there, with Paolo Castagna talking about implementing page-rank like algorithms over MR. Having had that explained to me by Paolo last week, I can say that if you remember your array notation it makes sense. It's also interesting how, in this world, a lot of the CS-related stuff that I used to know is actually worth remembering.

Johann's announcement

I've started organizing the next Hadoop meetup in London, UK. The date is April 14th and the presentations so far include:

• Michael Stack (Powerset): Apache HBase
• Isabel Drost (Neofonie): Introducing Apache Mahout
• Iadh Ounis and Craig Macdonalt (University of Glasgow): Terrier
• Paolo Castagna (HP): "Having Fun with PageRank and MapReduce"

Keep an eye on the blog for updates: http://huguk.org/

Help in the form of sponsoring (venue, beer etc) would be much appreciated. Also let me know if you want to present. Personally I'd love to see presentations from other Hadoop related projects (pig, hive, hama etc).

/Johan

Normally in open source apps you assume that everyone else has coding skills, often software engineering experience, but there's no requirement for having studied computer science. Its not normally needed, and so you avoid using too many CS-terms "turing-equivalent", "polynomial time", and such like. Which is why the Hadoop core-user mailing list is so interesting. On one hand, there are people trying to get the JAVA_HOME environment variable to point to a JDK installed on a system. On the other, there is a discussion on graph traversal with MapReduce, where an example sentence is

If there is a Hamiltonian path in the graph , your algorithm will find it, so your algorithm cannot be better than the best Hamiltonian path algorithm. Finding a Hamiltonian path is NP-complete, so map reduce will probably not help you. Perhaps your graphs have some extra structure that could be exploited? There are efficient solutions for DAGs, but you specifically said the graph may have cycles.

Lovely. There's an assumption there that the audience knows about Hamiltionian Paths and Cycles, and NP-completeness. After that, using DAG as an acronym is just a detail. This is why the Hadoop ecosystem is so very interesting. There's CS people doing graph work, engineers bringing up the system, and room for anyone else to play if they are prepared to check out the code and wait for the tests to finish.

It's amusing to see that currently google UK views all web sites on the planet as potentially harmful

Given this screenshot was taken on a windows laptop, it is technically correct, but I was surprised to see the same warnings appear when browsing on linux, which is a lot harder to subvert.

Lovely set of slides Programming the Virtual Infrastructure,, by Paul Anderson of Edinburgh Uni. This was his keynote at LISA last year

The key premise is that when the first computers were built, they were wired up and programmed at the machine code level; HLLs were written so that others could work with them, and to improve productivity, later portability.

Paul then argues that system configuration should take the same path, as with a highly virtualised infrastructure, you are now telling the infrastructure what to create and where. Instead of going MiniMRCluster mrCluster=new MiniMRCluster(), as you do in Hadoop to create an in-process cluster for testing, you can go

Cluster cluster=new Cluster();
cluster.add(new NameNode);
cluster.add(new DataNode);
cluster.add(new DataNode);
cluster.add(new DataNode);
cluster.start();

And then you are off with four virtualised hosts forming a small distributed filesystem.

The next issue becomes: what is the language to use. Paul (and we) are clearly in the declarative world; Sun's Project Caroline more procedural. One is good for monitoring, the other easier for classic-imperative-programmers to pick up. Where Paul does go it a bit further is that he argues that everything deployed in the infrastructure is in fact an agent, and that instead of describing its state, you have to consider describing/modelling its behaviour. That's a fairly interesting way to look at things. Having dabbled in Agent stuff two decades ago (!), what I remember from those days is that agents have beliefs that have to be viewed relative to that agent and the time they held them. There are no facts, only beliefs. That is probably a more accurate view of the world, (my laptop thinks it needs a proxy, that belief was valid at work, but not now it is at home), but it also complicates a lot. Question is, is it a necessary step?

I seem to have spent a few hours debugging RMI grief. Specifically SFOS-1079, "why do I get a java.rmi.ConnectException: Connection refused to host: 127.0.0.1;" error when trying to talk to a vmware image on a laptop in the same room in my house as my desktop PC

The answer lies deep in RMI, so deep, you can't see the source in the classic Sun JDK. Which is where I have to praise the OpenJDK team, as their distro does include everything I need to step deep into the internals. I now know the problem: if the far end gets very confused it returns 127.0.0.1 as the IP address to things talking to it. You can start talking to the registry OK, but not to anything it has registered.

So far I've done some ugly things to detect and warn on this early (searching for 127.0.0.1 in the toString() value of the stub), more diags needed. Then there is some root causes to deal with.

How do you get RMI so confused? Run in on a network without DNS

As a followup to the HDFS reliability topic, it's nice to see a new JIRA issue up, HADOOP-5034, which is about how to efficiently propagate deletion requests, as currently they get lower priority than block replication requests

As the bugrep says: "This brings a problem when a near-full cluster loses a bunch of DataNodes. In reaction to the DataNode loss, NameNode starts to replicate blocks. However, replication takes a lot of cpu and a lot of replications fail because of the lack of disk space. So the administrator tries to delete some DFS files to free up space. However, block deletion requests get delayed for very long time because it takes a long time to drain the block replication requests for most DataNodes."

I'm glad someone else is encountering and handling the "failure handling when your multi-petabyte filesystem is full" situation, because when the pager goes off, you know the problems will be serious. It also shows another aspect of the "state-space" where its safer to be following rather than leading. As our local High Availability expert says "No operating systems cope gracefully with the filesystem filling up"; Hadoop has just spread that problem across more machines. Which is good in terms of redundancy, but that NameNode is the SPOF here, and the point under load.

I'm excited to be talking about Application Architectures for the Cloud at ApacheCon EU at Easter.

This is not going to be a talk on Hadoop and MapReduce.

It's going be a sketching out of the architecture for big applications that you want to deploy on a datacentre, one probably hosted by a third party. In this world, machines come and go on demand and your app needs to be designed not just to cope with it, but to take advantage of it. It also needs to take advantage of the fact that the distributed filestore and MR engine close the loop in terms of feedback: start by assuming there is a big DFS there instead of a database.

Lots of the Java ecosystem needs to evolve in this world. All the logging tools now need to think about pushing facts out to the DFS, to have post-mortem analysis tools running over it and looking for recurrent errors across 500+ nodes. Same for the xUnit test runners: no more one-XML-file-per-test case, now you have 500 servers running the same test suite, and the most interesting problems are those that fail on 15% of the supposedly homogenous servers. That's frequent enough to matter, but not the 100% failure rate that is easy to debug.

There are lots of other bits in this story, and a new edition to my slideware will be Project Voldemort, assuming the J.K.Rowling lawyers haven't had it renamed by then. More details on High-Scalability

Tom White has put up a lovely document on HDFS reliability

I'd back up what he says about configuration. Only change options (replication, block size) if you are prepared to become the first person to test a cluster with those options set. Hadoop really needs something to push a cluster through a set of configurations under semi-random control, to see what happens.

One thing I'd add is my rule about networks: Programs work best in the network environment in which they were written. Sun has a well managed DNS infrastructure with not many laptops, so Java doesn't like changing network addresses or DNS playing up. Lots of Linux is written at home, hence proxy configuration is a nightmare that some apps simply don't support. And Hadoop?

It assumes DNS is working, IPAddresses are stable, and that nobody malicious is in the datacentre. There's probably an implicit assumption that all clocks are moving forward at the same rate -VM-hosted code can break lots of programs that way, though its usually a sign your VM-server farm is overloaded. These assumptions all hold for the well-managed datacentres of Yahoo!, Facebook! and similar. They don't all hold on, say, Amazon EC2, where anyone can run code in the same subnet for a few cents an hour, and so scan all your ports for no bandwidth charges. They don't hold in my house either, which causes problems for me, but for nobody else.

What could Hadoop do here? I think the network assumptions should be documented more, so we know what to set up/expect. I also think we (and it's a we :) could do better diagnostics, to identify whether things are good, and if not, what's wrong.