Sunday, March 16, 2008

REST: Unix programming for the Web

I've been giving some thought to REST-style architectures recently, and recently re-read some of The Art of Unix Programming by Eric S. Raymond. ESR notes that some of the characteristics of Unix-style programming include:

  • Do one thing, and do it well (attributed to Doug McIlroy)
  • Everything is a file.
  • Comprise complex systems by connecting smaller, simpler programs (e.g. Unix pipes).

Unix-style systems have had undeniable success and remarkable stickiness for a technology; I have an old copy of my father's System V manual from when he worked at Bell Labs (yeah, they actually printed out the man pages and bound them!), and I pretty much recognize everything in there. Sure, there are many new commands available, new kernels, new distributions, etc., but they are all very recognizably Unixy.

I've been thinking about how this philosophy applies to the Web 2.0 world. I think this list turns into:

  • Do one thing, and do it well.
  • Everything is a RESTful service.
  • Comprise complex systems by interconnecting smaller, simpler services.

For one thing, "do one thing, and do it well" isn't limited to Unix, this is a key part of abstracting and decomposing a technical problem. But we see it everywhere: I read my mail on Gmail, keep my to-do lists on Remember The Milk, store photos on Flickr, keep my browser bookmarks on, etc. All these sites adhere to this principle.

But taking things down a layer, what does this mean to someone architecting or implementing a Web 2.0 service? For one thing, I think this means that it would make sense to break your overall service down into individual microservices that are individually maintained and deployed; i.e. break things down to the smallest level where they still make coherent sense.

As for "everything is a REST service", the everything-is-a-file abstraction worked for Unix because there was a small set of common operations that applied to files (open, close, dup, read, write). Sounds a lot like doing REST over HTTP, with HEAD, GET, PUT, POST, DELETE.

Combining various small REST microservices into larger services is already being done (this is, after all, basically what iGoogle is) on a one-off basis. The main question is: what is the Web 2.0 equivalent of the pipe? Namely, is there an easily understood abstraction for composing webservices? Sounds like a topic that might be rife for some kind of logical calculus (like the relational calculus for databases or the pi-calculus for concurrent processes), e.g. the REST calculus. If there were a couple of easily understand and specifiable combinators, these could be pretty easily built into some language-specific libraries for use in quickly building some new macro-services.

I might try to interest some of my old colleagues from my programming language research days to see if they have anything to say about the matter....

Comments definitely welcome here. Is this a new idea? Are others espousing this? Is this even a good idea?


Bradley said...

"Everything is a RESTful service" - close but not quite right. "Everything is a RESTful resource" is more correct. Verbs such as HEAD, GET, PUT, POST, DELETE can then be applied to these resources. Just a minor quibble, interesting entry ;-)

Anonymous said...


O' Pipes and Filters for the Internet.

O'Reilly and Jon Udell have written numerous pieces on the subject matter going back to 1996.

BTW - check out Yahoo! Pipes. It's amazing.


Jon Moore said...

Thanks, Karl, the Yahoo! Pipes are pretty cool. I think I'm looking for something a little lower-level here, though that involves more than just XML translations on GETs.

I'm thinking of "command lines" like:

GET http://host:port/path | xslt transform.xslt | POST http://otherhost/path2

Actually, holy crap, now that I wrote that, I think that's the basics of a "REST shell". In fact, I bet you could write a compiler for it...

Matt said...

Jon, I had not considered visualizing resource modeling and composite web services/applications with something equally elegant like UNIX file abstractions and orchestration of simple filters. I will have to adopt that. What I have found most helpful, as you suggest, is a calculus. See Actor Model and check out Ahga's dissertation.

Ironically, it was in WS-* land and efforts in process execution engines (WS-BPEL) where pi-calculus is a necessity, that I developed my understanding of how a calculus for RESTful web resources may work (at least for me). This whiteboard picture of Actor Model for REST is what I keep in mind. The biggest take away from this is how natural parallelism shines through (as the value) of REST architecture in web resources. People claim that REST architecture is more scalable and Google demonstrates this with GData but perhaps a calculus proves this.

Actor Model Theory and Web Resources

Anonymous said...

wget http://host:port/path | xsltproc
transform.xslt | wget --post-data=string http://otherhost/path2


GET http://host:port/path | xslt transform.xslt | POST

the tools already exist. the command line already exists :)

One of the beauties of the Web and Unix. As soon as you start desigining with REST in mind, these things start to become leverageable in a real way.

Here goes a good article on the subject:

Note that command line I shared above isn't tested and sure isn't going to run, but walk by Mat's desk and I bet he could wip it out in 5 minutes or less :)

- Karl

Steve said...

As for the academic side of things, I'm not aware of anything that's been done with respect to RESTful resources (but I haven't looked very hard either). It looks like a really nice problem domain for a bit of formalization and some theory...


Anonymous said...

Check out NetKernel, a resource-oriented microkernel environment that has been under development for years. Dual-license. Very inline with what you are thinking about: