Through this open world I’m about to ramble,
Through ice and snows, sleet and rain,
I’m about to ride that mornin’ railroad,
P’raps I’ll die on that train.
- MAN OF CONSTANT SORROW

During the panel discussion at the recent British Computer Society Open Source event, there was discussion (and confusion) about Open Source versus Open Standards. I was asked “So, can you give us some examples of Open Standards”. I rattled off a few, but I thought I’d add a few more here. There is a lot more to be said on the topic, but a good place to start is to list the standards that I think are important.

If I get the time, I plan to turn this into a nice diagram that is much more easily digestible. If there are important standards that I’ve forgotten about that anyone interested in web sites should know about, please let me know in the comments. I’d avoided worrying about file formats (e.g. PNG, MPEG, PDF). And REST isn’t a standard – it is an architectural style that was developed in parallel with the HTTP/1.1 protocol. I’m sure there are many many important ones I’ve left out though.

This is a long and boring post with a record-breaking number of acronyms. So maybe you should stop reading now.

The Internet Plumbing

These standards are the plumbing of the Internet. Like the sewers under a big city, they are impossible to change and will be there forever. They’re infrastructure. Some people are saying Twitter has already become infrastructure, but I’m not convinced about that yet. These standards are split into layers – the link layer is about physical connection to a network and include standards such as Ethernet. The Internet layer routes packets of information across one or more network using the Internet protocol (IP). The Transport Layer is responsible for the reliable delivery of messages, and uses standards such as TCP or UDP. Finally, the Application Layer provides higher level application specific protocols such as DNS, HTTP (and WebDAV) for web servers, FTP, SMTP for mail servers, NTP for time servers, LDAP for user directories and so on. But I’m not here to talk about any of these. I want to talk about the standards that sit on top of these, specifically for web pages.

Making Web Pages

First 10 Years of the W3C - Click for large version

Let’s start with the standards we know and love that make up web pages. Of course we have HTML 4, XHTML and the eagerly awaited HTML 5. We make our HTML pretty using Cascading Style Sheets (CSS) and we interact with the page using the Document Object Model (DOM), which has a large number of associated standards. Note AJAX is not a standard, despite what you might hear. The XMLHttpRequest DOM API (which can be used to implement AJAX) is currently a last call working draft and may be a W3C standard soon. Another client side standard, Scalable Vector Graphics (SVG) never really took off and probably never will.

So we have standards to make interactive web pages that may or may not be semantically rich. But the world would be a better place if these pages can be accessed by as many people as possible. So we have accessibility standards as part of the Web Accessibility Initiative (WAI). These include Web Content Accessibility Guidelines (WCAG) for your web page and the imminent Accessible Rich Internet Applications (WAI-ARIA).

Excellent! Our HTML is neat, we’ve styled it, and all humans can interact with it. But what about the machines? They don’t understand our badly structured markup. If we want machines to be able to understand the content, we need to engage with the semantic web standards and Resource Description Framework (RDF). The UK public sector is keen on HTML+RDFa although this is not a W3C standard yet. You can query your RDF data set using SPARQL and define your ontology (formal representation of concepts and relations between them) using the Web Ontology Language (OWL, not WOL). While we’re at it, a related and very succesful standard which touches my world is the Dublin Core Metadata Standard, which is an ISO standard. I like this good introduction to Semantic Web standards if you want to read more.

The Biggest Standard of the Naughties

Extensible Markup Language (XML) is a hugely successful standard. If you judge the success of a standard by its adoption (which I do), it was the Hit of the Decade. XML Schema (XSD) has replaced the ill-thought-out DTD standard for defining XML structures. Other child standards include the node selection language XPath and query language XQuery. XSL Transformations (XSLT) is my favourite templating language. XML Inclusions (XInclude) joins XML documents together. They’ve also given us XForms to collect data – sadly it hasn’t taken off as I’d have liked.

Also XML related, the Web Services Standards have given us a wonderful way to make remote services play together. The Holy Trinity behind Web Services are SOAP (previously Simple Object Access Protocol) to define the message formats, Web Services Description Language (WSDL) to give service descriptions and Universal Description, Discovery and Integration (UDDI) to find the services. UDDI is actually an OASIS specification, not W3C, but it fits better here.

OASIS standards

OASIS (Organization for the Advancement of Structured Information Standards), in their own words, “drives the development, convergence and adoption of open standards for the global information society”. The OASIS standards that touch my world include, in no particular order:

The empty-handed painter from your streets
Is drawing crazy patterns on your sheets.
- IT’S ALL OVER NOW, BABY BLUE

Every now and again, certain tumultuous events coincide that makes us feel insignificant. A bit like a total solar eclipse, which is a rare and humbling thing. This week three seemingly unconnected occurances came to pass which made me stop and think. Firstly, CMIS was mentioned in the New York Times. Yes, the New York Times. Secondly, Jeff Potts released his “Getting Started with CMIS” tutorial. And, finally, I had the chance to drink beer and talk shit with the great David Nüscheler.

So, to honour all of this, I drew a picture.

Why, you may ask. Well, after chatting to David, I felt inspired to think (and blog) more about the JCR, CMIS and OSGi. But while doing my cursory research (that’s a strong word for what I do, but anyway), I discovered that there were quite a few things I knew nothing about, and a few products that I’d never heard of. I also coined the word Dignorance in honour of my newly found knowledge gaps.

So here is a small version of the picture. Click for a large one, or download it as PDF. Do whatever you want to with it. If you think it is useful, or think it needs fixing, leave comments here or get in touch. I’m sure it’s completely wrong all over the place. If you want the original Visio drawing to play with, let me know.

Next time I come to your office, I except to see this printed as A3 and stuck on your wall.

UPDATE 18 Dec 2009: I’ve written an article on CMS Wire which explains some of this. Have a look:

http://www.cmswire.com/cms/web-cms/wcm-field-notes-the-skinny-on-jcr-cmis-and-osgi-006279.php

Mama, put my guns in the ground
I can’t shoot them anymore.
That long black cloud is comin’ down
I feel like I’m knockin’ on heaven’s door.
- KNOCKIN’ ON HEAVEN’S DOOR

The interesting thing about cloud computing is that we’ve redefined cloud computing to include everything that we already do. I can’t think of anything that isn’t cloud computing with all of these announcements. The computer industry is the only industry that is more fashion-driven than women’s fashion. Maybe I’m an idiot, but I have no idea what anyone is talking about. What is it? It’s complete gibberish. It’s insane. When is this idiocy going to stop?

Okay, I see. It all makes sense now.

Right, that settles it. Let’s dig a bit deeper. Which services and applications currently live in the Cloud (again, from Wikipedia, which is really a good a place as any for this):

• SaaS Applications such as Google Apps, Salesforce – they’re in monster data centers.
• Hosted Social Networking Platforms, such as FaceBook – okay, another monster data center.
• Storage, such as Amazon S3 or Mobile me – more big data centers.
• Operating systems, such as Azure – Okay, so we can host our OS in monster data centers too and access it remotely. Hold on. Are we just going back to the dumb terminal/thin client model? Did we have The Cloud many moons ago and let it slip when we installed big fat operating systems on our home computers?

At least we’re seeing a pattern here. Excellent, we can make a definition – The cloud consists of big data centers with virtualisation. But these also allegedy live in the cloud:

• Peer-to-peer applcations, such as Skype or BitTorrent – now these are the opposite. Not centralised at all. In this case, we’re The Cloud.
• Open services such as OpenID – this is allegedly a “Cloud Service”. It’s distributed too. I can pick my provider.
• Proprietory services such as Google Maps or PayPal – again, “Cloud Services”. Sounds like a Cloud Service is anything that is accessed directly from a client’s browser as part of a mash-up. Server-to-server communication isn’t sexy enough for The Cloud.
• Google Analytics / Omniture / WebTrends – Yep, they must be Analytics in The Cloud. Client side JavaScript integration. Tick.
• What about Social Community in the Cloud, for example from Pluck ? JavaScript integration. Tick. But wait. Each Pluck client has their own hardware. It isn’t shared at all. Sorry, Pluck. You ain’t got no Cloud. Just racks and racks of tin.

Shock! Horror! It’s starting to sound like The Cloud is just another word for The Internet. You’ll be glad to know that many others are as confused as Larry and me. Richard Stallman , founder of the Free Software Foundation doesn’t get it. And Joel on Software, who rocks, doesn’t get it either. Get this, even the Cloud Computing Journal (CCJ) says it is all hype and agrees with Larry. They say:

The current fad of butchering the term “Cloud Computing” to bring sexy back to the *aaS (anything as a service) model is embarrassing. More embarrassing is the fact that I agree with Larry Ellison …

As Kas jokingly said to me on Twitter: You have to have a constant flow of new buzzwords in this business. Otherwise analysts can’t write”XYZ IsDead” articles 2 yrs later. Well I don’t want hype and buzzwords. I want standards. Maybe any protocol or standard with an HTTP binding is a Cloud Protocol? Wikipedia’s list of “Cloud Standards” is a joke at the time of writing. A mix of existing web standards and, well, nonsense: HTTP, XMPP, SSL, Atom, AJAX (what!?!), HTML 5, LAMP (somebody shoot me), XML, JSON, WebServices/REST (like they’re the same thing). What a mess that is. Maybe, just maybe, CMIS is going to give us Content In The Cloud.

Get it yet?

You may ask yourself, is anyone working on Cloud Computing standards? The same CCJ article states that:

C’mon, people. Don’t give into the generalist hype. Cloud computing is real. “THE Cloud?” Not so much.

Now I hear what he is saying, and there is some real work going on behind the scenes, but I don’t get what Cloud Computing really is either. The recent Open Cloud Manifesto seems very vague to me. It is by its own admission “meant to begin the conversation, not define it.” But what is the conversation it is meant to be starting? The Open Cloud Consortium is trying to do something but their website doesn’t explain much to me. They even have a testbed, but I don’t know what they’re testing. The Unified Cloud Interface Project is real and seems to be trying to merge Cloud Computing with the Semantic Web. I haven’t grokked this yet but it seems to me that if the Semantic Web standards (RDF/OWL/SPARQL) are losing out in the real world to Microformats , it’s going to be a while before any standards come out of this. There are many other expert groups that are putting their heads together and telling us that Cloud Computing is in its infancy and needs standards. But I’ve yet to find the detail outlining exactly what we’re trying to standardise. Whatever it is must be a pretty touchy subject, though, as different groups are already falling out with one another and refusing to endorse things.

It’s all too much for little old me. Doesn’t seem like The Cloud is going to help me much in the short term at all. So, what am I going to do about it? Absolutely nothing. I’m going to sit back and ignore The Cloud until some draft specifications start to emerge. Then I’m going to read them, fail to understand them, but understand what I’m failing to understand. And that’ll be a step in the right direction.

In this country, you gotta make the buzzwords first. Then when you get the buzzwords, you get the power. Then when you get the power, then you get the women.

UPDATE 24 April: Since posting this, some helpful experts have helped me enormously by helping to separate the wheat from the crap. The useful stuff is hard to find, so read the comments below. And follow @stevecla and @jamesurquhart on Twitter.

Sara, Sara,
Whatever made you want to change your mind?
Sara, Sara,
So easy to look at, so hard to define.
- SARA

So What Is CMIS?

• Dr David Choy from EMC (who really knows his stuff!) gives an friendly overview on YouTube: Part I and Part II
• The Alfresco CMIS wiki – http://wiki.alfresco.com/wiki/CMIS
• Kas Thomas from CMS Watch on CMIS
• The EMC CMIS Page
• The CMIS and Interoperability – AIIM 2009 presentation from Alfresco on SlideShare – very useful
• Laurence Hart’s Word of Pie blog
• [13 Apr 2009 UPDATED VERSION] Read version 0.6 of the specification (it’s longer than you think but a lot of prose)

I should also mention here that there is a fair bit of debate about whether CMIS is really RESTful. It’s hard to argue with Roy Fielding who defined the term. But a misuse of the term REST hasn’t stopped other APIs from gaining enormous popularity. This posting is only going to talk about Part I of the specification (the Domain Model), not Part II (the SOAP and APP bindings).

CMIS and the Java Content Respository

The CMIS goal has a lot in common with the Java Content Repository (JSR 170/283). The main difference that everyone cites is the fact the JCR interface is Java only, while CMIS allows access to any implementation. CMIS is much simpler, which may give it the kiss of life that the more functionally rich JCR seems to lack. CMIS is accessed via a remote API, while the JCR is accessed via Java methods, but I don’t think this difference is fundamental. The CMIS specification could have added a remote HTTP access protocol on top of the JCR to overcome the differences mentioned. Most of the contributors to the CMIS specification were also involved in the JCR, so the fact that this didn’t happen suggests to me that they felt something else was amiss. Get your chainsaws out ’cause I’m going to go out on a limb here and suggest that the main difference between CMIS and the JCR API lies in the query language choice – XPath versus SQL. Note that SQL is an optional extra in the JCR spec, while XPath currently isn’t an option in CMIS.

Below are some of my thoughts based on my limited understanding of the standard. I’m positive that the points I’m going to raise have been discussed in the CMIS committee meetings, and equally positive that I’m completely wrong on all of this. I’ve probably missed something obvious too. But I would be extremely grateful if those in the know could point me to any resources that answer the questions I have. My googling skills didn’t unearth anything.

The CMIS Relational View

In order to understand the choice of query language, one must first understand the virtual relational view of a CMIS repository. This consists of a collection of virtual tables. A virtual table exists for every queryable object type (content type if you prefer) in the repository. Each row in these virtual tables correspond to an instance of the corresponding object type (or of one of its subtypes). A column exists for every property that the object type has. The figure below, taken from the specification, tries to explain this. Someone needs to draw a prettier version of this.

Aside for another posting: One thing worth thinking about would have been the benefit of implementing an entity-attribute (or “skinny table”) relational view instead of a table per object. While the SQL queries against such a repository are dog-ugly, they would have many of the benefits of XPath I think.

So Why not XPath?

When reading the spec, I kept asking myself why the query language chosen is based on SQL (called CMIS SQL or CQL), not XPath like the JCR. The only reference I found to XPath and CMIS was in Russ Danner’s excellent blog . He says:

CMIS also specifies a SQL like query language. Unlike previously proposed standards that pushed XQUERY and XPATH, CMIS is adopting a well understood paradigm which I believe will only encourage its adoption.

So this is the opposite view to mine. My instinct screams XPath to me for the following reasons:

• XPath is a more natural way to search a hierarchy. ANSI SQL doesn’t provide this functionality although extensions like Oracle CONNECT BY and SQL Server’s Common Table Expressions make it possible. Most implementations would involve creating a denormalised table which flattened the graph. The CMIS specification adds 2 extensions (IN_FOLDER and IN_TREE) which I find slightly smelly.
• An XPath result set can naturally contain content of different types. In the SQL model, each content type needs to be added to the result set which would usually mean a whole lot of UNIONS and ensure that the columns selected from each virtual table are the same. Except that CSQL doesn’t do UNIONS. For example, if I want to find all objects that are green, I far prefer
  XPATH: /root//*[@color='green']
  to
  SQL: SELECT * FROM OBJ_TYPE_ONE WHERE ( IN_TREE( , ‘ID000XXXXXXXX) ) AND ( ‘green’ = ANY COLOR)
  which I’d have to issue for every object type as UNIONS are not supported. If I am querying multiple types I need to send multiple queries, and I can’t use ORDER BY across the types.
• With the standard as it is, I can achieve content of different types in the same SQL resultset, but only if they are sub-types. I’m not a big fan of deep type inheritance trees. And most of the content repositories that I deal with don’t support content type inheritance natively.
• I don’t need to change my XPath Queries if a new content type is added to the repository. Using SQL, I could dynamically generate the queries by using reflection (getTypes and getTypeDefinition) on the repository, but that’s an extra step.
• The Navigation Services API calls could probably be replaced with more XPath queries. Calls such as getChildren, getDescendants, getObjectParents, getFolderParents,
• Once I retrieve a document, I’d like to get the “folder breadcrumb”. I didn’t see an obvious way to do this. I think a multi-filed document might need the concept of a “primary folder” or, at least, an ordering of the folder-document containment relationship.
• The pagination functionality feels slighlty unwieldy, sending SKIPCOUNT and MAXCOUNT as optional parameters to the query function. Not that this is solved by XPath, but I thought I’d mention it anyway.

Maybe XPath is just too hard for the vendors?

So why didn’t they go with XPath? I think the biggest hint comes from the design goal is the CMIS specification:

However, it is an explicit goal that CMIS will NOT require major product changes or significant data model changes like other standards such as JSR 170 have required.

Wearing my developer hat, I think the API would be more useful if I could interogate it using XPath. However, from the point of view of the ECM vendor based on a relational database, maybe implementing an XPath search on their respository is just too damn hard! One would think that the vendors that support JCR already have done most of the heavy lifting. But not many vendors have implemented it, and maybe of those that have use Apache Jackrabbit.

So, in summary, my theory is that an XPath based query language is very difficult for the vendors to implement. Which means developers of CMIS clients are gonna have to bite the bullet and use CSQL. Which is still going to be great, and means we’re going to get far more CMIS enabled repositories than JCR ones. Which hopefully means the Day CMIS PlugFest is going to be a very busy event. But I do so love XPath, and here’s hoping that it makes it into a later version of the specification.

Rock on, CMIS.

“Oh no,” says the Sergeant. “I’ll have no such chat,
And neither will I take it from snappy young brats,
For if you insult me with one other word,
I’ll cut off your heads in the morning.”
- ARTHUR MCBRIDE

In the beginning there was the CMS Vendor Meme, which turned into a light hearted and entertaining exercise in which over 20 CMS vendors participated. As a response to this, Vignette posted a second meme which appears to have backfired. There aren’t going to be many (any?) responses, and Luis Sala has posted a rather biting response on his *personal* (not Alfresco’s) blog.

Before I comment on this, I’d like to say for the record that there is a lot about Vignette that I like. They were, without doubt, CMS pioneers. When Vignette V7 was released many moons ago, I genuinely believe it was the best Content Management System out there. Apart from a couple of major gripes, I still like the CMS and think certain aspects of it are uniquely attractive. VAP is one of the leading portals too. I hate the Dynamic Portal/Site integration but that’s just me. But I would still happily implement Vignette and do currently work with it.

Before reading this, make sure you read Luis’ post Vignette tries to start a WCM Vendor Meme (yawn!). I wonder if Vignette are going to respond to this, or just leave it at that. Vignette have lost their way a bit and I agree with a lot of the comments that Luis made. They have also been in the press quite a lot recently for a number of reasons, and I’m not going to repeat things that have been already said elsewhere. Some of the points in Luis’ post have been discussed in the CMSWire article Five Reasons to Choose Vignette (or Not…) by John Conroy and Irina Guseva. I am, however, going to add my thoughts to a few of Luis’ comments.

I would argue that one of the main reasons that happened was because Vignette, Interwoven and Documentum are entrenched in 7-15 year technologies and mindsets that have resulted in stagnation while the smaller, more agile vendors that ranked a little higher on the list can more successfully innovate and adapt to changing market conditions.

I think is completely true that the speed of development of the ECM vendors mentioned is much slower. Compare Vignette VCM 7.3.0.1 to 7.6 – not as much progress as one would hope in about 5 years. It isn’t only Vignette – we’re still waiting for TeamSite 7 for example having had a sneak peak 2 years ago at GearUp 2007. But I blame the technologies more than the mindsets. In the Vignette CMS case, I’m going to point the finger at an extremely complex J2EE implementation using some technologies that are pretty old by current standards. Aim some monitoring software like SpotLight at the VCM, change and publish a simple content type, and watch the party. Quite staggering just how much is going on behind the scenes. Not surprising that it is complex to extend and change. In addition, I think the larger players are hamstrung by backward compatibility promises as they tend to support much larger, business critical implementations. You’d think this means that their upgrade paths are easier and more reliable, but sadly recent history his taught us the opposite.

That’s probably one of the main reasons Vignette’s earnings continue to drop and, more alarmingly, Vignette Professional Services account for roughly 50% of their revenue. Very scary…

Yes, Vignette’s PS account for a large part of their revenue. From the agency/systems integrator (that’s where I live) perspective, we are very cautious of this model. It doesn’t make our lives any easier when we need to compete against the vendor’s PS team if we recommend their CMS for an implementation. More and more small vendors are moving to the Partner Channel only model, which makes life much more pleasant for everyone. And I appreciate that the vendors need to ensure that partners have the skill to implement correctly and want to certify some parts of the solution, but I believe the Professional Services should be contracted to the partner, not directly to the client.

All vendors must be very careful about claiming “massive scalability” as every implementation is unique and while the software may be capable of scaling in one use-case, it could die in sputtering, driveling fits in the other.

Of course. Bad implementation can kill any software. Let me tell you a story that depresses me a bit. 10 years ago, Vignette had what I consider an excellent caching model. They used “Components” which actually ended up as output cached HTML fragments. They used Server Side Includes (SSIs) to allow for an extremely flexible mix of cached and dynamic content on the same page. They didn’t have a very sophisticated dependency graph based de-cache back then, but no-one really did. They were very proud of it, claimed it had multiple patents, and I think it was the dog’s bollocks (that’s a good thing for the non-UK readers). The model vanished in later versions, and was replaced completely when Vignette Application Portal (VAP) become the recommended delivery mechanism.

A couple of years ago I was presenting at Vignette Village 2007 with a customer. One of the main themes of this Village was that Vignette had the fastest, most scalable delivery mechanism in the universe, ever. Another main theme was that the product roadmap was very driven by customer demand. “You asked, we listened” kind of thing. So I found it quite ironic that a major product launch at Village was a brand new component that sped up the delivery even more: Vignette High Performance Delivery (HPD). Now the fastest delivery mechanism in the world is even faster! And the architecture behind HPD is remarkably similar to the caching model they had in the late 90s!

I’m going slightly off topic here, but I don’t like the way the blurb on the site for this product says “HPD lets organizations deliver fresh, customized content online without either the high cost of hardware or the unacceptable cost of slow site performance“. There are other options. For example, Vignette really did a great job on the Atlanta 2004 Olympics site. It looked excellent and performed well under massive load. But this was all Akamai fronted. I’m a big fan of CDNs and ESIs for massively scalably delivery.

After wasting investing hundreds of millions buying OnDisplay, DataSage, Revenio, Epicentric, Intraspect, Tower Technology and most recently Vidavee, one could argue that Vignette could *possibly* address all those areas, but the dirty little secret is that even over a decade after some of those acquisitions occured, Vignette has positively and quite spectacularly failed in truly integrating all those services.

I think the key word here is “truly”. Some of the products are reasonably well integrated, but these are point integrations between product A and product B. I’d like to see a generic integration architecture instead. A single repository (or illusion thereof) for the different products would be the dream, but that seems unlikely. I think the word wasting is unfair, although some of the acquisitions have been worse than others. I’d also like to add that I believe Vignette are aware of these issues, do embrace and support enabling emerging standards (JSR-168/286, JSR-170/283, maybe the new favourite CMIS) which could help with their integrations, and still have some really smart people. But their current product stack makes this a really difficult problem which is going to take them a lot more time (which maybe they don’t have?) to solve.

The Vignette Application Portal is pretty much the only *simple* way to render content. While the sales and sales engineers might say that customers can create websites using any web framework and programming language, a realization of the effort involved will serve as a near-instant death-knell to such foolhardy notions.

I’m going to disagree with this on two points. I like the VCM Content API and have successfully implemented systems that use a non-Vignette (this doesn’t mean bespoke!) delivery framework. Surely one of the main benefits of the “decoupled” ECM approach is that one does have the option to use the ECM suite simply to manage content to be consumed by any delivery channel. My second disagreement is that VAP is a simple way to render content. For a public facing site that does not need a portal, I believe that using VAP (or any portal) is overkill and fraught with peril.

Customers foolish enough to buy from Vignette are faced with an 8 page-long pricesheet (please take a look!)

Actually, the PDF version is more like 30 pages. In response to the “Simple Price List” question in the initial meme, part of the response was “Look out for a pricing innovation coming soon to a price sheet near you … it will probably be simple enough for a five year old”. Vignette are aware of the over-complication and hopefully will address it. If they don’t, they’re in trouble.

Vignette’s product suite is so expansive and disjointed that the typical Vignette sales engineer cannot even fathom how to install them all. So Vignette has a “Sales Enablement Team” whose primary job function is to figure out how to install all these moving parts and set up a hosted VMWare environment so that they can demo it. I pity my brethren there who still have to run more than one Vignette app on their laptops.

*cough*

Anyway, enough of this now. Let me close my coffin on this near extinct meme with a final thought. I still believe Vignette has some really good software and some really good people. They’ve also got some really big architectural and software problems. I, for one, hope they can sort them out before it is too late. If they don’t, they wouldn’t be the first US Pioneer to die in Texas. And it is only 81 miles / 1 hour 22 mins drive from Vignette Corporate Headquaters to the Alamo.