I agree with absolutely everything you’ve said. I think the main point here though is that this list/survey/competition really is nothing to do with the product. It’s all about the vendors, and their perception of themselves. I trust the answers they volunteer not at all. In fact, I’d say the lower the better!

Maybe this kind of thing is more meaningful for a Plone/Drupal/Joomla!/Umbraco kind of ecosystem?
http://jonontech.com/2009/05/12/cms-celebrity-deathmatch-the-developers-speak/

Jon

http://francescociriaci.wordpress.com/2009/11/19/we-get-it-checklist-for-vendors-plone/#comment-63

A tangent about comparing enterprise vendors to commercial community support: In enterprise search, I had to suffer sales presentations at a former employer from FAST where they constantly maligned our primary choice of Solr as not having a single throat to choke — they never purchased FAST and started slowly adopting usage of Solr with self-support. Now that Solr has a more robust primary-vendor sales channel (via Lucid Imagination), it might make Solr more attractive to the same companies that need hand-holding, high-transaction-cost sales process, but only for companies that have broken tech acquisition process (I would much rather see enterprise F/OSS vendors like this succeed for the quality of their customers, not the lack thereof). You should judge vendors (or channels of vendors) on the availability of commercial support and pre-sale consulting options, not the availability of sales staff with a product-agenda that trumps the agenda of the customer.

Disclaimer: I dabble as an occasional Plone consultant, and have made money from employment at user companies related to Plone/Zope for the past 10 years.

Sean

@James H: you’re right of course. Commercial imperative demands products that tick boxes and demo well. But just try living with such a thing! Poacher or gamekeeper….

Imagine the poor customer listening to the 6th presentation in a row, all claiming to be the holy effing grail of CMS. And than it is your turn, and you admit there are some bugs and funny GUIs. I have won many tenders with this strategy. You can not imagine those eyes wide open when I said we have bugs.

The other is open source. What I tell customers is to go and download it, find out for yourself. If the IT guys are not that busy, they can even check ut the source, at least the API parts.

I always wonder how people dare to decide on a CMS based on a written proposal, still many companies do.

Lest we forget – most vendors are extremely experienced in filling in tender and RFI documents that get assessed for their merit – including evading the question in the first place and answering the one you would prefer – to coin the phrase ‘the devil is in the detail’ would put it mildly – and reading the responses to a number of vendors definately put a competitive smile on my face……..

Here was our CMS meme: Vendor Meme – Enano CMS

Thanks so much for doing the extra dilligence of putting the scores and commentary together. It was good and fun professional exercise.

-Dave Twitter @ektrondave

I’m also really interested to know how Ektron determined that they have the most WCMS installations. Maybe they’re counting eWebEditPro installations too. Regardless, if someone has counted I’d love to get my hands on the results.

The truth is, you probably could put ANY kind of software product on this matrix and ask the same questions, since quite frankly some of these products bear no resemblance other than the vendors call them either WCMS or CMS or ECM ‘solutions.’

Thumbs up on this study, it feels a little to CMS/ECM as Jon Stewart is to real news (actually, some of his news is more incisive than the real stuff).

I think it is quite a fine line, though. I err on the side of brutal honesty and, as a result, a couple of our Client Services Team (our word for sales people) would rather I didn’t go to their meetings. If the competition is all claiming something is easy, you might not even get your foot in the door if you say it might not be.

Seeing my team are the ones that have to deliver the thing, I’d rather not win something that we can’t deliver according to the client’s expectations. I don’t get commission

I think the question about ‘does the CMS vendor eat their own dog food’ could be extended to ask when was the last time the sales person actually tried to use the system they are selling for the things the customer is asking rather than using a tried and tested canned demo?

As in all sales scenarios that classic Latin phrase ‘caveat emptor’ (let the buyer beware) applies and the onus is on the prospective buyer to demand a full access trail and to speak directly to other customers.

The rise of open source and its associated community followings has forced proprietary vendors to become more transparent and CMS Watch has pushed this further by telling prospective buyers to look for solutions that have wide and active user bases and communities.

In the proprietary environment I believe there is a point when the CMS Vendor CEOs stop really caring about their product and customers and driving sales (at all cost) becomes the all consuming goal. When you look at the pressures they face in paying for the overheads and keeping the City and shareholders happy this is understandable but sometimes I don’t think they realise how these financially driven attitudes manifest themselves throughout the rest of their organisations at the expense of the passion for delivering a high quality product and really delivering what customers want.

The natural progression from this is acquisition and consolidation which, as we have seen played out countless times, often sees product development stall and leaves customers wondering when those long standing product ‘pain points’ are ever going to get resolved.

As an example: Here is some information about soft crops for images in our upcoming 5.0-3 release: http://www.escenic.com/products/roadmap/article5896.ece

a) The questions asked by the customer are too broad so the vendor can answer yes with a reasonably clear conscience. But they do sometimes push the boundaries a bit here. I don’t think vendors lie blatantly about the capabilities, but some of the sales guys will say yes to everyone without actually knowing if the product complies. Or even, sometimes, what the question means.

b) Poor implementations, either by an inhouse team, a partner or even the vendor themselves. A badly configured CMS, as you know, isn’t going to do things smoothly. And the deliver side functionality is very dependent on the implementation, not the CMS.

On the SDK question, I was interested to know if you thoughts any of the vendors that responded were being shifty, or if they all do really give it away without any strings attached.

Thanks again for your reply.

Let me comment briefly on the “Software does what we say it does” item, as well as the “free SDK” bit.

I included the item on software doing “what we say it does” not because I thought any vendor would answer No, but because this complaint comes up again and again in customer satisfaction research. Amazingly, a major cause of customer dissatisfaction is that software doesn’t do what that the customer thought it would do! Obviously, vendors cannot be expected to score themselves fairly on this question. It should shock no one that every vendor believes their software does what they say it will do. The Zen koan that everyone should be considering is: Why the disconnect? Why do customers so often say that they are disappointed by software not doing what they were led to believe it would do, when vendors uniformly say there’s no problem here?

As for vendors charging money for SDKs, this is something I actually do see, believe it or not, especially among DAM vendors. Sometimes it’s a case of “if you sign up for our $5000 developer training course, you get the SDK for free” (but otherwise you don’t). Whether the SDK comes with a SKU number and a price tag or not isn’t the point, in any case. One way or another, you pay to use it. The point is whether the SDK is kept hidden, under lock and key, out of view of people who haven’t paid Big Money to see it, or is made freely available to anyone and everyone.

My view is pretty simple. SDKs should be free and posted on the public-facing company web site (not a password-protected extranet). Hiding API doc, SDKs, etc., makes no sense to me. Unless, of course, you are ashamed of your APIs or SDK. In which case it makes perfect sense to keep it hidden.

I would expect the vendor to get the SDK docs right at release, but code samples and gotchas only start to surface properly once the product gets used in the real world.

I’m with Kas on the documentation though. For some reason, many vendors think it’s a great idea to be able to keep updating the documentation. However, that implies that they don’t think they can get it right at launch of a new version (unfortunately, they’re mostly right about that).

Paradoxically, another annoying habit of online documentation is that it’s usually about the latest version. Not all customers (in fact, very few) immediately update to the new version. Some wait a long time. They need documentation on the version *they’re* using, and many vendors don’t keep separate (matching) versions of the docs.

I’m still clinging on to the PDFs I have for older versions of many products. They’re usually long gone from the vendor’s sites…