How to Keep a CMS Vendor on their Toes
You say my eyes are pretty and my smile is nice
Well, I’ll sell it to ya at a reduced price
You don’t understand it – my feelings for you
You’d be honest with me, if only you knew
- HONEST WITH ME
We’ve all read plenty of CMS Evaluation RFP response documents. And we’ve all sat through many a long CMS Vendor demo. Maybe it’s just me, but they’re all starting to look pretty similar. Most vendors say yes to nearly everything, and the top products have interfaces and patterns that are converging. Many experts are agreeing that having long lists of requirements doesn’t help anyone anyway. So, how do you differentiate between these things?
When picking a CMS, I think the actual product is only about half the story. The other half revolves around the intangibles including the vendor themselves, their support, their roadmap and priorities and, very importantly, how much you trust them. So, to mix things up a bit, why don’t you ask the vendors a few of these questions during your next RFP or Q&A session. When you interview a candidate for a job, you always throw in a few odd questions to keep them on their toes. Let’s do the same to the vendors.
- Who was the last vendor to beat you in the last round of a selection exercise? Why do you think they won? – This doesn’t have a right answer, but every vendor has lost a few. It would be interesting to see how they answer this.
- If, in a few years time, we decided to move away from your product, how would I go about migrating all my content into a new system? – I want a system with a nice content export and a vendor willing to admit it is a possibility.
- How many active developers do you have on your developer forums? – This is something you can check. If you ask in the demo and you think they’ve made up a number, say “Show me”.
- All of these are important, but please rate these in order of your priority: a) Product Features b) Performance and Stability c) Usability d) Security – Again, no correct answer. I’d expect most to say c) as customers give this as the biggest selection criteria. But still interesting to hear what they say after they’ve repeatedly said they’re all important.
- How much would I expect to pay a contractor developer that is skilled with your CMS, and are they easy to find? – If a vendor says they don’t know, they’re probably talking horseshit.
- Assuming your CMS license cost 10 groats, how many more groats would you guesstimate we need for our entire implementation? – I’d expect them to say somewhere between 10 and 40, depending on the project. But fun to hear what they say. Implementing costs more than buying.
- Why are the URLs on your demo site/corporate site so ugly? – Most of the demo sites have crap, non-SEO friendly URLs. If the implementation of Friendly URLs is as easy as the vendor claims, what haven’t they done it on their own sites.
- If we selected your CMS, how would you recommend we went about selecting an implementation partner? – Three kinds of answers here: a) we’ll build it for you b) we have an extensive partner network or c) we recommend Company XYZ. I’m skeptical of a), but even if you like that option it’s a good discussion to have with the vendor.
- How important is accessibility to you, and why doesn’t your site (which is presumably built with your CMS) validate? – They almost never do. I did a W3C validation test here not so long ago. Most will blame the implementation. Say will say it wasn’t a requirement, which I don’t buy. They fact that they don’t validate 100% isn’t a disaster, but it has got to look like they’ve tried.
- Your product also includes modules for analytics/search/community/collaboration/outbound email. When should I use these instead of the existing products I own for these, and what benefits do your products give? - Many vendors seem to use features that I wouldn’t consider core to Content Management in order to differentiate. I like my CMS to be focussed, but I’ll concede that sometimes getting other components as part of the same product might be interesting. But ask things like “So, how does your analytics compare with Omniture” and see what they say.
Even better, during the meeting you could have a screen running a Twitter search for the vendor’s name. You’ll see a fair bit of “Looking at XYZ” and “XYZ Industry News”. But you’ll also see a lot of “A Pox on XYZ and all of their houses” from angry developers.
Assuming it isn’t illegal to ask this kind of thing, I’d love it if people gave it a try. Or do you already have similar questions you ask that I could add to the list? One day I might be sitting as the same meeting as you and we can both enjoy some questions we haven’t heard twenty times before.
RSS Feed
Follow Me
I can assure you that whenever a prospect has these “odd” questions (and it does happen), I will always answer them truthfully.
If, a vendor, doesn’t do this, it will come back as a boomerang and for sure they don’t want these kind of negative chatter going around. Especially with the power of social media these days!
I personally welcome any kind of questions and I would urge all who is thinking about buying a CMS, to really assess what is important for the organization and what will be important in a couple of years(this is a difficult one). Further more, organizations who are in a CMS selection, need to be aware that this needs to be done by someone who knows what a CMS is and what the capabilities of a CMS are, this way the chance of misinterpretations are minimal.
Great questions. I’d add another one though:
Tell us about 2 or 3 projects where you were not successful and the reasons for those. I am quite sure they will blame requirements or implementation team but if they are honest, it’ll help to know the kind of things they can’t do that well.
Great post, Jon! As you said, vendor and implementor should probably seldom (never?) be the same people, for a number of reasons. Successful implementors usually haven’t chosen CMS products for their offerings based on chance – they want something that they like as well. So, the partner layer between vendors and end-customers is actually a good first selection process. A bit of Darwinism, perhaps?
However, to complement your list, I have a question that I usually include whenever I have an advisory role in CMS selection projects: What kind of companies or websites is [product name] not primarily designed for?
Answers vary on this one, but I’ve found that this question often isn’t perceived as being “hostile” in any way, and you can learn a whole lot from the answer. Not only about the product, but about the vendor as well.
Interesting questions, but I would never advice buyers to ask them, except perhaps 3 and 5. I would rephrase #3 to something like
#3: How active is your developer community?
- Counting developers is like counting references – what matters is: Is it likely that somebody answers my questions. Again: Quality vs. quantity.
and for #5, I would ask for a listing of some recommended freelancers/contractors/implementation partners.
The other questions are interesting mostly for industry insiders, including those working at digital agencies. For buyers, I would call them a detour for CMS selection and probably a good source of confusion, at least for the uninformed buyers, some of which I am sure read this blog as well.
Surely these buyers need to have someone informed (like yourself) assisting them with the vendor selection process? If everyone doing the selection truly is uninformed, they’re going to get given the run around and will struggle to understand all of the very important intangibles. Getting a vendor to have a generic chat so you can start to understand the chemistry is surely useful. I do accept your point that some buyers might get confused if they don’t really understand why we’re asking the questions.
If those questions confuse the buyer, the buyer probably shouldn’t be in a CMS vendor meeting by themselves!
I often participate as an adviser in CMS selection processes simply to avoid the buyer becoming confused – the questions still need to be raised.
Doing a CMS selection is quite different than interviewing a candidate for a job. I do accept the point that the intangibles are important and often underestimated.
If the buyer has involved some external help, like myself, I would not advice them to ask the questions. The customer should expect the external expert to know the answer to a few of them (e.g. 2, 3, 5, 6) and for the rest, I can’t see how their answers changes much.
Here some easy ways of keeping a vendor on their toes:
- if you want to show us slides, you have to send the PPT at least 2 days in advance
- follow the agenda
- show us a demo
- send us a list of who’s coming to the meeting, including names and title and role in project
- ask about the upgrade to the next planned version; how difficult will it be? when?
- ask about what they see as the most significant challenges in your project
- give them no more than 60 minutes
Keep up the great posts.
Cheers!
Janus,I wish more buyers would adhere to the 60 minute rule. Unfortunately I’ve seen the trend go in the other direction. I think 60 minutes is more than enough time to present and demo in most situations. Basic CMS functions are commoditized, vendors should stick to presenting and demonstrating what makes them different and special.
Back to the original point. As a vendor, none of these questions listed here would give me much heartburn (but then again, I’ve been doing this for 10 years…) I might prefer not to answer some of the questions, but they are all reasonable. Maybe the lesson learned is to treat these sorts of questions as a test for the vendor. Any vendor should be able to respond to odd questions. If they won’t (or can’t) it says a lot about the vendor and/or the vendor’s sales team.
From my experience, I believe that many buyers don’t give enough time. An hours goes very quickly and it’s very difficult for a buyer, who isn’t familiar with CMS, to differentiate products in that time frame. At most you’ll be able to disqualify the poorest of solutions.
Secondly, it really worries me when I see a long agenda and a short amount of time allocated to conduct it. It raises flags with me that the buyer isn’t serious about the project to give it necessary investment of their time or they see the whole project as a buying a commodity. It can also indicate that they’ve already chosen a solution and is just going through the motions.
My recommendation to buyers is allow 1 hour of a end user focused demo / scenarios and a second hour for a more technical discussion (how easy is it to setup, how flexible are the templates, the architecture and other techie questions)
The whole timing question warrants a separate discussion. I think 60 minutes might be enough if it is a pure presenation without too many questions and interactions. However, I far prefer an interactive session which is impossible in 60 minutes. Nothing worse than being asked lots of questions and then being told to stop when the 60 minute buzzer goes off.
I think I like Piero’s suggestion.
When we’re interviewing a candidate for a senior job, it’s not unusual to have 3 or 4 interviews of 60 minutes each. And it’s a whole lot easier to replace a badly hiring candidate than a badly chosen CMS, I think.
Good stuff, we have a demo from a ‘big’ vendor next week I’m considering posing questions #1, 2, 7, 8 and 9.
I think this is a useful list, particularly inasmuch as it focuses more on working with the vendor (“VX?”) than with the tool. If you ask in the demo, though, you might not have the right people in the room to answer fully, but even partial answers will be revealing.
I like a lot of these. Given that a new CMS is a 3-5 year commitment, I’m surprised that there aren’t more questions about vendor intangibles in the RFPs I deal with, particularly around vendor financials… i.e. are they going to be around in 3+ years, or are they likely to get swallowed up in a recession-driven consolidation of the CMS market?
Great post. This is a topic we are passionate about and advocate for anything can cut through the scripted demo and get to real issues of: Is the CMS capable of addressing the organizations issues, how will it do it and once it is done will the end users actually be able to use it?
A couple of posts we did that compliment what you wrote here:
http://www.cmsmyth.com/blogs/cms_myth/archive/2009/04/14/get-your-cms-vendor-off-the-script.aspx
http://www.cmsmyth.com/blogs/cms_myth/archive/2009/05/30/does-your-cms-fit.aspx
I can tell you the answer to question #1 since I’ve heard it so many times. First off, you’ll never hear a truthful answer. (I’ve heard an honest answer maybe two or three times. And usually, that was after a couple of beers.) Secondly, they’ll have won deals against the perceived #1 and #2 in the market, which they’re up against “all of the time.” I.e., in search, for years, everybody was telling me they won a deal when they were on a shortlist with Autonomy, Verity, and/or Fast…
#9 isn’t really fair. Sure, vendors are sloppy with their own sites (why tie up professional services in work that doesn’t generate revenue directly?). It’s a bit like doctors with a secret smoking habit. They should know better but hey, they’re only human
And the one I would add to the list: my favorite question while sitting through vendor demos (before I knew them as well as I do now). With a smoke-and-mirrors demo of all the features, always ask “Cool. But how do we turn it off?”
When I asked Jon, just a second ago, I missed Janus’ comment about “ask about the upgrade to the next planned version; how difficult will it be? when?”.
This whole upgrade cacophony scares me. It’s not only the bought software (CMS in this case) that can be an issue. But the underlying OS (W2K -> W7?!), the language (PHP3 -> PHP5!), the plug-ins and 3rd party software. Some of that unsupported, “yanked” from somewhere for free (In my head is playing episode of Beavis and Butt-Head where Tom Anderson is walking around in a HUUGE DIY shop after hours, echoingly, asking ‘anyone here..?’).
Anyway back on topic, at this point, it might be good to ask for a technological road map that includes a) OS, b) Language/Dev platform, c) any third party software.
The main reason I didn’t include the upgrade question is because it is something I see asked a lot anyway. The vendors have canned answers for it. But getting the roadmap is always a good thing.
It is a great list … personally I have been using some of them and I watched lots of vendor’s dancing in writing and in person answering them.
What about one more question on the disclosure of the third party and OEM components in their system?
Very interesting reading. Makes me wonder about a few answers my company would come up with…
Feel free to answer some of them here as a test run. Use a fictional company name like zE Publish
zE Publish ? That’s a good name ! I’ll give it a try tomorrow morning
where to buy Premier and Stitched Chicago Bears jerseys ??…
[...]we adivce go to this website to see more about Chicago Bears jerseys and others. [...]…
where to buy Premier and Stitched Miami Heat jerseys ??…
[...]we adivce go to this website to see more about Miami Heat jerseys and others. [...]…
Thankfulness to my father ԝɦo shared ѡith me regarding this
weblog, this webpage iis actualloy remarkable.
Ѕtoƿ by my homepage … aftonbladet
It’s difficult to find knowledgeable people in this particular subject, however,
you sound like you know what you’re talking about! Thanks
I used to be recommended this web site via my cousin. I am now not positive
whether or not this submit is written by means of him as no one else recognise such
precise approximately my problem. You’re wonderful! Thanks!