CMS Evals: Leading a horse to water…

Part of my role of late has been working with clients on Software Evaluations for CMS software. Sometimes we are hired as a stand alone project to make a CMS recommendation. Sometimes we evaluate and recommend CMSs as part of redesign of the client’s Website. Sometimes we use a formal process where  we gather and prioritize key client needs, identify key decision criteria,  then match requirements to product offerings narrowing the list from 8 to 4 to a final 2 vendors to demo to the client. Or alternatively, we use a more informal/faster/less costly approach, that leverages the  knowledge we’ve accumulated on past projects and simply matches the tools we know with their requirements and circumstances.

While many things vary from project to project — the client’s needs, budgets, technology preferences, interest in Open Source, the decision making team — our role is to lead clients to the point where they can make a decisions (yes this is the leading the horse to water part of the metaphor). However, there are some things happen on each project that amaze me including that sometimes you just can’t get the horse to drink from the right pond (have I extended the metaphor too far?). Read about the things that amaze me again and again, after the jump.

Ease of Use – Just when I think that we can share that one tool is easier to use (either based on our experience or feedback from of other clients), the client completely disagrees… Perhaps like the old saying goes, “Ease of use is in the eye of the beholder”.

Price Doesn’t Matter – Most clients subscribe to some variation of this statement at least in the beginning. But in many cases, price is a factor. We typically go into an evaluation with an idea of what the client would like to spend, but don’t use this as a criteria to eliminate products until we know more about the likely options or once it is clear that there are simply better options with lower costs. But just as when shopping for a car or house, clients often want to see what they may not be able to afford so they can understand what they might be missing.

In a recent project, the client was very clear that they only had $X for the software license and thus we restricted the search to those tools that cost less $X or less and Open Source tools. We knew that based on their requirements, there were few tools that would fit their needs within the budgetary and technology preferences. After initial demos received a cool response, we suggested a demo of a tool that we’d indicated from the start would be a great fit, save for its cost. Turns out, they thought it was a great fit too and then made a case internally to spend more than they planned to get the capabilities they wanted/needed. Conclusion: Price is always a factor but just one of many. Don’t over-weigh this factor til you have considered your full set of options.

Establish and follow criteria for making a decision, but expect factors to change. In all projects, one of the first steps is to highlight must have features, internal business factors like staff skills, existing technologies, impact of change on the organization, implementation timeline, and budgets. From this list we arrive at the critical criteria that all agree Beaconfire will use to make a recommendation and that the client team will use to make a decision. Well, it turns out that no matter how good we get at eliciting these priorities, nor how well informed the client team is, in the end, the decision will almost always involve factors that emerge in importance during the project. I think this is largely a good thing since often new factors result from a natural evolution in thinking and education on the client side.

When selecting a CMS, the reality is that there are no perfect fits, just choices with a tradeoffs. Our job is to advise clients on how to balance the tradeoffs and successfully overcome deficiencies, to mitigate risks and ensure success. And success often hinges on business factors and impacts rather than technology ones.

The difficulty for us is when the ‘new factors’ result from a smooth/slight of hand sales demo or the focus on the latest shiny feature that doesn’t address the core business need. See the section on biases below for how we deal with this. Conclusion: Expect different factors to emerge but be clear that these result from evolving needs/priorities rather than less substantive causes.

Quality of the Demos really does matter - In every project, there is one factor that can’t be completely mitigated… The products that are demo’d well, regardless of the tools capabilities, gain an advantage. This is human nature and demonstrates the value of a good sales engineer. We try to even the playing field by having vendors review the same functionality in the same order by using a demo script. This can allow clients to see how two features compare more directly and expose the holes in some tools.

Demos also invariably reinforce any biases that exist and are rarely convincing to those who’ve made up their mind. However, every now and then a bad demo will sink a solid tool even for those predisposed to like it. The demo is also the first time a client will experience the tool, thus the strength of the demo is the single biggest factor affecting the perceived ease of use. A bad demo is a huge problem for those looking for an easy to use tool.

Impressions/Perceptions/Bias – Each client has some bias, or perception or other source of information that they bring to the evaluation in addition to what Beaconfire presents them with. Sometimes this is feedback from peers who the client respects. Sometimes it is good/bad press or buzz from the market that gives folks preconceptions about some tools. Sometimes there are just differences in outlook between internal departments/players centers (IT vs Marketing for example). One challenge for us is to both understand these issues so we can either work to overcome a bias that isn’t factual or to ensure that there is a balanced view from all sides of an issue. These perceptions/biases etc usually play out in a few ways:

• Beaconfire validates the bias/perception and the process moves smoothly toward a recommendation.
• Beaconfire refutes a bias and the process moves more slowly, thoroughly towards a conclusion:
• One outcome is where the client ultimately finds another reason way to reach a decision that validates their perception.
• The other outcome is when the client makes a decision contrary to their initial point of view.

Can we make them drink? I mean the ‘lead a horse to water’ metaphor not the ‘we drive our clients to drink’ metaphor :) Ultimately, Beaconfire’s role as honest broker is to ensure the client has all of the information they need to make an informed decision. In three of the last four projects, we led the clients through a process that revealed what we thought was a clear choice in the tools.  (When we say the choice is clear to us, that doesn’t typically mean the other choice is bad or wouldn’t work for them. Rather, most of the time the client has two solid options to choose from and we lay out ways that each tool would work for them, leaving the final choice to them.) As we neared the end of the project, it became clear that the clients were unsure of which tool to select or were holding back due to something they couldn’t quite articulate. It turns out that in each one, a factor mentioned above was at play.

Conclusion: Our job ends at leading the client ‘to water’ but that ‘drinking’ is up to them. They ultimately have to weigh the factors, the comparison information, and the recommendations we provide and own the selection of one tool or the other. Basically, we can lead the horse to water, but can’t make them drink from the pond we recommend.