• web-based card sorting to learn how your audience groups and labels content
• tree testing to validate your navigation
• task-based “first click” testing to get insight into how people navigate your site
• usability testing on clickable prototypes, to evaluate a new site structure / IA

And there are new tools being introduced all the time.

So, which approach is best for your website? Are some more effective than others? Do low-cost, rapid methods produce the same results as traditional usability testing studies?

If you’ve found yourself asking these questions and would like to hear opinions from a range of experts, check out the panel I’m proposing for SXSW 2012: UX Smackdown! User testing techniques in the ring. After registering for the SXSW panelpicker, you can give a “thumbs up” vote and leave comments to help this panel become a reality.

The panel speakers are:

• Aviva Rosenstein is lead UX researcher at Salesforce.com — you may recognize her from UIE’s recent Web App Master’s Tour!
• John Whalen of Brilliant Experience has over ten years of user-centered design experience and has most recently presented on the topic of persuasive design
• Kyle Soucy of Usable Interface has created interfaces for everything from web sites to touch screen devices. You may have seen her speak at a past UPA or IA Summit conference, or read one of her articles at UX Matters.
• Michael Summers of SUMMERS Consulting has watched over 1000 consumers make real end-to-end purchases on the web. He was one of the earliest adopters of eyetracking technology for UX research.

I would love to have the opportunity to pick their brains and get some UX testing tips — if you would too, head on over to the Panel Picker to vote and comment before Friday’s deadline!

• Your redesign went great – until you were running around the day before launch, trying to get content from your program staff and stuff it into your new CMS.
• Two months after your new site launched, you find a headline that says “Lorem ipsum…”  Oops.
• You leave the default autoresponders for all your most important actions, thinking “we’ll fix it later.”
• Your landing pages are ranking low in search, and have high bounce rates. You suspect your audiences just aren’t engaging with them, but haven’t figured out why.
• Your program staff are responsible for your web content, and they really know their stuff… but they don’t know much about writing for the web.
• Your design is beautiful, but once you start entering content, it’s just not fitting in the boxes that looked so perfect in the design phase.

These things could happen to you, if they haven’t already. Planning for content during a redesign often just means mapping the old content to the new, and then migrating it. But it should get more attention than that, because content is the most important part of your site.

I repeat: content is the most important part of your site.

Content — be it text, images, or video — is what people visit your site to find. Content — headlines, buttons, auto-responders, images — is what motivates people to become supporters or donors.  Content — page structure, metadata, alt tags — determines how your site will rank in search engines.  Content is important stuff.

That’s why Content Strategy, though an emerging field, is growing at lightning speed as people in the web community say, “yes, this is something we need.”  It’s been pioneered by folks like Kristina Halvorson, whose thoughts you can check out on the Brain Traffic Blog.

Content Strategy means deeply analyzing and revisiting your content throughout the redesign process, starting with a messaging strategy and letting that inform your content throughout the site.  It also means planning for content creation, curating good content, and actively improving your existing content before you migrate it.

The most natural time to do a content strategy project is during a redesign, but it can be a stand-alone effort as well. If it’s something you know your site needs, it doesn’t have to wait. Content isn’t always the first thing that’s looked at to improve a site’s performance, at least not holistically.  But in one way or another, content is often the solution, whether you need better SEO, a more emotional image for a landing page, or a more concise introduction to an important form. A content strategy project is your opportunity to look holistically at all these elements, and make sure they’re working together.

Have you done any content strategy for your organization’s site?  If so, what was your experience with it?

Through the IA process, we focus on two parallel inputs – our client’s needs and the needs of their priority audiences.  We identify the key information audiences want from the site and frequently get their input on how they’d categorize that content via open card sorts.  The card sort gives us an opportunity to learn how the priority audiences think about and group information on a site and what language they use to describe it.  In an open sort, participants are presented a list of content items and asked to group those items into whatever categories make sense to them.  Rebecca wrote a great post that explains more about card sorting last summer.

After the card sort, we bring that input back to the client in the form of a draft a sitemap and continue to refine the site structure to ensure all the content has a home.  Before we sign-off on the sitemap, we like to present the “new & improved” navigation to users and see if it is going to work for them.   In the past, we have tested the new navigation with a ‘closed card sort.’   Like an open sort, users are presented a list of content items but in a closed card sort, they are asked to sort those items into the pre-determined category (navigation element) that makes the most sense.  Closed sorting has worked well for our navigation validation – we’ve seen good results with the testing and solid performance on the navigation.

However, we recently discovered OptimalWorkshop’s TreeJack, a “tree testing” tool that allows us to test the navigation from a much more authentic starting point.  In a tree test, participants are asked to click through the site tree (navigation structure) to complete a series of tasks.  This type of process closely mirrors website users mental model “Where will I find X” when they come to a site.

In tree testing, we develop tasks that align with the priority content and features of the site or, if a particular navigation element is in question, we can create a task to test that. The testing results give us a quick overview of how many participants succeeded (got the page we identified as an answer for the task); whether they got there directly or wandered through the navigation; and how long it took.  In addition, the testing results show us the path that each participant took for each task which gives us good insight into the ‘why’s’ behind unsuccessful tasks. 

We’ve been impressed with the TreeJack tool.  Tests are easy to set-up and the test results are easy to understand and actionable.  It’s also nice that the tool is online, so it ready whenever your participants are, and no special software is needed.

Sounds fun, doesn’t it? Try it out here and let us know what you think: https://beaconfire.optimalworkshop.com/treejack/survey/BeaconfireWire

What I do know about is what knowledge is online. Since Sir Tim Berners-Lee (Yes. He was knighted.) created the World Wide Web to link documents together on the then nascent Internet, knowledge became more than monolithic documents or books that were linked loosely via citations and references. Instead of specifying in APA, MLA, Chicago, or Turabian style where the source of a particular knowledge was, one could directly link it using something called “HyperText”, or what some know as “Hyper Text Markup Language”. Today, all websites that you see online are built with a combination of HTML, some JavaScript, and possibly some Flash or Java.

Never mind what the knowledge is built with. Let’s examine very briefly the trend of tagging and keywording content. “Tagging” or “keywording” existing content can help web site users browse through indexes of such tags or keywords similar to how a book may be indexed by the editors and publishers. What was the first representation of such archaic systems on the Internet? Search Engines. Search engines were of two varieties. Directories such as Yahoo or Excite and Search Indexes such as Alta Vista. Both had their place. If you wanted to browse through a human edited directory, you had good reason to. People had taken the time to classify and organize the websites using their God given intellect. If you wanted to search, you also had good reason to.

Since the computer was able to search all of the content in a set of knowledge automatically, search indexes could be created using complex algorithms. One of the reasons Google succeeded in the search indexing industry was because they figured that relevant pages would link to each other and the more relevant websites would have higher density of connections. Similar sites around similar sets of knowledge would coalesce using the network effect to create centers of knowledge. Further more, content that is properly tagged and keyworded can heighten the quality of the search because the search engine has the power to correlate keyword usage and the keywords themselves.

What is OpenCalais?

The Calais Web Service:  The Calais web service automatically attaches rich semantic metadata to the content you submit. Using natural language processing, machine learning and other methods, Calais categorizes and links your document with entities (people, places, organizations, etc.), facts (person “x” works for company “y”), and events (person “z” was appointed chairman of company “y” on date “x”). (OpenCalais 2009)

Now that you have a bird’s eye view of how knowledge is stored and sought online, let’s talk about OpenCalais. OpenCalais is a toolkit that can be used by Content Management systems to automatically find the best set of keywords to represent a particular piece of content. Instead of having to pour your blood, toil, sweat, and tears, you can have a machine do it for you. OpenCalais is not for consumers and must be integrated into an existing piece of online content management software, or be integrated into a custom solution.
If you have a 5-10 page website, you probably don’t need to use OpenCalais. If you have a catalog of any kind, be it books, products, article, or news you should look into integrating this tool so that your website becomes easier to search and navigate on your website as well as through external search engines. OpenCalais looks at your content and finds out if there are references to people, cities, companies, industry terms, and a number of other existing knowledge centers.

A test using the OpenCalais Document Viewer

To demonstrate what OpenCalais can extract from your content, I submitted the content of this article above this sentence to get this screenshot. I’m only showing the highlight of one found term. Similarly, the other terms that were found are connected to vast amounts of knowledge already exist intent on the internet. Those knowledge centers have relevant information and hence if your content is relevant, it is given a high content relevancy rating for each word that it finds. Apparently, my article on “Online Knowledge” only gets a rating of 35%. It’s better than nothing. I’ll try again next time. You can try yourself using their Document Viewer. If you want help in implementing this great tool, let us know by contacting us through our website’s Contact Us page.

After completing card sorting exercises and drafting a new information architecture with NTEN, we moved on to the next step: usability testing with wireframe prototypes!

Testing with Wireframes

We most often use usability testing in two ways:

1. To evaluate an existing site and identify areas of improvement
2. To evaluate a new enhancement, redesign, or information architecture — something that doesn’t actually exist yet

Our project with NTEN falls into the second category. So, how do you test something that doesn’t yet exist? We used wireframes:

By quickly sketching out the new navigation and some key content pages, we were able to produce an interactive prototype. One of the first things you’ll notice about our wireframes is that they aren’t exactly pretty. They’re nearly entirely in black and white, with no styling elements.

There are a few reasons for this:

• It allows us (and the participants) to focus exclusively on the navigation and content.
• It allows us to test early in a project, if the visual design hasn’t been determined yet.
• It’s quick! And easy to edit on the fly, thanks to the lack of images and css.

It’s also helpful to do usability testing with high fidelity prototypes later on, once the visual design has been developed. The use of color and styling can have a huge effect on the weight of different elements on a page, such as navigation, which is why usability testing can be even more valuable as an iterative process.

The Recap

We conducted 12 usability testing sessions at the Bridge Conference last month. Using the wireframe prototypes, participants were asked to complete a series of tasks, some specific (“Renew your membership”), and others more open-ended (“Think of a nonprofit tech topic you’d like to know more about and find information on it”). We quickly began to identify areas of the new navigation that work extremely well and ones that still need some adjusting. Participants also gave feedback on content and we learned about how they use the NTEN website as it exists today.

What We Learned

Some examples of our takeaways from usability testing:

• Members of NTEN want the community (such as discussion forums and networking opportunities) brought to the surface and featured more prominently
• As we learned in the card sorting, about half consider webinars to be events and half don’t — everyone seems to have an opinion on this, one way or the other!
• The current account management section has room for improvement and can help visitors by giving time-sensitive information (such as upcoming events that you’ve registered for) priority on the page.
• The “Get Involved” navigation item should be replaced with “Community” — this is the terminology that visitors use and are looking for. Plus, “Get Involved” could mean a variety of things.
• Visit NTEN’s blog entry below for more.

NTEN’s Perspective

Check out NTEN’s blog entry on the testing process, complete with video!

Wireframe Testing: Failing Informatively

Card sorting is a common technique used to gather input from representative audiences about how they think about your information. Traditionally, participants are given cards representing pieces of content (or sections of your website) and asked to organize those cards in a way that makes sense to them. The process can also take place remotely, using online tools, as we’ve detailed below. When re-structuring any website, card sorting is an important step to ensure that the new site structure is as intuitive as possible for visitors.

Preparation

To create the list of items (or “cards”) to be sorted, we used two inputs:

• A content inventory, which is a list of all content/pages on the site. The list is created in a spreadsheet which clearly shows the current structure of the site and which main section each item is currently categorized into.
• A list of the website’s audiences and their primary goals or tasks. This helps to ensure that every key task is represented by a content item in the card sort, and it is crucial for usability testing (which will be a later step in our process).

We ended up with a list of 47 items that represented key content from NTEN’s website. The list was then loaded into Websort. Websort provides virtual card sorting, allowing participants to drag and drop cards into categories, then label those categories.

There are some limitations to online card sorting – participants can’t write or comment on the individual cards, there isn’t any communication between the facilitator and sorter (aside from one comments field), and the sorter can only create sub-categories using a labeling workaround. But online sorting also has major benefits: Anyone can participate, regardless of geographic location or scheduling constraints. The time commitment for facilitation is minimal after the sort has been set up, compared to the many hours involved in facilitating traditional in-person sorts. And lastly, the data is immediately ready to manipulate.

For NTEN, doing a virtual sort was a success: In the first four days, over 120 people participated.

One Sort, Two Audiences

We created two separate instances in Websort: One for active members of NTEN who have participated in an event, and one for those who belong to NTEN or have heard of it, but aren’t necessarily active. NTEN then distributed the links via email, blog, and Twitter to the two different groups. Although the cards used for the two sorts were exactly the same, splitting it into two instances would allow us to identify differences in the way the two audiences think about the content.

Results & Analysis

Card sort results can give insight into the following:

• What groups are created by participants?
  Websort uses a clustering algorithm to provide quick visual overviews of how items have been grouped together.
  

  These visuals make it easy to see not only what common groups are emerging, but which items were most difficult to sort.

• How did participants label those groups?
  Despite creating common groupings, participants labeled the groups differently. For example, some labeled by type of content (“Research” or “Webinars”). Others labeled by what they would use the content for (“Learn”).
• Do active vs. inactive members think about the content differently?
  We saw a couple of minor differences in the way active vs. inactive members grouped content. For example, active members more often gave the Non-Profit Technology Conference its own category, while inactive members grouped it under Events.

To read the full list of observations, download the summary document.

Next Steps

Next, we’ll create a revised information architecture for the NTEN website, which shows how the content should be reorganized. Then, usability testing will be conducted to confirm the new structure and make any necessary changes. We’ll be providing more updates along the way, so stay tuned!

Creative Decisions:

Analytics packages can tell you a lot about the technology your audience uses. Some stats worth looking at include:

• Resolution;
• Browser;
• Flash Version;
• Java.

Are 75% of your users at 1024×768? Then maybe it’s time to break out of that limiting 800×600 design. Do 30% of your users still in IE6? Avoid transparent images. 5% of your users don’t have Flash or Java? Make sure you offer alternatives to that nifty Slideshow on the homepage (which you should do anyway, but that is another post for another day).

IA Decisions

What do you think is your most popular page (after the homepage)? Are you sure? Looking at your analytics may surprise you. If they do, then take a closer look at your popular pages.  Use your analytics to figure out how it got so high, and learn about how you can bring up your most important pages. Something as simple as the order of navigation can make significant changes in your top content reports.

Usability Decisions

What brings people to your site? What keeps them there? You can get a lot of this information by looking at the following:

• The keywords that bring people to your site;
• The keywords that keep people there;
• The keywords that drive people away;
• The words users enter when they conduct a site search;
• The page a user is on when they conduct a search.

Do you have a high bounce rates among users who search for “scholarships” on your site? Maybe your scholarships page is difficult to maneuver. Do a lot of visitors use the “search” function when they are on your Get Involved page? Then maybe your Get Involved Page doesn’t have the content most users would expect. This kind of data gives you insights of dozens of user interviews. Use it to transform your site’s design and information architecture in a way that your users are able to find exactly what they need.

Shortening urls provides some important benefits to the person sharing the link, which is why they’ve become so popular.  Aside from taking up less space (thus making them more tweetable), many of the services provide tracking data, giving you details about how your content is being used.  That can be really fun for the casual user, and valuable for organizations.  Some people think they look tidier than long urls.

But there are two (or maybe three) other parties affected by the shortened url, and they aren’t so well served by it.

The reader, who comes across a shortened url in a post or tweet, may have no idea where the link is supposed to go.  All they get is the context of the link, which may be as little as “check out this cool link”.  Without clicking on it, they have no way to know where it goes.  (Not totally true – there exist Firefox plugins that will expand them for you – but it is true for most users.)  From a shortened url, you can’t tell whether the destination is the New York Times, a non-profit, a personal blog, or something far less reputable.  For you, as a user, all the clues you use to decide whether to click on a link may be missing.  That’s a problem for the person posting the content – if you don’t trust them, you’re much less likely to click.  Furthermore, it’s less likely to catch your interest.

The benefit of longer, SEO-conscious urls is that you should be able to tell what you’ll find before you click: you know the domain, the page name, and possibly other information.  Take this post, for example: http://www.beaconfire.com/blog/2009/05/26/url-shorteners-how-to-stay-out-of-trouble tells you that a company called Beaconfire is providing the content (even if you’re not familiar with who we are), that it’s a post from our blog, that it was published in May 2009, and that the topic is url shorteners.  Armed with that information, you can decide whether you’re interested enough to click on it.  Run it through a service like bit.ly, and all that information is lost – but at least the link is short?

There’s also the content provider – the site that’s being linked to.  At the very least, they’re losing their branding, since their domain is no longer displayed.  A lot of SEO experts are also concerned that the shortening services themselves – which parse the link and redirect you to the destination – may rob the destination content of its search ranking, that search bots may not follow the link and give credit to the destination.  Is this a serious concern?  It’s an open question, and every service works a little bit differently.  Some do much better than others, so it’s worth doing your homework on which url shortener will best preserve your SEO.  (In all cases, you lose the benefit of having keywords in the link text, which is typically just the url.)

The third party affected by shortened urls is the internet as a whole.  Joshua Schachter has written a fantastic and detailed explanation of why shortened urls are hurting the web – the gist is that these services are middlemen in an already complex system.  They are numerous, fragile, and unreliable, and if they someday disappear, we’ll have a lot of links that no longer point to their content, and the internet will be broken.  If that seems like an overly philosophical concern, consider how quickly sites often come and go on the internet.

If it were up to me, I’d say these services aren’t worth the trouble.  But, unless the whole world wakes up tomorrow and decides that Twitter is just no longer cool, I think shortened urls will be around for a while.

So, how can you embrace social media without the downsides of shortened urls?

Most importantly, don’t use them when you don’t need them.  If you’re twittering to promote your own campaign, create a friendly url that will fit easily in a tweet, and you’ll promote recognition for your site.  This is especially true if you’re not using any tracking features from the service.

If you do need to use them, look for a service that lets you customize the url to some extent, as well as following good SEO practices with their redirects.  Check this list for an easy feature comparison. Be sure to provide plenty of context about why users should click through.

One service, kl.am, even lets you append Google Analytics campaign variables for your short url so you can track them by source in Analytics.  I can’t speak to kl.am’s other features, but this seems like a nice one – often, appending source codes is part of what makes urls long and unsightly.

The biggest lesson, to me, is not to use shortened urls frivolously.  If you don’t need them, don’t use them.  Period.  If you do need to keep your url short, consider your readers and your brand, and use these services wisely.

• NTEN Blog
• Gavin’s Digital Diner
• Confessions of a Non-Profit Executive Director
• Wild Apricot
• Michael Stein’s NON-PROFIT TECHNOLOGY BLOG
• Zen and the Art of Nonprofit Technology

Step 2: Aggregate them all via a Yahoo! Pipe

• The Pipe (feel free to clone)
• The Pipe’s RSS Output

Step 3: Consume the feed with Wordle

Step 4: Mess around with screenshots in Photoshop for 15 minutes because people still hate Java Applets

Here’s a screenshot of his homepage above the fold a few months back (via nowitis):

And here’s a screenshot of the same from today:

Notice the difference? Analysis of how the space is used, and lessons for your nonprofit, below the fold…

Before we cut the page up and start looking at calls to action and space utilization, let’s look at the purpose of the site. There are two primary target audiences for barackobama.com: activists/small donors and undecided voters. There are also a number of secondary audiences, including the media, big donors, wavering Democratic and Republican voters, and bloggers, but for now, let’s concentrate just on our two primary audiences. These goals haven’t really changed much for the two versions of the site – the universe of undecided voters shifted from Dem primary voters to general election voters, and the activist pool had to expand to include a call for former activists for other candidates, but they’re still largely similar.

The goal for the activists/small donors is to get them to take actions that will help the campaign, including donating and volunteering, as well as to keep them fired up for future asks to take these actions. All four items in the central “big story” area apply to these users, along with the “vote for change” plug, the “get involved” button, the MyBO bar at the top, and of course the big red donate button.

The goal for undecided voters, of course, is to convince them to vote Obama. For this group, two of the “big story” items apply (the “Honor” ad and the plug for hurricane victims), as well as the state homepages and the tagline near the top that says what the campaign is all about.

One item that straddles the two categories is the “En Espanol” button near the top right – ultimately, I’d say that it currently caters more to undecideds, given that most of the content on the page focuses on persuasion rather than action, but note that there is a Spanish “donate” button as well.

Scrolling down there’s a bevy of other actions as well:

We can see that there are more activist-themed widgets on the bottom than those oriented for undecided voters, but did you notice how many of them repeat content at the top of the page?

This is where I see a real lesson for non-profits. The overcrowded nature of the homepage didn’t crop up overnight. It probably appeared bit by bit as various departments saw their items creeping down the page, and that they were no longer hitting their goals. So they leaned on the web team to give them a plug near the top of the page as well – the states, MyBo, action center, and voter registration all moved their way up in between the two screenshots (and Biden moved in, but I think we can call that unrelated).

The result of this is the tragedy of the commons: everyone said, “Well, it’s just one tiny button – how much harm can it do?” until the page became so filled with calls to action that it’s difficult for users to understand what to do. Non-profit web teams need to have the executive backing to be able to look at the big picture and say “no” to some initiatives that clamor for top billing on your website, otherwise this kind of clutter sets in, and is impossible to remove without a redesign.

I don’t doubt that many on the Obama web team would like to remove some of these items from the homepage. But the moment they remove the store, for example, they can set a very short timer until the finance director bursts into the campaign manager’s office to show stats on how merch sales have plummeted since the store disappeared. Meanwhile, the six other departments may see incremental gain from removing competition for eyeballs, but not enough so that they’ll really individually fight to keep the store from reappearing, at which point everything starts over just like it was, except that now the store has been established as unmoveable – the same would happen for any item that was cut or moved below the fold. And when the next redesign cycle comes along, all the department heads will say “The homepage is too crowded – make it less busy,” leaving unsaid, “But don’t you dare touch my stuff!”