Lisa Pecunia posted an entry about User Ratings and their credibility to her blog (http://making.grouvia.com/2009/10/22/ratings-are-overrated/#comment-25) and to the User Experience group of Linked In. I think she has a point that is worth some reflection. As an overall system, I think that the ratings disparity will correct itself. Here’s why.
In user research sessions, I have met some people who tend to trust a source until it is proven untrustworthy. I have met others tend to distrust a source until it is proven trustworthy, particularly when the source has a vested interest in a positive rating (ok, I’m in the latter group). Both groups will lower their trust for a given source when they have an experience that belies that trust. For example, you trusted the rating that your VA had received in the rating system, and were disappointed with the results you experienced. There could be many reasons for this disparity. But in the end, you put less trust in that system because it had betrayed your initial trust level. You may try to correct that system by assigning a slightly lower rating, but in the future you will definitely have less trust in that system’s ratings.
On the other side of the ratings system, the people supplying the ratings practice a certain level of authenticity (the quality or condition of being trustworthy or genuine). Some companies supply the ratings as they are submitted. Amazon doesn’t need to care if people don’t like the book. They can find another one. An online shoe retailer, on the other hand, has to care if somebody says a certain shoe brand is crap. That may affect a lot of purchases, particularly if the customer repeats this assertion in a number of places online. The shoe retailer has to decide to what degree they should “adjust” the ratings by omitting negative outliers. Rating by rating, they are building the perceptions of their authenticity.
The authenticity of a given source of ratings is not known until an experience reveals it. You had expected a degree of authenticity from the VA ratings, because a lack of authenticity defeats the purpose of the system. The people supplying the ratings were (apparently) not authentic. Their lack of authenticity was discovered through experience and you adjusted your trust level. Others probably have had the same experience. Together, a large group is lowering its trust, and the perception of authenticity is decreasing overall.
Companies used to be able to control the conversation that impacted the perception of authenticity, thus engendering trust by spending large amounts of advertising dollars. But not anymore. That conversation is out of their hands. In aggregate people will continually adjust their trust level, and communicate those adjustments to others, which will cause harm to the reputation of authenticity of a given source. In user research sessions I have found that lack of trust is one of the most difficult barriers to overcome in a transaction-based web site, so a decrease in aggregate trust will be felt in the bottom line.
This bottom line impact will take a while to surface. It will eventually be recognized as a problem by the companies that lack authenticity in their ratings. But by then it will be as difficult to recover trust in the ratings system as it is in every other context.