Customer Testimonials Hold More Judgement in the Cutting Than in the Collecting

Twenty people were asked, eight replied, and three of those were published. A common shape. What the reader sees is those three.

The collecting has a procedure. Ask about the state before and after. Have them include figures. Get the publication cleared. All of it is organised, and all of it can be executed as written.

What has no procedure is the step from eight down to three. Which to publish, where to cut, in what order to arrange them. The criteria for that work are, in most cases, left to the writer’s discretion.

Where discretion is wide, meaning moves. Not one false word, no exaggeration, permission obtained — and what the reader receives changes anyway. Correctness in the collecting does not stop that movement. The reason is simple: the discipline of collecting only reaches the acquisition step.

Questions that can be settled here, and questions that cannot, are worth separating. Three can be settled. First, how many judgements the writer makes before anything is published. Second, why the overall impression moves even when every statement is factual. Third, how far the form of the question determines the answer that comes back.

Two questions get no answer here. How many testimonials is the right number, and how long any one of them should be. No figures for those. They depend on the nature of what is being sold and on how much the reader already holds about the field. Borrow a rule of thumb from outside and you borrow the selection criterion along with it.

An oversight here surfaces late. The reasons the five were dropped are recorded nowhere, so the object of any later inspection does not exist. What remains is the three that were published, and no amount of rereading those three reveals the habit in how the others were dropped. Publish repeatedly and the count grows while the bias in selection accumulates in the same direction, uninspected. And the bias becomes visible only when someone whose conditions did not fit signs up and gets no result — by which point the cause cannot be traced from the published side.

Pick three different testimonials out of the same eight and the published content changes. Both sets are in the customers’ own words, and both are covered by permission. What changed was the selection alone. And with no criterion for selecting, what survives is whichever testimonials draw the strongest response. Response is measurable, so the drift runs towards the measurable side.

Before any of that, one boundary: the design of what to ask and how is carried by how to run a customer interview. What is handled here is what happens after the answers are in.

What Practice Shares About Customer Testimonials Is a Procedure for Collecting

The handling of customer testimonials has settled, on the practice side, into a procedure for acquisition.

A customer testimonial means the words of someone who actually used the product or service, published as they were given. The user’s words are said to be received more readily than the seller’s own explanation.

A collecting procedure is given. Prepare a questionnaire. Ask what the problem was before and what changed after. Have specific figures included. Speak to them directly if possible. Obtain permission to publish.

Sample questions come with it. “What issues did you have before?” “What decided it for you?” “Tell us what changed afterwards.” “Who would you recommend this to?” Advice follows about arranging them from easiest to answer.

There is advice on presentation too. Real names and photographs are trusted more. Add affiliation and role. Summarise anything too long. Place it where it will be read.

Cautions are listed as well. Do not have anyone write something untrue. Do not exaggerate. Do not publish without permission. All of these are correct cautions.

Every item is sound as a description of practice. Users’ words do work; asking about before and after does function as a way to draw out change. Obtaining permission is a baseline.

Under all of this sits one premise. The correctness of a testimonial is organised as something secured during collection. No falsehoods, permission obtained, no exaggeration — then it may be published. It is assembled as a question of acquisition.

This is not to say the discipline of acquisition is unnecessary. Publishing a manufactured testimonial does not work at all: words someone was told to write are not their words. That discipline stays assumed throughout what follows.

The trouble is that once it is fully organised as acquisition, the work that comes after collection drops out of view. Collect ten and publish three, and the judgement that dropped seven is recorded nowhere.

Of course you pick the good ones — that objection exists. Of course. The problem is not the selecting; it is that the criterion for selecting is invisible to the reader. Being invisible, it gets received as the shape of the whole.

And the better organised the procedure, the harder the selecting step is to see. Having followed the procedure operates as proof of the whole thing being sound. What the procedure secures is acquisition; beyond that, it secures nothing. The boundary of what it secures is not written inside the procedure, so the writer who follows it most faithfully is the one least likely to notice there is an unsecured range.

Four Judgements Enter Before a Testimonial Is Published

Publishing a single testimonial takes at least four judgements on the writer’s part.

The first is who gets asked. Unless everyone is asked, a selection has already entered. And even when the request goes to everyone, satisfied customers find it easier to answer, so what comes back is skewed.

The second is which ones to publish. You choose from what arrived. The reasons for the ones left out are, in most cases, not recorded.

The third is where to cut. Pull a section out of a long answer and the surrounding text drops. Take only the second half of “I was sceptical at first, but three months in something changed” and you have a different statement.

The fourth is how to arrange them. The testimonial placed first governs how the rest get read.

Only the first of those four belongs to collecting. The remaining three happen afterwards. And the procedure that exists covers, in most cases, only the first.

Twenty asked, eight returned, three published. The reader sees three, and how those three represent the twenty cannot be known unless it is written.

So would publishing all of them solve it? The selection problem shrinks; a different problem appears. Long lists do not get read, so the reader ends up reading the top few anyway. The judgement about ordering remains. And the gap between the eight who answered and the twelve who did not is not closed by publishing everything.

Not every selection is a problem. Shortening for readability, correcting typos — that work does not move meaning. What moves it is dropping a part that bears on the judgement being described. The test is to put the dropped part back and read the two versions against each other. No change in meaning means editing; a change means selection.

There is a record-side answer to the number of judgements: when selecting, leave one line for each item dropped and why. Kept that way, the bias in selection can be inspected later. Without it, the object of inspection does not exist.

Of the four, the last three repeat with every publication. A selection habit adopted once gets used again next time. Keep selecting on the same criterion and the bias compounds rather than adds. The more rounds of publication, the more the testimonials on display resemble one another.

And the object of inspection is not the published page. However many times the three on display are reread, the five that were dropped do not appear in them. Try to raise precision by looking only at the visible side and the way things get dropped never gets examined at all.

Arranging Only Factual Statements Still Moves the Overall Impression

Even when every individual statement is factual, the arrangement as a whole can produce an impression that is not.

The mechanism is on the sampling side. When the visible set has been drawn out of the whole under a particular condition, reading that set correctly still does not tell you about the whole. What happens when you estimate from survivors alone is handled in how to read a drop-off rate.

With testimonials, the invisible side is threefold. People who did not buy. People who bought and did not use it. People who used it and did not answer. What gets published is the words of those who passed all three.

That the choice of whether to answer shapes the distribution itself has been observed on the public-review side. Hu and colleagues, in 2009, showed that ratings of online product reviews form a J-shape — thin in the middle, heavy at both ends — and attributed it to the decision about whether to post (Hu, Zhang & Pavlou, 2009, Communications of the ACM, 52(10), 144–147). Strongly satisfied and strongly dissatisfied people post; the people in the middle do not.

The object of that analysis was reviews posted voluntarily by an open population, not testimonials solicited by a business. What it showed is a direction: the decision about whether to answer determines the shape of the set that accumulates.

When they are solicited, only one side is left standing. A dissatisfied customer drops out by not replying to the request. Strong satisfaction is what remains in hand, and arranged on a page, the reader takes that set for the whole. The writer has not written a single false word. What is moving is the sample.

So should dissatisfied testimonials be published too? Nothing that demanding is required; the range can simply be stated. How many were asked, how many replied, how many were published. Those three figures alone let the reader discount for themselves. A reader who can discount is a reader holding material for judgement.

Sometimes the figures cannot be given. While the counts are small, publishing them risks identifying people. In that case what gets written is the range alone: “gathered from a few of the people we work with.” Even the range tells the reader this is not the whole.

The cost is explicit. State the range and the published effect falls, because the reader discounts. What is left after the discount is what actually arrived. Where the writing prevented the discount, what arrived will be discounted later instead. And a later discount does not stop at that one occasion. It reaches back over everything read before it.

Evaluative Language Is Assumed by the Writer

The evaluative language inside a testimonial belongs to the person who said it. The decision to publish it, though, belongs to the writer.

Publishing “they handled everything so carefully” means the writer agrees with that evaluation and has arranged it accordingly. Nobody publishes an evaluation they disagree with. The reader therefore receives it as part of the writer’s own claim.

The discipline of separating observation from evaluation is handled in how to write from first-hand information. Evaluative language can only be written once the outcome is known, and interpretation enters at the moment of writing. With testimonials, that discipline lands on the quoting side.

Concretely, two ways of writing sit side by side. “They handled everything so carefully,” and “the day after I sent the question, three options came back with the conditions for each.” The second is observation, and the reader can evaluate it themselves.

A testimonial made only of evaluative language cannot be assessed by the reader. The standard behind “careful” belongs to that person. With the standard unavailable, whether it applies is unavailable too.

Adding numbers does not change that state. Suppose three testimonials read “the response was fast, which helped.” What the reader wants to know is how fast. If all three are evaluation only, three of them carry no more information than one.

There is a worry that publishing observation alone keeps the feeling from coming through. Feeling can be published. “Not being able to settle while I waited, and then the relief when it came back the next day” is a description of a state. Evaluation and state are different things, and the state is something only the person can write.

Nor does every evaluative word have to go. A single testimonial containing both observation and evaluation reads more naturally. What becomes a problem is arranging testimonials in which observation is at zero. Even an evaluation-only testimonial conveys temperature when there is one. The problem is a set in which not one of them carries an observation.

And whether observation can be drawn out at all is decided by how the question is asked. Ask “how was it?” and evaluation comes back; ask “in what situation did you use it?” and observation comes back. The asking side is handled in how to run a customer interview.

Testimonials containing observation run long. Being long, fewer of them fit. Prioritise the count and evaluation is all that survives; prioritise the content and the count falls. Which to take depends on what that page is trying to hand over. Count looks like a quantity of trust, but however many unassessable items are lined up, the reader’s judgement does not advance a step.

A Statement of Change Requires an Object of Comparison

“It changed” cannot be assessed on its own.

Change is written as the gap between two points in time. What it was, what it became. With the before unwritten, the gap is unavailable.

And a bias operates on how the before gets written. In a context meant to convey change, the pull is towards drawing the before low. That structure is the same one handled in writing from personal experience.

With testimonials, the bias applies twice. The person writes in the contrastive form, and the writer selects the part where the change is largest. The published gap is the output of two rounds of selection.

So what is needed is specificity about the prior state. Not “nothing was working at all,” but “two enquiries a month, and nothing moved past that.” With specifics, the reader can compare their own position.

Suppose a testimonial says “revenue tripled.” From a hundred thousand, that is three hundred thousand. With no period stated, six months and three years are indistinguishable. That a figure without conditions cannot be assessed is the same point handled in the principles of sales copy.

May the writer supplement the figure the customer gave? Supplementing is possible. Ask them, and have the period or the prior state added. What the writer cannot do is fill it in by inference. Where confirmation is unavailable, the figure does not get used.

Some domains produce no figures for change — the quality of decisions, or which work gets taken on. There, the content of the change gets placed as a description of behaviour. “I used to take enquiries in the order they arrived; now I have criteria for declining.” There is no figure in that sentence, yet before and after are clearly separated. Separated, the object of comparison holds.

Conversely, publishing only the domains that yield figures makes just one part of the work visible. Behavioural descriptions can even be easier to assess than figures, because the reader can check directly whether they are doing the same thing.

The burden shifts to the person answering. Writing the prior state specifically takes an act of recall. As the burden rises, the response rate falls. What is lost there can be covered by taking the record earlier — noting the state at the point of starting. With that on file, the recall is not needed at all. A record taken when its use was still unknown is what pays off here.

By the Time a Testimonial Is Requested, the Answer Has Already Been Given a Shape

The answer that comes back is built along the shape of the question.

Ask “tell us what changed after you started” and change comes back. Someone for whom nothing changed finds it hard to answer. People who find it hard to answer do not answer, so the replies lean towards the side where change occurred.

And the choice of words moves the answer too. There is a well-known experiment on the memory side. Loftus and Palmer, in 1974, showed people footage of a car accident and then asked how fast the cars were going when they “hit” versus when they “smashed into” each other, and the estimates that came back differed with the word (Loftus & Palmer, 1974, Journal of Verbal Learning and Verbal Behavior, 13(5), 585–589).

The reach of that experiment is narrow. What it handled was memory for an event seen on film, not answers to a customer questionnaire. What it showed is a direction: words contained in the question affect the answer that comes back.

The implication for practice is plain. “Where did you feel the effect?” puts the existence of an effect in place as a premise. A question with a premise in it returns an answer along that premise.

Put two questions side by side. Asking “what was useful” and asking “how was it to use” produce answers of different width. The second also returns the features that went unused and the parts that did not suit.

What is wrong with asking about the good parts? What is wrong arrives at the publishing step. Arrange testimonials collected by asking about good parts as an overall assessment, and the shape of the question is invisible to the reader. Being invisible, it gets received as unprompted comment.

Making the question neutral does not resolve it either. Even with a neutral question, the person answering is aware of who asked. Since it cannot be removed entirely, what remains is to state which question they were answering. State it and the reader can estimate the skew themselves. An estimable skew does less damage than a hidden one.

The stating is simple. One line — “we asked customers for their impressions after use” — gives the reader the context.

Ask widely and the number of answers unusable for publication rises. Material for fixing the product is sometimes inside those unusable answers. Ask solely for publication and that material never accumulates. Separating the two purposes has to be done in advance. Without separating them, what accumulates is answers usable on the page, and the material for fixing things is missing without anyone noticing that it is missing.

Consent to Publish Is Consent to a Range

“May we publish this?” contains no range.

At least four ranges are in play. Where it will appear. In what form (real name, anonymous, with affiliation). For how long. Whether parts may be quoted.

Of the four, the first two are usually confirmed. Duration and partial quotation are, in most cases, not.

Duration becomes a problem once time passes. Someone who consented three years ago will not necessarily make the same call now. Their situation may have changed as well.

Partial quotation becomes a problem when meaning moves. Consent to the full text does not include consent to the shape a fragment takes. Take only the second half of “I was sceptical at first, but sticking with it changed things.” The quoted part is their own words. What they consented to, even so, was publication of the whole.

There is also the reaction that confirming at that level of detail means nothing will come in. Adding items to confirm does add work. But the added work is one-off, and smaller than the work of being asked to remove something later. Settle duration and quotation rights at the start and everything after is operation.

Not everything can be settled in advance. Publication venues get added; formats change. What can be settled in that case is the undertaking to notify before changing anything. With that in place, the later confirmation is something you can raise from your side.

And a request to be removed goes through without a reason being required. Make it a process that asks why, and the request itself stops happening.

The reason it stops happening sits in the relationship. People who offer a testimonial are, in most cases, well disposed towards the writer. Being well disposed, they find it hard to say take it down. Leave that difficulty in place and what remains of consent is the form of it.

So a confirmation initiated from your side is required. Once a year, show each person the testimonial still on display and ask whether it should stay. Asked directly, removal becomes an answer rather than a request. The direction of the burden reverses.

Everything settled in detail reduces what can be published. Set a duration and the expired items come down. What is lost there gets replaced by asking again. That is work, but it holds the state in which the testimonials on display are current ones. A page of three-year-old testimonials and a page of this year’s convey a different present to the reader.

A Testimonial That Works Too Well Attracts People It Does Not Suit

The larger the result in a testimonial, the stronger the response it draws.

Two kinds of people are mixed into that response. People in the same conditions, for whom the same route is passable. People in different conditions who were drawn by the size of the result.

More of the second causes trouble later. Conditions differ, so the result does not repeat. The failure then gets read as a fault in the method, rather than as the mismatch it was.

The range a single case can show is handled in writing from personal experience. It can show that a thing can happen; it cannot show that it generally happens. The same constraint applies to testimonials.

So selecting which testimonials to publish needs an axis besides the size of the result. Testimonials whose conditions can be stated. Testimonials matching where most readers are. Testimonials whose route is reproducible.

Suppose there is one where the numbers moved sharply in six months, and one where, over a year, the kind of work being accepted changed. The first draws more response. The second may match the conditions of more readers.

Is it not natural to publish whichever draws response? The response for that round does rise. Included in the rise are people whose conditions do not fit. They spend time, get no result, and leave. Counted as a cost to both sides, it can exceed the increment in response for that round.

This is not to say a large-result testimonial should never be published. Attach the conditions and the reader can judge. Where a condition cannot be attached — where even the customer cannot identify it — that is what gets written. Write “why it worked, even they cannot pin down,” and the case reads as one instance of something being possible rather than as a template for reproduction.

And adding one testimonial from someone it did not suit does work. “It was not right for people in this situation” helps the reader decide. Whether it can be published is confirmed with that person.

That single item also acts on the page as a whole, because the presence of a case that did not fit changes how the others are read. It hands over the material for judging that these are not only the selected ones. One item lowers the discount applied to everything else.

Testimonials with conditions attached select their readers. Selecting, they lower the response for that round. Look at the lowered figure and strip the conditions, and you are back where you started. Which figure to watch has to be decided in advance. Undecided, it gets decided at the moment the lower figure appears, and that decision tends to tilt towards the figure.

Before Publishing, Inspect How the Testimonials Were Selected

Whether the testimonials on display have become material for the reader is legible from the selecting side rather than the published page. Three confirmations, all executable before publication.

The first confirmation is to line up three numbers. How many were asked, how many came back, how many were published. Lined up, the width of the selection becomes visible. Where it is wide, the range gets stated alongside. Even where circumstances prevent publishing the figures, hold them on your own side. Select without holding them and the criterion you selected on is unavailable to you as well.

The second confirmation is to strip every evaluative word out of each item and see what is left. Where observation remains, the reader can assess it themselves. Where nothing remains, that item is the writer’s own claim restated in someone else’s mouth.

The third confirmation is to put the surrounding text back and read the two versions against each other. No change in meaning means editing. A change means the cut has moved the content. Where it has, confirm that the person consented to publication in that form.

The effect of publishing sits outside all three. Effect is an outcome, which is a different axis from whether something is material.

One verdict stays outside all three. None of them tells you whether the testimonial fits the reader’s conditions. Collected accurately, quoted accurately, range stated — and still not applicable to this reader. That verdict cannot be reached from the published side.

How many to publish, how long each should be, whether to use real names. None of that gets decided here. Select on borrowed figures and you have selected on the criteria of whoever produced them. Having selected on borrowed criteria, why those three cannot be explained even by the writer. An unexplainable selection is the form least likely to be inspected.

Of the three numbers twenty, eight and three, the procedure covers only the first. The remaining two — the narrowing from eight to three, and how those three were cut — have no criteria, and that is where meaning moves. The four judgements, the skewed sample, the shape of the question. What all three share is a position that cannot be inspected from the published page.

One step gets reordered. Before selecting, settle which readers, in which position, the page is for. Settled, the selection criterion is settled with it. Unsettled, the criterion becomes the size of the result and nothing else. And after selecting, leave one line for each item dropped and why. With those lines, the bias in the next round of selecting can be inspected.

The asking itself is handled in how to run a customer interview, and the limits of using other people’s judgements as a cue are in where social proof works. Where the material is your own experience, writing from personal experience is the neighbouring piece. Judgement gathers at the cutting stage because the material sits outside you. Where the work of handling someone else’s experience as material is placed in the whole is handled in verbalisation and transmission.