What the Drop-Off Rate Observes Is Only the Side That Stayed, and Those Who Left Need a Way of Being Counted

Open the drop-off rate screen and what is lined up there is the record of the people who stayed. For those who left, there is the single fact that they left, and nothing else.

The reason for leaving, what they came expecting, where it matched the expectation and where it stopped matching — none of it is in the records. And none of it can be fetched afterwards. Without contact details, the route to that person disappears the moment they leave.

For those who stayed, the reverse happens. Information thickens as time passes, and asking brings answers back. The material for improvement gathers only from this side.

Keep improving with material arriving from one side only and the destination is settled. The place becomes a better one for the people who stayed. The reason those who left did so is never touched.

This article can answer three questions.

The first is what the drop-off rate counts. It counts behaviour, not state. This section carries through to what that difference does to judgement.

The second is what goes askew when you improve while seeing only the side that stayed. This is not a matter of impressions but can be handled as a matter of how the sample is taken. Being handleable means the direction of the skew can also be estimated.

The third is what conditions, concretely, a place must meet to be one where those who left can be counted. There are three conditions, and each of them trades against short-term figures. Whether it is worth paying depends on what the record of departures teaches.

Conversely, one question gets no value returned. The passing line for a drop-off rate. To what percentage counts as healthy, this article offers no number. The same value emerges from two groups with entirely different compositions. Judging on the value alone does not hold together as a form.

Further, this indicator carries a condition of scale. On a page with few visits, the proportion swings tens of points on the movement of a few people. Read the swing as the effect of an improvement and a treatment gets applied against a change where nothing actually happened.

Methods for reducing drop-offs are not lined up here either. The reducing operation works regardless of the breakdown. The same restraint acts on those leaving having achieved their purpose and on those leaving in disappointment. As it acts, the figure improves. From an improved figure, which of the two decreased cannot be read.

The Drop-Off Rate and the Bounce Rate Differ Only in the Denominator

The two indicators count the same thing with different denominators. The difference is the denominator alone, and the range of what can be read from either stops in the same place.

The drop-off rate is the proportion in which a given page became the last page of a visit. The calculation divides departures from that page by that page’s impressions.

The bounce rate is the proportion of visits entering the site that left on the first page alone without viewing another. The calculation divides visits ending on one page by visits beginning from that page.

For the same page, treatment differs between being used as an entrance and being passed through midway. Two figures exist in order to separate that difference. Separated, both figures still answer only what proportion ended there, and beyond that they stop in the same place.

The improvement directions offered run roughly to three.

First, correcting a mismatch between content and search intent. Where content differs from what was expected, people leave on the spot.

Second, placing guidance to what comes next. Where no destination is shown after reading, it ends there.

Third, removing difficulty of reading. Display speed, text size, quantity of advertising. Factors causing departure before reading sit outside the content too.

All three are apt as practical advice. Mismatched intent means departure; no destination means an ending. Slow display means not being read.

The three converge on one treatment. Each treats drop-off as an event to be reduced. The framing is that being left is a loss and retaining is an improvement.

Departure through mismatched intent is a state that genuinely should be improved. The three pieces of advice work correctly against it. What they do not work on are the drop-offs that fall outside this framing.

Someone who left having achieved their purpose and someone who left in disappointment are tallied as the same single unit. Tallied as the same unit, the treatment is the same too. A retaining treatment gets applied to both.

Once retention becomes the aim, design shifts toward reducing exits. More guidance to what comes next, related content lined up, another entrance placed at the position of finishing. Each is a reasonable measure; stacked up, a place gets built where finishing a read is difficult.

The three pieces of advice do not specify an order. The order of starting gets settled, in practice, from the side with least effort. Display speed changes with a setting; correcting a mismatch between content and intent has no visible end. The order of least effort does not coincide with the order of greatest effect.

Drop-Off Means the Page Was Closed, Not the Relationship Ended

A drop-off is the record that a visit ended on that page. It contains nothing beyond that.

The distinction has substance. Someone who read to the end in satisfaction is tallied as a drop-off too. They got the information they needed, achieved their purpose, and closed. That person is the best kind of reader, and at the level of the indicator they are someone who left.

Conversely, someone who did not drop off is not necessarily in a good state. Someone who moved to the next page without finding what they were looking for has not dropped off. Circulation appears to be occurring while, in fact, they are lost.

So this indicator is a record of behaviour, not of state. Opposite states sit beneath the same behaviour.

From here comes the limit of judgement. Judging good or bad from the drop-off rate alone is impossible in principle. Judgement requires information on what the visit was for, and that information is not contained in the record.

Could combining dwell time and scroll depth separate them? Some of it separates. But an ambiguity remains even combined. Someone who stayed long and left may have read closely, or may have kept searching without finding. However many behavioural records you stack, the purpose is not recovered.

Dwell time carries a further awkward property. In many measurement schemes, dwell time is calculated as the difference between the timestamps of the first and last records. In a visit with only one record, that difference is zero. Someone who read a single page and closed it is therefore sometimes tallied as zero dwell time, however long they were reading.

Because of that property, judgements combining bounce and dwell time miss twice over. The relation whereby people who bounced show shorter dwell times comes out of how the calculation works, not out of reader behaviour.

That does not mean it need not be looked at. Which pages people tend to leave from is useful information in itself. What this indicator shows is a place, not a reason. Knowing the place settles where to go and investigate what is happening. Fix without investigating and what got fixed is the reason you guessed at.

When a fix made on a guess shows up in the figures, a confirmation remains that the guess was right. Figures move for several reasons, so that confirmation is unreliable. Even so, it remains as a record in the form of fixed it and it improved.

Records of that form stack up. Stacked, they get used as grounds the next time the same symptom appears. That fixed it last time is not an account of a cause but a record of a past operation.

Fixing on a guess is unavoidable in itself. Investigate everything and nothing ever gets touched. Leave a record that the fix was a guess and room remains to doubt it when the same symptom next appears.

What Can Be Observed Is Only the Side That Stayed

Information about those who left becomes unobtainable at the moment of leaving. Without contact details, there is no route for asking why that person left.

Information about those who stayed, by contrast, thickens as time passes. This asymmetry skews the material for improvement. What gathers to hand is the opinion of those who stayed. And those who stayed are, by definition, the people who fit the current design.

During the Second World War the statistician Abraham Wald wrote a method for handling this structure (Wald, 1943, “A Method of Estimating Plane Vulnerability Based on Damage of Survivors”, Statistical Research Group, Columbia University). A method for estimating the vulnerability of aircraft as a whole from damage data on those that returned.

The title of the paper states the structure itself. What is to hand is data on the aircraft that survived, and the object of estimation is the whole, including those that did not.

This work circulates as a one-line anecdote about armouring the places with least damage, but what is actually written is a statistical method for estimating the probability of being hit across the whole from surviving data. The anecdotal form is a summary of the conclusion, not the method.

Moved to business it runs like this. A survey of remaining customers teaches the reasons for remaining. It does not teach the reasons for leaving. When satisfaction surveys return high figures while drop-offs do not stop, this structure is usually at work.

Surveys are sent to those who left as well, so surely they arrive. Sometimes they can be sent. But they can only be sent to parties whose contact details you hold. They do not reach those who left before handing anything over.

And even when they arrive, only a portion of those who left respond. Someone who takes the trouble to answer still has some interest remaining. Someone who has entirely lost interest does not respond either. Even among those who left, answers come back only from the side closer to you.

The skew itself cannot be removed. In principle, the whole of those who left is unreachable. What changes is whether you read on the premise of the skew, and the conclusions that come out of that.

And saying account for the skew is still vague. What, concretely, counts as having accounted for it? The counting has to widen from those who stayed to those who left.

Inferring the Whole from a Selected Sample Requires the Selection Process

Accounting for the skew means making the condition that separates the staying side from the leaving side into an object of observation.

In a 1979 paper the economist James Heckman formulated the state of a non-randomly selected sample not as a statistical error but as an error in how the model is specified (Heckman, 1979, Econometrica, 47(1), 153–161). Where only the selected side is to hand, the relation estimated from that sample contains what the selection introduced.

What the paper shows is a method for handling this state. Specify the process of selection itself as an equation and build it into the estimation. Built in, an estimate about the whole can be made from the selected sample. Not built in, it cannot.

Put differently, whether correction is possible turns on whether the process of selection can be observed. If who remained under what conditions and who left under what conditions is known, there is a path from the records of the staying side to an inference about the whole. If it is not known, there is no path.

This formulation addresses estimation in econometrics; it is not a claim that business records fall under the same treatment. What carries over is only the direction: knowing that your sample is skewed is not enough, and nothing can be corrected without observing the condition that produced the skew.

That direction bears directly on practical design. The stance of discounting because these are the opinions of those who stayed is a state of knowing about the skew. Knowing alone does not settle how much to discount. A discount whose size is not settled ends up as an adjustment by taste.

It gets settled when the act of leaving remains as a record on your side. How many, when, and at which point they left. With that recorded, the condition separating the staying side from the leaving side can be read. Read it, and the range within which the opinions of those who stayed may be generalised gets settled too.

Every reason for leaving cannot be known. What can be done is to make the act of leaving remain as a record. Made to remain, the skew becomes something that can be handled.

The order matters. What is needed first is the design of the records, not the design of the survey. However you refine what to ask those who left, without a record that they left you cannot identify whom to ask.

Aiming to Reduce Drop-Offs Produces a Design That Blocks the Exits

Place lowering the drop-off rate as a target and design moves in one direction. And that direction works independently of the other party’s purpose.

Place another entrance at the position of finishing. Line up related content. Insert another element midway to arrest the gaze. All of these lower the drop-off rate.

They are displayed to someone who found what they were looking for as well. If that person proceeds to the next thing, they have left their original purpose and are circulating.

Circulation is not bad in itself. The problem is that retaining someone who achieved their purpose appears, at the level of the indicator, as an improvement.

Placing the measured stage as the target makes that stage the object of optimisation — the structure handled in the cost of acquiring a customer appears here in another form. Drop-offs are being measured, so drop-offs decrease. Whether the decrease consists of departures in disappointment or departures in satisfaction is not contained in the indicator.

Carried to its limit, this design changes the nature of the place itself. A place that is hard to leave cannot hold records of those who left. Without records, the reasons for leaving are unknown too.

Further, difficulty of leaving extends dwell but guarantees nothing about the quality of dwelling. Someone dwelling in a state where no exit is visible is not building a reason to come back.

Is having people circulate really so bad? It is not. The judgement is not whether circulation occurs but whose purpose it occurs for.

If the other party proceeds holding the next question, that is their purpose. If they proceed along the options you displayed, that is yours. In the records both appear as the same behaviour.

There is one way to tell them apart. Look at what happened on the next page. Where they proceeded on their own question, the next page gets read about as much. Where they proceeded on your display, the next page is closed at once. A state where movement occurred but reading did not.

This design also changes judgements on the writer’s side, because writing in a form that hands over to the next thing produces better figures than writing in a form that can be finished. Once handing over becomes the aim, what remains for someone who read only that piece falls outside consideration.

Placing guidance to what comes next is itself a necessary design. Where a continuation exists and is not shown, it cannot be found. The dividing line is whether the guidance states what lies ahead or urges going there. The former leaves the option of not going. The latter treats a finished read as incomplete. The same one line, in the same position and of the same length, splits into these two.

How to Count the State of Leaving Having Achieved a Purpose

There is a reading that does not treat being left as a bad event. Where the purpose of a visit was achieved and it ended, the drop-off is a record of completion.

Came from a search, learned what they wanted to know, closed. That person is likely to return when the same kind of question arises.

Whether they return cannot be observed within that visit. It can be observed when the next visit occurs. So the goodness or badness of a drop-off cannot be judged at the time of the drop-off; it is judged afterwards. The time required for the judgement depends on how often the question recurs in that field.

From here comes a condition on the period. Evaluate the drop-off rate over a short period and good and bad drop-offs cannot be distinguished in principle. They become distinguishable once records of return visits have gathered.

And recording a return visit requires knowing it is the same person. That is knowable only where contact details are held or where records continue. Holding neither, a good drop-off is never observed as a good drop-off.

This connects to the problem, handled in the premises of list building, of whether a route of arrival is held. In a place holding no route, everyone who left becomes the same person who left.

Nor does holding a route make everything visible. Even with a route, the reasons for leaving are not known automatically. Only one thing is certain: in a place holding no route, the single unit of someone who left and the single unit of someone who achieved their purpose can never be distinguished. Improve without distinguishing them and the same treatment gets applied to both.

And a treatment that works on both is superfluous for one of them. For someone who achieved their purpose, guidance to what comes next either does not register or gets in the way. Getting in the way leaves no record, so the cost of the treatment is never tallied.

Does asking for contact details not itself increase drop-offs? It does. A judgement gets inserted about whether to hand them over, and people leave there. This is a cost to be acknowledged as fact. The cost of people leaving where a judgement is inserted arises each time a stage is added. What necessarily happens on the side that places stages is handled in the funnel as a whole.

On that footing, the content of the cost is that those who left judged it was not worth handing over. That judgement is accurate. Retain them and all that grows is the number of people dwelling with the judgement suspended.

Place a point that asks for a judgement and it shows up in the figures as the place with the most drop-offs. That makes it tempting to treat that point as an object of improvement.

Let improvement head toward a form that asks for no judgement and the distinction between those who left before handing over and those who decided not to hand over disappears again. The point of judgement is the place that produces drop-offs and, at the same time, one of the few places where the reason for leaving becomes clear.

While the Counts Are Small, the Proportion Cannot Be Read

The drop-off rate is a proportion, so it swings widely when the denominator is small. And the swing is indistinguishable from the effect of an improvement.

On a page visited by ten people a day, seven leaving makes seventy per cent. Four leaving the next day makes forty. The figure moved thirty points, but what moved was three people.

Take that swing for the result of a measure and a hand gets laid on a month in which nothing changed. If the figure happens to come back after the treatment, that treatment counts as having worked.

Since proportions produce extreme values more readily the smaller the denominator, the worst page is usually the page with fewest visits. Sort by worst and the top is filled with small numbers.

Few counts does not mean not looking. What to look at is the actual number, not the proportion. Read seven people left rather than seventy per cent dropped off and the weight of the judgement matches the actual state.

Sort by actual number and only high-traffic pages come to the top, so priorities cannot be set — that reading is available. They can. Priority is settled neither by proportion nor by actual number but by what that drop-off is halting in the business. Ten people leaving just before an application and a hundred leaving on a reference page differ in meaning at the same ten units.

This is the same problem as the structure handled in where to begin doubting when the conversion rate is poor, where the whole is settled as a product of several proportions. Where people left cannot be judged without seeing where that stage sits in the whole.

The habit of sorting by proportion works hardest in the period when a business is young. Visits are few, so every page has a small denominator and every one produces extreme values. What must most be avoided then is making large design changes on the grounds of an extreme value. Change it and what had been working can no longer be traced afterwards.

The smallness of a denominator carries a property easily missed. At a scale where the proportion cannot be read, bad and nothing has happened yet wear the same face. The drop-off rate of a page visited by three people takes an extreme value whether the content is good or bad.

What is needed at this stage is not improving the proportion but holding the design until the scale at which the proportion becomes readable. During that holding, what may be changed is the set of recorded items. Add records after the scale arrives and the period before it cannot be recovered.

Nor does it mean no judgement is possible until the scale arrives. Individual cases can be read regardless of scale. If one person stopped just before an application, what that one person saw before stopping can be confirmed independently of any proportion.

Reasons for Leaving Cannot Be Asked After Leaving

The reason someone left is easiest to confirm when you ask before they leave.

Asking before they leave is done in practice. Placing a field asking for a reason within a cancellation procedure is exactly that.

Asking within a cancellation procedure has two limits, however.

First, the answers become short. Someone in the act of stopping wants to be done with it, so they pick one option and finish. The options are ones you made, so reasons you did not anticipate never get chosen.

Second, the answers are not necessarily accurate. People reconstruct the reasons for their own behaviour after the fact. Among those who answered the price is high are people for whom price was not the reason. The reason easiest to state gets chosen.

Asking is necessary in itself. On that footing, taking the answer as the cause leads you to fix the wrong place.

More reliable is the record of behaviour immediately before leaving. Which pages they viewed, where they stopped, what they did not do. Behaviour is not reconstructed.

Does interpretation of behavioural records not also introduce your own guesswork? It does. The difference is whether the guessing can be recognised as guessing. A reason someone stated gets treated as fact; a behavioural record is understood as interpretation. A guess you are aware of becomes an object of verification.

Having behavioural records does not settle everything either. Records teach only about the items recorded. About unrecorded items, nothing can be known retroactively.

So the moment you settle what to record, the range of what can be asked afterwards is settled. That decision is usually made without thought, left at the defaults.

Default items are the commonly used ones, so a state results in which every business can ask only the same things. Impressions, dwell time, destination. These can be taken regardless of the kind of business, so they answer no question specific to a business.

Answering a specific question requires recording specific items. Whether someone actually tried what you provide, for instance, is not in the default items. Putting it in requires you to define and place it.

Deciding on recorded items usually gets left to the end of the design. What to build is settled, how to present it is settled, and only then is measurement considered. As an order, this is the latest possible. By then, there is effectively no option other than taking the defaults.

And only the items you defined and placed become what can be asked afterwards. The design of records is a hard-to-reverse decision, made before any analysis.

The Conditions for a Place Where Those Who Left Can Be Counted

A place where those who left can be counted is one where the act of leaving remains as a record on your side. There are three conditions.

First, an act of leaving exists. In a place where doing nothing constitutes a drop-off, the point of leaving cannot be identified. Cancelling a subscription, withdrawing, terminating a contract. Only with an explicit act does leaving become an event.

Second, that act is easy. If it is hard, people who want to leave stay instead. Those who stay are tallied as continuing, so the fact of leaving disappears from the records altogether.

Third, you do not turn adversarial afterwards. Apply strong retention at the moment of departure and the person leaves without stating a reason. Left without a statement, the record holds only no reason given.

All three work in the direction of lowering short-term figures. Make leaving easy and drop-offs rise; refrain from retention and continuation rates fall. And the lowered figures appear in the monthly aggregate, while the quantity of information that should have risen is aggregated nowhere. The cost is measured and the return is not. This asymmetry is what makes the three conditions hard to hold.

The reason for holding them anyway is that only a record of being left teaches who the current design does not fit. From those who stayed, this information does not emerge.

How a thinly treated party would actually have fared cannot, in principle, remain in the records — the same structure seen from the side of the indicator appears in customer lifetime value. This section is its counterpart on the side of the place.

Make leaving easy and surely more people leave. They do. On that footing, the increase includes people who were going to leave anyway. If only the timing moved earlier, the difference in eventual retention stays small.

Where the difference appears is in learning the reason early because they left early. Being left after a year with no reason, and being left after three months in a form that gives a reason, provide different material for the next design.

Meeting the three conditions does not mean learning every reason. Parties who perform no explicit act of leaving and simply stop responding occur at some rate in any format.

For that layer, even the point of leaving cannot be identified. What can be done is to draw a line yourself once a period of no response has continued.

Drawing the line has meaning in itself. Without it, that party remains on the list as continuing forever. The remaining figure keeps growing beyond the actual state. A figure larger than the actual state works only in the direction of distorting judgement.

Whether the Records Can Answer About Those Who Left

The confirmation runs on something other than the proportion of drop-offs. It runs on the structure of the records.

The first indicator is whether you can name several people who left recently. If you cannot, being left is not recorded as an individual event. A state where only a proportion is to hand is a state where those who left exist only inside an aggregate.

The second indicator is how much of the recorded reasons for leaving falls outside the options you prepared. If everything fits into one of the options, not a single unanticipated reason has been captured. Free text appearing at some rate is the evidence that the records are working.

The third indicator is the number of steps required to leave. Count them and it becomes clear. If more steps are required than for applying, part of the figure is being produced by difficulty of leaving. If the same or fewer, the current continuation holds for reasons other than friction.

The proportion of drop-offs does not enter the three indicators. Since the question is whether those who left can be counted, what needs confirming is the side of the records.

The three carry a limit. All of them work only in formats where an act of leaving exists. In a place where people arrive from a search, read, and close, those who left are anonymous from the outset. What can be taken there is only placing a route that builds a relationship before they leave.

Naming people is possible only while the counts are small. That is so, and once counts grow, grasping everyone individually becomes impossible. But what the first indicator asks about is not everyone; it is the most recent few. A state where even a few cannot be named is a problem of the structure of the records, not of the counts.

And even after counts grow, the moment you stop reading individually, the information on reasons for leaving vanishes into the aggregate. An aggregate teaches what proportion left; what that person came expecting and failed to obtain cannot be expressed at the granularity of an aggregate.

To a question shaped as a passing line, this article returns no number. The same value emerges from a group of people who left having achieved their purpose and from a group who left in disappointment. Handing over a line amounts to saying it is acceptable to judge with the two crushed into one figure.

What remains in hand is the record of those who stayed. This asymmetry cannot be removed. It becomes handleable only when the act of leaving is made to remain as a record. There is no way to know all the reasons people left. There is, however, a way to make the leaving remain as an event.

The next time you settle the recorded items, what those items can answer is only the range you defined. If contact details remain, you can ask after someone leaves. But what comes back is only from the side closest to you. Material without a skew can be built only before they leave.