Post hoc, ergo propter hoc

Economists have a handy phrase to describe a fairly common fallacy: "Post hoc ergo propter hoc" meaning "After, therefore because".

Wikipedia has an example of how this might look in the wild:

A tenant moves into an apartment and the building's furnace develops a fault. The manager blames the tenant's arrival for the malfunction. One event merely followed the other, in the absence of causality. (link).

In tech this might manifest as part of a feature launch. Say we launch a feature on a Monday and on Tuesday our retention goes way up. We're tempted to think that our feature launch caused the increase in retention.

This might be a fine assumption if our metrics are generally constant but more often than not this is just another case of correlation not meaning causation. Just because these events happened together (albeit with a lag) doesn't mean that they were causally linked.

With how volatile the world has been this year, this is even more relevant. Our metrics are moving up and down for reasons totally unrelated to our individual actions.

Usually, I'd deal with this type of risk by running a controlled experiment. We randomly split some users into two equal groups and give them different product experiences. If the users behave differently, then we know that the difference is caused by the difference in the product experience and not some arbitrary outside force.