Human error and cognitive bias

Human error comes in three kinds: skill, rule and understanding

People are not naturally good statisticians, and an attacker does not need to break a cryptographic system when a predictable slip in judgement gets the same result for free. The lecture opens with a figure of 80 to 90% of security breaches caused by human error. It varies a great deal by source and methodology but sits in the right neighbourhood of every industry estimate checked against it. Two things drive it: predictable failures in the mechanics of doing a task, and predictable biases in how people weigh risk, reward and effort.

Human error in any equipment-operating task falls into three categories the lecture borrows from safety research:

• Manual skill fails. A skill performed often enough becomes automatic, and automatic skills slip under distraction or fatigue. Typosquatters exploit this by registering a domain one keystroke from the real one and waiting for a typing slip to send the victim there: github.com to glthub.com, google.com to gougle.com, amazon.com to amozon.com, netflix.com to netfllix.com, linkedin.com to linkedin.co.
• Following the wrong rule. A rule that is normally safe gets applied somewhere it should not be. “Trust HTTPS because it is secure” is the security version: a padlock icon proves the connection is encrypted, not that the site behind it is trustworthy, and attackers can obtain certificates too.
• Cognitive reasons. The person does not understand the problem, pretends to, or understands it and ignores it to get the job done. An employee who finds new update procedures cumbersome keeps using the old ones, or postpones a patch because installing it now interrupts their work.

These are failures at three levels (skill, rule, and understanding or intent), and an attacker designs around whichever one a target is most likely to make.

Recall

A worker slips back into their old, familiar routine despite having been trained on a new, safer procedure, because the new one feels cumbersome. Which of the three categories of human error is this?

Cognitive reasons. The worker understands the new procedure and chooses to ignore it, rather than making a skill slip or following a rule that was wrong for the situation.

People misjudge risk in predictable ways, and scams are built on them

Exam detail

Rational choice theory says people select whichever option maximises their own satisfaction, like a shopper comparing cereal brands on price, taste and nutrition. Expected utility theory narrows this to decisions made under risk and uncertainty, where the outcome is not guaranteed: an investor weighing several assets considers both the potential return and the probability of getting it, and picks whichever combination maximises expected utility. Both theories assume people compute this correctly. Kahneman and Tversky’s prospect theory showed they mostly do not.

Loss aversion. Losses are felt more intensely than an equivalent gain. A message reading “Your PayPal account has been frozen, and you need to click here to unlock it” exploits this: the threatened loss of account access pulls harder than any equivalent framing of a gain would.

Framing effect. The same offer, presented as avoiding a loss rather than securing a gain, produces a stronger reaction. “Don’t miss out, sale ends soon!” outperforms “Get it now on sale!” for an identical deal. Only the frame changed.

Anchoring effect. A judgement is built on the first number seen and adjusted only slightly from there. A car marked $30,000 looks expensive on its own; the same car marked “was $60,000, now $30,000” gets bought, because the buyer anchors on the size of the discount rather than the final price. Fake antivirus pop-ups use the same trick (“$1,000 per year, we are giving it away for $500”), and so do lottery scams, where the size of the promised jackpot makes a small “processing fee” look negligible.

Availability heuristic. Information that comes to mind easily gets treated as the best guide to what is likely, regardless of the real statistics. Someone who watched a shark-attack documentary the night before hesitates to swim the next morning: the vivid recollection outweighs the actual, very low, risk. “Spray and pray” SMS scams around tax time exploit the same effect, arriving when a recent tax return is fresh in the recipient’s memory so an ATO-themed message feels immediately plausible.

Impatience and pattern-seeking bend judgement further

Present bias and hyperbolic discounting. People weight an immediate cost far more heavily than a larger, merely delayed one. Postponing a software update because installing it now is inconvenient, even though leaving it unpatched carries a bigger long-term risk, is present bias in action. Hyperbolic discounting is the model decision scientists use to quantify it: the discount applied to future costs grows so steep that avoiding the present interruption dominates the choice, whatever the deferred cost turns out to be.

The privacy paradox is the same effect applied to privacy: people say they care about it, then click “Accept” on a cookie banner, skip the privacy policy, and leave default sharing settings alone, because the immediate cost of reading and deciding outweighs a data-exposure risk that is diffuse and delayed.

Compare

Clustering illusion

Seeing a pattern in data that is genuinely random. The “hot hands” belief that a basketball player who just scored is due to score again, and the gambler’s fallacy that a coin “owes” tails after several heads, are both patterns the mind supplies where none exists. A phishing email’s fake logo and familiar layout register as “looks right” the same way: the mind is pattern-matching, not verifying.

Confirmation bias

Filtering new evidence to fit a belief already held, and dismissing what contradicts it. Once a reader has decided a PayPal email is genuine, small discrepancies in the URL or the logo get overlooked, not because they pattern-match as correct, but because the reader has already stopped checking.

Zero-risk bias. People prefer eliminating one risk completely over reducing total risk by a larger amount, because certainty feels better than a bigger but merely probabilistic improvement. Asked to rank hazard-site cleanup plans, participants preferred a plan that cleaned one site perfectly while leaving others untouched over a plan that reduced risk more overall across every site. The same bias shows up in security engineering: insisting on an information-theoretically perfect protocol for every channel, including low-risk internal ones, when a “very good but not perfect” alternative would cover far more ground for far less cost.

The response is to design around the bias, not to train it away

None of these biases are unique to careless people. They are how human judgement works under uncertainty, for everyone. The lecture’s response is not “train people to stop being biased” but to design the system so the secure choice does not depend on overriding the bias in the moment, the subject of 02-03.

Pitfall

The biases above and the persuasion techniques in 02-03 share examples and even some names. “Respect to authority” reads like a bias, but the lecture places it among Cialdini’s six techniques, not this list. If a question asks for one of the ten items above, an answer drawn from Cialdini’s six is off by one section, even though the lecture itself says the two lists are closely related.