Concepts · Entry 3
How detection works
Signatures, heuristics, behavioural monitoring, sandboxing and cloud lookups, and why every one of them trades one kind of mistake for another.
Short answer
No single method is sufficient. Exact matching is precise but blind to anything new; general rules catch new things and misfire on clean files; watching behaviour catches what static inspection misses but only once something is already running. Products layer all of them, and the layering is what the differences between products largely consist of.
Five methods, used together
Every mainstream scanner uses several detection methods simultaneously and decides by combining their outputs. Understanding them individually explains the behaviour people actually notice: why one product flags a file another ignores, why a scan finds something that was there for weeks, and why a trusted internal tool is sometimes quarantined without warning.
1. Signatures and exact matching
The oldest method. A researcher or an automated system analyses a malicious file and produces an identifier for it: a cryptographic hash of the whole file, or a pattern describing a distinctive sequence of bytes within it. The scanner compares what it encounters against its local database of those identifiers.
The strength is precision. A hash match is effectively certain, produces almost no false positives, and costs very little to check. The weakness is equally clear: a signature exists only after someone has analysed that specific file, and changing a single byte changes a hash. Automated systems on the attacking side generate large numbers of variants for exactly this reason, which is why signature databases are large and updated many times a day, and why an out-of-date installation is meaningfully weaker than an updated one.
2. Heuristics and static analysis
Instead of asking whether this is a known bad file, heuristic analysis asks whether this file has the characteristics of a bad file, without running it. The scanner inspects structure: how the executable is packed or obfuscated, which system functions it imports, whether it contains encrypted sections, whether its declared structure is internally inconsistent, whether it resembles known families in ways a hash would not capture.
Machine learning models are the current form of this idea. A model trained on very large collections of malicious and clean files scores a new file on how closely it resembles each group. The score is a probability, not a verdict, which is the key point: heuristic detection generalises, and everything that generalises is sometimes wrong. Detection names containing markers such as generic, heur or ML come from this layer, and they are the names most worth checking before acting on.
3. Behavioural monitoring
Some things can only be recognised in motion. Behavioural monitoring observes running processes and evaluates sequences of actions: a process that enumerates document folders and begins rewriting every file in them, a program that copies itself into a startup location and then contacts an unfamiliar server, a document that spawns a scripting host which downloads an executable.
This layer is the answer to file-less techniques, where the malicious activity is carried out by legitimate system tools driven by a script, so that there is no suspicious file on disk to inspect. It is also the layer that most directly targets ransomware, since mass encryption is a distinctive behaviour even when the program performing it is unknown. The trade is timing: behaviour can only be judged once something has begun, so the question becomes how much happens before the pattern is recognised.
4. Sandboxing and emulation
Rather than waiting for a real execution, the product can run a suspicious file in a constrained, disposable environment and watch what it attempts. Consumer products typically use lightweight local emulation, while heavier analysis happens in the vendor's infrastructure.
Malware authors respond by writing code that checks whether it is being observed — testing for virtualised hardware, unusually small disks, an absence of ordinary user files, or simply waiting several minutes before doing anything, because automated analysis rarely waits that long. This back-and-forth is why sandbox results supplement other layers rather than replacing them.
5. Cloud lookups and reputation
When the local engine meets something it does not recognise, it can ask the vendor's servers. What is sent is normally a hash and metadata rather than the file itself, though products differ on whether unknown files are uploaded for analysis, and that is a privacy question as much as a technical one — scanning and privacy deals with it directly.
The answer that comes back is typically a reputation: how widely this file has been seen, how long it has existed, whether it is digitally signed and by whom. A file signed by a well-known publisher and present on millions of machines is treated differently from an unsigned executable seen twice in the last hour. Reputation is useful and it has a known blind spot: targeted software that exists on very few machines looks statistically identical to something merely new.
Two kinds of mistake
Every tuning decision in a scanner is a trade between two errors, and neither can be driven to zero.
| False negative | False positive | |
|---|---|---|
| What happened | Something malicious was not identified | Something clean was identified as malicious |
| Immediate cost | The code runs and does what it was written to do | A wanted file is blocked, quarantined or deleted |
| Secondary cost | Discovery may be delayed by weeks | People learn to dismiss warnings, weakening the true ones |
| Usual cause | A new variant, an unpatched flaw, or a technique with no file to scan | A general rule matching an unusual but legitimate file |
| What reduces it | More sensitive rules and more layers | More conservative rules and stronger reputation data |
The two rows at the bottom point in opposite directions, and that is the whole difficulty. A vendor that tunes for maximum detection accepts more false positives; one that tunes for quiet operation accepts more misses. Independent testing organisations such as AV-Comparatives and AV-TEST publish their testing methodologies alongside their results, and reading the methodology is more informative than reading any single score, because it states what was measured and under what conditions.
If a file you trust is quarantined
- Note the exact detection name and file path before doing anything, since both disappear from view once the item is restored or removed.
- Check whether the name indicates a generic or heuristic match rather than an exact one.
- Confirm where the file actually came from — the original vendor's own download page, or somewhere else.
- If you have reason to believe it is clean, restore from quarantine rather than disabling protection, and report the file to the vendor so the rule can be corrected.
- Add an exclusion only for that specific file, never for an entire drive or user folder.
Why updates matter more than settings
Two different things update, and conflating them causes confusion. Definition updates arrive frequently, sometimes many times a day, and carry new signatures and rules. Engine updates arrive rarely and change the scanning software itself, including its behavioural logic. A product with current definitions and an old engine is partially current; one with neither is running on an obsolete picture of the world.
The same principle applies far beyond the scanner. Operating system and browser updates close the specific flaws that exploit kits rely on, which is why the ACSC's published guidance at cyber.gov.au places patching among the foundational controls rather than treating it as maintenance.
One engine, several brands
Worth knowing when comparing products: not every vendor builds its own detection engine. Engine licensing is common in the industry, and two differently branded products may share a substantial part of their detection logic while differing in interface, pricing, bundled extras and update cadence. Vendors do not always publicise the arrangement. The practical implication is that feature lists and brand names say less about detection capability than they appear to, and that running two products in parallel for extra coverage frequently produces conflicts rather than redundancy.