URL Normalization: Avoid Incorrect IOC Matches

Two different strings can represent the same destination, while two visually similar URLs can identify different resources. For an IOC matching engine, URL normalization must remove justified differences without erasing distinctions that matter to the investigation.

An overly strict rule misses matches. An overly aggressive rule connects unrelated events. The task is to define equality for the control being used, then preserve the raw string needed to explain each decision.

Define what must match

Start by naming the scope of the indicator. A domain, an origin consisting of a scheme, host and port, a complete URL and a path prefix represent different rules. Do not silently reduce a URL IOC to its domain to simplify indexing.

Specify what data the sensor actually provides. A DNS log cannot see an HTTP path. A proxy might retain only part of a request. A browser can produce an already transformed URL. The engine must compare observed fields without filling in missing pieces as though they were known.

This step complements the wider IOC management pipeline. The question here is narrower: which transformations justify treating two observations as a match for the same indicator?

Keep evidence separate from the comparison key

Store at least the raw value, source, timestamp, comparison value and normalizer version. Also retain the parsing result and warnings. A rejected entry must remain explainable, subject to appropriate confidentiality controls.

The calculated key supports searches. The raw value helps explain an error, replay a conversion and distinguish a producer’s transformation from one performed by your platform. Without this separation, changing the normalizer implicitly rewrites the history of your observations.

Do not automatically copy complete URLs into every diagnostic log. Parameters can contain a token or a personal address. Provide restricted storage for the evidence and a redacted version for everyday views, without treating the two representations as identical.

Handle case one component at a time

RFC 3986, section 6, distinguishes several forms of comparison and normalization. The scheme and host are handled differently from other components. Lowercasing the entire string can change a path or query data.

The following fictional examples use a reserved domain:

https://PORTAIL.EXAMPLE/Compte?id=AbC
https://portail.example/Compte?id=AbC
https://portail.example/compte?id=abc

The first two forms differ only in hostname case. The third also changes the path and parameter value. Your engine must not merge them based on an assumption that “URLs are case-insensitive.”

For internationalized names, use the same library and convention on both sides of the comparison. Preserve representations useful to the analyst, but keep visual similarity distinct from technical equality of the name.

Avoid successive decoding

Percent encoding needs particular care. Decoding once for display and a second time before matching can change the value being compared. Reserved characters and separators must be transformed with their role in mind.

Compare these two fictional paths:

https://portail.example/fichiers/a%2Fb
https://portail.example/fichiers/a/b

They are not interchangeable by default. The first contains an encoded separator whose treatment also depends on the server chain. Preserve that distinction until your policy establishes precisely which equivalence it intends to recognize.

Similarly, separate HTML decoding of a link extracted from a message from parsing the URL itself. & can come from the document containing the link. Blindly applying this conversion to an already extracted string adds another transformation. Record which layer each operation belongs to.

Preserve query and fragment semantics

Sorting parameters can appear to simplify deduplication. However, an application can interpret their order or repeated keys. An application signature can also depend on the transmitted representation. Preserve order unless equivalence is documented for your specific use.

The fragment after # is part of the URL representation and can affect client-side behavior. An HTTP request log generally does not observe it. Your policy must specify whether it compares a complete navigation URL or a target sent to the server.

The WHATWG URL Standard describes parsing and serialization operations for web URLs. Choose a parser whose behavior matches your source, then test edge cases. Mixing several parsers in successive stages can produce different results for an ambiguous input.

Build a test corpus that catches false equivalence

Prepare two groups of tests: pairs that must match and pairs that must remain distinct. Each case should state the matching scope and reason for the decision. A positive test alone will never catch normalization that turns every URL on a domain into the same key.

Include malformed data, explicit ports and relative URLs. A relative URL requires a known base; arbitrarily attaching a domain invents an indicator. Rerun the corpus whenever the library or normalization rules change.

Explain an alert using the exact transformation

When an event matches, show the rule’s scope and the transformations applied. The analyst needs to distinguish exact equality, limited normalization and a broader domain-level rule.

Before an external lookup, examine sensitive data using the guide to suspicious URLs and redirects. You can then consult a report on the relevant public indicator, preserving the original URL separately if it contains secrets.

A useful conclusion states what actually matched. “Same domain” and “same observed URL” call for different actions. Your normalizer must preserve that distinction through to the final decision.