DNS over HTTPS Security: Detect DoH Abuse

DNS over HTTPS (DoH) protects resolver traffic inside HTTPS. That improves confidentiality on untrusted networks and reduces on-path manipulation. In an enterprise, however, unmanaged DoH can also bypass the resolver where filtering, logging and threat-intelligence policies live.

The security objective is not to ban encrypted DNS. It is to provide an approved private path, prevent silent policy bypass and detect malware that disguises command-and-control traffic as resolver requests.

What DoH Changes

RFC 8484 standardizes DNS queries over HTTPS. Network devices that previously observed port 53 may now see only HTTPS to a resolver. This changes visibility in three ways:

• the query and response are encrypted in transit;
• DoH shares port 443 with ordinary web traffic;
• applications can select a resolver independently of the operating system.

Legitimate browsers, operating systems and privacy tools support DoH. Malware can use the same protocol or imitate its shape. Context matters more than the presence of encryption.

Start With Resolver Governance

Define approved resolvers for managed devices. They should provide appropriate privacy, retention, filtering, availability and legal controls. Configure browsers and operating systems through enterprise policy so they use the intended path.

Document exceptions for developers, mobile users and security tooling. An exception should name the application, owner, destination and expiry. Block or challenge known public resolvers only after confirming that managed clients have a reliable alternative.

Detection Without Universal TLS Decryption

Combine several telemetry sources:

Destination intelligence

Maintain an allowlist of approved resolver hostnames, IPs and certificates. Alert on endpoints contacting newly observed or low-reputation destinations with resolver-like patterns.

Endpoint process context

Collect the process responsible for the connection. DoH from an approved browser is different from an unsigned executable in a temporary directory contacting the same service.

Traffic behavior

Look for frequent small requests, regular intervals, unusual upload/download ratios and long-lived activity inconsistent with the process. These are hypotheses, not proofs.

DNS and domain correlation

Compare endpoints that bypassed the resolver with later connections to unknown domains. Enrich destinations through domain reputation and IP reputation.

Selective TLS inspection may be justified for managed high-risk segments, but it should follow policy, legal review and data minimization. Do not make mass decryption the default answer to a governance problem.

Detecting DoH as a C2 Channel

Malware may call a public resolver normally to retrieve malicious records, or operate a custom DoH endpoint that carries arbitrary data. Investigate when:

• a non-browser process makes repeated HTTPS calls to a resolver;
• the endpoint ignores corporate resolver policy;
• request timing resembles beaconing;
• the destination domain is new, uncommon or linked to malicious infrastructure;
• the same process creates files, persistence or suspicious child processes.

Correlate network evidence with endpoint behavior and C2 detection guidance. A network-only signature is fragile because legitimate and malicious DoH can use identical protocol features.

Incident Response

Preserve the process tree, destination, certificate, request timing and device policy state. Isolate the endpoint when supporting evidence indicates compromise. Block the malicious destination and related infrastructure, then determine whether the attacker used a public resolver, a custom endpoint or protocol mimicry.

Review policy drift across the fleet. One compromised device may reveal that many endpoints silently bypass the intended resolver.

Metrics and Privacy

Measure the percentage of managed endpoints using approved resolvers, unapproved DoH destinations, policy failures, alerts confirmed by endpoint evidence and time to remediation. Also measure data collection: security teams should know what DNS and endpoint fields they retain and why.

Conclusion

DoH is a privacy protocol, not a malicious indicator. Secure it with governed resolvers, endpoint-aware telemetry and risk-based investigation. Preserve visibility through correlation rather than reflexive decryption, and use DNS security and threat intelligence to keep policy effective across encrypted and traditional resolution.