Bulk IP and Domain Lookups: Designing Indicator Enrichment That Survives Real Volume

Individual indicator lookups are fine when a human is asking the question. The problem starts when the list comes from a machine: a proxy log extraction, a firewall export, the indicator set from an incident report, or a feed you want to validate before trusting it. At that point you are not doing lookups, you are running a job — and the design decisions that make a job work are different from the ones that make a single lookup work.

This guide covers how to structure batch enrichment so it stays fast, stays inside quota, and does not silently drop half your indicators when something fails.

Where the Volume Comes From

Batch enrichment shows up in a predictable set of situations:

• Retrospective hunting. A new campaign is published with indicators, and you query weeks of logs for matches. The extraction produces thousands of candidate addresses and domains, almost all irrelevant.
• Incident scoping. One confirmed intrusion generates every destination the compromised host contacted. Sorting attacker infrastructure from ordinary traffic is the whole task.
• Firewall and proxy log review. Periodic review of what your network actually talked to, which is the only reliable way to find the connection nobody alerted on.
• Feed validation. Before ingesting a third-party feed into blocking controls, checking it against an independent source tells you its false-positive rate.
• Attack surface assessment. Enumerating an organization's own domains and addresses — during onboarding, an acquisition, or an audit — and checking each for exposure.
• Blocklist hygiene. Existing block entries age badly. Re-checking them periodically finds the addresses that have since been reassigned to something legitimate.

The common shape: a list produced automatically, mostly noise, where the value is in the small subset that turns out to matter.

Reduce Before You Send

The largest efficiency gain in any batch pipeline happens before the first request.

Deduplicate. Raw log extractions are extraordinarily repetitive — a single busy address can account for thousands of rows. Reducing to a unique set routinely cuts volume by an order of magnitude and costs nothing in coverage.

Filter what you already know. Several categories can be removed without a lookup:

• private and reserved address ranges, which have no external reputation to check;
• your own ASNs and address blocks;
• addresses and domains belonging to infrastructure you have already reviewed and allowlisted;
• entries you enriched recently enough that the answer has not plausibly changed.

Cache aggressively, with a sensible TTL. Reputation changes, but not minute to minute. Caching results for hours means the same indicators reappearing in tomorrow's batch are answered locally. Registration data changes even more slowly — WHOIS records are typically cached for around 24 hours for exactly this reason.

Normalise. Strip ports, lowercase domains, resolve punycode consistently, and decide up front whether example.com and www.example.com are one indicator or two. Inconsistent normalisation defeats deduplication and inflates the request count without adding information.

Only after those four steps does the size of your actual job become clear, and it is usually a fraction of the raw extraction.

Design Around Two Separate Limits

Batch APIs impose limits in two dimensions, and conflating them causes most integration problems.

Per-request batch size is how many indicators fit in one call. The isMalicious bulk API accepts up to 100 entities per request on Pro, with smaller evaluation batches on Free. Larger jobs are chunked across multiple requests.

Total volume over time is your monthly quota and your burst rate — on Pro, a monthly reputation-check allowance and a per-minute API burst ceiling. A job can be perfectly chunked and still fail because it tried to run 400 requests in 30 seconds, or succeed all week and then exhaust the monthly allowance on the 28th.

Practical consequences:

• Chunk to the batch limit, not to a number you picked. Requests of 100 where 100 is allowed means a tenth of the calls of requests of 10.
• Rate-limit yourself deliberately rather than discovering the ceiling through failed requests. A small delay between chunks is cheaper than retry logic.
• Track quota consumption as a metric, and alert before exhaustion. A pipeline that silently stops enriching is worse than one that fails loudly, because nobody notices until an incident.
• Prioritise within the job. If a batch exceeds what you can process, enrich indicators that appeared in alerts before those that only appeared in logs.

Handle Partial Failure Explicitly

A batch of 100 does not succeed or fail as a unit in practice. Individual entries can be malformed, unresolvable, or of a type the endpoint does not handle, and a pipeline that treats any per-entry problem as a whole-batch failure will retry expensive work repeatedly.

• Key results back to inputs explicitly. Do not rely on response ordering to match results to submitted indicators.
• Separate "no data" from "error." An address with no reputation history is a valid answer and should not be retried. A timeout should be.
• Retry with backoff, and only the failures. Re-submitting a whole chunk because two entries failed wastes quota on 98 successful lookups.
• Record what you did not enrich. An indicator that silently never got checked is the one that turns up in the post-incident review.

Interpreting Bulk Output Without Creating an Outage

The temptation with batch enrichment is to wire the output straight into blocking. Resist it, at least initially.

A reputation verdict is evidence, not an instruction. Automated blocking on score alone will eventually take out a CDN edge address, a mail provider, or a SaaS endpoint the business depends on — and it will do so at scale, because that is what automation does. The safer pattern separates the two decisions:

• Enrich and prioritise automatically, for everything. Attaching context to every indicator costs nothing and makes every downstream decision better.
• Block automatically only within reviewed categories, behind a confidence threshold, and always behind an allowlist of infrastructure the business cannot lose.
• Route the ambiguous middle to a human, with the enrichment already attached so the analyst is making a judgment rather than gathering data.

It also helps that bulk results return full threat context per entry rather than a bare verdict — the reason an indicator scored the way it did is what lets an analyst overrule it correctly.

A Pipeline That Works

1. Extract indicators from the source, keeping the original context — which log, which host, which alert.
2. Normalise into consistent indicator types.
3. Deduplicate to a unique set.
4. Filter private ranges, own infrastructure, allowlisted entries, and anything already cached.
5. Chunk to the per-request batch limit.
6. Submit with deliberate pacing inside the burst ceiling.
7. Merge results back onto the original context, so a verdict is attached to the host that made the connection rather than floating free.
8. Cache everything with an appropriate TTL.
9. Route by confidence: high-confidence malicious to controls, ambiguous to analysts, clean to the record.
10. Measure quota consumption, cache hit rate, and the proportion of indicators that turned out to matter — that last number tells you whether your filtering is too loose or too tight.

Getting It Running

The isMalicious bulk API accepts JSON arrays, newline-delimited text, and CSV, and detects the type of each entry so domains, IPs, and URLs can be mixed in a single request instead of being split by type. Each entry returns full threat context rather than a verdict alone, which is what makes the routing decisions above possible.

For implementation specifics — request shape, chunking behaviour, and the exact limits on your plan — see the bulk endpoint documentation and the rate limit reference. Teams wiring this into detection tooling usually start from the SIEM integration path, and developer documentation covers the SDK route if you would rather not hand-roll the HTTP layer.

The underlying point is unglamorous: the bottleneck in indicator enrichment is almost never the lookup. It is the hundreds of duplicate rows, the missing cache, the untracked quota, and the retry storm caused by two malformed entries. Fix those and enrichment stops being the slow step in triage.