TAXII Threat Feeds: Build a Continuous SIEM Integration

A TAXII connection becomes useful when it runs repeatedly without duplicating indicators, skipping pages, or turning a temporary feed failure into missing detections. The hard part is not the first successful poll. It is preserving a trustworthy cursor, mapping STIX consistently, and making every partial run recoverable.

isMalicious exposes TAXII 2.1 discovery at https://api.ismalicious.com/taxii, then an API root, collection catalog, and collection objects. This guide focuses on the continuous path into a SIEM or TIP. For the broader choice between OpenCTI, MISP, and direct SIEM ingestion, read the existing STIX/TAXII operational guide.

Define the ingestion contract first

Write down the boundary before configuring a connector. The producer owns collection delivery and object context. Your consumer owns polling state, schema mapping, deduplication, retention, and any action taken from the data. A feed outage must pause ingestion, not erase the last good dataset or mark unseen indicators as safe.

Choose where the first durable copy will live. In a direct SIEM workflow, use a dedicated threat-intelligence index or lookup table rather than the main alert index. In a TIP-first design, ingest into OpenCTI or MISP and publish a smaller, curated view to the SIEM. The threat feeds overview helps place TAXII beside simpler list and API delivery patterns.

Define four records:

• connector configuration, with server URL and selected collection ID;
• secret reference, never the credential itself;
• ingestion checkpoint, committed only after a complete window;
• run manifest, with start time, end time, page count, object count, status, and error class.

These records make the pipeline auditable without exposing indicator payloads in routine logs.

Authenticate and enumerate collections

Start from the isMalicious STIX/TAXII page, then confirm the current paths in the API reference. Send the Base64 form of apiKey:apiSecret in X-API-KEY and request the TAXII 2.1 media type.

curl --silent --show-error \
  --header "X-API-KEY: $ISMALICIOUS_BASE64_CREDENTIAL" \
  --header 'Accept: application/taxii+json;version=2.1' \
  'https://api.ismalicious.com/taxii'

curl --silent --show-error \
  --header "X-API-KEY: $ISMALICIOUS_BASE64_CREDENTIAL" \
  --header 'Accept: application/taxii+json;version=2.1' \
  'https://api.ismalicious.com/taxii/api-root/collections'

Persist the collection ID selected from the live catalog. Do not infer availability from a blog example. Treat 401 as a credential problem, 403 as an access problem, and 404 as a stale collection or route configuration. None should be retried in a tight loop.

The OASIS TAXII 2.1 standard defines discovery, API roots, collections, media types, and pagination semantics. Authentication policy remains specific to the service, which is why the isMalicious API reference must stay part of your connector configuration.

Establish a baseline and a durable checkpoint

For the first synchronization, request:

GET /taxii/api-root/collections/{collectionId}/objects

Omit added_after and added_before during this baseline. The current service can include records without parseable intelligence timestamps when no date filter is active; adding a time boundary would exclude them. Consume the response objects, inspect more, and pass the opaque next value on the following request until more is false.

Write each page to staging with an idempotency key based on the STIX object id and its version signal, commonly modified when present. Commit the page and its raw-response hash together. Do not advance the durable window checkpoint after page one. Advance it only when the final page has been validated and committed.

For later pulls, use added_after with an overlap before the previous successful boundary. The overlap catches records that arrive late or carry timestamps near the cutoff. It also creates duplicates, which is expected and safe when ingestion is idempotent. Keep the same filters while following next; changing the window mid-pagination breaks the meaning of the cursor.

Map STIX into a SIEM without losing context

Keep the raw STIX object in inexpensive storage, then map a compact set of fields into the SIEM. Useful columns include STIX id, type, spec_version, indicator pattern, pattern_type, created, modified, labels, confidence when present, and source collection. Store relationships or external references outside the hot lookup when they would inflate every alert.

Parse indicator patterns rather than stripping punctuation with regular expressions. A STIX pattern can contain an IP, domain, URL, or hash expression and may include operators your first parser does not support. Send unsupported or malformed patterns to a dead-letter queue. Do not silently coerce them into a different observable.

Normalize the extracted value according to type, but preserve the original pattern for evidence. Add ingestion metadata in your own namespace: feed_received_at, collection_id, connector_run_id, and mapping_version. This makes schema migrations and replay possible.

Teams using OpenCTI or MISP should let the TIP retain relationships, then export a purpose-built SIEM lookup. A direct Microsoft Sentinel integration can still use the same staging, checkpoint, and dead-letter design.

Operate the polling loop safely

Schedule according to your detection latency target and the rate at which your downstream system can index data. Prevent overlapping runs with a distributed lock or scheduler guarantee. One slow poll must finish or fail before the next one begins.

Use bounded connect and read timeouts. Retry network errors, 503, and 504 with exponential backoff and jitter. Honor Retry-After when returned. If a page repeatedly fails, leave the durable checkpoint unchanged, retain the run manifest, and resume from the last committed state. An invalid next token should restart the same bounded window, relying on idempotency to discard duplicates.

Separate fetching from promotion. The fetcher writes staging. A validator checks schema, patterns, timestamps, unexpected volume changes, and duplicate behavior. A publisher updates the active lookup atomically only after validation. Alerts continue to use the last known good lookup while a new run is incomplete.

The step-by-step vendor-neutral TAXII ingestion playbook is a useful implementation companion once this control model is agreed.

Validate, measure, and recover

Test the connector with a small selected collection before connecting it to production correlation. Exercise a bad credential, denied access, missing collection, timeout, busy service, malformed object, duplicate page, and crash between page commit and checkpoint commit. The expected result is either a complete new version or the previous good version, never a half-updated lookup.

Track feed age, time since the last successful full window, run duration, pages received, objects accepted, duplicates, dead-letter objects, retry count, and promotion status. In the SIEM, separately measure indicator matches, unique affected assets, analyst-confirmed matches, false positives, and exceptions. These are operational measurements, not claims about the size or effectiveness of the provider's internal corpus.

Keep at least the current and previous published lookup versions. Recovery then becomes a pointer change rather than a rebuild under pressure. Document who can pause polling, promote a staged version, restore the previous version, and rotate the credential.

Extend the feed without weakening the control

Once one collection runs cleanly, add collections one at a time and keep separate checkpoints. Different collections may need different retention or promotion policies. Reuse the same raw archive, mapper versioning, validation, and atomic publication pattern.

Feed ingestion supplies baseline coverage. Alert-time enrichment still answers questions about an indicator that is absent or needs fresher context. Connect both through the same evidence model, then let analysts or policy decide what reaches prevention. This is how a TAXII feed becomes a continuous security control rather than a recurring import job.