TAXII Pagination: Sync Without Losing Feed Additions

A TAXII client can report “synchronization successful” while missing objects. This often happens when it uses a STIX object's modified value as added_after, or advances its checkpoint before saving the final page.

To verify your integration, separate two problems: reading every page of a request, and knowing where to resume after a failure. A pagination cursor addresses the first. A durable checkpoint and storage that can safely replay the same data address the second.

Intelligence timestamps and transport timestamps differ

In TAXII 2.1, added_after selects objects added to a collection after a specified time. That time is independent of the timestamps inside STIX objects. The manifest exposes date_added; the X-TAXII-Date-Added-First and X-TAXII-Date-Added-Last headers describe the response boundaries. These rules appear in the TAXII 2.1 specification.

Consider this fictional example, where A and B stand for object identifiers:

An import starting after October 11 must be able to retrieve A despite its older content. If a local filter automatically rejects objects whose modified timestamp precedes the previous run, A disappears. Make this a priority case in your synchronization tests.

Store timestamps at their original precision. Converting them to whole seconds to simplify a database can shift a recovery boundary. Keep a readable representation too, but do not use that display value as a technical key.

Read every page before confirming the run

When more is true, a TAXII 2.1 response may provide next. The client passes that token in the following request with the original options. Without a token, the specification provides a fallback: use X-TAXII-Date-Added-Last as added_after, retaining the other filters. The OASIS interoperability test document describes both paths.

Treat next as an opaque value. Do not decode it to construct a page number or reuse it for another collection. The API root, collection, and filters must remain associated with the current run.

An operational procedure can follow these steps:

1. Load the last confirmed checkpoint and the task's filters.
2. Request the first page and verify its status and structure.
3. Durably save the objects and any validation rejections.
4. Continue according to more and the continuation information received.
5. Confirm the new checkpoint once the entire run has been processed.

This defines a processing contract, without prescribing a particular transaction implementation. Saving each page separately is appropriate if replay causes no additional effects. You need to be able to explain what happens between a successful write and a process failure.

Test the boundaries where objects disappear

Build a test collection containing several objects with exactly the same addition timestamp. Set a small page size so that the group spans multiple pages. Verify that the sequence of responses returns every expected identifier.

This test is essential for timestamp-based continuation: a strictly “after T” condition does not, by itself, retrieve other items dated T. If the server cannot traverse that boundary correctly, blindly repeating the request will not solve the problem. Document the discrepancy and clarify the behavior with the provider before declaring the import complete.

Also test an empty page following a response that indicated more items were available. A collection can change while it is being read. The client must record this result and finish or report the anomaly according to the observed contract, without entering an infinite loop. Add protection against a repeated token that produces no progress.

An overlap window between runs can help replay boundary items. It also increases the duplicates your storage must absorb. It cannot repair expired retention or broken pagination that makes some objects inaccessible.

Choose which versions to retain explicitly

The TAXII match[version] filter can request first, last, or all; last is the default, as described in the version filtering section. For versioned STIX objects, a specific value corresponds to their modified timestamp. The same identifier can therefore refer to several states of the intelligence.

Decide whether you need a current view or a history. For a current view, you can retain the latest state while archiving each import's provenance. For a history, request the relevant versions and verify that the server actually retains them. No parameter can recreate a version the provider has deleted.

Deduplicating on the identifier alone can discard a useful revision. For STIX objects that carry modified, test a key that includes it. For types without that field, define a rule appropriate to the type instead of adding an invented date.

Simulate failure before enabling the production feed

Deliberately interrupt the client after it saves a page but before it confirms the checkpoint. On restart, it should reread what is necessary without creating another alert for every object already known.

Repeat the exercise with an invalid object. Two choices are defensible: stop the window for correction, or durably isolate the rejected object and continue under an explicit policy. Silently ignoring it makes the “imported” counter misleading. Logs should distinguish received, saved, already present, and rejected objects.

The continuous TAXII-to-SIEM integration guide covers the connector's operating environment. The differences between Indicator and Observed Data then help you verify what happens to objects after transport.

Before adding more collections, replay these cases against one: a late addition, identical timestamps, a new version, an interruption, and a validation rejection. Keep the expected results with the connector. They provide a concrete check whenever the STIX/TAXII pipeline changes, beyond counting successful HTTP responses.