API pagination is how a client pulls a large result set out of a REST API in bite-sized pages instead of one enormous response. When you ask a historian or cloud data API for months of tag history, the answer can run to millions of rows, far too many to return at once, so the API hands them back a page at a time and tells you how to ask for the next one. Doing this correctly, without dropping or duplicating rows across a range that may take hundreds of requests to walk, is the difference between a trustworthy bulk export and a full of holes one. This page covers cursor versus offset paging, resuming after a failure, respecting rate limits, and stitching the pages into a gap-free series.
API pagination for bulk data pulls in one line: API pagination splits a large query result into sequential pages, each request returning a limited number of rows plus a marker for fetching the next page. For SCADA bulk pulls it lets you extract a long tag-history range without overwhelming the server or the client, provided you page in a stable order, resume from the last marker after failures, and back off when rate limited.
There are two dominant ways an API lets you walk through pages. Offset pagination is the familiar skip-and-take style: you ask for rows 0 to 999, then 1000 to 1999, and so on, incrementing an offset each time. It is simple to reason about, but it has a serious weakness for time-series data. If rows are being inserted while you page, or the underlying order shifts, the offsets slide, and you can silently skip rows or read some twice. It also gets slower deep into a large set, because the database still has to count past all the skipped rows to reach your window.
Cursor pagination avoids these problems by paging on a stable value rather than a position. Instead of asking for the next thousand rows by number, you ask for rows after a specific point, typically the last timestamp and key you already received, encoded as an opaque cursor or page token the API returns with each page. Because the cursor points at actual data rather than a count, new inserts before your position do not shift what next means, and the query can use an index to jump straight to the continuation point. For a monotonically increasing tag-history stream keyed on timestamp, cursor paging is almost always the right choice.
The practical rule for bulk SCADA pulls is to prefer cursor or keyset pagination whenever the API offers it, and to always page in a deterministic order such as ascending timestamp with a tie-breaker on tag or a unique row identifier. A stable, total ordering is what guarantees that after is unambiguous, so no two pages overlap and no gap opens between them. Offset paging can work for small, static result sets, but for a long history range that may still be receiving writes, it is a liability.
A bulk pull that spans hundreds of pages will sometimes fail partway through, from a dropped connection, a timeout, or a transient server error, and the integration must be able to pick up where it left off rather than starting over. This is exactly why the page token matters beyond just fetching the next page: the client persists the last successfully processed cursor, so on restart it resumes from that token and continues forward. With offset paging this resume is fragile because the meaning of an offset can drift; with a cursor tied to real data, resuming is exact.
Large pulls also collide with rate limits, because a client hammering an API for page after page as fast as it can will trip the server's request ceiling. The API signals this, commonly with a 429 status and often a header telling you how long to wait, and the client's job is to respect that rather than retrying instantly and making things worse. Exponential backoff with jitter is the standard response: on a rate-limit or transient error, wait a short interval, then a longer one if it recurs, adding a little randomness so many clients do not all retry in lockstep. Honoring an explicit retry-after value the server sends is better still.
Backoff and resume work together to make a long pull robust. A transient failure triggers a backed-off retry of the same page from the last good cursor, so no data is lost and the server is not overwhelmed. The client should distinguish retryable errors, like timeouts and rate limits, from permanent ones, like an invalid query or an authentication failure, and only back off and retry the former. Bounding the total number of retries prevents an integration from spinning forever on a genuinely broken request.
The reason all this care matters is that the end product has to be a continuous, gap-free tag history, not a pile of pages with unknown holes. Because each page continues exactly where the last ended, following the cursor, the pages concatenate into one ordered series with no overlaps and no missing spans, as long as the ordering was deterministic throughout. If any page were fetched out of order or an offset had slipped, the stitched result could hide a gap that only surfaces much later as a suspicious flat line in a trend or a missing hour in a report.
It helps to verify the seam between pages rather than trusting it blindly. Checking that the first record of each page truly follows the last record of the previous page, in the agreed ordering, catches a broken cursor or a server-side quirk early. Recording the range actually covered, the first and last timestamps ingested, lets a later reconciliation confirm the whole requested window was retrieved. For a pull that also needs late-arriving corrections, a subsequent incremental page beginning at the last cursor picks up anything added since, extending the series without re-fetching what you already have.
For a platform like Merobix backfilling historical telemetry from a customer's existing historian or cloud API, this disciplined pagination is what makes a migration or a large export trustworthy. Cursor-based paging with persisted tokens, exponential backoff on rate limits, and page-boundary checks turn a query too big to return in one shot into a complete, verifiable series that lands in the new system exactly as it existed in the old one, without the silent gaps that undermine confidence in the data.
Use cursor or keyset pagination when the API supports it, especially for time-series data that may still be receiving writes. Cursor paging continues after a stable value like the last timestamp, so new inserts do not shift what the next page means and no rows are skipped or duplicated. Offset paging is fine only for small, static result sets.
Persist the page token or cursor of the last successfully processed page as you go, so on restart you resume from that marker instead of beginning again. Because a cursor points at real data rather than a numeric position, resuming from it is exact and does not risk skipping or repeating rows. This is one of the main reasons cursor paging is preferred for long pulls.
Back off and retry rather than hammering the endpoint. On a rate-limit response, wait before retrying, honoring any retry-after value the server provides, and use exponential backoff with jitter if it keeps happening so many clients do not retry in lockstep. Only retry retryable errors like rate limits and timeouts, not permanent ones like a bad query or failed authentication.
Merobix reads your field devices into a cloud SCADA - the real thing behind these terms, live in days from any browser.