---
title: "Pagination"
description: "Explains client-side pagination for all Reporting API tables and pagination for the LMPI table, with sample scripts for each."
contentType: "guide"
url: "https://developer.mindtickle.com/docs/reporting/pagination/"
---

The Reporting API supports pagination, so you can send successive API requests and fetch a larger dataset. Client-side pagination with the `skip` and `top` options works for all Reporting API tables. The `LearnerModulePerformancesIncremental` (LMPI) table has its own pagination option, which fetches up to 5,000,000 records.

## Client-side pagination

Client-side pagination is available for all customers by default.

### How client-side pagination works

The Reporting API supports client-side pagination with the `skip` and `top` options to fetch the data in batches when the data is large and cannot be fetched in a single go. Every time you run the script, the data is sorted on the primary key and the script skips the records that are already fetched. When no records have been fetched at the start, `skip` is `0`, and the script runs until there are no records left to fetch.

Records inserted during an export may be absent until a later run. Treat each export as a changing dataset and reconcile successive exports by the table's keys; do not assume a consistent snapshot across requests.

### Best practices

- You do not need to apply sorting on top of pagination, because sorting is provided within the pagination implementation.
- When you use the `top` option to fetch top records, the value must not be greater than 300,000 for the `CourseObjects`, `QuickUpdateObjects`, and `AssessmentObjects` tables, and not greater than 100,000 for the `LearnerModulePerformances` table.

### How the skip value changes

Consider the following three consecutive queries. The difference between them is the value of the `skip` option. When you use the `skip` option in a query, the full data is sorted after filtering on the primary key of the table, and records are skipped based on the `skip` value. In the first query there are no records fetched yet, so `skip` is `0`. In the subsequent queries, the `skip` value increases as records are fetched.

First query, with `top=100000` and `skip=0`:

```text
https://admin.mindtickle.com/Odata.svc/CourseObjects?$format=csv&$filter=StartTime ge '2021-01-01 00:00:00' and StartTime le '2021-12-31 23:59:59'&$top=100000&$skip=0
```

Second query, with `top=100000` and `skip=100000`, because 100,000 records are already fetched:

```text
https://admin.mindtickle.com/Odata.svc/CourseObjects?$format=csv&$filter=StartTime ge '2021-01-01 00:00:00' and StartTime le '2021-12-31 23:59:59'&$top=100000&$skip=100000
```

Third query, with `top=100000` and `skip=200000`, because 200,000 records are already fetched:

```text
https://admin.mindtickle.com/Odata.svc/CourseObjects?$format=csv&$filter=StartTime ge '2021-01-01 00:00:00' and StartTime le '2021-12-31 23:59:59'&$top=100000&$skip=200000
```

The script runs consecutive queries until the `X-MTAPI-RecordCount` token value in the header is `0`. That is, the script execution completes when there are no more records to fetch.

After processing each query, a CSV file is created, with the file name in a predefined format. Each run uses a timestamp, including microseconds, in its file names. A page is first written to a temporary `.csv.part` file and then renamed to `.csv` after the write completes. A failed run leaves earlier completed pages in place and does not automatically resume; keep output from separate runs apart to avoid importing overlapping exports. The file name format is:

```text
table_name + str(datetime.now().strftime("_%Y_%m_%d_%H_%M_%S_%f_")) + str(first_row_num) + '_to_' + str(last_row_num) + '.csv'
```

Where `table_name` is `CourseObjects`, `first_row_num` is `1`, `last_row_num` is `100000`, and `str(datetime.now().strftime("_%Y_%m_%d_%H_%M_%S_%f_"))` is `2022_01_17_12_58_18_123456`. For those values, the file name is `CourseObjects_2022_01_17_12_58_18_123456_1_to_100000.csv`.

### Client-side pagination script

Use this standalone Python sample to export CSV pages. Install `requests`, set your Reporting credentials and regional base URL, and edit `options` for your query. Keep `$format` set to `csv`. The request timeout is 10 seconds to connect and 120 seconds without received data, not a total export deadline. Each page gets at most three attempts for connection failures, timeouts, and HTTP 429, 500, 502, 503, or 504 responses. Other failures stop the script. A completed page is saved before pagination advances; `Finished Processing` appears only after normal completion.

```python
import time
from datetime import datetime
from pathlib import Path
from urllib.parse import urlencode

import requests
from requests.auth import HTTPBasicAuth

MAX_ATTEMPTS = 3
RETRY_DELAY = 60
REQUEST_TIMEOUT = (10, 120)


def get_query_url(base_url, options):
    return f"{base_url}?{urlencode({key: value for key, value in options.items() if value is not None})}"


def fetch_page(url):
    for attempt in range(1, MAX_ATTEMPTS + 1):
        try:
            with requests.get(
                url,
                auth=HTTPBasicAuth(username, password),
                timeout=REQUEST_TIMEOUT,
                allow_redirects=False,
            ) as response:
                response.raise_for_status()
                if response.status_code != 200:
                    raise ValueError("Expected HTTP 200; check the configured Reporting URL")
                count = int(response.headers.get('X-MTAPI-RecordCount', ''))
                if count < 0:
                    raise ValueError("X-MTAPI-RecordCount must be nonnegative")
                return response.content, count, response.headers.get('X-MTAPI-NextToken')
        except (requests.Timeout, requests.ConnectionError, requests.HTTPError) as error:
            if isinstance(error, requests.HTTPError) and error.response.status_code not in {
                429, 500, 502, 503, 504
            }:
                raise
            if attempt == MAX_ATTEMPTS:
                raise
            print(f"Request failed; retrying this page in {RETRY_DELAY} seconds")
            time.sleep(RETRY_DELAY)


def save_csv_page(content, file_name):
    target = Path(file_name)
    temporary = target.with_suffix('.csv.part')
    try:
        temporary.write_bytes(content)
        temporary.replace(target)
    finally:
        temporary.unlink(missing_ok=True)


table_name = 'CourseObjects'
base_url = f'https://admin.mindtickle.com/Odata.svc/{table_name}'
username = 'username'
password = 'password'
options = {
    '$format': 'csv',
    '$top': 10000,
    '$select': None,
    '$filter': None,
    '$orderby': None,
}


def run():
    total_rows = 0
    iteration = 0
    export_time = datetime.now().strftime('_%Y_%m_%d_%H_%M_%S_%f_')
    while True:
        url = get_query_url(base_url, {**options, '$skip': total_rows})
        content, count, _ = fetch_page(url)
        if count == 0:
            break
        last_row = total_rows + count
        file_name = f'{table_name}{export_time}{total_rows + 1}_to_{last_row}.csv'
        save_csv_page(content, file_name)
        total_rows = last_row
        iteration += 1
        print(f'Total rows after iteration {iteration} = {total_rows}')
    print('Finished Processing')


if __name__ == '__main__':
    run()
```

## Pagination for the LMPI table

The Reporting API supports pagination for the [`LearnerModulePerformancesIncremental`](/docs/reporting/tables-performance-and-completion/) (LMPI) table. With the pagination script you can pull up to 5,000,000 records.

- Pagination with the `$pagination` option is supported only for the `LearnerModulePerformancesIncremental` table.
- If your API requests do not exceed 100,000 records, you do not need to implement pagination and can use the `LearnerModulePerformancesIncremental` table normally. You must implement pagination if your API requests exceed 100,000 records.
- With the pagination script you can pull up to 5,000,000 records across successive requests. Each generated CSV file can contain a maximum of 100,000 rows. For example, if there are 4,251,795 records in total, 43 CSV files are created: 42 files with 100,000 records each, and 1 file with the remaining 51,795 records.

The date filter must be within the last 30 days. The dated URLs below illustrate query syntax; replace their dates before use. Obtain the initial data dump before starting incremental extraction.

To use pagination, add the `$pagination=true` option to the OData query and apply the date filter. For example:

```text
https://admin.mindtickle.com/Odata.svc/LearnerModulePerformancesIncremental?$format=csv&$filter=Date ge '2025-03-17'&$pagination=true
```

Once you run the query, look at the value returned by `next_token`, which is `resp.headers.get('X-MTAPI-NextToken')`, to know whether there is more data to be fetched:

- If `next_token` returns a query URL, that is the value is not `None`, there is more data to be fetched. Use the query URL returned by `next_token` to run another query. Run consecutive queries in the script until the `next_token` value is `None`.
- If `next_token` is `None`, all data has been fetched.

The example here accepts `next_token` as a full query URL and rejects other formats. If your response uses a different format, confirm how to use it with [Mindtickle Support](mailto:support@mindtickle.com) before continuing. The continuation URL includes an `Id` filter based on the table's primary columns, `ModuleId`, `UserId`, and `SeriesId`, and the `LastUpdatedTime` column, with `LastUpdatedTime` taking priority. This filter excludes already fetched records; it does not guarantee snapshot consistency while data changes.

### Running a request with pagination

1. Run the following first query with the date filter and the `$pagination=true` option. Replace `START_DATE` with a date in YYYY-MM-DD format within the last 30 days.

   ```text
   https://admin.mindtickle.com/Odata.svc/LearnerModulePerformancesIncremental?$format=csv&$filter=Date ge 'START_DATE'&$pagination=true
   ```

2. The following is returned as output:

   1. A CSV file is created with the name in the `file_name` format.
   2. If `next_token` returns a query URL, that is the value is not `None`, there is more data to be fetched. Use the query URL returned by `next_token` to run another query. The query URL contains an additional `Id` filter.

3. If there are more records to be fetched, that is `next_token` is not `None`, use the query URL returned by `next_token` in step 2 to run another query. Keep its date filter unchanged. The following schematic example uses the same `START_DATE` as step 1; use the actual returned URL rather than constructing the `Id` cursor yourself:

   ```text
   https://admin.mindtickle.com/Odata.svc/LearnerModulePerformancesIncremental?$format=csv&$filter=Date ge 'START_DATE' and Id gt '1641851706001567027402678076815628'&$pagination=true
   ```

4. Repeat step 3 until the `next_token` value is `None` and there are no more records left to be fetched.

### LMPI pagination script

This sample accepts full-URL continuation tokens. It stops if a continuation URL is repeated, malformed, or points outside the configured table, preventing credentials from being sent to another destination.

This standalone sample uses the same CSV output, request timeouts, retries, and failure handling described in [Client-side pagination script](#client-side-pagination-script). Set your credentials, regional base URL, and query options before running it. The sample starts 7 days before the current UTC date to stay within the 30-day window; this is an example range, not a recommended synchronization schedule.

```python
import time
from datetime import datetime, timedelta, timezone
from pathlib import Path
from urllib.parse import urlencode, urlsplit

import requests
from requests.auth import HTTPBasicAuth

MAX_ATTEMPTS = 3
RETRY_DELAY = 60
REQUEST_TIMEOUT = (10, 120)


def get_query_url(base_url, options):
    return f"{base_url}?{urlencode({key: value for key, value in options.items() if value is not None})}"


def fetch_page(url):
    for attempt in range(1, MAX_ATTEMPTS + 1):
        try:
            with requests.get(
                url,
                auth=HTTPBasicAuth(username, password),
                timeout=REQUEST_TIMEOUT,
                allow_redirects=False,
            ) as response:
                response.raise_for_status()
                if response.status_code != 200:
                    raise ValueError("Expected HTTP 200; check the configured Reporting URL")
                count = int(response.headers.get('X-MTAPI-RecordCount', ''))
                if count < 0:
                    raise ValueError("X-MTAPI-RecordCount must be nonnegative")
                return response.content, count, response.headers.get('X-MTAPI-NextToken')
        except (requests.Timeout, requests.ConnectionError, requests.HTTPError) as error:
            if isinstance(error, requests.HTTPError) and error.response.status_code not in {
                429, 500, 502, 503, 504
            }:
                raise
            if attempt == MAX_ATTEMPTS:
                raise
            print(f"Request failed; retrying this page in {RETRY_DELAY} seconds")
            time.sleep(RETRY_DELAY)


def save_csv_page(content, file_name):
    target = Path(file_name)
    temporary = target.with_suffix('.csv.part')
    try:
        temporary.write_bytes(content)
        temporary.replace(target)
    finally:
        temporary.unlink(missing_ok=True)


def get_next_url(token, current_url, seen_urls):
    if token is None:
        return None
    continuation = urlsplit(token)
    expected = urlsplit(base_url)
    if (
        continuation.scheme != 'https'
        or continuation.netloc != expected.netloc
        or continuation.path != expected.path
        or not continuation.query
        or continuation.fragment
    ):
        raise ValueError('Expected a full continuation URL for the same Reporting table')
    if token == current_url or token in seen_urls:
        raise ValueError('The continuation URL repeated; pagination stopped')
    return token


table_name = 'LearnerModulePerformancesIncremental'
base_url = f'https://admin.mindtickle.com/Odata.svc/{table_name}'
username = 'username'
password = 'password'
start_date = (datetime.now(timezone.utc) - timedelta(days=7)).date().isoformat()
options = {
    '$format': 'csv',
    '$filter': f"Date ge '{start_date}'",
    '$select': None,
    '$pagination': 'true',
}


def run():
    url = get_query_url(base_url, options)
    seen_urls = set()
    total_rows = 0
    iteration = 0
    export_time = datetime.now().strftime('_%Y_%m_%d_%H_%M_%S_%f_')
    while url is not None:
        content, count, token = fetch_page(url)
        next_url = get_next_url(token, url, seen_urls)
        if count:
            last_row = total_rows + count
            file_name = f'{table_name}{export_time}{total_rows + 1}_to_{last_row}.csv'
            save_csv_page(content, file_name)
            total_rows = last_row
        seen_urls.add(url)
        url = next_url
        iteration += 1
        print(f'Total rows after iteration {iteration} = {total_rows}')
    print('Finished Processing')


if __name__ == '__main__':
    run()
```

## Related

- [Best practices](/docs/reporting/best-practices/): The data limits that make pagination necessary.
- [Use OData](/docs/reporting/use-odata/): The query options pagination builds on.
- [Tables: performance and completion](/docs/reporting/tables-performance-and-completion/): The tables you paginate, including the LMPI table.