Pagination

The Reporting API supports pagination, so you can send successive API requests and fetch a larger dataset. Client-side pagination with the skip and top options works for all Reporting API tables. The LearnerModulePerformancesIncremental (LMPI) table has its own pagination option, which fetches up to 5,000,000 records.

Client-side pagination

Client-side pagination is available for all customers by default.

How client-side pagination works

The Reporting API supports client-side pagination with the skip and top options to fetch the data in batches when the data is large and cannot be fetched in a single go. Every time you run the script, the data is sorted on the primary key and the script skips the records that are already fetched. When no records have been fetched at the start, skip is 0, and the script runs until there are no records left to fetch.

Records inserted during an export may be absent until a later run. Treat each export as a changing dataset and reconcile successive exports by the table’s keys; do not assume a consistent snapshot across requests.

Best practices

• You do not need to apply sorting on top of pagination, because sorting is provided within the pagination implementation.
• When you use the top option to fetch top records, the value must not be greater than 300,000 for the CourseObjects, QuickUpdateObjects, and AssessmentObjects tables, and not greater than 100,000 for the LearnerModulePerformances table.

How the skip value changes

Consider the following three consecutive queries. The difference between them is the value of the skip option. When you use the skip option in a query, the full data is sorted after filtering on the primary key of the table, and records are skipped based on the skip value. In the first query there are no records fetched yet, so skip is 0. In the subsequent queries, the skip value increases as records are fetched.

First query, with top=100000 and skip=0:

https://admin.mindtickle.com/Odata.svc/CourseObjects?$format=csv&$filter=StartTime ge '2021-01-01 00:00:00' and StartTime le '2021-12-31 23:59:59'&$top=100000&$skip=0

Second query, with top=100000 and skip=100000, because 100,000 records are already fetched:

https://admin.mindtickle.com/Odata.svc/CourseObjects?$format=csv&$filter=StartTime ge '2021-01-01 00:00:00' and StartTime le '2021-12-31 23:59:59'&$top=100000&$skip=100000

Third query, with top=100000 and skip=200000, because 200,000 records are already fetched:

https://admin.mindtickle.com/Odata.svc/CourseObjects?$format=csv&$filter=StartTime ge '2021-01-01 00:00:00' and StartTime le '2021-12-31 23:59:59'&$top=100000&$skip=200000

The script runs consecutive queries until the X-MTAPI-RecordCount token value in the header is 0. That is, the script execution completes when there are no more records to fetch.

After processing each query, a CSV file is created, with the file name in a predefined format. Each run uses a timestamp, including microseconds, in its file names. A page is first written to a temporary .csv.part file and then renamed to .csv after the write completes. A failed run leaves earlier completed pages in place and does not automatically resume; keep output from separate runs apart to avoid importing overlapping exports. The file name format is:

table_name + str(datetime.now().strftime("_%Y_%m_%d_%H_%M_%S_%f_")) + str(first_row_num) + '_to_' + str(last_row_num) + '.csv'

Where table_name is CourseObjects, first_row_num is 1, last_row_num is 100000, and str(datetime.now().strftime("_%Y_%m_%d_%H_%M_%S_%f_")) is 2022_01_17_12_58_18_123456. For those values, the file name is CourseObjects_2022_01_17_12_58_18_123456_1_to_100000.csv.

Client-side pagination script

Use this standalone Python sample to export CSV pages. Install requests, set your Reporting credentials and regional base URL, and edit options for your query. Keep $format set to csv. The request timeout is 10 seconds to connect and 120 seconds without received data, not a total export deadline. Each page gets at most three attempts for connection failures, timeouts, and HTTP 429, 500, 502, 503, or 504 responses. Other failures stop the script. A completed page is saved before pagination advances; Finished Processing appears only after normal completion.

import time
from datetime import datetime
from pathlib import Path
from urllib.parse import urlencode

import requests
from requests.auth import HTTPBasicAuth

MAX_ATTEMPTS = 3
RETRY_DELAY = 60
REQUEST_TIMEOUT = (10, 120)


def get_query_url(base_url, options):
    return f"{base_url}?{urlencode({key: value for key, value in options.items() if value is not None})}"


def fetch_page(url):
    for attempt in range(1, MAX_ATTEMPTS + 1):
        try:
            with requests.get(
                url,
                auth=HTTPBasicAuth(username, password),
                timeout=REQUEST_TIMEOUT,
                allow_redirects=False,
            ) as response:
                response.raise_for_status()
                if response.status_code != 200:
                    raise ValueError("Expected HTTP 200; check the configured Reporting URL")
                count = int(response.headers.get('X-MTAPI-RecordCount', ''))
                if count < 0:
                    raise ValueError("X-MTAPI-RecordCount must be nonnegative")
                return response.content, count, response.headers.get('X-MTAPI-NextToken')
        except (requests.Timeout, requests.ConnectionError, requests.HTTPError) as error:
            if isinstance(error, requests.HTTPError) and error.response.status_code not in {
                429, 500, 502, 503, 504
            }:
                raise
            if attempt == MAX_ATTEMPTS:
                raise
            print(f"Request failed; retrying this page in {RETRY_DELAY} seconds")
            time.sleep(RETRY_DELAY)


def save_csv_page(content, file_name):
    target = Path(file_name)
    temporary = target.with_suffix('.csv.part')
    try:
        temporary.write_bytes(content)
        temporary.replace(target)
    finally:
        temporary.unlink(missing_ok=True)


table_name = 'CourseObjects'
base_url = f'https://admin.mindtickle.com/Odata.svc/{table_name}'
username = 'username'
password = 'password'
options = {
    '$format': 'csv',
    '$top': 10000,
    '$select': None,
    '$filter': None,
    '$orderby': None,
}


def run():
    total_rows = 0
    iteration = 0
    export_time = datetime.now().strftime('_%Y_%m_%d_%H_%M_%S_%f_')
    while True:
        url = get_query_url(base_url, {**options, '$skip': total_rows})
        content, count, _ = fetch_page(url)
        if count == 0:
            break
        last_row = total_rows + count
        file_name = f'{table_name}{export_time}{total_rows + 1}_to_{last_row}.csv'
        save_csv_page(content, file_name)
        total_rows = last_row
        iteration += 1
        print(f'Total rows after iteration {iteration} = {total_rows}')
    print('Finished Processing')


if __name__ == '__main__':
    run()

Pagination for the LMPI table

The Reporting API supports pagination for the LearnerModulePerformancesIncremental (LMPI) table. With the pagination script you can pull up to 5,000,000 records.

• Pagination with the $pagination option is supported only for the LearnerModulePerformancesIncremental table.
• If your API requests do not exceed 100,000 records, you do not need to implement pagination and can use the LearnerModulePerformancesIncremental table normally. You must implement pagination if your API requests exceed 100,000 records.
• With the pagination script you can pull up to 5,000,000 records across successive requests. Each generated CSV file can contain a maximum of 100,000 rows. For example, if there are 4,251,795 records in total, 43 CSV files are created: 42 files with 100,000 records each, and 1 file with the remaining 51,795 records.

The date filter must be within the last 30 days. The dated URLs below illustrate query syntax; replace their dates before use. Obtain the initial data dump before starting incremental extraction.

To use pagination, add the $pagination=true option to the OData query and apply the date filter. For example:

https://admin.mindtickle.com/Odata.svc/LearnerModulePerformancesIncremental?$format=csv&$filter=Date ge '2025-03-17'&$pagination=true

Once you run the query, look at the value returned by next_token, which is resp.headers.get('X-MTAPI-NextToken'), to know whether there is more data to be fetched:

• If next_token returns a query URL, that is the value is not None, there is more data to be fetched. Use the query URL returned by next_token to run another query. Run consecutive queries in the script until the next_token value is None.
• If next_token is None, all data has been fetched.

The example here accepts next_token as a full query URL and rejects other formats. If your response uses a different format, confirm how to use it with Mindtickle Support before continuing. The continuation URL includes an Id filter based on the table’s primary columns, ModuleId, UserId, and SeriesId, and the LastUpdatedTime column, with LastUpdatedTime taking priority. This filter excludes already fetched records; it does not guarantee snapshot consistency while data changes.

Running a request with pagination

1. Run the following first query with the date filter and the $pagination=true option. Replace START_DATE with a date in YYYY-MM-DD format within the last 30 days.

   https://admin.mindtickle.com/Odata.svc/LearnerModulePerformancesIncremental?$format=csv&$filter=Date ge 'START_DATE'&$pagination=true

2. The following is returned as output:

   1. A CSV file is created with the name in the file_name format.
   2. If next_token returns a query URL, that is the value is not None, there is more data to be fetched. Use the query URL returned by next_token to run another query. The query URL contains an additional Id filter.

3. If there are more records to be fetched, that is next_token is not None, use the query URL returned by next_token in step 2 to run another query. Keep its date filter unchanged. The following schematic example uses the same START_DATE as step 1; use the actual returned URL rather than constructing the Id cursor yourself:

   https://admin.mindtickle.com/Odata.svc/LearnerModulePerformancesIncremental?$format=csv&$filter=Date ge 'START_DATE' and Id gt '1641851706001567027402678076815628'&$pagination=true

4. Repeat step 3 until the next_token value is None and there are no more records left to be fetched.

LMPI pagination script

This sample accepts full-URL continuation tokens. It stops if a continuation URL is repeated, malformed, or points outside the configured table, preventing credentials from being sent to another destination.

This standalone sample uses the same CSV output, request timeouts, retries, and failure handling described in Client-side pagination script. Set your credentials, regional base URL, and query options before running it. The sample starts 7 days before the current UTC date to stay within the 30-day window; this is an example range, not a recommended synchronization schedule.

import time
from datetime import datetime, timedelta, timezone
from pathlib import Path
from urllib.parse import urlencode, urlsplit

import requests
from requests.auth import HTTPBasicAuth

MAX_ATTEMPTS = 3
RETRY_DELAY = 60
REQUEST_TIMEOUT = (10, 120)


def get_query_url(base_url, options):
    return f"{base_url}?{urlencode({key: value for key, value in options.items() if value is not None})}"


def fetch_page(url):
    for attempt in range(1, MAX_ATTEMPTS + 1):
        try:
            with requests.get(
                url,
                auth=HTTPBasicAuth(username, password),
                timeout=REQUEST_TIMEOUT,
                allow_redirects=False,
            ) as response:
                response.raise_for_status()
                if response.status_code != 200:
                    raise ValueError("Expected HTTP 200; check the configured Reporting URL")
                count = int(response.headers.get('X-MTAPI-RecordCount', ''))
                if count < 0:
                    raise ValueError("X-MTAPI-RecordCount must be nonnegative")
                return response.content, count, response.headers.get('X-MTAPI-NextToken')
        except (requests.Timeout, requests.ConnectionError, requests.HTTPError) as error:
            if isinstance(error, requests.HTTPError) and error.response.status_code not in {
                429, 500, 502, 503, 504
            }:
                raise
            if attempt == MAX_ATTEMPTS:
                raise
            print(f"Request failed; retrying this page in {RETRY_DELAY} seconds")
            time.sleep(RETRY_DELAY)


def save_csv_page(content, file_name):
    target = Path(file_name)
    temporary = target.with_suffix('.csv.part')
    try:
        temporary.write_bytes(content)
        temporary.replace(target)
    finally:
        temporary.unlink(missing_ok=True)


def get_next_url(token, current_url, seen_urls):
    if token is None:
        return None
    continuation = urlsplit(token)
    expected = urlsplit(base_url)
    if (
        continuation.scheme != 'https'
        or continuation.netloc != expected.netloc
        or continuation.path != expected.path
        or not continuation.query
        or continuation.fragment
    ):
        raise ValueError('Expected a full continuation URL for the same Reporting table')
    if token == current_url or token in seen_urls:
        raise ValueError('The continuation URL repeated; pagination stopped')
    return token


table_name = 'LearnerModulePerformancesIncremental'
base_url = f'https://admin.mindtickle.com/Odata.svc/{table_name}'
username = 'username'
password = 'password'
start_date = (datetime.now(timezone.utc) - timedelta(days=7)).date().isoformat()
options = {
    '$format': 'csv',
    '$filter': f"Date ge '{start_date}'",
    '$select': None,
    '$pagination': 'true',
}


def run():
    url = get_query_url(base_url, options)
    seen_urls = set()
    total_rows = 0
    iteration = 0
    export_time = datetime.now().strftime('_%Y_%m_%d_%H_%M_%S_%f_')
    while url is not None:
        content, count, token = fetch_page(url)
        next_url = get_next_url(token, url, seen_urls)
        if count:
            last_row = total_rows + count
            file_name = f'{table_name}{export_time}{total_rows + 1}_to_{last_row}.csv'
            save_csv_page(content, file_name)
            total_rows = last_row
        seen_urls.add(url)
        url = next_url
        iteration += 1
        print(f'Total rows after iteration {iteration} = {total_rows}')
    print('Finished Processing')


if __name__ == '__main__':
    run()

Related

• Best practices: The data limits that make pagination necessary.
• Use OData: The query options pagination builds on.
• Tables: performance and completion: The tables you paginate, including the LMPI table.