Why is the GET API to gather unmapped values not return the full set of unmapped values?

Yes, your understanding is correct. When using the Reltio Reference Data Management (RDM) API to retrieve unmapped lookup values, the limit parameter in a GET request is capped at 1000 records per call. To retrieve more than 1000 unmapped values, you can employ one of the following methods:(docs.reltio.com)

Option 1: Use offset Pagination

For datasets with fewer than 10,000 unmapped values, you can paginate through the results using the offset parameter. Calculate the offset as offset = limit * page_number, where page_number starts from 0.

Example:

• First Page:

  GET https://rdm.reltio.com/unmapped/<RDM_Tenant_ID>?filter=equals(type,'AWARDS_CATEGORY')&sort=code&order=asc&limit=1000&offset=0

• Second Page:

  GET https://rdm.reltio.com/unmapped/<RDM_Tenant_ID>?filter=equals(type,'AWARDS_CATEGORY')&sort=code&order=asc&limit=1000&offset=1000

Continue incrementing the offset until no more records are returned.

Option 2: Use _dbscan for Large Datasets

For datasets exceeding 10,000 unmapped values, utilize the _dbscan endpoint, which supports cursor-based pagination using a scrollId.

Step-by-Step:

1. Initial Request:

   POST https://rdm.reltio.com/unmapped/<RDM_Tenant_ID>/_dbscan?filter=equals(type,'AWARDS_CATEGORY')&limit=1000
   

Note: The request body should be empty.

2. Subsequent Requests:

   Use the scrollId returned from the previous response in the body of the next request.

   POST https://rdm.reltio.com/unmapped/<RDM_Tenant_ID>/_dbscan?filter=equals(type,'AWARDS_CATEGORY')&limit=1000
   Body:
   {
     "scrollId": "<previous_scrollId>"
   }
   

Repeat this process until the response no longer contains a scrollId, indicating that all records have been retrieved.

References:

• Unmapped Values API Documentation(docs.reltio.com)
• Reltio Support Article on Unmapped Values(support.reltio.com)

If you need assistance with automating this process or handling the responses, feel free to ask.