Problem
What is Internal Error In Elastic Search while using offset and max parameters in GET entities APIs? How to resolve the Issue?
Solution
The customer was facing an error when trying to use the GET “Entities” API and when it results in fetching records beyond an offset of 10,000. Below are the details of the request:
Method - GET
Header - Authorization
Content-Type : Application/json
Parameters
filter=(gt(createdTime,"1548958099000") and (equals(type,'configuration/entityTypes/HCP') or equals(type,'configuration/entityTypes/HCO'))
select=uri,crosswalks,type
max=100
offset=10000
Whenever the offset exceeds 10,000, Reltio sends back an error response saying “Internal Error occurred while operating with the search system, message – Internal error in Elastic search. Please view the application log for details”. Even if we change the “max” parameter, the error is thrown at an offset of 10k.
See error screenshot below:
There is a limitation of 10000 in API response using max and offset.
For example, if you run the below query, with max and offset = 10+9990 which is equal to 10000, then we are getting the response.
{tenantURL}/entities?filter=(gt(createdTime,"1548958099000") and (equals(type,'configuration/entityTypes/HCP') or equals(type,'configuration/entityTypes/HCO'))&select=uri,crosswalks,type&max=10&offset=9990
But if the combination exceeding 10000 like max=10&offset=9991 or max=11&offset=9990, its throwing error.
{
"severity": "Error",
"errorMessage": "Search system access error",
"errorCode": 1002,
"errorDetailMessage": "Internal error occurred while operating with search system, message - Internal error in Elastic Search. Please view application log for details"
}
Workaround
Here we can go for _scan API. We can use the cursor to get the next set of data.
References
https://docs.reltio.com/activitylogapi/searchwithcursor.html
Comments
Please sign in to leave a comment.