What is Internal Error In Elastic Search while using offset and max parameter in GET entities APIs? How to resolve the Issue?

Problem

What is Internal Error In Elastic Search while using offset and max parameters in GET entities APIs? How to resolve the Issue?

Solution

The customer was facing an error when trying to use the GET “Entities” API and when it results in fetching records beyond an offset of 10,000. Below are the details of the request:

 

Method - GET
Header - Authorization
        Content-Type : Application/json
Parameters
        filter=(gt(createdTime,"1548958099000") and (equals(type,'configuration/entityTypes/HCP') or equals(type,'configuration/entityTypes/HCO'))
       select=uri,crosswalks,type
        max=100
        offset=10000

 

Whenever the offset exceeds 10,000, Reltio sends back an error response saying “Internal Error occurred while operating with the search system, message – Internal error in Elastic search. Please view the application log for details”. Even if we change the “max” parameter, the error is thrown at an offset of 10k.

See error screenshot below:

cid:image001.png@01D4BEE0.10AF1B50

There is a limitation of 10000 in API response using max and offset.

For example, if you run the below query, with max and offset = 10+9990 which is equal to 10000, then we are getting the response.



{tenantURL}/entities?filter=(gt(createdTime,"1548958099000") and (equals(type,'configuration/entityTypes/HCP') or equals(type,'configuration/entityTypes/HCO'))&select=uri,crosswalks,type&max=10&offset=9990



mceclip0.png


But if the combination exceeding 10000 like max=10&offset=9991 or max=11&offset=9990, its throwing error.

{
"severity": "Error",
"errorMessage": "Search system access error",
"errorCode": 1002,
"errorDetailMessage": "Internal error occurred while operating with search system, message - Internal error in Elastic Search. Please view application log for details"
}

mceclip1.png


mceclip2.png

 

Workaround

Here we can go for _scan API. We can use the cursor to get the next set of data.

References

https://docs.reltio.com/activitylogapi/searchwithcursor.html

 

 

 

Was this article helpful?
0 out of 0 found this helpful

Comments

0 comments

Please sign in to leave a comment.