Why I am seeing TOO_MANY_CONCURRENT_REQUESTS_WITH_NEW_CROSSWALK error when I am performing an update?

Question:

Please note that the Team did a data load of 1382515, out of which 80567 failed.
 
The error produced in the error log
 

CommonException: Code: TOO_MANY_CONCURRENT_REQUESTS_WITH_NEW_CROSSWALK; Message parameters:
 [configuration/sources/VeevaCRM/Address/791510e32af05a6c10d4b39e0cf8e03f]. Too many 
requests are trying to update the data for the following crosswalks: 
configuration/sources/VeevaCRM/Address/791510e32af05a6c10d4b39e0cf8e03f..
 Too many concurrent requests are trying to update the data with same crosswalks. 
Please try again later.

 
Answer:

When you see the error message TOO_MANY_CONCURRENT_REQUESTS_WITH_NEW_CROSSWALK, the error happens when simultaneous requests contain the same crosswalk that is new for the system.

It happens when we try to save data into storage and see that another thread already modified it. To resolve a conflict we will try to repeat the operation and if that attempt fails, we throw an exception.

When we load files, some entities interfere with each other. It can be directly or indirectly. Indirectly, it means that you are updating addresses belonging to one HCP (and HCP has a reference attribute "Address"). In this case, some batches might be dropped as the threshold of retry attempts was exceeded.
 
Reltio's data loader looks through the entire file and forms the one entity update in case we have two in the file. But, even there, we can have a "Too many requests..." error (because of indirect updates).

Two approaches can be used by the platform to resolve them:

1. Apply our custom logic, which processes data so that the result should be similar to the execution of a single request containing all changes. This is the default behavior.

2. Repeat the request one more time. So, the result of resolving a conflict will be the same if two requests are executed one by one. However, we do not guarantee any order in which these requests will be executed. This behavior can be enabled in a physical configuration using the repeatAllOperationsOnConflict optional parameter. We use this approach when the logic of a partial override of nested attributes can be applied to the request for which a conflict was found. In general, it means that the request is a partial override, and the enableNestedPartialOverride option is true in a physical configuration.

Not every two simultaneous requests that update the same object cause a conflict. One request starts execution a little bit earlier and finishes quickly. Another request could spend some time on operational needs, so when it starts execution, the first one will already finish its execution. So, there will be no conflict, and the outcome will be the same if requests are executed individually.

When our system retries some request, it resets the time for it. So if some request was initiated a few milliseconds earlier than the other, but the conflict was founded for the first one and we repeat it, then in the history, we will see that this change was applied later.

A repeatAllOperationsOnConflict optional parameter can be enabled on your physical tenant configuration to resolve conflicts arising from multiple requests.  This parameter was introduced in 2021.3.   The new optional parameter is enabled for the tenant, and our platform will repeat requests when simultaneous updates cause a conflict.

For example, five retries during 1 minute means it will retry only if it fails again and again five times.

• To get the current set of optional parameters, perform the following API

GET http://host:port/reltio/tenants/{tenantId}/optionalParameters/

• The below request will upsert Optional Parameters in the given Tenant configuration. If optional parameters are not present, it will add and if the same parameters are already present then it will overwrite them.

PUT http://host:port/reltio/tenants/{tenantId}/optionalParameters

Request Body :

{  “optionalParameters”: {
        “tenantSize”: “<tenantSize>”,
        “provisionedBy”: “<provisedBy>”,
        “repeatAllOperationsOnConflict”: true
    }

}