Why I am seeing TOO_MANY_CONCURRENT_REQUESTS_WITH_NEW_CROSSWALK error when I am performing an update?

Question:

Please note that the Team did a data load of 1382515, out of which 80567 failed.
 
The error produced in the error log
 

CommonException: Code: TOO_MANY_CONCURRENT_REQUESTS_WITH_NEW_CROSSWALK; Message parameters:
 [configuration/sources/VeevaCRM/Address/791510e32af05a6c10d4b39e0cf8e03f]. Too many 
requests are trying to update the data for the following crosswalks: 
configuration/sources/VeevaCRM/Address/791510e32af05a6c10d4b39e0cf8e03f..
 Too many concurrent requests are trying to update the data with same crosswalks. 
Please try again later.

 
Answer:

When you see the error message TOO_MANY_CONCURRENT_REQUESTS_WITH_NEW_CROSSWALK, the error happens when simultaneous requests contain the same crosswalk that is new for the system.

It happens when we try to save data into storage and see that another thread has already modified it. To resolve a conflict, we will try to repeat the operation, and if that attempt fails, we throw an exception.

When we load files, some entities interfere with each other. It can be done directly or indirectly. Indirectly, it means that you are updating addresses belonging to one HCP (and HCP has a reference attribute "Address"). In this case, some batches might be dropped as the threshold of retry attempts was exceeded.
 
Reltio's data loader looks through the entire file and forms the one entity update in case we have two in the file. But, even there, we can have a "Too many requests..." error (because of indirect updates).

Two approaches can be used by the platform to resolve them:

1. Apply our custom logic, which processes data so that the result should be similar to the execution of a single request containing all changes. This is the default behavior.

2. Repeat the request one more time. So, the result of resolving a conflict will be the same if two requests are executed one by one. However, we do not guarantee any order in which these requests will be executed. This behavior can be enabled in a physical configuration using the repeatAllOperationsOnConflict optional parameter. We use this approach when the logic of a partial override of nested attributes can be applied to the request for which a conflict was found. In general, it means that the request is a partial override, and the enableNestedPartialOverride option is true in a physical configuration.

Not every two simultaneous requests that update the same object cause a conflict. One request starts execution a little bit earlier and finishes quickly. Another request could spend some time on operational needs, so when it starts execution, the first one will already finish its execution. So, there will be no conflict, and the outcome will be the same if requests are executed individually.

When our system retries some request, it resets the time for it. So if some request was initiated a few milliseconds earlier than the other, but the conflict was founded for the first one and we repeat it, then in the history, we will see that this change was applied later.

A repeatAllOperationsOnConflict optional parameter can be enabled on your physical tenant configuration to resolve conflicts arising from multiple requests.  This parameter was introduced in 2021.3.   The new optional parameter is enabled for the tenant, and our platform will repeat requests when simultaneous updates cause a conflict.

For example, five retries during 1 minute means it will retry only if it fails again and again five times.

• To get the current set of optional parameters, perform the following API

GET http://host:port/reltio/tenants/{tenantId}/optionalParameters/

• The below request will upsert Optional Parameters in the given Tenant configuration. If optional parameters are not present, it will add and if the same parameters are already present then it will overwrite them.

PUT http://host:port/reltio/tenants/{tenantId}/optionalParameters

Request Body:

{ “optionalParameters”: {
        “tenantSize”: “<tenantSize>”,
        “provisionedBy”: “<provisedBy>”,
        “repeatAllOperationsOnConflict”: true
    }

}

Enabling repeatAllOperationsOnConflict is essentially a retry-on-conflict mechanism. It helps reduce rejects like TOO_MANY_CONCURRENT_REQUESTS_WITH_NEW_CROSSWALK, but at the cost of more work per conflict, which can impact performance under certain conditions.

What it does

• When multiple requests try to create/update the same crosswalk or entity at the same time, the platform throws:

  • TOO_MANY_CONCURRENT_REQUESTS_WITH_NEW_CROSSWALK

• If repeatAllOperationsOnConflict = true in the physical config:

  • Instead of failing immediately, the platform repeats the whole operation when it detects that conflict.

  • Data Loader/ROCS jobs that previously had many rejects can see fewer rejects, because those records are retried automatically.

So functionally, it trades errors for retries.

Performance implications

Per-conflict cost increases

• Without this flag:

  • Conflicting request → fails quickly → 1 attempt, 1 write try.

• With this flag:

  • Conflicting request → detects conflict → repeats full operation (and possibly multiple times, depending on contention).

• Net effect:

  • For each conflict, CPU, DB writes, and queue work increase (because the same row is processed more than once).

  • If conflicts are rare → negligible impact.

  • If conflicts are frequent → can significantly increase total work.

Can increase pressure on DynamoDB / throttling in bad patterns

• Tenants will see the following in LogDNA.

  • TOO_MANY_CONCURRENT_REQUESTS_WITH_NEW_CROSSWALK

  • plus DATABASE_THROTTLING / ProvisionedThroughputExceededException

• After enabling repeatAllOperationsOnConflict:

  • The TOO_MANY_CONCURRENT… errors reduced,

  • But database throttling and performance issues existed because the root cause is one of the following:

    • high concurrency,

    • same crosswalk updated in parallel,

    • DB at or near capacity limits.

When the underlying problem is high concurrency + hot crosswalks, automatic retries can:

• Increase the number of write attempts during peak contention,

• Prolong the time the system spends under throttling,

• Potentially make jobs slower (more retries, more backoff) even if fewer records end in final error.

Throughput / duration of data loads

For a Data Loader or ROCS job:

• If conflicts are low:

  • Enabling the flag has minimal performance impact.

  • It may reduce manual re-runs of reject files.

• If conflicts are common (e.g., same IDs in many rows, many jobs hitting same entities):

  • The job may:

    • run longer 

    • consume more credits/CP,

    • increase DynamoDB write load and lock contention.

  • You may see:

    • more time with queues busy,

    • more DB throttling if close to capacity.

When it’s beneficial vs risky

Good use cases:

• Initial load or delta loads where:

  • You see occasional TOO_MANY_CONCURRENT_REQUESTS_WITH_NEW_CROSSWALK rejects,

  • DynamoDB is not throttling heavily,

  • The tenant isn’t already near the resource or object size limits.

• Goal is to:

  • eliminate the need to repeatedly reload large reject files,

  • smooth over rare race conditions.

Risky scenarios:

• You already see:

  • DATABASE_THROTTLING / ProvisionedThroughputExceededException

  • Lots of hot entity/crosswalk contention,

  • Large objects and heavy match/merge activity.

• In that context, this flag:

  • doesn’t fix the bottleneck (capacity + design),

  • and can increase total work during the same stress window.

Best‑practice guidance

If you enable repeatAllOperationsOnConflict:

Treat it as a safety net, not a primary scaling tool.

• Root fixes are still:

  • Lowering loader concurrency (threads/batch size),

  • Avoiding the same crosswalk in many simultaneous jobs/files,

  • Sequencing feeds that touch the same entities.

Monitor closely right after enabling:

• Data Loader job durations and OPS,

• API error mix:

  • Did TOO_MANY_CONCURRENT_REQUESTS_WITH_NEW_CROSSWALK go down?

  • Any increase in DATABASE_THROTTLING or latencies?

• DynamoDB throttling metrics / CP consumption.

If you see heavy throttling or long job times:

• Keep the flag enabled,

• But reduce contention:

  • Reduce parallelism of Data Loader jobs,

  • Break files so that the same crosswalks/entities aren’t hammered at once,

  • Consider staging/ordering updates (e.g., load parents then children, or sources in sequence).