Why are some entities missing from the export?

Question:

In PROD we do hourly exports and ingest into Snowflake and there are some critical EoD batches happening for the down stream channels. We noticed few changed entities are NOT coming in Reltio exports. 

• Update to the profile happened around 03/04/2023 12:56:50 PM EST
• Our export ran around 3/04/2023 1 PM EST and did not pickup this entity

Answer:

We can see this behavior if there is identified consistency issue between secondary and primary storage.  We can divide consistency in the following groups:

• CRUD operations generated events that were not delivered to secondary storages which cause secondary storage inconsistencies. In this case, we usually suggest a full consistency task execution.

POST  https://<environment>-dataload.reltio.com/reltio/api/<tenantId>/esCassandraConsistencyCheck?fixInconsistency=true&fixVersionConflicts=true&compareVersions=true&distributed=true&taskPartsCount=6 

• Secondary storages are not up to date because not all events delivering and events are in queues. Customers often treat this as inconsistency, but this issue resolves automatically.  We would suggest that 3 to 5 minutes would be a good rule of time to expect a final update of an event.

It is also possible to perform the following to check the consistency of any events on entity updates that occured within a period of time and then perform the export.

• This step will make sure that any changed entities where the change occured from <startTime> to <endTime> is consistent.  It does add time to your hourly process. But, I am giving you a solution to avoid the sporatical issue that appears to have occured. 

POST {ApplicationURL}/api/{tenantId}/findChangedEntitiesDuringLastDataload?distributed=true&taskPartsCount=4&updatedSince=<startTimeStamp>&updatedTill=<endTimeStamp>&entityType=<entityType>

•  Followed by an export with the same dates applied in updatedSince and updatedTill.