Question
We are seeing that snowflake connector sending data even if reltio data is not changing. We refreshed the data on 15th after that we have not loaded anything into reltio. I am attaching the S3 file which we got today. The relevant entity ids are '1EHGDbd', '2B8ShUQ'
Result in Snowflake
[
{
"deleted": false,
"potential_matches": [
{
"matchScore": 0,
"matchRules": [
"configuration/entityTypes/Organization/matchGroups/Rule1_1",
"configuration/entityTypes/Organization/matchGroups/Rule3"
],
"entityId": "1EHGDbd"
}
],
"manual_matches": [],
"entityId": "2B8ShUQ",
"not_matches": [],
"version": 2000000000,
"timestamp": 1698080394342,
"insertedTime": 1698080397771,
"objectType": "match"
},
{
"deleted": false,
"potential_matches": [
{
"matchScore": 0,
"matchRules": [
"configuration/entityTypes/Organization/matchGroups/Rule1_1",
"configuration/entityTypes/Organization/matchGroups/Rule3"
],
"entityId": "2B8ShUQ"
}
],
"manual_matches": [],
"entityId": "1EHGDbd",
"not_matches": [],
"version": 2000000000,
"timestamp": 1698080394342,
"insertedTime": 1698080397771,
"objectType": "match"
}
]
Answer
We can see that you did have activity on the two entities that you described. Here we see that 1EHGDbd had the following event that was processed.
{
"type": {
"name": "POTENTIAL_MATCHES_REMOVED"
},
"timestamp": 1698189101285,
"updatedBy": "IncrementalMatchThread",
"entity_uri": "entities/1EHGDbd",
"entity_type": "Organization",
"label": "MAESTRO SWISS CHOCOLATE SDN. BHD.",
"affectedEntityURIs": [
"entities/2B8ShUQ"
]
},
To store matches data type correctly we need to listen all matches events. If potential matches are removed we need to remove it from snowflake as well, for this we are listening all potential matches events
timestamp | eventId | eventType1 | queueAction |
---|---|---|---|
2023-10-24 20:51:21.717000 UTC | 1698179932126_DRPNa1m | M1_ENTITY_CHANGED | CREATED |
2023-10-24 20:51:21.746000 UTC | 1698179932126_DRPNa1m | M1_ENTITY_CHANGED | SENT |
2023-10-24 22:25:48.219000 UTC | 1698185149944_HBB5opL | ENTITY_CHANGED | CREATED |
2023-10-24 22:25:48.605000 UTC | 1698185149944_HBB5opL | ENTITY_CHANGED | SENT |
2023-10-24 23:11:41.091000 UTC | 1698179932126_DRPNa1m | M1_ENTITY_CHANGED | RECEIVED |
2023-10-24 23:11:45.995000 UTC | 1698189101285_WCnlb4i | ENTITIES_MATCHES_CHANGED | CREATED |
2023-10-24 23:11:47.293000 UTC | 1698189101285_WCnlb4i | ENTITIES_MATCHES_CHANGED | SENT |
2023-10-24 23:11:47.397000 UTC | 1698179932126_DRPNa1m | M1_ENTITY_CHANGED | PROCESSED |
2023-10-24 23:11:51.591000 UTC | 1698189101285_WCnlb4i | ENTITIES_MATCHES_CHANGED | RECEIVED |
2023-10-24 23:11:52.478000 UTC | 1698189101285_WCnlb4i | ENTITIES_MATCHES_CHANGED | RECEIVED |
2023-10-24 23:11:52.843000 UTC | 1698189101285_WCnlb4i | ENTITIES_MATCHES_CHANGED | DATAPIPELINE_PROCESSED |
2023-10-24 23:11:53.772000 UTC | 1698189101285_WCnlb4i | ENTITIES_MATCHES_CHANGED | PROCESSED |
2023-10-24 23:39:24.629000 UTC | 1698185149944_HBB5opL | ENTITY_CHANGED | RECEIVED |
2023-10-24 23:39:27.323000 UTC | 1698185149944_HBB5opL | ENTITY_CHANGED | PROCESSED |
Comments
Please sign in to leave a comment.