Reltio SQS Event Delivery Behavior: Delays, Ordering, and Duplicate Messages

Overview

Customers using Reltio external queues (AWS SQS) for delta data extraction may observe the following behaviors:

• Events arriving later than the corresponding update timestamps in Reltio
• Events for the same object appearing out of order
• Duplicate messages for the same event
• Missing objectVersion in some payloads

This article explains why these behaviors occur and provides guidance for downstream system design.

Key Concepts

1. Asynchronous Event Processing

Reltio publishes events to external queues asynchronously. This means:

• Event generation, processing, and delivery are decoupled
• Delivery timing is not guaranteed to match the transaction time

Impact:

• Events may be delayed (in some cases up to several minutes)
• Arrival order in SQS may not match the update order in Reltio

2. Non-FIFO Delivery Model

Reltio Behavior

Reltio does not guarantee FIFO (first-in-first-out) ordering of events.

AWS SQS Standard Queue Behavior

SQS standard queues:

• Do not guarantee ordering
• Use at-least-once delivery

Impact:

• Events may arrive out of order
• The same message may be delivered more than once

Event Ordering Guidance

Do NOT use:

• updatedTime
• createdTime
• SQS arrival order

These are not reliable for sequencing in concurrent or asynchronous systems.

Recommended approach:

• Use objectVersion to determine the ordering of events for the same object

Note: If objectVersion is missing from payloads; contact Reltio Support to validate configuration and payload settings.

Understanding Duplicate Messages

Duplicate messages can originate from multiple valid scenarios:

1. SQS Redelivery (Expected)

SQS standard queues provide at least once delivery.

Common causes:

• Message not deleted before visibility timeout expires
• Consumer failures or retries

Result: Consumers may receive the same message multiple times.

2. Event Reprocessing (Same eventId)

The same event may be reprocessed internally within Reltio.

Possible triggers:

• Retry mechanisms
• Queue re-consumption
• Consumer restart or offset misalignment

Characteristics:

• Same eventId
• Same checksum
• Event processed multiple times

3. Background Consistency Republish (Expected)

Reltio includes a background consistency mechanism that ensures all events are fully processed.

If an event is detected as incomplete or inconsistent:

• The system republishes the event
• The same eventId may be reused

Purpose: Ensure data consistency across systems

Impact: Duplicate messages may be observed downstream

4. Multiple Valid Events (Merge Scenarios)

Certain operations (e.g., entity merges) generate multiple legitimate events.

Example event types:

• ENTITY_CHANGED
• MERGE_ENTITY_CHANGED

Characteristics:

• Different eventId values
• Represent distinct operations

Impact: These are not duplicates but separate valid events

Best Practices for Downstream Systems

To correctly process Reltio event streams, downstream systems should:

1. Implement Idempotency

• Ensure that processing the same event multiple times does not create inconsistencies
• Use eventId or entity identifiers to deduplicate

2. Handle Out-of-Order Events

• Do not assume events arrive in order
• Always reconcile using objectVersion

3. Use Version-Based Processing

• Apply changes only if the incoming objectVersion is newer than the last processed version

4. Manage SQS Consumption Properly

• Delete messages after successful processing
• Configure appropriate visibility timeout
• Monitor retries and dead-letter queues (DLQ)

Troubleshooting Checklist

If unexpected behavior persists:

1. Verify SQS configuration:
   • Visibility timeout
   • Retry policies
2. Validate consumer logic:
   • Idempotency implementation
   • Proper message deletion
3. Check event payloads:
   • Presence of eventId
   • Presence of objectVersion
4. Review event patterns:
   • Same eventId → likely reprocessing
   • Different eventId → separate events
5. Engage Reltio Support if:
   • objectVersion is consistently missing
   • Suspected true duplicate publishing with evidence of repeated SENT events for the same eventId

Proper downstream handling using idempotency and version-based sequencing ensures reliable data processing despite these conditions.