Overview

When consuming Reltio data in Databricks, it is important to distinguish among an inactive entity, a merge loser, and a permanently deleted entity.

These scenarios are represented differently in the Databricks data share. Downstream consumers should therefore not rely solely on the deleted or deletedTime columns to determine whether an entity is no longer active or available.

Reltio Data Sharing with Databricks provides separate supported streaming tables, including:

• entity_<entity_type>
• links
• merges
• activities

The links and merges datasets are separate from the entity tables and should be used when processing merge lineage.

Problem

A downstream Databricks consumer may expect that whenever an entity is no longer active in Reltio:

deleted = true
deletedTime = <timestamp>

However, this is not a reliable assumption.

Testing performed for support case #139046 demonstrated three different outcomes depending on how the entity became inactive or was removed.

Scenario 1: End-Dated / Inactive Entity

An entity can become inactive without being deleted.

Reltio provides system-managed startDate and endDate properties to control object activeness. When the current date is later than the entity's endDate, the entity is considered inactive.

An entity can also be affected by a source crosswalk's deleteDate. Reltio describes deleteDate as representing a source record that has been removed or made inactive; when populated, the corresponding crosswalk is treated as end-dated.

Expected Databricks behavior

For the case validated in ticket #139046:

deleted = false
deletedTime = NULL
endDate = <inactivation timestamp>

This is expected because the entity was inactivated, not explicitly deleted.

Recommended downstream logic

Use endDate to identify entities that are no longer active.

Example:

SELECT
    id,
    endDate,
    updatedTime
FROM <catalog>.<schema>.entity_person
WHERE endDate IS NOT NULL
  AND endDate < unix_timestamp() * 1000;

Key point

Do not interpret:

deleted = false
deletedTime = NULL

as meaning the entity is necessarily active.

Always evaluate endDate when determining activeness.

Scenario 2: Entity Becomes a Merge Loser

Reltio merge operations create a surviving or winner entity and a loser entity. The attributes, crosswalks, roles, and other information from the losing entity are consolidated into the surviving entity. Reltio's merge API documentation also distinguishes the winner and loser URIs in the resulting merge event.

For Databricks consumers, merge information should be obtained from the dedicated merge-related datasets.

Reltio explicitly lists both:

links
merges

as supported Databricks streaming tables.

Expected Databricks behavior

Example

Winner: 1wOyDnN
Loser:  q96RykT

The following behavior was confirmed:

• The winner remained in entity_person.
• The loser was no longer present in entity_person.
• The merge was present in the merges table.
• The loser-to-winner mapping was present in the links table.

Therefore, a downstream consumer should not expect the merge loser to remain in entity_person with:

deleted = true
deletedTime = <timestamp>

Recommended downstream logic

Use merges for merge event/history information.

For example:

SELECT
    winnerId,
    loserId,
    to_timestamp(timestamp / 1000) AS merge_event_ts,
    to_timestamp(insertedTime / 1000) AS merge_ingest_ts
FROM <catalog>.<schema>.merges
WHERE loserId = '<loser_id>'
   OR winnerId = '<winner_id>';

Use links to determine loser-to-winner mappings:

SELECT
    objectType,
    winnerId,
    loserId,
    deleted,
    version,
    to_timestamp(timestamp / 1000) AS merge_event_ts
FROM <catalog>.<schema>.links
WHERE winnerId = '<winner_id>';

Key point

For merge processing:

entity table ≠ merge history

Use:

merges → merge event / lineage
links  → loser-to-winner mapping

rather than relying on deletedTime.

Scenario 3: Permanently Deleted Entity

Reltio provides an Entities API operation to delete an entity:

DELETE {TenantURL}/{entity object URI}

The official API documentation describes this operation as deleting the entity from the tenant.

Expected Databricks behavior

The entity 1wOxBrV was explicitly deleted through the API. After deletion, a query against entity_person returned:

No rows returned

The entity was therefore no longer represented in the current entity snapshot. There was no retained entity row containing:

deleted = true
deletedTime = <timestamp>

for this tested behavior.

Recommended downstream logic

If a downstream process must identify entities that have completely disappeared, compare the entity IDs from a previous snapshot with those in the current snapshot.

Conceptually:

SELECT previous.id
FROM previous_entity_snapshot previous

LEFT ANTI JOIN current_entity_snapshot current
    ON previous.id = current.id;

The resulting IDs represent records that existed previously but are no longer present in the current entity dataset.

Depending on the application, additional validation against merge data may be appropriate to prevent a merge loser from being incorrectly classified as an independent hard deletion.

Recommended Decision Logic

A downstream application can use the following decision model:

Is the entity present in entity_<entity_type>?
│
├── Yes
│   │
│   ├── endDate is NULL or in the future
│   │      → Current/active entity
│   │
│   └── endDate is in the past
│          → Inactive/end-dated entity
│
└── No
    │
    ├── Entity appears as loserId in merges/links
    │      → Merge loser
    │
    └── Entity is not represented by merge lineage
           → Potential deletion/removal
           → Validate using previous/current snapshot comparison

Fields and Tables to Use

Important Considerations for Downstream Consumers

Do not build downstream deletion logic equivalent to:

WHERE deleted = true

or:

WHERE deletedTime IS NOT NULL

and assume that it captures every entity that is no longer active.

Instead, distinguish the business state being detected:

Inactive?
    → endDate

Merged?
    → merges + links

Physically absent/deleted?
    → snapshot comparison

This distinction prevents downstream applications from incorrectly retaining inactive records, misclassifying merge losers as hard deletes, or failing to detect records that have disappeared from the current entity snapshot.

Reltio recommends querying the supported Databricks streaming tables rather than their underlying landing tables. This includes using entity_<entity_type>, links, and merges for downstream analytics and data engineering workloads.

Example Validation Queries

Check an entity's current state

SELECT
    id,
    deleted,
    deletedTime,
    endDate,
    updatedTime
FROM <catalog>.<schema>.entity_person
WHERE id = '<entity_id>';

Check whether the entity became a merge loser

SELECT
    winnerId,
    loserId,
    to_timestamp(timestamp / 1000) AS merge_event_ts
FROM <catalog>.<schema>.merges
WHERE loserId = '<entity_id>';

Check the loser-to-winner mapping

SELECT *
FROM <catalog>.<schema>.links
WHERE loserId = '<entity_id>';

Find currently inactive entities

SELECT
    id,
    endDate,
    updatedTime
FROM <catalog>.<schema>.entity_person
WHERE endDate IS NOT NULL
  AND endDate < unix_timestamp() * 1000;

Resolution / Expected Behavior

The behavior described above is expected for the scenarios validated in support ticket #139046:

End-dated entity: remains identifiable through endDate.

Merge loser: identify through merges and links; the surviving entity is represented in the current entity dataset.

Hard-deleted entity: may no longer have a row in the current entity snapshot; detect disappearance using previous/current snapshot comparison.

Accordingly, deleted and deletedTime should not be treated as a universal deletion-detection mechanism for Databricks consumers.