Why is initialSource an array in Delta Share entity tables?

In Delta Share entity streaming tables, initialSource inside the crosswalks column is an array of entity URIs (ARRAY<STRING>), so parse it as a list, not a single value.

Question

A Delta Share entity streaming table shows initialSource in the crosswalks column as a list, and some records hold more than one value. The same entity returned by the Reltio REST API shows a single value. Why do the two differ, and is the list expected?

Answer

In Delta Share streaming tables, crosswalks is an array of structs, and initialSource inside each struct is serialized as an array of entity URIs, for example ["entities/<id>"]. A one-element array is the normal form, so a list in the column does not by itself mean anything is wrong.

The REST API returns a simplified view of the same crosswalk, which is why you see a single value there. Always read initialSource from Delta Share as ARRAY<STRING>, never as a scalar string.

When it holds more than one value

Multi-value arrays are most likely to appear on entities that have been merged, where the surviving profile's crosswalk can trace back to more than one entity URI. Crosswalk updates and relation crosswalks may also contribute. Engineering has not confirmed which of these cases produce more than one value.

To check a specific record:

1. Take one entity that initialSource has several values in Delta Share.
2. Compare its crosswalks with GET /entities/{id} from the REST API.
3. Review that entity's merge history to see whether the extra URIs match merged entities.

Suggestions on how to parse it in Databricks

Important: Warranty Disclaimer - This SQL is provided to you free of charge, and on an "AS IS" basis, without any technical support or warranty of any kind from Reltio including, without limitation, a warranty of merchantability, fitness for a particular purpose and non-infringement.

Define initialSource as ARRAY<STRING>. Use the example that matches how your table stores crosswalks. Replace the <catalog>, <schema> and <entity_streaming_table> placeholders with your own names.

Option 1: crosswalks is stored as a JSON string

WITH parsed_entities AS (
  SELECT
    uri AS entity_uri,
    from_json(
      crosswalks,
      'ARRAY<STRUCT<
        type: STRING,
        value: STRING,
        sourceTable: STRING,
        initialSource: ARRAY<STRING>
      >>'
    ) AS parsed_crosswalks
  FROM `<catalog>`.`<schema>`.`<entity_streaming_table>`
),
exploded_crosswalks AS (
  SELECT entity_uri, explode(parsed_crosswalks) AS cw
  FROM parsed_entities
)
SELECT
  entity_uri,
  cw.type AS crosswalk_type,
  cw.value AS crosswalk_value,
  cw.sourceTable AS source_table,
  cw.initialSource AS initial_source_all,
  size(cw.initialSource) AS initial_source_count
FROM exploded_crosswalks;

Option 2: crosswalks is a native ARRAY<STRUCT<...>> column

SELECT
  uri AS entity_uri,
  cw.type AS crosswalk_type,
  cw.value AS crosswalk_value,
  cw.initialSource AS initial_source_all,
  size(cw.initialSource) AS initial_source_count
FROM `<catalog>`.`<schema>`.`<entity_streaming_table>`
  LATERAL VIEW explode(crosswalks) AS cw;

To list one row per URI in the array, add LATERAL VIEW explode(cw.initialSource) s AS initial_source_uri to either query. Do not rely on the position of a URI in the array: the order has not been documented.

Limits

• The Delta Share field reference lists crosswalks as ARRAY<STRUCT> but does not describe initialSource in detail.
• The SQL above is a customer-side customizable example. Reltio Customer Engineering cannot support custom queries; for help adapting them, contact Reltio Professional Support.
• This article covers Delta Share (Data Sharing). It does not cover the Data Pipeline for Databricks, where initialSource is controlled by the serializeInitialSourcesInCrosswalks setting.