How to Check and Troubleshoot Match Token Generation for an Entity

Reltio matching tools provide two dedicated API endpoints to check token generation for a particular entity. These endpoints help identify root causes for common match token issues, specifically:

• Overcollisioned tokens 

• A large number of tokens generated per entity 

1. Get match tokens

   • Purpose: Displays the list of tokens generated for a specific entity.

   • Use Case: Identifies potential issues with the total count/volume of tokens.

2. Get match tokens detailed

   • Purpose: Displays the list of tokens and explains the exact composition and mechanism of how each token was generated.

   • Use Case: Performs in-depth root cause analysis for overcollision and token proliferation.

When an overcollisioned token is produced, analyze the token structure and sample entities

1. Obtain examples of the overcollisioned token and the entity IDs producing it.

2. Call the Get match tokens detailed endpoint for an entity ID to inspect the resulting JSON.

3. Example Analysis:

   • Token value: denmark:a-s.

   • The token contains two attribute values separated by a colon (:):

     • First part (denmark): Generated from the exact attribute City ].

     • Second part (a-s): Generated from the fuzzy attribute Organisation/Name.

   • Finding: The token generation process produced a-s from the organization name, which functions as a noisy word (e.g., standard business entity abbreviations/suffixes).

4. Resolution:

   • Apply an organization names cleanser to strip noisy words before the tokenization process begins.

This occurs when a match rule consists solely of low-cardinality attributes:

1. Obtain examples of the overcollisioned token and the entity IDs producing it.

2. Execute the Get match tokens detailed endpoint.

3. Example Analysis:

   • Token value: denmark:123456.

   • Composition breakdown.

     • First part (denmark): Generated from exact attribute City .

     • Second part (123456): Generated from exact attribute Zip.

   • Finding: The token relies exclusively on two low-cardinality attributes. In a large dataset, tokens generated under this rule will collide too often.

4. Resolution:

   • Add high-cardinality attributes to the match rule, such as Emails, Phone, or a combination of First and Last names.

A rule may produce a total token count that exceeds the operational limit.

1. Obtain examples of entities producing token counts beyond the platform threshold (by default, 300 tokens).

2. Run the Get match tokens detailed endpoint to inspect the combinatorial token parts.

3. Example Analysis:

   • The rule contains several parts contributing to token generation, multiplying out to:
     $$\text{Total Tokens} = 31 \times 1 \times 1 \times 14 = 434 \text{ tokens}$$
     This total (434) exceeds the default threshold of 300.

   • Review why individual parts produce high counts (e.g., Part 1 produces 31 token parts, Part 4 produces 14 token parts).

   • Looking at the attribute generating 14 token parts (City), the configuration specifies:

 Because fuzzy token generation is enabled on `City`, it generates multiple variations [[Match tokens generation check](https://reltio.jira.com/wiki/spaces/RCTL/pages/2459468374/Match+tokens+generation+check)].

     {
       "attribute": "configuration/entityTypes/Organization/attributes/Addresses/attributes/City",
       "class": "com.reltio.match.token.FuzzyTextAndNumberMatchToken"
     }

4. Resolution:

   • Change token generation for City from fuzzy to exact (since City should typically be a cleansed value).

   • This reduces the City token multiplier from 14 down to 1:
     $$\text{New Total} = 31 \times 1 \times 1 \times 1 \times 1 = 31 \text{ tokens}$$
     This brings the total well below the 300-token threshold.

Documentation links

Diagnostic endpoints

1. Get match tokens: Get Match Tokens for an Entity. It documents GET /entities/{entityId}/matchTokens.
2. Get match tokens detailed: Generate Match Tokens for an Existing Entity. It documents GET /entities/{entityId}/matchTokensDetailed, and it’s the page to use for all three scenarios.

Related pages

• Overview of the entity-level token APIs: Working with Entities
• Token generation for an entity or configuration you supply (useful for testing a fix, such as switching City from fuzzy to exact, before changing the tenant config): Generate Match Tokens for a Specific Entity
• Scenario 1 (noisy words, organization names cleanser): Remove Noise Words and Match cleansers
• Scenarios 2 and 3 (low-cardinality attributes, fuzzy vs. exact token classes): Design your Match Tokenization Scheme and Match Token Generation
• The 300-token default in Scenario 3: Inspections for Tokens. It lists thresholdManyTokens as 300 for all match groups, or 150 for a single match group.