Why am I not getting any events passed to my Snowflake connector?

Question

We are configuring the Snowflake Data Warehouse connector to publish MDM data into Snowflake through the console. We successfully created the pipeline, but it is not publishing any data to the blob storage even though there are changes in Reltio. 

Answer

  • Validate that streaming is enabled in the physical tenant configuration.
"streamingConfig": {
      "streamingEnabled": true, <<<===Make sure that it is set to true
      "streamingAPIEnabled": true, <<<===Make sure that it is set to true
      "analyzeOvChanges": false,
      "emptyStartEndRelationCrosswalks": false,
      "largeObjectsSupport": false,
      "skipPayload": false,
      "payloadType": "SNAPSHOT",
      "ovOnly": false,

 

  • Check the pipeline queue has been defined in the physical storage configuration.  If it is not set, follow the instructions in https://docs.reltio.com/en/integrate/reltio-data-pipelines/reltio-data-pipeline-for-snowflake/configure-a-snowflake-pipeline-in-the-console
 {
                  "payloadType": "SNAPSHOT",
                  "type": "queue",
                  "provider": "dph",
                  "name": "<environment>-datapipeline-events_<tenant>",
                  "dtssQueue": false,
                  "groupingQueue": false,
                  "dataPipelineQueue": true,
                  "enabled": true,  <<<<=====  Make sure that it is set to true
                  "format": "KRYO"
              }

 

"adapters": [
            {

                "type": "snowflake",
                "enabled": true,
                "dataFilteringEnabled": false,
                "name": "Snowflake",
                "serializeInitialSourcesInCrosswalks": false,
                "stagingDataRetentionInDays": 14,
                "cloudProvider": "AZURE",
                "azureConfig": {
                    "azureCredentials": {
                        "authMethod": "KEYS",
                        "storageAccountName": "<accountName>"
                  }
                },

                "ovOnly": false,
                "stagingBucket": "<queuebucket name>",
                "attributeFormat": "STANDARD",
                "useLegacyUpdatedTime": false
          }
        ]
POST https://<environment>-data-pipeline-hub.reltio.com/api/tenants/<tenant>/adapters/Snowflake/validate
  • Check to make sure that events are processing.
    • Check external queue

 

Was this article helpful?
0 out of 0 found this helpful

Comments

0 comments

Please sign in to leave a comment.