Question
We are configuring the Snowflake Data Warehouse connector to publish MDM data into Snowflake through the console. We successfully created the pipeline, but it is not publishing any data to the blob storage even though there are changes in Reltio.
Answer
- Validate that streaming is enabled in the physical tenant configuration.
"streamingConfig": {
"streamingEnabled": true, <<<===Make sure that it is set to true
"streamingAPIEnabled": true, <<<===Make sure that it is set to true
"analyzeOvChanges": false,
"emptyStartEndRelationCrosswalks": false,
"largeObjectsSupport": false,
"skipPayload": false,
"payloadType": "SNAPSHOT",
"ovOnly": false,
- Check the pipeline queue has been defined in the physical storage configuration. If it is not set, follow the instructions in https://docs.reltio.com/en/integrate/reltio-data-pipelines/reltio-data-pipeline-for-snowflake/configure-a-snowflake-pipeline-in-the-console
{
"payloadType": "SNAPSHOT",
"type": "queue",
"provider": "dph",
"name": "<environment>-datapipeline-events_<tenant>",
"dtssQueue": false,
"groupingQueue": false,
"dataPipelineQueue": true,
"enabled": true, <<<<===== Make sure that it is set to true
"format": "KRYO"
}
"adapters": [
{
"type": "snowflake",
"enabled": true,
"dataFilteringEnabled": false,
"name": "Snowflake",
"serializeInitialSourcesInCrosswalks": false,
"stagingDataRetentionInDays": 14,
"cloudProvider": "AZURE",
"azureConfig": {
"azureCredentials": {
"authMethod": "KEYS",
"storageAccountName": "<accountName>"
}
},
"ovOnly": false,
"stagingBucket": "<queuebucket name>",
"attributeFormat": "STANDARD",
"useLegacyUpdatedTime": false
}
]
- Ensure the adapter is valid by using the following API and checking for a status of 200. If not, walk through the connector setup again at https://docs.reltio.com/en/integrate/reltio-data-pipelines/reltio-data-pipeline-for-snowflake for the correct cloud platform with the customer.
POST https://<environment>-data-pipeline-hub.reltio.com/api/tenants/<tenant>/adapters/Snowflake/validate
- Check to make sure that events are processing.
- Check external queue
-
- Run the following API for a specific entity to validate the processed event across the pipeline.
-
GET https://<environment>-data-pipeline-hub.reltio.com/api/tenants/<tenantid>/monitoring/_entityMonitoring?entityId=<entityID>
Comments
Please sign in to leave a comment.