How can I apply a filter to only allow only specific entity types to be passed into the DPH cloud pipeline queue?

Question

We need you to alter the physical configuration on our snowflake connectors to filter ONLY "VIN" entity types so we do not receive unnecessary data

Answer

• Make sure that dataFilteringEnabled is turned on in the physical tenant.  The dataFilteringEnabled property is a configuration setting that enables data filtering for the Snowflake adapter. When enabled, filtering applies to all adapters.

"dataPipelineConfig": {
      "enabled": true,
      "activityLogEnabled": true,
      "messagingName": "mpe-03-datapipeline-events",
      "adapters": [
          {
              "type": "snowflake",
              "enabled": true,
              "dataFilteringEnabled": true,   <<<<<<<<<<<<<<<<<<================
              "name": "VINPROD",
              "serializeInitialSourcesInCrosswalks": false,

• Ensure the L3 is changed for the specific entity type and attributes.  In the L3 configuration, dataPipelineConfig enables data filtering and specifies which attributes are received for different entity, interaction, and relation types. For example, it can be configured to receive only specific attributes from entityTypes/Employment or interactionTypes/ExhibitionEvent for a tenant.

 {
          "uri": "configuration/entityTypes/VIN",
          "label": "VIN",
          "description": "VIN",
          "typeColor": "#80D557",
          "typeImage": "images/defaultImage/no-loc.png",
          "typeGraphIcon": "images/graphIcon/organization-icon.png",
          "dataPipelineConfig": {
                "enabled": true
            },

• Make sure that the following is applied to the pipeline queue.  In a queue definition, objectFilter is a property that allows filtering messages based on the entire object content, such as type, attributes, and timestamps. For CRUD events, it filters by the entire object content, while for MERGE events, it filters by specific properties of the 'Winner' entity, like type and timestamps.

 {

                  "payloadType": "SNAPSHOT",

                  "type": "queue",

                  "provider": "dph",

                  "name": "mpe-03-datapipeline-events_",

                  "dtssQueue": false,

                  "groupingQueue": false,

                  "dataPipelineQueue": true,

                  "enabled": true,

                  "objectFilter": "equals(type,'configuration/entityTypes/VIN')",

                  "format": "KRYO"

              }