In the Data Loader, how can we use an EXISTING Job Definition, upload a NEW file to the EXISTING Job Definition, and then EXECUTE the EXISTING Job Definition?

Question

I am currently facing an issue while attempting to trigger a data loader job using the API provided by Reltio. I believe there might be a misconfiguration in the API call or the file upload process.

POST https://361-dataloader.reltio.com/dataloader/api/<tenantId>/project/<projectId>/jobs/run

During my attempts, I've observed that the API response contains a sourcePath different from the actual file I uploaded. The response shows:

"sourcePath": "eW6FFLP30ff8i9P/2690432e-af54-40f6-bcf8-b0013a5ee84b/Individual_LoanServicing.csv"

However, the expected sourcePath should be:

"sourcePath": "eW6FFLP30ff8i9P/2690432e-af54-40f6-bcf8-b0013a5ee84b/LoanServicing2.csv"

This discrepancy in the sourcePath might be causing the issue I'm facing. On the Reltio UI data loader status page, I've noticed that the job has errored out with the following status message: "ERROR: Job LoanServicing2 cancelled as no new files are available for processing."

If possible, could you provide guidance on the recommended and accurate procedure for uploading a file and successfully loading the data into Reltio using API calls?

Answer:

The following is the full set of steps to create a project, upload file to an existing job location and execute that job definition.

Step 1: Create new Dataloader Project Job Definition

POST https://console.reltio.com/dataloader/services/dataloader/<tenantId>/project

--header 'Environment: <environment>' \
--header 'Module: dataloader' \

Request Body

{
"loadType": "ENTITIES",
"checkForUpdates": true,
"enableAutoMerge": true,
"mappingDetails": null,
"name": "IssueReplication1",
"schedulingDetails": null,
"storageDetails": null,
"additionalAttributes": {
"alwaysCreateDCR": false
},
"updateType": "UPDATE_MERGED",
"progressStatus": "Upload",
"environment": "<environment>"
}

Step 2: Upload File from storage location

POST https://developer.reltio.com/services/dataloader/api/<tenantId>/storage/<storageId>/upload?replace=true&projectId=<project>&delimiter=,

--header 'Environment: <environment>' \
--header 'Module: dataloader' \
--form 'file=@"<fileName>"'

You will get the error message "ERROR: Job New data load job_200323 cancelled as no new files are available for processing" if you trying to replace an existing file. 

Step 3: Apply Mapping

PUT https://console.reltio.com/dataloader/services/dataloader/<tenantId>/project/<projectId>

Request Body

{
"checkForUpdates": true,
"enableAutoMerge": true,
"name": "IssueReplication1",
"updateType": "UPDATE_MERGED",
"storageDetails": {
"storageId": "<storageId>"
},
"mappingDetails": {
"mappingId": <mappingId>
},
"schedulingDetails": null,
"progressStatus": "Map",
"id": "c5870fdd-630d-4f8f-9076-10a005645611",
"environment": "test-usg",
"loadType": "ENTITIES",
"additionalAttributes": {
"alwaysCreateDCR": false
}
}

Step 4: Execute Data loader Job

POST https://developer.reltio.com/services/dataloader/api/<tenantId/project/<projectId>/jobs/run
--header 'Environment: <environment>' \
--header 'Module: dataloader' \

Request Body

{
"name": "IssueReplication1.0"
}'

Step 5: Replace File in EXISTING Job Definition

POST https://developer.reltio.com/services/dataloader/api/<tenantId>/storage/<storageId>/upload?replace=true&projectId=<projectId>&delimiter=,

Response body

--header 'Environment: <enviornment>' \
--header 'Module: dataloader' \
--form 'file=@"<file>"'

STEP 6: Execute Existing Job with the new file

POST https://developer.reltio.com/services/dataloader/api/<tenantId>/project/<projectId>/jobs/run
--header 'Environment: <environment>' \
--header 'Module: dataloader' \

Request body

{
"name": "IssueReplication2.0"
}'

Resources

https://developer.reltio.com/private/swagger.htm?module=Data%20Ingestion#/Data%20Load%20Job