Why is SFDC no longer synchronizing with an error message of "MUTUAL_AUTHENTICATION_FAILED:"?

Question

SFDC has stopped synchronizing data from Reltio to Salesforce.  If we check the logging messages using the following API.

GET {SFDCConnectorURL}/logging?tenant=<tenantId>&from=<date when the synchronization stopped>&logLevel=ERROR

We see the following message.  What does this mean and why has SFDC stopped synchronizing?

{
  "insertId": "........Lc4FoZ1uOcS3zdx.58LFVg.s",
  ...
    "message": "env:361<Vg93ZrGQ5rWtHqF+00D4C0000008yhHUAQ> <w7>: CNR_E219 Salesforce operation connApexGet (345 ms) failed: MUTUAL_AUTHENTICATION_FAILED: This session could not be mutually authenticated for use with the REST API\n    at HttpApi.getError (/usr/src/app/node_modules/jsforce/lib/http-api.js:250:13)\n    at /usr/src/app/node_modules/jsforce/lib/http-api.js:95:22\n    at tryCallOne (/usr/src/app/node_modules/promise/lib/core.js:37:12)\n    at /usr/src/app/node_modules/promise/lib/core.js:123:15\n    at flush (/usr/src/app/node_modules/asap/raw.js:50:29)\n    at process._tickCallback (internal/process/next_tick.js:61:11)"
  },
 
  "timestamp": "2021-06-02T22:36:39.822999954Z",
  "severity": "ERROR",
  "logName": "projects/sfdc-connector-prod/logs/winston_log",
  "receiveTimestamp": "2021-06-02T22:36:39.885938916Z"
}

Answer

We should check the status of the SDFC connector using the following API

GET {SFDCConnectorURL}/status

If the following is shown in the status.

{
 "module": "SalesForce",
 "syncEnabled": true,
 "status": "failure",
 "connectivity": {
 "status": "failure",
 "error": "This session could not be mutually authenticated for use with the REST API"
                }
            },

This would indicate that the salesforce account password has expired or is no longer valid.