This incident has now been resolved, and we will prepare a postmortem summary in the week ahead.
In short, we noticed some problems with the primary database yesterday evening causing an increase in API latency and transaction processing stopping. We began the recovery process within 5 minutes of the first notice, and in the process of recovering from it, inadvertently removed the disk storage for the primary database master. As a result, transaction processing was stopped for another minute while the system failed over to the backup master and a data recovery snapshot was taken.
We have recovered the data from the primary database from the backups and the database replication logs, and believe that no transactional data is missing, but during that time the reporting and transaction summary information for the affected transactions would have been unavailable, and any automated reports sent would have been out-of-date.
All systems are operating normally once more and the data should be available, and Ingresso would like to apologise to our partners for the impact of this incident.
Posted Dec 09, 2019 - 10:20 GMT
Monitoring
We have identified and fixed main source of the problem, we are currently monitoring API functionality to verify that all operations are back to normal.
Posted Dec 08, 2019 - 18:32 GMT
Identified
We have identified issue with our internal API, all connected distributors might experience intermittent problems with data retrieval and transaction processing. We are investigating the issue and working on a resolution