Reporting is back up. At approximately 6:45 am HST our Solr server experienced an error writing to AWS Elastic Block Storage. This issue unfortunately failed to trigger our monitoring systems. Retreaver is architected so that a failure in our search servers will not affect call processing or any other functionality. Once aware of the failure, the server was rebooted to restore access to EBS, and the search index was updated. We regret that our monitoring did not flag the error and we apologize for the interruption to our customers. We're working to improve our monitoring so that this type of incident can be handled more expediently in the future.
Posted Jan 11, 2018 - 13:31 EST
Monitoring
Reporting is back up and we're fixing the issue of missing calls in the search index - zero data has been lost. We'll be fully reindexed shortly. Our apologies for the inconvenience.
Posted Jan 11, 2018 - 13:18 EST
Investigating
Calls continue to be processed as normal. We are working on a fix.