We're back and fully recovered. All asynchronous webhooks have been fired.
Posted Feb 28, 2017 - 18:20 EST
Update
We have resumed firing pixels/webhooks. We'll update this issue as soon as we're caught up.
Posted Feb 28, 2017 - 17:46 EST
Update
We're still unable to write to S3, so asynchronous webhooks continue to be delayed. Users will be unable to upload audio prompts, download reports, or listen to call recordings until this is resolved. All other systems are go. We'll update this issue shortly.
Posted Feb 28, 2017 - 16:24 EST
Update
We're still waiting for Amazon to fully recover. Prompt recordings which are not cached by our telephony provider are unable to be played and may lead to call handling failures. We suggest temporarily switching your campaigns to use text-to-speech. Pixel fires and call recordings are currently delayed. Once S3 comes back online we'll get caught up. You may also experience slowness in our UI since we rely on a couple third-party libraries such as New Relic and Hubspot which we now know are hosted on S3.
Posted Feb 28, 2017 - 16:03 EST
Update
S3 is back up and we're working on recovery. Webhooks will start firing momentarily.
Posted Feb 28, 2017 - 15:36 EST
Update
Prompt recordings which are not cached by our telephony provider are unable to be played and may lead to call handling failures. We suggest temporarily switching your campaigns to use text-to-speech. Pixel fires and call recordings are currently delayed. Once S3 comes back online we'll get caught up. You may also experience slowness in our UI since we rely on a couple third-party libraries such as New Relic and Hubspot which we now know are hosted on S3.
Posted Feb 28, 2017 - 13:43 EST
Update
Webhooks/pixel fires are currently delayed due to an outage at Amazon S3. Our HAR files, which log web traffic between our server and the webhook servers, are stored on S3. We're actively monitoring the situation and will update shortly.
Posted Feb 28, 2017 - 13:24 EST
Monitoring
We've brought additional capacity online to deal with the high volume of traffic, asynchronous webhooks/pixel fires will be caught up shortly. Synchronous fires are not experiencing an issue and all other systems continue to operate normally.