Sat 7 September: RESOLVED: Soup Mail is now operating normally.
We are now reviewing the incident and planning changes to our server recovery processes to reduce the likelihood and impact of similar incidents in the future.
8:15pm: the reindexing process is nearing completion and we are performing tests to ensure Soup Mail is operating optimally.
4:25pm: the reindex is continuing but is slower than expected. We currently estimate full recovery in around 2-3 hours’ time.
1:00pm: the reindex is progressing steadily but slowly. We still estimate full recovery in around 5-6 hours’ time.
10:07am: a full recovery is currently expected within 8 hours. Emails will progressively become available during this time.
The team is assessing ways in which this can be expedited. Emails added via sync may take slightly longer than usual to save to Soup Mail during this period.
10:00am: the reindexing process is progressing steadily, and we are monitoring to get a clear idea on when it will be complete.
09:50am: reindexing occurs from oldest to newest, so as emails gradually become visible, it is expected that it is older emails that will be visible initially.