PushPushGo behind the scene

Application crash - summary of activities

Last updated - November 22 2023 4:12 PM
Dawid Mędrek
Reading time - 2 min
PushPushGo behind the scene

Application crash - summary of activities

Last updated - November 22, 2023
author push notification Dawid Mędrek
2 min Read
push communication and marketing tips

During the night of Tuesday 9 of March and Wednesday 10 of March, a fire broke out in the OVHCloud server room, resulting in damage to two data centers, which directly caused the inability to send notifications, collect subscribers and the complete suspension of the PushPushGo application

The situation has now been brought under control and full operation of the app has been restored. As PushPushGo, we have made every effort to restore functionality as quickly as possible despite the serious and unforeseen situation. 

Thanks to the hard work of our IT team, we were able to do this in under 24 hours, which as you can see from the current market situation in relation to the outage, is an unexpectedly good result. 

I would like to thank all our customers for their patience and understanding in the face of the situation we found ourselves in. Your words of support have been invaluable to us, thank you.

Below are some highlights from yesterday's outage and answers to the most frequently asked questions.

 

Timeline

00:30 CET

The first minor incident occurred - we received a notification from Page Duty - Several servers were down. Our Administrators have started working to fix the infrastructure.

00:47 CET

First notification of fire in SGB2 - as published by OVH. Disaster recovery plan implemented.

04:10 CET

90% of the infrastructure has stopped working - (firefighting action) data center SGB1, SGB2, SGB3, SGB4, where we had a large part of the repositories. The other data centers were not affected. 

06:00 - 18:00 CET

Full infrastructure restoration and application source code recovery which consisted of preparing and running a copy of the database (to accommodate full load traffic),

Around 14:00 CET first update:

  • Ability to log into the application,
  • Collection of subscribers has resumed,
  • We do not process dispatches (intentional),
  • Infrastructure ready.

18:30 CET

Sending campaigns available again - senders/requesters started running again.

FAQ

  • During what period new subscriptions were not collected?

    Subscriptions for notifications were not possible from 00:30 to 13:00 CET.

  • Have the planned campaigns from yesterday been sent?

    According to our announcements yesterday, no campaigns scheduled between 00:30 and 18:00 CET were sent.

  • Has any data been lost?

    No historical data regarding subscribers, mailings nor billing has been lost.

author photo
Dawid Mędrek

CEO at PushPushGo

Try out push notifications from PushPushGo

Create an account and start testing!

web and mobile push notifications

Push your business forward with PushPushGo

Test for free Start a 14-day trial!