At approximately 4:30 AM CDT, September 4, 2018, users were no longer able to login to Measure Desktop, Measure Mobile, RFMS Mobile or RFMS Online Services. Users who were already logged into these applications could use them locally, but could not use cloud services such as product search, project sync, exporting to RFMS, or any other functions that required access to their RFMS database.
At approximately 10 PM CDT, September 4, 2018, access to these applications and their related services was fully restored.
Upon investigation early in the day, it was discovered that our provider of hosted data services for Measure and our mobile apps, Microsoft Azure, suffered a severe weather event at their South Central US data center located in San Antonio, TX. This weather event included lightning strikes immediately around the data center. To prevent data loss or corruption, the data center was powered down. As a result, traffic to this data center was diverted to other data centers around the world. However, the heavy amount of this diverted traffic overloaded the other data centers, causing their services to begin failing, thus affecting clients globally. Microsoft used Twitter and a Status page to provide updates every hour or two. By approximately 10 PM CDT, Microsoft engineers were able to balance traffic flow enough for normal operations to resume.
Throughout the day, clients of Microsoft Azure from around the world replied to Microsoft's Twitter posts to indicate that they were unable to use the Azure services critical to the operation of their business. RFMS faced the same issue.
At no time during this incident was client data vulnerable to loss, hacking or corruption.
RFMS Apps developers were able to implement a workaround at approximately 2 PM CST that allowed users to login to Measure Desktop to use it locally, but access to cloud services such as product search or projects saved in the cloud was not available.
Users who were already logged into Measure Mobile prior to the incident were able to use the app in offline mode throughout the day - however, products stored in the cloud or in their RFMS database could not be searched, projects could not be accessed from or saved to the cloud, nor could they be exported to RFMS.
RFMS Mobile was completely inaccessible throughout the incident.
RFMS Online Services
The RFMS Online Services website was completely inaccessible throughout the incident.
We empathize with you, our clients, as we understand that our applications are a crucial part of your business operations. Being without these for a full business day is unacceptable to us. Please be assured that we will do everything in our power to mitigate such an issue in the future.
If you have additional questions about this incident, please log a ticket with us.