What happened
Information Systems & Computing (ISC) was conducting a planned network upgrade over the weekend. The upgrade was scheduled for January 4th from 3-3:30am. This update would cut off all network connectivity for this thirty minute period for most, if not all PMACS servers including the ones that host Way to Health. However, the servers did not come back online until 10:30am. Additionally, based on our logs, the offline period began as early at 1:00am, not 3:30am as had been communicated. This meant that while the servers themselves were running, they could not communicate to any external internet service to collect incoming data, or send out messaging during that time period.
The estimated offline period for Way to Health was: 1/4/2020 1:00am - 10:30am.
What does this mean for your clinical program or research trial?
In most cases, incoming data was queued and then collected late. Outbound messaging was queued and sent late, or failed to send entirely. A detailed description by incoming data and outgoing messaging is listed below.
Incoming data (Epic feeds)
Epic feeds were unaffected because they are within the internal network and the servers were not shut down. Only communication with the external world was cut off.
Incoming data (MMS/SMS)
Incoming text and picture messages were not delivered to the Way to Health platform. Participants received an automated response from Twilio that read “Due to a system error, your message was not received. Please try again in a few minutes.”
Those queued messages are accessible by the Way to Health Team. If you wish to receive an Excel file for your project, please place a service request via our Help Desk. The volume of inbound messages during this window of time was very low.
Incoming data (Fitbit, Withings, Hidrate Spark, Propeller?)
Data feeds for these devices were queued by the vendor during the offline window and sent to Way to Health when internet access was restored. You can view this in Manage Data and filter on a Time Stamp date of 1/3/2020, sorting by “Received in WTH” date/time to get a full history of incoming data.
Outbound messaging (SMS)
Outbound SMS messages were queued by Way to Health during the offline window. When internet access was restored, the platform attempted to send out the entire queue of messages. Because of the large volume across the platform, some messages that originally failed were sent, while others remained in a failed state. To view a list of all failed messages for your study or clinical program, got to Manage Participants > Text Messages. Filter by timestamp between 1/4/2020 12:00am and 1/4/2020 10:30am, and by status of “Failed”. You can also export a full list of text messages from Reports and find the Text Messages export under General Category.
Outbound messaging (email)
Outbound email messages were queued by Way to Health during the offline period. When internet access was restored, the platform sent those emails.
Linking late data with feedback messaging
Because the evaluation time point for most daily device data (e.g. steps, fluid intake) occurs during the overnight hours, it is very possible that participants received late incorrect messaging on January 4th regarding their progress in the study or program. For example, an 8am morning message about reaching a goal may have evaluated the participant as non-compliant ,or not reaching their goal, simply because the data didn’t come through until the next morning when Way to Health came back online. The system had already evaluated and the message was queued for delivery prior to the new device data coming in after 10:30am that morning.
Most study and clinical teams have standard processes for manually adjusting data as needed (e.g. adjusting points, levels, financial incentives). You should continue to follow these processes. If you need assistance, please place a service request via our Help Desk.
Lessons learned and next steps
We are evaluating a range of options to see what can be done to mitigate issues like this going forward. We also intend to work with PMACS and Corp IS to come up with some approaches and strategies to avoid such scenarios.
We do apologize for the inconvenience that this must have caused you. Please do not hesitate to reach out to us with any requests for help or support.