ManageWP rotates the keys used for synchronization (communication with websites added to ManageWP dashboard) in regular time intervals. This is done primarily for security purposes, so even if an unauthorized person somehow gained the access to those keys, they would be likely to expire before malicious or unauthorized action could be performed against those websites.
On April 11, one of the queries that writes new key data in our database failed, and the exception wasn’t managed properly. As a consequence entity manager instance associated with a persistence context has shut down, effectively preventing any further write commands.
This issue was rectified as soon as it was discovered, but unfortunately it caused 4,200 websites under our management to go out of sync, i.e. use new keys, while old keys remained in our databases. As a result, communication to them was disrupted, which manifested as those sites being disconnected from users’ dashboards. To fix this, you have to deactivate/reactivate the ManageWP Worker plugin, and click the Reconnect website button on your ManageWP dashboard. And if you’re hiding the Worker plugin, you have to log in via FTP and rename the wp-content/plugins/worker folder, in order to deactivate the plugin and unhide it.
We are making several changes that will prevent this kind of behavior in the future: failsafes that will limit the damage to a singel website, and better logging that will allow us to resolve the sync issue without your involvement.
We apologize deeply for the service disruption you had. We are aware how important this service is to you – our customers and your businesses. We will do everything we can to learn from this event and use it to improve our stability even further.