Protect your sites from hackers and boost performance with Sucuri’s Junior Dev Security Bundle — now $500 off.

Weekend Downtime for a Major Infrastructure Upgrade

UPDATE (Oct 9, 5am EDT): The upgrade is complete and we are back up, you can log into your ManageWP dashboard.

This weekend we’ll be taking down ManageWP Orion for some major upgrade that’s long overdue. The 48h downtime will start on Saturday, October 8th at 3am EDT, and last until Monday, October 10 at 3am EDT.

Why the Whole Weekend?

ManageWP Orion is on Amazon cloud infrastructure. It allows us to infinitely scale it without sacrificing performance. Unfortunately, the database does not scale that easily, and in the past several months it caused a couple of short downtimes. So far we’ve been MacGuyvering fixes, but for a permanent solution we need to take it down and perform a major upgrade.

The upgrade will ensure a smooth and consistent experience in the next year (we’ll figure something else to tinker with by then). The actual upgrade should take less than 20 hours, but we’d rather underpromise and overdeliver, than the other way around.

If you want to go into more technical details, we’re mainly looking to resolve the scaling issue where we would be able to add more servers without a drop in performance. Right now the database scales inefficiently, i.e. if we double the server power, we only get a 10% increase in the capacity. We need to redesign the distribution process and dump/reimport the database, which will take up a huge chunk of the downtime.

What’s Going to Be Down?

We’re putting the Orion dashboard in maintenance mode. Classic dashboard will be unavailable as well, since it partially relies on the Orion database. The website itself will be up, but signup and registration will be disabled.

It’s important to note that ManageWP services will not be running during the upgrade. It effectively means that Uptime Monitor will not notify you of up/down events, and that scheduled backups and checks will not run. If you’re not already storing backups on your own off-site storage (it’s good practice to have backups on two separate locations), I recommend downloading a backup to your computer, so you could be ready to restore your website during the ManageWP downtime.

Needless to say, the upgrade will not affect your websites in any way.

How Can I Reach You If Orion Is Down?

Our Customer Happiness team is here for you. Our contact form will be working, and you can reach us directly at helpdesk@managewp.com

What’s the Downtime in My Time Zone?

worldtimebuddy.com is an awesome time zone conversion tool. Just click the link and add your city/time zone to see when the downtime starts.

How Will I Know When Orion Is Up?

We’ll update this article when the dashboard is back online, and also post regular updates on Twitter.

I Hate You.

This upgrade is really important for both you and us, and the sooner we do it, the better for everyone. I understand that this might interrupt your daily flow, and we’ll be happy to give you a bear hug full of love when we meet in real life.

Thank you for your patience and understanding!

Nemanja Aleksic Avatar

55 responses

  1. kenny@phnx21designs.com Avatar
    kenny@phnx21designs.com

    Thanks for the detailed heads-up! Btw, the “I hate you.” subhead… LMAO!! 🙂

  2. Erwin Avatar
    Erwin

    Cool Guys. Go for it! Looking forward to that ‘bear hug’ 😉

  3. brianjosephking@gmail.com Avatar
    brianjosephking@gmail.com

    Thanks for the details. It would have been nice to get an email about this so that I could have planned better. Also, in the future if backups aren’t going to even be available in some capacity it would be good to be able to download these in some alternate method or transfer the most recent backup directly from ManageWP to my own S3/Google Drive/Dropbox account and avoid having to download to my local machine first.

    I’ll excuse you for now because I love the “I HATE YOU” section, but I do hope you consider the above feature request very seriously.

    Cheers, keep up the good work.

  4. Torben Heikel Vinther Avatar
    Torben Heikel Vinther

    What others say 🙂
    P.S. I LOVE YOU

  5. ttownsend Avatar
    ttownsend

    I have to 2nd the Email request…. I have been out of loop for a few days and just logged in today….If I had not logged in today I would have missed this. On a desperate request. I would really like to have access to Orion from a Mobile App…this would help resolve the above email issue too as you could Opt for Push Notifications….I would be very interested from a technical perspective on your AWS issues. I am currently working with firm that Big Data is pretty much their middle name hah….migrating to AWS. Have you looked into Hadoop or Redshift clusters as a way to deal with your issues ?

  6. ttownsend Avatar
    ttownsend

    BTW you guys ROCK 🙂

  7. justin.matt.web@gmail.com Avatar
    justin.matt.web@gmail.com

    I love that you’re providing updates, but a warning directly inside of ManageWP would have been extremely helpful.. we’re now delaying all of our work because of the pretty much nonexistent warning.

  8. Pat Avatar
    Pat

    Hey folks,
    How can you turn down the system without any notifications?
    Never experienced this from you before but it really made my weekend more troublesome for the work I have planned myself.
    As you failed to notify us, a link on the maintenance page would be appreciated.
    No, no hugs here and the “hate you” statement is quite childish.

    Irony, just a feeling from the past from something abbreviated GD…

    Sad.

  9. todd@972creative.com Avatar
    todd@972creative.com

    Reading this article only after experiencing the [rather unknown] planned down time is frustrating. You guys tend to be “notice-happy” in the menu bar. I’d suggested for system outages to enable a much more prominent alter (think ‘HelloBar’ or ‘EU Cookie Notice’ styles). Someone also suggested [in the comments] to link to this article from the “undergoing maintenance” message on the dashboard. That would be helpful.

  10. gil@tech-center.com Avatar
    gil@tech-center.com

    I have to delay what I’m doing. Yep wasn’t notified but that’s fine. It’s for our good I hope. Hopefully more notification in the future.

  11. alexander@virtuallycreative.ca Avatar
    alexander@virtuallycreative.ca

    We better get an email letting us know the service is back up…We shouldn’t have to check a blog post to see when things are back to normal.

    With almost all of us being developers, we understand the importance of upkeep, but the lack of effort in communicating to your userbase was completely unacceptable. With the amount of developers relying on your platform, it would be nice to know you care enough to at least try to inform us, as best as you can, that this level of upkeep needs to occur and to give us a heads up so important project items weren’t planned.

    So, which department do I send the invoice for all the money I lost this weekend from not being able to access your service?

  12. stein.david@gmail.com Avatar
    stein.david@gmail.com

    Cool – i’m going to stop paying for while this upgrade happens – my business bank account and finances are undergoing an infrastructure overhaul. That’s cool, right? But seriously, I agree w/ previous posters – some notice, other than hidden dashboard messages would be nice. I remember getting many emails when Orion was launching and you needed us to update our address etc and this seems like i’m not the only one who was surprised to log in and find the site not just down but down for 2 days?… Did I miss this?

  13. Jay Clark Avatar
    Jay Clark

    This is NOT the way it is done. You do all of your changes in a Developmental site, get the bugs out and then Migrate the old Database into the new structure. The only downtime is while the DB is upgrading. If we could do the Nielson Rating database that way with minimal downtime… unless your DB is 20 TB or greater, you could do the same thing. Even 12 hours would have been acceptable.

    The thing is we are addicted to ManageWP and I do not have a password to log in to any website with. Thankfully, my Employees do. This is a rookie move.

  14. stein.david@gmail.com Avatar
    stein.david@gmail.com

    How about a full screen modal pop up or something that doesn’t let you access the Orion dashboard until you read the message and click – ok, I have read that Manage WP will be offline for a really long time (in developer’s time)? Thank you for upgrading though – glad to know Manage WP will be getting better. It’s pretty good now so getting better is a good thing. The down time isn’t the issue – it’s the lack of notification – that wrecks a weekend’s worth of work and income or at least cuts it in half. Knowing before hand would allow us to schedule accordingly.

  15. Karin Avatar
    Karin

    Love you guys – and your humor, Nemanja! I didn’t see the warning, either, but I must admit, I rarely check the little bell thing. Normally there’s nothing there that I need to worry about. Fortunately for me, I don’t have anything huge going on this weekend, but I can understand if I had planned to do something major, this would be very disappointing. But – live & learn. Just realize that all these folks that are mad – are only so because your product is so terrific that they depend on it! You are a real important piece of my business and I appreciate you. May the switchy thing go easily and well for you! blessings!!

  16. Seth Avatar
    Seth

    Um… Shame about the typo. Perhaps during the maintenance the techies can correct MacGuyvering to MacGyvering. And, in a spare five minutes, sit back and think “Gee, this is the dumbest not-cool move we have ever done.” If I said to my boss, hey I am gonna take your website down for the weekend, I will outta door before I finished the sentence.

    Also, perhaps the techies’ boss could consider this: your clients are techies who have to maintain sites. And WP updates can be hit and miss, and blow-up sites. That, after all, is why we subscribe to the service: to minimize the pain. So when is the best time to do an update in case it goes awol? The weekend! So, having zip notice I find myself in my office unable to do my weekend’s tasks. Thanks guys!

  17. Alexander Avatar
    Alexander

    Hi Guys, I love your service…in the meanwhile it’s an important part of my own business. Cross my fingers for the update that all would work fine. Alexander P.S. Try pushcrew.com as notification service…i was not very happy to see this morning that i cant migrate the prototypes for my customers

  18. david.bolthouse Avatar
    david.bolthouse

    Well, I guess we have to learn the hard lesson that we cannot trust your service will always be available. Not good. You should have updated offline then moved it over instead of blocking us out for a weekend, when many of us work. I log in every day and had no clue you were doing this all weekend until I really needed to get in here this morning and found it not available.

  19. Mundo Doido Avatar
    Mundo Doido

    Hi Guys, I love your service… Thanks for all.

  20. Dan Avatar
    Dan

    You’ve got to be kidding me?! Why am I only finding out about this now when I try and log in? In what world do you live in where it didn’t occur to you to email us and make sure we all knew this was going to happen?! And then put massive notices in big red letters in the dashboard in the preceding week??! It’s all very well saying, hey, make sure you download a backup just in case, but I didn’t know you were going to do this so that is 1000% useless. I am absolutely STUNNED by how you guys have handled this. I get that you needed to make the update, but it’s not acceptable that I had to find out about it when I try and log in.

  21. Amanda Avatar
    Amanda

    I’m disappointed there was not a better notification effort about this downtime prior to attempting to log in and finding it down. This is the kind of event that businesses who rely on your service need time to plan for. I was going to move my website from staging to production using the clone tool this weekend (the entire reason I purchased backups at all – I already have another backup system). Now I will not be able to do so until Monday and it will affect my marketing schedule. This is a letdown.

  22. ccarter@emicrobranding.com Avatar
    ccarter@emicrobranding.com

    It seems that you guys are getting pretty adept at generating hate mail. Reminds me of early Facebook. But here’s the deal. You’ve been working hard as hell for a very long time build a platform and a service that people absolutely love and depend on. So you’re not a start up anymore. Stop acting like it.

  23. Todd Lash Avatar
    Todd Lash

    +1 On the Email Request. I had planned a migration this weekend and my client is counting on me. I had no idea the service would be offline.

    Not Happy.

  24. Paul Avatar
    Paul

    I’m adding my voice to the list of the unimpressed – just so you’re aware of the breadth of impact your inadequately announced outage has caused. Having had one test on your service for a week or so, I was planning to add all of my remaining sites to your service this weekend. Given how essential your service is to what we offer to clients, losing trust in your judgement from this experience make’s me second-guess whether that’s a sensible thing to do.

  25. Fred Sessoms Avatar
    Fred Sessoms

    You are awarded winner of WTF of the Year

  26. billc_lists@greenbuilder.com Avatar
    billc_lists@greenbuilder.com

    Knuckleheaded move, as I’m sure you’re now aware. (not that I haven’t made a few in my day…) And now you know: downtime for planned maintenance means you have to let everyone know in one or more very obvious ways, well in advance.

    I’m just happy that it’s so far been a fairly quiet weekend and there haven’t been any critical security fixes to deploy so far.

  27. Richard Avatar
    Richard

    I don’t know why every one is so upset to the point of getting vulgar about it. Surely, there could have been other ways to do the update and other ways to notify us, but ever since the new price structure and the blazing, wonderful Orion came out, I’m happy to be able to just use it with my team.

  28. Nemanja Aleksic Avatar
    Nemanja Aleksic

    A quick update: Things are going according to plan, and we’re currently in the verification stage. If everything checks out, we’ll be getting ManageWP back online in the next few hours.

    We will also email all ManageWP users when the service is back.

    Stay tuned!

  29. Nemanja Aleksic Avatar
    Nemanja Aleksic

    …and we’re back!

    I’ve updated the blog post, and I’ve emailed all Orion users about the completed upgrade.

    Thank you for your patience, understanding, and most of all for your feedback how we could handle this better in the future.

    You rock 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *