Weekend Downtime for a Major Infrastructure Upgrade

UPDATE (Oct 9, 5am EDT): The upgrade is complete and we are back up, you can log into your ManageWP dashboard.

This weekend we’ll be taking down ManageWP Orion for some major upgrade that’s long overdue. The 48h downtime will start on Saturday, October 8th at 3am EDT, and last until Monday, October 10 at 3am EDT.

Why the Whole Weekend?

ManageWP Orion is on Amazon cloud infrastructure. It allows us to infinitely scale it without sacrificing performance. Unfortunately, the database does not scale that easily, and in the past several months it caused a couple of short downtimes. So far we’ve been MacGuyvering fixes, but for a permanent solution we need to take it down and perform a major upgrade.

The upgrade will ensure a smooth and consistent experience in the next year (we’ll figure something else to tinker with by then). The actual upgrade should take less than 20 hours, but we’d rather underpromise and overdeliver, than the other way around.

If you want to go into more technical details, we’re mainly looking to resolve the scaling issue where we would be able to add more servers without a drop in performance. Right now the database scales inefficiently, i.e. if we double the server power, we only get a 10% increase in the capacity. We need to redesign the distribution process and dump/reimport the database, which will take up a huge chunk of the downtime.

What’s Going to Be Down?

We’re putting the Orion dashboard in maintenance mode. Classic dashboard will be unavailable as well, since it partially relies on the Orion database. The website itself will be up, but signup and registration will be disabled.

It’s important to note that ManageWP services will not be running during the upgrade. It effectively means that Uptime Monitor will not notify you of up/down events, and that scheduled backups and checks will not run. If you’re not already storing backups on your own off-site storage (it’s good practice to have backups on two separate locations), I recommend downloading a backup to your computer, so you could be ready to restore your website during the ManageWP downtime.

Needless to say, the upgrade will not affect your websites in any way.

How Can I Reach You If Orion Is Down?

Our Customer Happiness team is here for you. Our contact form will be working, and you can reach us directly at helpdesk@managewp.com

What’s the Downtime in My Time Zone?

worldtimebuddy.com is an awesome time zone conversion tool. Just click the link and add your city/time zone to see when the downtime starts.

How Will I Know When Orion Is Up?

We’ll update this article when the dashboard is back online, and also post regular updates on Twitter.

I Hate You.

This upgrade is really important for both you and us, and the sooner we do it, the better for everyone. I understand that this might interrupt your daily flow, and we’ll be happy to give you a bear hug full of love when we meet in real life.

Thank you for your patience and understanding!

Nemanja Aleksic

Head of Growth at ManageWP. Marketing Manager at GoDaddy. WordCamp Belgrade organizer. But first and foremost, a father, a husband and a puck stopper.

55 Comments

  1. Nemanja Aleksic

    Author

    …and we’re back!

    I’ve updated the blog post, and I’ve emailed all Orion users about the completed upgrade.

    Thank you for your patience, understanding, and most of all for your feedback how we could handle this better in the future.

    You rock 🙂

  2. Nemanja Aleksic

    Author

    A quick update: Things are going according to plan, and we’re currently in the verification stage. If everything checks out, we’ll be getting ManageWP back online in the next few hours.

    We will also email all ManageWP users when the service is back.

    Stay tuned!

  3. Richard

    I don’t know why every one is so upset to the point of getting vulgar about it. Surely, there could have been other ways to do the update and other ways to notify us, but ever since the new price structure and the blazing, wonderful Orion came out, I’m happy to be able to just use it with my team.

  4. billc_lists

    Knuckleheaded move, as I’m sure you’re now aware. (not that I haven’t made a few in my day…) And now you know: downtime for planned maintenance means you have to let everyone know in one or more very obvious ways, well in advance.

    I’m just happy that it’s so far been a fairly quiet weekend and there haven’t been any critical security fixes to deploy so far.

    1. Nemanja Aleksic

      Author

      It’s one of those lessons that one hopes to learn from someone else’s experience.

  5. Fred Sessoms

    You are awarded winner of WTF of the Year

  6. Paul

    I’m adding my voice to the list of the unimpressed – just so you’re aware of the breadth of impact your inadequately announced outage has caused. Having had one test on your service for a week or so, I was planning to add all of my remaining sites to your service this weekend. Given how essential your service is to what we offer to clients, losing trust in your judgement from this experience make’s me second-guess whether that’s a sensible thing to do.

  7. Todd Lash

    +1 On the Email Request. I had planned a migration this weekend and my client is counting on me. I had no idea the service would be offline.

    Not Happy.

  8. ccarter

    It seems that you guys are getting pretty adept at generating hate mail. Reminds me of early Facebook. But here’s the deal. You’ve been working hard as hell for a very long time build a platform and a service that people absolutely love and depend on. So you’re not a start up anymore. Stop acting like it.

  9. Amanda

    I’m disappointed there was not a better notification effort about this downtime prior to attempting to log in and finding it down. This is the kind of event that businesses who rely on your service need time to plan for. I was going to move my website from staging to production using the clone tool this weekend (the entire reason I purchased backups at all – I already have another backup system). Now I will not be able to do so until Monday and it will affect my marketing schedule. This is a letdown.

  10. Dan

    You’ve got to be kidding me?! Why am I only finding out about this now when I try and log in? In what world do you live in where it didn’t occur to you to email us and make sure we all knew this was going to happen?! And then put massive notices in big red letters in the dashboard in the preceding week??! It’s all very well saying, hey, make sure you download a backup just in case, but I didn’t know you were going to do this so that is 1000% useless. I am absolutely STUNNED by how you guys have handled this. I get that you needed to make the update, but it’s not acceptable that I had to find out about it when I try and log in.

  11. Mundo Doido

    Hi Guys, I love your service… Thanks for all.

  12. david.bolthouse

    Well, I guess we have to learn the hard lesson that we cannot trust your service will always be available. Not good. You should have updated offline then moved it over instead of blocking us out for a weekend, when many of us work. I log in every day and had no clue you were doing this all weekend until I really needed to get in here this morning and found it not available.

  13. Alexander

    Hi Guys, I love your service…in the meanwhile it’s an important part of my own business. Cross my fingers for the update that all would work fine. Alexander P.S. Try pushcrew.com as notification service…i was not very happy to see this morning that i cant migrate the prototypes for my customers

  14. Seth

    Um… Shame about the typo. Perhaps during the maintenance the techies can correct MacGuyvering to MacGyvering. And, in a spare five minutes, sit back and think “Gee, this is the dumbest not-cool move we have ever done.” If I said to my boss, hey I am gonna take your website down for the weekend, I will outta door before I finished the sentence.

    Also, perhaps the techies’ boss could consider this: your clients are techies who have to maintain sites. And WP updates can be hit and miss, and blow-up sites. That, after all, is why we subscribe to the service: to minimize the pain. So when is the best time to do an update in case it goes awol? The weekend! So, having zip notice I find myself in my office unable to do my weekend’s tasks. Thanks guys!

    1. Nemanja Aleksic

      Author

      I know this doesn’t make it any better, but I 100% agree that we’ve dropped the ball on this one. We should have handled it much better, the way you deserve it to be handled.

  15. Karin

    Love you guys – and your humor, Nemanja! I didn’t see the warning, either, but I must admit, I rarely check the little bell thing. Normally there’s nothing there that I need to worry about. Fortunately for me, I don’t have anything huge going on this weekend, but I can understand if I had planned to do something major, this would be very disappointing. But – live & learn. Just realize that all these folks that are mad – are only so because your product is so terrific that they depend on it! You are a real important piece of my business and I appreciate you. May the switchy thing go easily and well for you! blessings!!

    1. Nemanja Aleksic

      Author

      Thanks Karin,

      They have every right to be mad, and we’re owning up to our mistake. And yeah, I agree that as long as you care about ManageWP, you’ll keep sending us feedback and we’ll keep learning and improving.

    2. justin.matt.web

      I didn’t even know there was a bell until I started hearing about it here. Thanks Nemanja for owning up to the mistakes of ManageWP. We still love you, just kinda heart broken. We expect cake when the update is finished 😉

  16. stein.david

    How about a full screen modal pop up or something that doesn’t let you access the Orion dashboard until you read the message and click – ok, I have read that Manage WP will be offline for a really long time (in developer’s time)? Thank you for upgrading though – glad to know Manage WP will be getting better. It’s pretty good now so getting better is a good thing. The down time isn’t the issue – it’s the lack of notification – that wrecks a weekend’s worth of work and income or at least cuts it in half. Knowing before hand would allow us to schedule accordingly.

    1. Nemanja Aleksic

      Author

      I completely agree. I know that I can’t give you your time back and undo this mess, but for what is worth, I apologize on behalf of the ManageWP team, and I can guarantee that we’ll give a much more advance warning through multiple channels.

      Based on the feedback the main things missing here were email and a blocking modal a week in advance.

  17. Jay Clark

    This is NOT the way it is done. You do all of your changes in a Developmental site, get the bugs out and then Migrate the old Database into the new structure. The only downtime is while the DB is upgrading. If we could do the Nielson Rating database that way with minimal downtime… unless your DB is 20 TB or greater, you could do the same thing. Even 12 hours would have been acceptable.

    The thing is we are addicted to ManageWP and I do not have a password to log in to any website with. Thankfully, my Employees do. This is a rookie move.

  18. stein.david

    Cool – i’m going to stop paying for while this upgrade happens – my business bank account and finances are undergoing an infrastructure overhaul. That’s cool, right? But seriously, I agree w/ previous posters – some notice, other than hidden dashboard messages would be nice. I remember getting many emails when Orion was launching and you needed us to update our address etc and this seems like i’m not the only one who was surprised to log in and find the site not just down but down for 2 days?… Did I miss this?

  19. alexander

    We better get an email letting us know the service is back up…We shouldn’t have to check a blog post to see when things are back to normal.

    With almost all of us being developers, we understand the importance of upkeep, but the lack of effort in communicating to your userbase was completely unacceptable. With the amount of developers relying on your platform, it would be nice to know you care enough to at least try to inform us, as best as you can, that this level of upkeep needs to occur and to give us a heads up so important project items weren’t planned.

    So, which department do I send the invoice for all the money I lost this weekend from not being able to access your service?

  20. gil

    I have to delay what I’m doing. Yep wasn’t notified but that’s fine. It’s for our good I hope. Hopefully more notification in the future.

  21. todd

    Reading this article only after experiencing the [rather unknown] planned down time is frustrating. You guys tend to be “notice-happy” in the menu bar. I’d suggested for system outages to enable a much more prominent alter (think ‘HelloBar’ or ‘EU Cookie Notice’ styles). Someone also suggested [in the comments] to link to this article from the “undergoing maintenance” message on the dashboard. That would be helpful.

    1. todd

      Just thought of another idea – the message that appears on the dashboard should probably include a countdown to when you expect to be back online.

      1. Nemanja Aleksic

        Author

        Thanks for your feedback, Todd. The timer wouldn’t work, since we’re hoping we’ll be back online much sooner than what we announced, but other ideas make much more sense.

        As for the menu bar, I’m trying to figure out how to make it work with the current system: would you expect critical information at the top left (Fresh News) or the top right (the little bell)?

        1. Henk van den Bor

          A notification front and center Popup Sumo style would have been more then appropriate..
          The bell thingy was absolutely unclear!

          1. Nemanja Aleksic

            Author

            That would probably the best move, since we already have modals and it wouldn’t require additional dev hours. Thanks!

          2. todd

            I agree. System outages should have their own style alerts. They are more critical then any other news you’d want to share.

        2. justin.matt.web

          I’d like to also state that a Fresh News or side notification is not the way to go. If it’s this immediate, it needs to be addressed clearly top or bottom center in red/orange and maybe even an animation or countdown timer leading up to it. Also, listing the day itself, versus the actual date or # of hours makes it less apparent. So saying 48 HOUR DOWNTIME would be great. Heck.. even just that exact phrase bottom center and a few words “48 HOUR DOWNTIME STARTS 10/8/2016”

  22. Pat

    Hey folks,
    How can you turn down the system without any notifications?
    Never experienced this from you before but it really made my weekend more troublesome for the work I have planned myself.
    As you failed to notify us, a link on the maintenance page would be appreciated.
    No, no hugs here and the “hate you” statement is quite childish.

    Irony, just a feeling from the past from something abbreviated GD…

    Sad.

    1. Nemanja Aleksic

      Author

      Hi Pat,

      Have you seen the dashboard notifications we put up earlier this week?

      Also, what do you mean by “a link on the maintenance page”? I’m trying to figure out ways to communicate this better in case we need to do a major downtime in the future.

      1. Pat

        Nope, unfortunately I haven’t seen that notification.
        Most information I can relate to that section are changelogs, which is great but if this kind of info also is put there, well it may drown in the info flow.
        I would like to see
        – email notification
        – large info banner in the Orion interface, not hidden in menus
        (and if this was announced with a banner, I better get new glasses)

        This is the look of the maintenance page:
        http://screencast.com/t/BDsaIdEhyb3b

        Plenty of room for a message saying that it is down the whole weekend.
        And perhaps a link to this blog post, https://managewp.com/managewp-downtime-this-weekend-for

        When I see the current maintenance page, I think it is a quick thing that is done in short time.
        Not the whole weekend.

        After a few hours I searched around and found this blog post.

        Well, I can just shake my head and say it was NOT good.
        Pat

        1. Nemanja Aleksic

          Author

          That’s a major screw up on our part, we’ll update the maintenance page ASAP. Thanks for pointing this out!

  23. justin.matt.web

    I love that you’re providing updates, but a warning directly inside of ManageWP would have been extremely helpful.. we’re now delaying all of our work because of the pretty much nonexistent warning.

    1. Nemanja Aleksic

      Author

      I agree that we should have conveyed it through email as well, and more in advance. We did have the dashboard notification, both on Orion and on Classic, tho.

      For what it’s worth, I apologize for the trouble the downtime has caused you and we’ll make sure this won’t happen again.

      1. justin.matt.web

        I saw a screenshot of the notification after the fact, it was not very addressing. It had a green background, light text and didn’t seem like a notice for anything, so most people likely just skipped over it. Next time just make it more apparent. Give it a yield sign, stop sign, or some other sort of sign that addresses something important. Heck if it was red/orange, I would have paid more attention. Anyhow I understand, I just want it to be known that most people will look at a notice like that and instantly think that it’s a 30 minute or 2 hour downtime like most every product we use.. but a 48 hour downtime, that should have been addressed more appropriately. You’ll see from the comments here there is a lot of concern and that’s simply because ManageWP is how some of us operate our businesses now.

        1. Nemanja Aleksic

          Author

          Agreed. I mentioned in one of my other replies that next time we’ll put up a blocking modal in the dashboard one week ahead, so people couldn’t miss it.

          1. justin.matt.web

            Okay thanks, be sure to also have it addressed somewhere after the blocking modal. Having it visible at all times prior to the downtime would help for people that are just in a rush or maybe forgetful as well.

  24. ttownsend

    BTW you guys ROCK 🙂

  25. ttownsend

    I have to 2nd the Email request…. I have been out of loop for a few days and just logged in today….If I had not logged in today I would have missed this. On a desperate request. I would really like to have access to Orion from a Mobile App…this would help resolve the above email issue too as you could Opt for Push Notifications….I would be very interested from a technical perspective on your AWS issues. I am currently working with firm that Big Data is pretty much their middle name hah….migrating to AWS. Have you looked into Hadoop or Redshift clusters as a way to deal with your issues ?

    1. Nemanja Aleksic

      Author

      Mobile app is something we need to get back to at some point since the old app was only compatible with Classic. As we’re working on making our API public, we’ll probably figure out a lot of creative ways to use it in 2017.

      We’re changing sharding configuration on MongoDB and upgrading resources on the MySQL database. Orion database is built for speed, while Hadoop/Redshift would be more suitable for data warehousing and mining.

  26. Torben Heikel Vinther

    What others say 🙂
    P.S. I LOVE YOU

  27. brianjosephking

    Thanks for the details. It would have been nice to get an email about this so that I could have planned better. Also, in the future if backups aren’t going to even be available in some capacity it would be good to be able to download these in some alternate method or transfer the most recent backup directly from ManageWP to my own S3/Google Drive/Dropbox account and avoid having to download to my local machine first.

    I’ll excuse you for now because I love the “I HATE YOU” section, but I do hope you consider the above feature request very seriously.

    Cheers, keep up the good work.

    1. Nemanja Aleksic

      Author

      That’s a very good point, Brian. I’ve passed it on to the Product team. The downtime was initially planned for the next weekend, but we’ve rescheduled at the last moment due to the increased load on the infrastructure.

      BTW you can still send out the backup directly to your S3/GD/Dropbox – simply connect the storage account and tell your backup to upload to 3rd party storage. We’ve written about it way back in the beta:
      https://managewp.com/backup-external-destinations-now-in-orion

      Please note that this feature incurs extra charges – $0.13/GB to be exact.

      1. brianjosephking

        My point is I don’t want to incur the additional charge as I don’t normally need it. However, because you’re the ones taking the service I do need offline, you should incur the charges.

        1. Nemanja Aleksic

          Author

          I agree 100%. Hopefully we won’t have a downtime like this in the future, but if it does happen, I’ll push for a free offsite backup, if technically possible (I don’t know if we can isolate this particular charge for such a limited period).

          Thank you for caring enough to help us do it better next time.

    2. justin.matt.web

      It just would have been nice if you sent an email, posted it inside the dashboard, gave a warning. For companies that rely on ManageWP to some extent, you’ve actually just hurt our earnings, livelyhoods, etc. the entire weekend we’re going to be going through hassles because of this. If we had a warning prior to.. that would not have been an issue at all. I understand there’s nothing you can do at this point, the wheel has already been set in motion, but please consider this for the future.. these types of things can RUIN launches.

      1. Nemanja Aleksic

        Author

        I completely agree, and I mentioned in my other reply that we pushed dashboard notifications a few days ago – I assume you missed it since it was under “Fresh News”.

        Next time something like this comes up, we’ll push notifications through all channels at least a week ahead, so everyone could know about it and adjust accordingly.

  28. Erwin

    Cool Guys. Go for it! Looking forward to that ‘bear hug’ 😉

    1. Nemanja Aleksic

      Author

      I’ll see if I can email you a Bear Hug IOU, just in case we forget it 🙂

  29. kenny

    Thanks for the detailed heads-up! Btw, the “I hate you.” subhead… LMAO!! 🙂

    1. Nemanja Aleksic

      Author

      Hehe, thanks! Most people around the office don’t share my sense of humor, it’s nice to see that there are a couple of people out of 7 billion that get me 😀

Leave a Reply

Your email address will not be published. Required fields are marked *

Over 65,000 WordPress professionals are already using ManageWP

Add as many websites as you want for free, no credit card required. Sign up and start saving time!

Have questions? Get in touch!

Over 65,000 WordPress professionals are already using ManageWP

Add as many websites as you want for free, no credit card required. Sign up and start saving time!



Have questions? Get in touch!

Over 65,000 WordPress professionals are already using ManageWP

Add as many websites as you want for free, no credit card required. Sign up and start saving time!



Have questions? Get in touch!

Over 65,000 WordPress professionals are already using ManageWP

Add as many websites as you want for free, no credit card required. Sign up and start saving time!



Have questions? Get in touch!