ManageWP Clone Feature Just Got Even Better

2017 is kicking off with some fantastic development news. We have released a huge improvement of the ManageWP Clone feature that internally we like to call Clone V3. It is on a completely new level technically, and I brought in Milos Colakovic, our Software Architect, who is the brilliant mind behind the new ManageWP Clone feature, to help me write this article.

 

ManageWP Clone Feature

What Makes This ManageWP Clone Feature Even Better

Before handing the reins over to Milos, who will take you through the technical details, lets have a look at what it all means on the surface. Most of you might not have noticed any changes, this is because we haven’t actually changed any of the interface, the UI has stayed the same, and all of the work took place under the hood. Having said that there are some obvious improvements.

Visually you will notice that the cloning process is much faster than before. On average the new ManageWP Clone feature will clone 4 times faster. This means for a website that has 10k files and a 10MB database, cloning with our previous version took 55 seconds (which is already very fast!), but with Clone V3 it only takes 14 seconds!

Another important visual improvement is that now you can expect a more precise clone time indication. That means you will know exactly how long said clone will take to complete, our message progress has been improved greatly.

ManageWP Clone Feature

Staying on the topic of messages, Milos has also worked on making clone related messages more informative. For example, now you can track your clone progress more clearly. You might see something like this.

Uploading files, we're at {{progress}}/{{total}} ({{percent}}%) .

This way you will know exactly what is happening during your clone process.  Error messages have also been made clearer. Here is an example.

The destination website has HTTP authentication enabled for realm \"{{realm}}\". Please, provide us with HTTP authentication credentials so we can continue.

This will help you take appropriate steps in fixing any issues.

A final note for us non tech experts. It’s a standalone tool, which means it can clone any type of website, and it’s not restricted to WordPress websites. It’s also scalable, at any one time it can clone around 100 websites. 

Tech Spec For New ManageWP Clone Feature

For those of you who are interested in the gritty technical details. Here is what Milos had to say.

ManageWP Clone Feature

The website clone/restore feature, along with the backup, is, in my opinion, our strongest feature. That’s why it has to work exceptionally. While maintaining the previous version I got many ideas on how parts of it could be improved, so I pushed the idea of making it into a separate service for easier maintainability.

There are also personal reasons to why I wanted to create a new clone backend from scratch (in Golang). I love working on software people use to its maximum, this is a really great opportunity (there’s no other migration tool that does it like we do) and I can’t stand unstable software (the previous version had some very specific issues, a lot of them related to PHP – how could I admit defeat to a language?)

The trick to faster clone/restore process is in keeping as little transient data as possible in the memory (e.g. using streams). That way the work can easily be parallelized.

As an example, here’s what happens in the life cycle of a single table restore process from a ZIP archive:

  1.  Spawn 3 threads:
    –  1st reads the data from the provided URL and feeds it to 2nd and 3rd thread
    –  2nd reads the ZIP archive and hashes files as they are coming in (ZIP format stores files sequentially and are independent, i.e. they share no data) and marks where each raw file begins in the containing archive
    – 3rd writes the original ZIP archive to a temporary location, so they can be read if required during the remaining steps
  2. Send an HTTP POST request to the website to start uploading the table backup
  3. Write file metadata to the request body (file path, expected size, modification time, etc.)
  4. Open the ZIP archive, seek to where the compressed table backup starts,
    – attach decompressing stream filter
    – attach table-backup-specific stream filters (e.g. prefix changing stream filter, which changes the database prefix in the backup file itself on-the-fly)
    – attach base64 stream filter, so some web-servers don’t get too suspicious of the payload
    – write the payload to the HTTP connection opened in step 2 until we reached the maximum upload size (the file upload will be continued if it’s cut off)
    – go to step 3 if there’s room for more files
  5. Read the response and process the results.

All the above steps happen in parallel in up to 20 connections, depending on your destination website’s server load.

This was a great new experience for me, but there’s plenty more where that came from – fixes and improvements are coming out daily, even if you don’t notice them. Stay tuned for more.

Wrapping Up

Have a go and test the new ManageWP Clone feature and let us know what you think. Like always we are working on improving all of our features to make you more efficient and save you time. Drop any questions you have below, and we will get back to you. Happy cloning!

 

Nevena Tomovic

Nevena wears many hats in ManageWP. One of them is being responsible for translating our team's geek talk and product features into value that normal people understand. She believes that "learning is a treasure that will follow its owner everywhere", so she tries to learn something new every day.

9 Comments

  1. jappleton

    Hi, can the backup or cloning tool also back up an extra database on same hosting. A client has a bespoke booking system integrated with Worpdress and all data is in a separate database. What they really want is a working clone that they could reference from in case main website goes down they can still see their bookings… maybe set it to auto clone once or more each day… possible with Manage WP?

  2. David

    I noticed that Clone leaves a few plugins in place from my original vanilla install. I am in WHM using QuickInstall to put into place my basic client site. Then I use Clone to override that with my Starter Site. I notice it leaves several plugins in place from the QuickInstall process. Any way to prevent this? It seems minor but in cases where I deal with many installs every second I can shave off is helpful. At any rate, I see in your article you are considering adding some pre-Clone settings, perhaps this could be one. Or is there a reason not to do it? Thanks!

  3. Kerry

    Hi…so far all is well…I just wondered how long after the clone has uploaded all the files and database should it stay on 99% Removing Cloner Files? I know I am probably being a little impatient but it has been doing so for more than an hour now…Many thanks

  4. Frank Gomez

    I’m interested in having clone replace everything in the destination database (except for the URL of course). In the past the clone tool just added more tables to the existing database.. Is is it possible to remove some or all existing tables with new clone tool?

    1. Milos Colakovic

      The clone process first removes all the tables that are present in the backup (you can see their list under the “Content” tab when you view a backup). With table prefixes WordPress officially supports having multiple WordPress installations sharing the same database, so we have to be extra careful and not touch tables that we aren’t 100% sure we should. WordPress itself is not bothered by extra tables, in fact, it completely ignores them.
      I’m not the one to make promises here, but we will get more options for users to fine-tune the migration process; specifically what to skip or delete. “Clean-up the destination before migration” is a reasonable checkbox to have.

  5. jeff arnold

    Does the cloning feature allow for cloning a single install into a multisite install? If not, do you know of any rock-solid cloners that do?

    1. Milos Colakovic

      The short answer is no.
      Multisite installations use different internal structure, they don’t have their own core/plugin/theme files (or even users), meaning they all borrow from the “main site”. That would mean that cloning a website with a different version of a single plugin or theme could break all the websites in the network since they have to be installed on the main site. Add to that, that cloning a vulnerable website could potentially open up the whole network to online attacks and you get a good reason to avoid multisite installations when possible. Nowadays they take more than what they give back when you already have a tool for multiple website management.
      I doubt that there is a tool that does that (at least I’m not aware of one) since it would be very limiting and would still require a lot of manual work.

  6. info

    The clone tool was certainly impressive before this update. Having used it recently, it is noticeable just how quick and efficient it is so a big thanks

    1. Nevena Tomovic

      Author

      Thanks for your comment, glad you are enjoying our Clone tool. 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *

Over 65,000 WordPress professionals are already using ManageWP

Add as many websites as you want for free, no credit card required. Sign up and start saving time!

Have questions? Get in touch!

Over 65,000 WordPress professionals are already using ManageWP

Add as many websites as you want for free, no credit card required. Sign up and start saving time!



Have questions? Get in touch!

Over 65,000 WordPress professionals are already using ManageWP

Add as many websites as you want for free, no credit card required. Sign up and start saving time!



Have questions? Get in touch!

Over 65,000 WordPress professionals are already using ManageWP

Add as many websites as you want for free, no credit card required. Sign up and start saving time!



Have questions? Get in touch!