ManageWP Clone Feature Just Got Even Better

2017 is kicking off with some fantastic development news. We have released a huge improvement of the ManageWP Clone feature that internally we like to call Clone V3. It is on a completely new level technically, and I brought in Milos Colakovic, our Software Architect, who is the brilliant mind behind the new ManageWP Clone feature, to help me write this article.

What Makes This ManageWP Clone Feature Even Better

Before handing the reins over to Milos, who will take you through the technical details, lets have a look at what it all means on the surface. Most of you might not have noticed any changes, this is because we haven’t actually changed any of the interface, the UI has stayed the same, and all of the work took place under the hood. Having said that there are some obvious improvements.

Visually you will notice that the cloning process is much faster than before. On average the new ManageWP Clone feature will clone 4 times faster. This means for a website that has 10k files and a 10MB database, cloning with our previous version took 55 seconds (which is already very fast!), but with Clone V3 it only takes 14 seconds!

Another important visual improvement is that now you can expect a more precise clone time indication. That means you will know exactly how long said clone will take to complete, our message progress has been improved greatly.

Staying on the topic of messages, Milos has also worked on making clone related messages more informative. For example, now you can track your clone progress more clearly. You might see something like this.

Uploading files, we're at {{progress}}/{{total}} ({{percent}}%) .

This way you will know exactly what is happening during your clone process.  Error messages have also been made clearer. Here is an example.

The destination website has HTTP authentication enabled for realm \"{{realm}}\". Please, provide us with HTTP authentication credentials so we can continue.

This will help you take appropriate steps in fixing any issues.

A final note for us non tech experts. It’s a standalone tool, which means it can clone any type of website, and it’s not restricted to WordPress websites. It’s also scalable, at any one time it can clone around 100 websites. 

Tech Spec For New ManageWP Clone Feature

For those of you who are interested in the gritty technical details. Here is what Milos had to say.

The website clone/restore feature, along with the backup, is, in my opinion, our strongest feature. That’s why it has to work exceptionally. While maintaining the previous version I got many ideas on how parts of it could be improved, so I pushed the idea of making it into a separate service for easier maintainability.

There are also personal reasons to why I wanted to create a new clone backend from scratch (in Golang). I love working on software people use to its maximum, this is a really great opportunity (there’s no other migration tool that does it like we do) and I can’t stand unstable software (the previous version had some very specific issues, a lot of them related to PHP – how could I admit defeat to a language?)

The trick to faster clone/restore process is in keeping as little transient data as possible in the memory (e.g. using streams). That way the work can easily be parallelized.

As an example, here’s what happens in the life cycle of a single table restore process from a ZIP archive:

1.  Spawn 3 threads:
   –  1st reads the data from the provided URL and feeds it to 2nd and 3rd thread
   –  2nd reads the ZIP archive and hashes files as they are coming in (ZIP format stores files sequentially and are independent, i.e. they share no data) and marks where each raw file begins in the containing archive
   – 3rd writes the original ZIP archive to a temporary location, so they can be read if required during the remaining steps
2. Send an HTTP POST request to the website to start uploading the table backup
3. Write file metadata to the request body (file path, expected size, modification time, etc.)
4. Open the ZIP archive, seek to where the compressed table backup starts,
   – attach decompressing stream filter
   – attach table-backup-specific stream filters (e.g. prefix changing stream filter, which changes the database prefix in the backup file itself on-the-fly)
   – attach base64 stream filter, so some web-servers don’t get too suspicious of the payload
   – write the payload to the HTTP connection opened in step 2 until we reached the maximum upload size (the file upload will be continued if it’s cut off)
   – go to step 3 if there’s room for more files
5. Read the response and process the results.

All the above steps happen in parallel in up to 20 connections, depending on your destination website’s server load.

This was a great new experience for me, but there’s plenty more where that came from – fixes and improvements are coming out daily, even if you don’t notice them. Stay tuned for more.

Wrapping Up

Have a go and test the new ManageWP Clone feature and let us know what you think. Like always we are working on improving all of our features to make you more efficient and save you time. Drop any questions you have below, and we will get back to you. Happy cloning!