2017 is kicking off with some fantastic development news. We have released a huge improvement of the ManageWP Clone feature that internally we like to call Clone V3. It is on a completely new level technically, and I brought in Milos Colakovic, our Software Architect, who is the brilliant mind behind the new ManageWP Clone feature, to help me write this article.
What Makes This ManageWP Clone Feature Even Better
Before handing the reins over to Milos, who will take you through the technical details, lets have a look at what it all means on the surface. Most of you might not have noticed any changes, this is because we haven’t actually changed any of the interface, the UI has stayed the same, and all of the work took place under the hood. Having said that there are some obvious improvements.
Visually you will notice that the cloning process is much faster than before. On average the new ManageWP Clone feature will clone 4 times faster. This means for a website that has 10k files and a 10MB database, cloning with our previous version took 55 seconds (which is already very fast!), but with Clone V3 it only takes 14 seconds!
Another important visual improvement is that now you can expect a more precise clone time indication. That means you will know exactly how long said clone will take to complete, our message progress has been improved greatly.
Staying on the topic of messages, Milos has also worked on making clone related messages more informative. For example, now you can track your clone progress more clearly. You might see something like this.
Uploading files, we're at {{progress}}/{{total}} ({{percent}}%) .
This way you will know exactly what is happening during your clone process. Error messages have also been made clearer. Here is an example.
The destination website has HTTP authentication enabled for realm \"{{realm}}\". Please, provide us with HTTP authentication credentials so we can continue.
This will help you take appropriate steps in fixing any issues.
A final note for us non tech experts. It’s a standalone tool, which means it can clone any type of website, and it’s not restricted to WordPress websites. It’s also scalable, at any one time it can clone around 100 websites.
Tech Spec For New ManageWP Clone Feature
For those of you who are interested in the gritty technical details. Here is what Milos had to say.
The website clone/restore feature, along with the backup, is, in my opinion, our strongest feature. That’s why it has to work exceptionally. While maintaining the previous version I got many ideas on how parts of it could be improved, so I pushed the idea of making it into a separate service for easier maintainability.
There are also personal reasons to why I wanted to create a new clone backend from scratch (in Golang). I love working on software people use to its maximum, this is a really great opportunity (there’s no other migration tool that does it like we do) and I can’t stand unstable software (the previous version had some very specific issues, a lot of them related to PHP – how could I admit defeat to a language?)
The trick to faster clone/restore process is in keeping as little transient data as possible in the memory (e.g. using streams). That way the work can easily be parallelized.
As an example, here’s what happens in the life cycle of a single table restore process from a ZIP archive:
- Spawn 3 threads:
– 1st reads the data from the provided URL and feeds it to 2nd and 3rd thread
– 2nd reads the ZIP archive and hashes files as they are coming in (ZIP format stores files sequentially and are independent, i.e. they share no data) and marks where each raw file begins in the containing archive
– 3rd writes the original ZIP archive to a temporary location, so they can be read if required during the remaining steps - Send an HTTP POST request to the website to start uploading the table backup
- Write file metadata to the request body (file path, expected size, modification time, etc.)
- Open the ZIP archive, seek to where the compressed table backup starts,
– attach decompressing stream filter
– attach table-backup-specific stream filters (e.g. prefix changing stream filter, which changes the database prefix in the backup file itself on-the-fly)
– attach base64 stream filter, so some web-servers don’t get too suspicious of the payload
– write the payload to the HTTP connection opened in step 2 until we reached the maximum upload size (the file upload will be continued if it’s cut off)
– go to step 3 if there’s room for more files - Read the response and process the results.
All the above steps happen in parallel in up to 20 connections, depending on your destination website’s server load.
This was a great new experience for me, but there’s plenty more where that came from – fixes and improvements are coming out daily, even if you don’t notice them. Stay tuned for more.
Wrapping Up
Have a go and test the new ManageWP Clone feature and let us know what you think. Like always we are working on improving all of our features to make you more efficient and save you time. Drop any questions you have below, and we will get back to you. Happy cloning!
jappleton
Hi, can the backup or cloning tool also back up an extra database on same hosting. A client has a bespoke booking system integrated with Worpdress and all data is in a separate database. What they really want is a working clone that they could reference from in case main website goes down they can still see their bookings… maybe set it to auto clone once or more each day… possible with Manage WP?
David
I noticed that Clone leaves a few plugins in place from my original vanilla install. I am in WHM using QuickInstall to put into place my basic client site. Then I use Clone to override that with my Starter Site. I notice it leaves several plugins in place from the QuickInstall process. Any way to prevent this? It seems minor but in cases where I deal with many installs every second I can shave off is helpful. At any rate, I see in your article you are considering adding some pre-Clone settings, perhaps this could be one. Or is there a reason not to do it? Thanks!
Kerry
Hi…so far all is well…I just wondered how long after the clone has uploaded all the files and database should it stay on 99% Removing Cloner Files? I know I am probably being a little impatient but it has been doing so for more than an hour now…Many thanks
Frank Gomez
I’m interested in having clone replace everything in the destination database (except for the URL of course). In the past the clone tool just added more tables to the existing database.. Is is it possible to remove some or all existing tables with new clone tool?
Milos Colakovic
The clone process first removes all the tables that are present in the backup (you can see their list under the “Content” tab when you view a backup). With table prefixes WordPress officially supports having multiple WordPress installations sharing the same database, so we have to be extra careful and not touch tables that we aren’t 100% sure we should. WordPress itself is not bothered by extra tables, in fact, it completely ignores them.
I’m not the one to make promises here, but we will get more options for users to fine-tune the migration process; specifically what to skip or delete. “Clean-up the destination before migration” is a reasonable checkbox to have.
jeff arnold
Does the cloning feature allow for cloning a single install into a multisite install? If not, do you know of any rock-solid cloners that do?
Milos Colakovic
The short answer is no.
Multisite installations use different internal structure, they don’t have their own core/plugin/theme files (or even users), meaning they all borrow from the “main site”. That would mean that cloning a website with a different version of a single plugin or theme could break all the websites in the network since they have to be installed on the main site. Add to that, that cloning a vulnerable website could potentially open up the whole network to online attacks and you get a good reason to avoid multisite installations when possible. Nowadays they take more than what they give back when you already have a tool for multiple website management.
I doubt that there is a tool that does that (at least I’m not aware of one) since it would be very limiting and would still require a lot of manual work.
info
The clone tool was certainly impressive before this update. Having used it recently, it is noticeable just how quick and efficient it is so a big thanks
Nevena Tomovic
Thanks for your comment, glad you are enjoying our Clone tool. 🙂