This post is my notes delving into Dev Ops with minimal success; however, I’m posting it because I’m hopeful that some of the research I found and mistakes I made will help someone else.
What am I attempting to do?
My goal with this endeavor was to set up an identical server on Digital Ocean to the one I have on RackSpace. The Rackspace server is old (10+ years) and running an old Ruby on Rails application configured by an outside team of developers that I can no longer contact.
Background
Obviously, you must use the right technical terms during online research to find the right instructions. I believe the key here is using the search term “image.” I started with search terms “migrate” and “move” – which were close but not quite what I needed. (Examples of not quite articles, for migrating – Migrate your current VPS or maybe deploying using Dokku – and tutorial from 2015 or Doing it the hard way). These might be useful further along, but I wanted to focus on server setup – the environment first.
My current understanding of the term “image” is that the image captures the current state of a server, including all files, applications, and settings, essentially creating a “clone” that can be used to spin up new servers with identical configurations. This is what I wanted!
My goal
For this blog post, my goal is to create and import a custom image of a 10-year-old Ruby on Rails application’s server from Rackspace and import it to DigitalOcean.
Process
Armed with resources directly from RackSpace and Digital Ocean, I confidently forged forward. There were many hiccups.
Step 1. Create the Image in RackSpace
Rackspace Resource: Create an image of a General Purpose v1 Cloud Server
This part went smoothly. In my Rackspace Cloud account, I can successfully see a current image of my active server and an image from when it was first configured 10 years ago.
Step 2. Download the Saved Image
This step proved more difficult. From within the Rackspace admin area, there was no way to directly export the Image file, instead I could only share it to another Rackspace user or account.
Customer service provided me the following resource.
Follow this instruction from RackSpace to put a image into your Cloud Files. Read the entire document FIRST. I went step-by-step and then thought, “Ah, this is likely out-of-date, you can download directly now.” So, I downloaded a VHD file directly, yep a 125 mb file, and then struggled, struggled, struggled. Eventually, I decided to go back to this document and start again and THEN I read all the way through and realized why I needed to use Cyberduck, too. Headslap!
Rackspace documentation: Export and download a saved image
So, I started following the instructions step-by-step. I made it through the pre-requisites and found my UUID for the Image; I made a cloud container and went to Pitchfork and made it to Step 5, my Image was Processing.
Stupid Mistake
This is where I made a stupid mistake. I saw the files were starting to populate, and they were each 125mb. I wondered why there were so many, but I thought something must have gone wrong or maybe I requested the process too many times. I also noticed that there was actually a DOWNLOAD option right from my Cloud Container.
I thought, “Oh, maybe you don’t need Cyberduck after all. Maybe this download option is new…” And I downloaded a single 125mb file and moved on.
This was a BIG MISTAKE. If I had kept reading the instructions, I would have seen that I needed CyberDuck to download the 0kb file, which compiles the dozens of 125mb files and ultimately provides me with one very large file! Steps 9 and 10 were crucial to the process.
So, my first tip: READ THE FULL INSTRUCTIONS. And expect a large VHD file, not a 125mb VHD file. My full VHD file was 17GB.
So, after lots of problems with my incomplete VHD file, I circle-backed and successfully downloaded a full VHD file using Cyberduck.
Alternatively, Rackspace support also provided another option for image download using Swiftly.
Digital Ocean doesn’t allow VHD Custom Images
This brings me to my next issue: Digital Ocean doesn’t allow VHD uploads. So, once I had a VHD file, I couldn’t use it without conversion. The Digital Ocean error message reads, “The file you are trying to upload ends in an extension that we do not support. We only accept image files that end in gz, bz2, vmdk, vhdx, qcow, qcow2, vdi, raw, img. (.zip is not supported).”
So, now I tried, tried and tried again to convert my VHD to a compatible format. AI options gave me links to lots of online converters but none actually made this conversion. I kept finding that I could use QEMU to convert virtual machine images on my Mac, but I always ran into an error. Eventually, I learned that my VHD file was incomplete and that was 99% of the problem with the conversions. (see above for why my file was incomplete)
What error did I receive when the VHD file was corrupt/incomplete/too small?
qemu-img: Could not open 'bf28f879-386c-4754-bef9-1df1e554c5fa.vhd': Failed to lock byte 100The “Failed to lock byte 100” was the error I always received when trying to make conversions from an incomplete VHD file.
I did not follow the Rackspace instructions completely. Instead I downloaded one 125mb vhd file and tried to use it. The full vhd file was 19.18GB. Once I downloaded the full VHD file via Cyberduck, I could use gemu-image to convert it.
What error did I receive when I was using the wrong argument string?
I also learned that when converting from VHD, you don’t type VHD in Terminal; you type VPC.
I kept getting errors such as “Unknown driver, ‘vhd'”
The solution was using vpc in the command instead of vhd for the file type.
This command works for converting a file named bf28f879-386c-4754-bef9-1df1e554c5fa.vhd from VHD format to Qcow2 format.
qemu-img convert -f vpc bf28f879-386c-4754-bef9-1df1e554c5fa.vhd -O qcow2 bf28f879-386c-4754-bef9-1df1e554c5fa.qcow2
or example to convert to raw
~$ qemu-img convert -f vpc bf28f879-386c-4754-bef9-1df1e554c5fa.vhd -O raw bf28f879-386c-4754-bef9-1df1e554c5fa.rawQemu-img File Format strings to use:
Everything I was reading said to use the file type or file extension as the string argument, so I was using VHD and kept getting errors. However, this chart says that VHD (Hyper-V) uses VPC!
Upload Image to Digital Ocean
So, now that I had a qcow2 image file, I thought it was going to be smooth sailing, but it wasn’t.
The Upload Image blue button didn’t work on Digital Ocean because my file was too large at 17GB.
I then put my file in Google Drive, but the link didn’t end in the file extension, so I couldn’t use “Import via URL.” The error message reads, “The URL you have entered ends in an extension that we do not support. We only accept image files that end in gz, bz2, vmdk, vhdx, qcow, qcow2, vdi, raw, img. (.zip is not supported).”
I then tried uploading the file into a shared hosting plan I owned, but that hosting plan had a small space limit, and the file was again too large and truncated.
Finally, I uploaded the large file to another server I had access to and created a link that ended in the image extension. I used the Import via URL button, and the image file showed up and was processing. But again and again, the file never stopped processing, it just eventually disappeared and the dashboard would be blank/empty again. There was never a warning or failed error displayed.
After some research, I’m assuming that my very outdated server has something in my qcow2 or vhd file that doesn’t meet the image requirements for upload on Digital Ocean (likely it is missing the installation of SSHD or cloud-init).
Additional Resources:
So this is where I stopped and decided to document my experience.
In a Nutshell
In conclusion, on RackSpace, I created and downloaded a VHD image file – which, unfortunately, is not a supported image file type on Digital Ocean. A Digital Ocean Dev Ops conversion tutorial was well-written but didn’t work for me. I spent hours trying to find a way to successfully convert my VHD file to another format on my Mac. Ultimately, I learned that my VHD file was incomplete (broken), and I used the wrong terminal commands for conversion. After fixing those issues, I failed because my qcow2 file kept erroring and would not fully process on Digital Ocean.
I’ve decided to keep my outdated rails app going on the Rackspace server even though it costs more than I’d like it to monthly.
Have tips to share?
Can you commiserate? Did this help you? Please feel free to leave a comment.
