Preparing a new Ubuntu VM Server

Here is my crib sheet on setting up a new Ubuntu server, including notes on enabling Virtualisation.  These notes include discussion of Kimsufi / OVH dedicated hosting, but the principles apply to all dedicated servers.  I use Kimsufi because they are the only dedicated hosting provider that I can afford for non-profit purposes...
See also:-
    Ubuntu Server Guide:  https://help.ubuntu.com/11.10/serverguide/C/
    Kimsufi KS series: Dedicated Hosting Prices:   http://www.kimsufi.co.uk/ks

UPDATE: I've revised this page recently for Ubuntu 12.04.  All seems similar.


1) Install base OS (Ubuntu 11.10 == November 2011)
This summary is based on Ubuntu Server 11.10.  Normally I would use an LTS version of Ubuntu, but 10.04 LTS is missing the ability to shut down the guest VM's when the host server is rebooted.   So if you intend to host any VM's, then I would suggest using 11.10 until 12.04 LTS is released in April 2012.

In the case of Kimsufi / OVH hosting, this is simply a matter of using the web control panel or the MOMI desktop application to reinstall the OS.  Choose "raw" OS distribution on the assumption that you want to control things yourself.   This will still include the RTM monitoring application, which is simply a CRON job that pushes status updates to the OVH servers every few minutes so that the web control panel shows up-to-date information about your boxes.

After the base install is finished on a Kimsufi / OVH box, you might want to change the random root password that was set during the automatic provisioning process.  If you look at the shadow file before and after, it seems likely that the automatic provisioner uses an older version of crypt(3) for generating the hash.  Setting a new password will replace the password hash with a "$6$" SHA one which should be harder to break by brute force. (On the other hand- the opposite viewpoint is: If an atttacker is in a position to read /etc/shadow, then your box has already been owned anyway.  And given direct physical access - or access to the diagnostic netboot image via the web management GUI - they can just change the root password anyhow.)

If you have your own box, then just download the Ubuntu Server .ISO media, burn to a CD or DVD, and boot from that.  Then do a basic install.

2) Enable password-less SSH logins
Typing in a password every time is insecure and tedious.  So straight after installation, edit /etc/ssh/sshd_config and make sure root logins are enabled (true by default for Kimsufi/OVH installs) :-
    PermitRootLogin yes

Use scp to copy the public SSH key from your management workstations (e.g. ~/.ssh/id_rsa.pub) into /root/.ssh/authorized_keys2 on the server.  Now execute "/etc/init.d/ssh restart" so that this takes effect.  Then log out, and log back in to test passwordless logins. 

Once passwordless logins are confirmed working, edit /etc/ssh/sshd_config to disable password authentication.  This will stop lames trying to brute-force your SSH password all day long…
    PasswordAuthentication no
Run "/etc/init.d/ssh restart" again so that this takes effect.

Also in /etc/ssh/sshd_config … consider changing the listening TCP port to a non-default value:
    Port 22022
BUT, this seems to break virt-manager client (see below) so you might want to avoid that if your server will be hosting VMs.

Also in /etc/ssh/sshd_config … consider enabling a legal warning text for logins:
    Banner /etc/issue.net
where that file contains:
    WARNING: COMPUTER MISUSE ACT 1990
  You will commit a criminal offence if you act outside your authority in relation to this computer.

To restart SSH after any config changes, you need to say:-
    /etc/init.d/ssh restart

If you'd like to change the messages shown at login time, change to the /etc/update-motd.d directory.  On Kimsufi OVH systems, it's probably worth commenting out everything in /etc/update-motd.d20-ovh-informations (or just stick an exit near the top).   You need to reboot afterwards to test.

3) Patch the box
    apt-get -y update
    apt-get -y dist-upgrade
    reboot

4) Fix hostname, hosts file, DNS…Edit /etc/hostname and set your desired hostname in Fully Qualified Domain Name format.
Example:-
    www7.example.com
   
Edit /etc/hosts and add your hostname (by itself and in FQDN format) and your IP address.
Example:-
    1.2.3.4 www7 www7.example.com

Edit /etc/resolv.conf.  Delete whatever crap is in it.  Make it use Google's nice fast name servers:-
    nameserver 8.8.8.8
    nameserver 8.8.4.4

UPDATE: It seems the DNS settings on Ubuntu Server should really be set up from /etc/network/interfaces, not from /etc/resolv.conf.  The syntax is as follows:
    dns-nameservers 8.8.8.8 8.8.4.4
    dns-search example.org

Edit /etc/network/interfaces and make sure IPv4 and IPv6 addresses, netmasks and gateways are correctly set.

Unless you want to act as a DNS server, remove bind:-
    apt-get remove bind9   

Reboot.

5) Revert to stock Linux kernel If the hosting server uses your vendor's custom kernel (e.g. for Kimsufi/OVH servers) you need to revert to a standard kernel and standard grub settings. Otherwise you won't get regular security updates, and perhaps the VM hosting won't work so well.

Full details:
    http://neuro.me.uk/2009/09/20/revert-to-standard-ubuntu-kernel-on-ovh-or-kimsufi-servers/

Summary: (based on Ubuntu 11.10 on Kimsufi 2010 box)
    root@server:~# uname -a
    Linux server.fqdn 2.6.38.2-grsec-xxxx-grs-ipv6-64 #2 SMP Thu Aug 25 16:40:22 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

    root@server:~# rm /etc/grub.d/06_OVHkernel <-- This removes the grub template for OVH's Dirty Kernel

    root@server:~# apt-get -y install linux-server grub
    …
    Could not find /boot/grub/menu.lst file. Would you like /boot/grub/menu.lst generated for you? (y/N) y
    Searching for splash image ... none found, skipping ...
    Found GRUB 2: /boot/grub/core.img
    Found kernel: /boot/vmlinuz-3.0.0-14-server
    Found GRUB 2: /boot/grub/core.img
    Updating /boot/grub/menu.lst ... done

    root@server:~# ln -sf "/dev/sda1" /dev/root
    root@server:~# grub-install --recheck --root-directory=/ /dev/sda
    Probing devices to guess BIOS drives. This may take a long time.
    Installing GRUB to /dev/sda as (hd0)...
    Installation finished. No error reported.
    This is the contents of the device map //boot/grub/device.map.
    Check if this is correct or not. If any of the lines is incorrect,
    fix it and re-run the script `grub-install'.

    (fd0)    /dev/fd0
    (hd0)    /dev/sda

    root@server:~# grub
    Probing devices to guess BIOS drives. This may take a long time.

       [ Minimal BASH-like line editing is supported.   For
         the   first   word,  TAB  lists  possible  command
         completions.  Anywhere else TAB lists the possible
         completions of a device/filename. ]
    grub> root (hd0,0)
    root (hd0,0)
    grub> find /boot/grub/stage2
    find /boot/grub/stage2
     (hd0,0)
    grub> setup (hd0)
    setup (hd0)
     Checking if "/boot/grub/stage1" exists... yes
     Checking if "/boot/grub/stage2" exists... yes
     Checking if "/boot/grub/e2fs_stage1_5" exists... yes
     Running "embed /boot/grub/e2fs_stage1_5 (hd0)"...  21 sectors are embedded.
    succeeded
     Running "install /boot/grub/stage1 (hd0) (hd0)1+21 p (hd0,0)/boot/grub/stage2 /boot/grub/menu.lst"... succeeded
    Done.
    grub> quit
    quit
    root@server:~# sync
    root@server:~# reboot
    …
    root@server:~# uname -a
    Linux server.fqdn 3.0.0-14-server #23-Ubuntu SMP Mon Nov 21 20:49:05 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

uname -a shows that the standard Ubuntu kernel is now running - good.

6) Install other Ubuntu daemons:

For accurate time keeping...

    ntpdate fr.pool.ntp.org        -> to fix the time initially
    apt-get install openntpd
    vi  /etc/openntpd/ntpd.conf
    ->    set a single server (cdns.ovh.net) if using Kimsufi hosting,
        otherwise use your country's server pool e.g. fr.pool.ntp.org
    /etc/init.d/openntpd restart
   
NOTE: it is NOT recommended to run NTP daemons on VM guests, just on VM hosts.

For a basic web server with a database and PHP, you might want to install the LAMP daemons (linux apache mysql php).   You can do this from the SSH command line like so:
        tasksel
Just select LAMP Server.  It will ask you for a MySQL root password.   That's it!   A working default config is installed.   But you probably only want to do that in VM guests, not in the VM host.

7) Install Virtualisation Support on Ubuntu Server

Documentation: https://help.ubuntu.com/11.10/serverguide/C/virtualization.html

We will install Ubuntu's default VM hosting package, KVM, on the server.

We can manage the VM's remotely via a secure SSH connection - either via the command line or via a GUI interface.  For the GUI option, prepare an Ubuntu Linux client (desktop, laptop or VM) for remote VM management by installing the Ubuntu desktop operating system.    On this client machine, install the Ubuntu packages virt-manager and virt-viewer.
      sudo apt-get install virt-viewer virt-manager

OK, back to your host server.    First, check that it supports KVM :-
    kvm-ok
    The program 'kvm-ok' is currently not installed.  You can install it by typing:
    apt-get install cpu-checker

    apt-get install cpu-checker
    …

    kvm-ok
    INFO: /dev/kvm does not exist
    HINT:   sudo modprobe kvm_intel
    INFO: Your CPU supports KVM extensions
    KVM acceleration can be used

    modprobe kvm_intel
    kvm-ok
    INFO: /dev/kvm exists
    KVM acceleration can be used

Good.  Now install the virtualization server packages…
    apt-get install kvm libvirt-bin virtinst

At this point I would probably reboot for luck.   Then you're ready to get started.   You can use the virsh command line client to manage virtual machines on the server.  But for initial setup you might want to use the GUI tool.   You'll need to have a recent Ubuntu workstation for this.

• Make sure that the management workstation's public SSH key is in the authorised hosts file on the server (as detailed earlier).   
• Make sure you can SSH from the management workstation onto the server. 
• THE VM  MANAGEMENT GUI WILL NOT WORK unless you can successfully SSH from your management workstation to your server on port 22.

From the management workstation, start up the virt-manager program.  This is a GUI application that makes it pretty easy to build new VM guests.  Start the Virtual Machine Manager GUI (if it's not in the menu, type virt-manager)….

    In virt-manager, use File / Add Connection to define a connection to your server host…
        Hypervisor type = QEMU/KVM
        Tick: Connect to remote host
        Method: SSH
        Username: root
        Hostname: your.server.fqdn

Once a connection is established, double-click on it to review the server's connection details.   Resist the urge to mess with the 192.168.x.x IP range (that's not actually used for production as far as I can tell).   But you will need to go to the Storage tab, and press the + button to add a new storage pool (because the default one is in a silly place with little free space).  .. 

• Create a directory, owned by root, called /home/VMs.     
• Use the + button in the GUI to add this as the storage pool.   
• Use the X button in the GUI to delete the default one.   
• Alternatively you could do some diddling with symlinks if you prefer.   
• The Add Storage Pool dialog appears.   Enter a pool name (default will do) and select "dir: Filesystem Directory" as the storage type unless you want to do something advanced (such as using raw disk partitions or iSCSI targets).  Press Forward and the storage pool will be allocated.

At this point you SSH onto the server host, and fetch an .ISO image to install the VM guest from.  For example:-
    cd /home/VMs/
   wget http://www.ubuntu.com/start-download?distro=server&bits=64&release=latest -O ubuntu11.10.x64.server.iso

The install CD will only take a minute to download from the lovely fast 100 megabit connection of a Kimsufi box.  Awesome!

In the GUI window Storage tab, press the Refresh icon (next to 'Volumes') so that the .ISO media file shows up in the list.

OK, back to the main Virtual Machine Manager window…
    - Right-click on the Connection, and select "New".  The 'Create a new virtual machine' dialogue appears…
    - Enter a name for the VM, click 'Local install media' (unless you want to import an existing VM image).  Press Forward. 
    - Select the .ISO file from the browse list.
    - Select the OS type and Version from the drop-down menu.  Press Forward.
    - Confirm RAM and CPU allocations.   Press Forward.
    - Confirm the storage size (default 8 GB).  Press Forward.
    - Tick "Customize configuration before install".    Also, expand the Advanced Options section.
        - Change 'Virtual default network (NAT)' to 'Specify shared device name'.
            - As 'Bridge Name', enter 'br0'
        - Tick 'Set a fixed MAC address' (if you are using Kimsufi / OVH hosting)
            - Copy the MAC address for this VM from the Kimsufi web manager Virtual MAC for VPS service dialog.
    - Press Finish. 
    - There will be a delay of one or two minutes.
    - A new GUI dialog will appear, allowing you to review and customise the VM installation parameters.  Similar to VMware.
    - Press Begin Installation to proceed.
    - A VNC VM server console window appears.   Click through the text-based menus to install Ubuntu Server as you normally would on your own PC.
    - IPV6 and DHCP auto configuration will fail.  Select 'Do Not Configure The Network At This Time'.
    - In the disk partitioner, select 'Guided - use entire disk'.
    - Select OpenSSH Server when prompted for which packages to install.

Once installed, the VM will reboot with amazing speed.  You will be able to control it through the VNC-based console window in Virtual Machine Manager.  Your first action will be to log in on this console and fix the /etc/network/interfaces file (see section 8 below - especially if using Kimsufi / OVH hosting), then reboot the VM.

You can also connect to the VM guest console of an existing VM without using virt-manager. 
Syntax:
    virt-viewer --connect 'qemu+ssh://root@server.example.com/system' vm1
where vm1 is the name of your VM guest.   Note that virt-viewer is quite slow - being based on VNC - but it's good enough to let you onto the VM guest console so that you can fix network settings in order to fix things so that SSH works (see next section).   But you may prefer to enable a proper text-mode virtual serialconsole - see later section in this guide ("HOWTO: Enable text-mode VM guest console").

8) Fixing the IP Routing for the VIrtual Machines (Kimsufi OVH Hosting only)
OVH uses a weird system whereby you can request extra IP addresses (failover IPs) which can then be used by your VM guests.  However, those IP's don't lie within the same IP subnet as your host server.   So your host server has to run its network interface in Bridge Mode, otherwise the OVH switching/routing infrastructure won't route traffic to and from those extra IP addresses.

So on Kimsufi / OVH hosting, you need to start off by requesting those extra "failover" IP addresses - which will incur a small extra monthly fee.  Use the Kimsufi Web Manager to do this.  It takes a few minutes to work for each extra IP.  You need to request a failover IP address and after that, you need to request a Virtual MAC.   In the web manager, use "Associate a virtual MAC to an IP address".

References:
    http://help.ovh.co.uk/BridgeClient
    http://help.ovh.com/DedieMac

For this to work, on the hosting server, we need to edit /etc/network/interfaces.  Comment out what's there and replace the 'eth0' references with 'br0' references.  This is necessary for your VM guests to successfully route traffic to the Internet (because of the unusual routing config at OVH).

For my VM server machine's /etc/network/interfaces, I ended up with:-

    auto lo
    iface lo inet loopback
   
    #auto eth0
    #iface eth0 inet static
    #    address 91.121.7.8
    #    netmask 255.255.255.0
    #    network 91.121.7.0
    #    broadcast 91.121.7.255
    #    gateway 91.121.7.254
   
    #iface eth0 inet6 static
    #   address 2001:41D0:0001:5A08:dead:beef:cafe:f00d
    #    netmask 56
    #   gateway 2001:41D0:0001:5AFF:00FF:00FF:00FF:00FF
   
    auto br0
    iface br0 inet static
        address 91.121.7.8
        netmask 255.255.255.0
        gateway 91.121.7.254
        bridge_ports eth0
        bridge_fd 9
        bridge_hello 2
        bridge_maxage 12
        bridge_stp off

        # dns-* options are implemented by the resolvconf package, if installed
        dns-nameservers 8.8.8.8 8.8.4.4
        dns-search example.org
   
    iface br0 inet6 static
        address 2001:41D0:0001:5A08:dead:beef:cafe:f00d
        netmask 56
        gateway 2001:41D0:0001:5AFF:00FF:00FF:00FF:00FF

By the way, the example above includes IPV6 support, which you might or might not need.  But it's the way of the future of course.   The gateway is perhaps a bit non-standard at OVH: it generates warnings about not matching the 56-bit net mask, but this seems to be by design, and it does work OK.

So… Edit your /etc/network/interfaces file accordingly and reboot the box.  If you can't log back in after 2 minutes, you've got something wrong, and you'll have to boot the box from a rescue image so that you can edit the file again and fix it.  On the Kimsufi hosting boxes, you can net boot into a recovery image via the web manager - it just takes a while (it always does a full FSCK first, annoyingly).

Note that some web servers may not start when you reboot.  The Mathopd web server failed to listen on 0.0.0.0 presumably because it started before br0 was fully initialised.  That could probably be fixed by tweaking the daemon startup order, but a quick fix was to change the ListenAddress in Mathopd from 0.0.0.0 to 91.121.7.8.   However, for maximum security, you should probably avoid running web servers (etc) in the VM hosting server OS.   It is better to run them in the VM guests, so that any compromise of such a service only destroys the VM guest, not everything on the physical box.

When we've done an initial installation of a VM guest (see above), we will need to go onto the guest VM console and edit the guest's /etc/network/interfaces file to match.  My first guest had the following config in /etc/network/interfaces…
   
    # The loopback network interface
    auto  lo
    iface lo inet loopback
   
    # The primary network interface
    auto  eth0
    iface eth0 inet static
        hwaddress ether 02:00:00:ab:af:f5
        address   178.32.50.87
        broadcast 178.32.50.87
        netmask   255.255.255.255
        post-up   route add            91.121.7.254 dev eth0
        post-up   route add default gw 91.121.7.254
        post-down route del            91.121.7.254 dev eth0
        post-down route del default gw 91.121.7.254

        # dns-* options are implemented by the resolvconf package, if installed
        dns-nameservers 8.8.8.8 8.8.4.4
        dns-search example.org

    iface eth0 inet6 static
        address 2001:41D0:0001:5A08::3
        netmask 56
        gateway 2001:41D0:0001:5AFF:00FF:00FF:00FF:00FF

That is, as far as the Guest VM is concerned, his network is /32, and his default route is the same router that the host VM sees directly connected on eth0 (even though you would think that the router was not in the right subnet).  Of course you will need to set the IP addresses and MAC (hwaddress) to match your own allocations from Kimsufi/OVH.

If troubleshooting is necessary to get guest networking working, use the VM Guest console, and remember to check the usual files are all correct:-
    /etc/networks
    /etc/network/interfaces
    /etc/hostname
    /etc/hosts
    /etc/resolv.conf  (possibly)
    /etc/udev/rules.d/70-persistent-net.rules (possibly)
    /root/.ssh/* (allowed keys - careful!)
    /etc/ssh/*   (config settings and host key)

When the VM guest is working, SSH into it, and execute:-
    apt-get install acpid
This ensures that it responds to virtual power off events.  In other words, it will close down cleanly when you reboot the host server (e.g. to apply new security updates).

You may also want to disable the non-privileged user account created during setup, after setting a root password and enabling passwordless SSH logins as for the host server. Of course you'll now need to set up the basic things on the VM guest, just as you did on the VM host.

HOWTO: Enable text-mode VM guest console

You can set up a virtual serial port as a much nicer (much faster) text-based alternative to the VNC-based console offered by virt-manager / virt-viewer.   Like the VNC console, it will still work when SSH doesn't (e.g. because you've got something wrong in a config file).

Reference:
     http://blog.codefront.net/2010/02/01/setting-up-virtualization-on-ubuntu-with-kvm/

In ubuntu 11.10, everything detailed in the Reference was already set up by default, except the definition for /etc/init/ttyS0.conf.

Getting a virtual serial console to your VM from the Host Server

    - SSH onto the VM server.  Edit the VM’s settings:
        virsh edit billiejean

    - In the [devices] block, add (if not already present, as is the case on recent versions):
        serial type='pty'
              target port='0'/
        /serial

    - SSH into the VM guest and create a file /etc/init/ttyS0.conf containing:-
       start on stopped rc RUNLEVEL=[2345]
           stop on runlevel [!2345]
       respawn
           exec /sbin/getty -8 38400 ttyS0 xterm-256color

  - Again on the VM guest SSH session, run vi /etc/default/grub and override the lines shown below (these are near the top) until they read as follows.
    GRUB_DEFAULT=0
    #GRUB_HIDDEN_TIMEOUT=0
    GRUB_HIDDEN_TIMEOUT_QUIET=true
    GRUB_TIMEOUT=5
    GRUB_CMDLINE_LINUX_DEFAULT=""
    GRUB_CMDLINE_LINUX="console=tty1 console=ttyS0,115200n8"
    GRUB_TERMINAL=serial
    GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"

      Now save your changes, and run update-grub .

    - Restart your VM:
        virsh stop billiejean
        virsh start billiejean

     - Verify the console works by opening a console to the VM from the server:
             virsh console billiejean
    You may have to hit “Enter” before you see any console output.  After editing files you may find you need to execute "clear ; reset" on the command line if the display won't scroll properly.

At least now you will get to see if the kernel panics when you reboot after making some system change.

Once in a blue moon, system updates will change something in Grub which might stop the boot process completing.  So I would always take a VM Snapshot before applying system updates...

HOWTO: Take a rapid VM Snapshot before applying system updates or changes

I don't like to apply a load of security updates unless I can roll back the system quickly in the event of problems.   Happily this is pretty easy with Ubuntu Server VM's.   However, you will first need to convert your VM image files from the default RAW format to the compressed QCOW2 format.  This is a bit slower than RAW, but it does offer Snapshots.   A Snapshot is a point in time, stored inside the image file, that you can roll back to using the virsh command.

To convert a VM image file to QCOW2 (Copy On Write) format:-
    qemu-img convert -f raw vm21.img -O qcow2 vm21.img.qcow2

This command will take some minutes to execute (depending on the size of the VM).    The QCOW2 file will be smaller than the original, as it does not store unused areas of the virtual disk.

To take a snapshot, use a syntax like this:-
    virsh snapshot-create vm3

If your VM3 VM image is in RAW format rather than QCOW2 format, the snapshot-create command will fail with:
    error: Requested operation is not valid: Disk 'vm3.img' does not support snapshotting

To change the VM definition to use QCOW2, first use qemu-img as shown above to make a QCOW2 version of the VM.   Then use 'virsh edit vm3' (for example) rather than editing the file from Linux directly.  That way, the VM software will know about the change.  You want to find a line below "disk .." like this:-
    driver name='qemu' type='raw'/
    source file='/home/VMs/vm21.img'/
and change this to:    driver name='qemu' type='qcow2'/
    source file='/home/VMs/vm21.img.qcow2'/

To manage the snapshots, you can use 'virsh snapshot-list', 'virsh snapshot-restore', 'virsh snapshot-delete' (etc).

Note! The Snapshots are not external files.  They are stored within the QCOW2 file itself.  So, they are an easy way to image a virtual box before applying system updates (etc) so that you can quickly revert back.  But for an external backup, the easiest thing is probably to shut down the VM, and take a copy of the whole QCOW2 file so that you can then restart the VM, and then copy the backup file to another server.

Note! "virsh snapshot-create vm23" took over 20 minutes on a running (but idle) guest VM.  During this time, the VM guest did not respond via the network.  It is very much quicker to say: 
    virsh
        shutdown vm23
        (wait for 15 seconds for it to close down)
        snapshot-create vm23
        start vm23

Booting a VM is much faster than booting a real box.  So there's no advantage that I can see in snapshotting a running box.

HOWTO: Clone an existing VM

Let's say you've built a nice VM guest environment and you want to be able to make a copy of it, so that you can use it as a template for other instances.  There are several ways to make the clone…

First, obtain a new IP address (and MAC address if necessary).   For example on Kimsufi/OVH hosting, you need a new "failover" IP and a corresponding new virtual MAC ready (see above).

Now, SSH onto the server and issue the shell command "virt-clone" (see below).
    - I don't recommend using the GUI (virt-manager) as you end up with daft filename (vm1-clone) that then needs changing.
    - You might try the clone command in "virsh" (but close down the virt-manager GUI first, or it will get confused).
    - With virt-clone, you can copy the VM image and change its MAC address in one go:-
     virsh shutdown vm1
     virt-clone -o vm1 -n vm2 -f /home/VMs/vm2.img --mac 02:00:00:47:f4:da
     virsh start vm1

PROBLEM: If you clone a .img.qcow2 file (as opposed to a raw .img file), it won't boot.  This is because virt-clone marks it as a RAW file, not a QCOW2 file.
SOLUTION: Using the virt-manager GUI, open the VM and fix this...  "View" / "Details" / "VirtIO Disk 1" / "Advanced" / "Storage Format" (change from RAW to QCOW2).

NOTE: If you feel you have to edit the XML stuff by hand:
    - There are config files in /etc/libvirt/qemu and /var/lib/libvirt.  These are best edited using "virsh edit vmname".
    - You may have to restart /etc/init.d/libvirt-bin daemon after any changes made by hand in external editors.

QUESTION: How do we then edit the /etc files to personalize the cloned VM?
i.e.  
    /etc/networks
    /etc/network/interfaces
    /etc/hostname
    /etc/hosts
    /etc/resolv.conf  (possibly)
    /etc/udev/rules.d/70-persistent-net.rules (possibly)
    /root/.ssh/* (allowed keys - careful!)
    /etc/ssh/*   (config settings and host key)

ANSWER: Well you have three choices…

    A. Boot the clone (alone to avoid IP clash), edit over SSH, then reboot;
or
    B. Boot the clone (alone to avoid IP clash), and use the remote console for editing the files.
or
    C. Mount the clone's filesystem from the VM host, and edit/fix before booting it
    example:
     modprobe nbd max_part=63
     qemu-nbd -c /dev/nbd2 /home/VMs/vm2.img
     mkdir /mnt/vm2
     mount /dev/nbd2p1 /mnt/vm2
        .. edit files in /mnt/vm2/etc/...
        umount /mnt/vm2
     qemu-nbd -d /dev/nbd2
Option C is probably best.