Shrink and optimise an existing QCOW2 image

A virtual disk image is a block device in a file. There are a number of different disk image formats to choose from when setting up a virtual machine. QEMU Copy On Write version 2 (QCOW2) is the default virtual disk image format for the Quick Emulator (QEMU). Features such as thin provisioning, snapshots and compression make QCOW2 one of the most versatile virtual disk formats available.

In the following example, the overall objective is to shrink and optimise an existing image for random read and write operations. You can also grow disk images using the same approach.

These instructions specifically use Debian 12 with a GNOME desktop as the host, but they should also be applicable to other Linux distributions such as Ubuntu or Linux Mint. The virtual machine in this case is a Windows 10 guest using NTFS as a file system.

Before you begin

Shut down the virtual machine and delete all existing snapshots from the image file.

Never modify images in use by a running virtual machine.

Step 1

Install the necessary tools for working with virtual disk images on the host.

$ sudo apt-get install --yes libguestfs-tools gnome-disk-utility

Step 2

Only root can access /var/lib/libvirt/images. Use the following command to gain the necessary privileges.

$ sudo su

Step 3

Continue by creating a directory in which to keep your virtual machine backups.

# mkdir /var/lib/libvirt/backups

Step 4

Now create a backup of the virtual machine with the name windows by copying its QCOW2 image file to the backups directory.

# cp /var/lib/libvirt/images/windows.qcow2 /var/lib/libvirt/backups/windows-backup.qcow2

Step 5

Sparsify the image file to convert any free space within the disk image to free space on the host.

# virt-sparsify --in-place /var/lib/libvirt/images/windows.qcow2

Step 6

Rename the sparsified image file.

# mv /var/lib/libvirt/images/windows.qcow2 /var/lib/libvirt/images/windows-sparsified.qcow2

Step 7

Check the disk size of the sparsified image file. The disk size should be smaller than the virtual size. In this particular case, the disk size is 26.7 GiB and the virtual size 64 GiB.

# qemu-img info /var/lib/libvirt/images/windows-sparsified.qcow2
image: /var/lib/libvirt/images/windows-sparsified.qcow2
file format: qcow2
virtual size: 64 GiB (68719476736 bytes)
disk size: 26.7 GiB
cluster_size: 2097152
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false

Step 8

Determine which partition to resize by obtaining more detailed information about the contents of the sparsified disk image.

# virt-filesystems --long -h --all -a /var/lib/libvirt/images/windows-sparsified.qcow2

On the virtual device /dev/sda, the size of the partition /dev/sda2 is 63G. It appears to offer the greatest scope for resizing, as the overall disk size in Step 7 is only 26.7 GiB in total.

Name       Type        VFS   Label            MBR  Size  Parent
/dev/sda1  filesystem  ntfs  System Reserved  -    50M   -
/dev/sda2  filesystem  ntfs  -                -    63G   -
/dev/sda3  filesystem  ntfs  -                -    530M  -
/dev/sda1  partition   -     -                07   50M   /dev/sda
/dev/sda2  partition   -     -                07   63G   /dev/sda
/dev/sda3  partition   -     -                27   530M  /dev/sda
/dev/sda   device      -     -                -    64G   -

Step 9

Load the network block device (NBD) kernel module.

# modprobe nbd max_part=8

Step 10

Connect the sparsified image.

# qemu-nbd --connect=/dev/nbd9 /var/lib/libvirt/images/windows-sparsified.qcow2

Step 11

The partition /dev/sda2 listed in Step 8 is equivalent to /dev/nbd9p2 connected as a network block device. Use GNOME Disks to shrink /dev/nbd9p2 to its Minimal Size.

The use of a graphical utility is intended to reduce the risk of introducing errors.

Select the correct partition and from the pop-up menu choose the option to resize.
Select Minimal Size and resize the partition.

Step 12

Disconnect the resized image.

# qemu-nbd -d /dev/nbd9

Step 13

Unload the NBD kernel module.

# modprobe -r nbd

Step 14

Create a target image larger than the resized source image. In this example, the size of the target image is 32G and its format QCOW2 with full preallocation and a cluster size of 2M.

# qemu-img create -f qcow2 -o preallocation=full -o cluster_size=2M /var/lib/libvirt/images/windows-target.qcow2 32G

Step 15

Copy the source image to the target image and specify the partition to expand in the process.

# virt-resize --expand /dev/sda2 /var/lib/libvirt/images/windows-sparsified.qcow2 /var/lib/libvirt/images/windows-target.qcow2

Step 16

Confirm the actual disk size of the target image.

# qemu-img info /var/lib/libvirt/images/windows-target.qcow2
image: /var/lib/libvirt/images/windows-target.qcow2
file format: qcow2
virtual size: 32 GiB (34359738368 bytes)
disk size: 32 GiB
cluster_size: 2097152
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false

Step 17

Obtain detailed information about the contents of the target disk image.

# virt-filesystems --long -h --all -a /var/lib/libvirt/images/windows-target.qcow2

The partition /dev/sda2 of the virtual device /dev/sda is now 31G in size. The overall disk size in Step 16 is now only 32 GiB in total.

Name       Type        VFS   Label            MBR  Size  Parent
/dev/sda1  filesystem  ntfs  System Reserved  -    50M   -
/dev/sda2  filesystem  ntfs  -                -    31G   -
/dev/sda3  filesystem  ntfs  -                -    530M  -
/dev/sda1  partition   -     -                07   50M   /dev/sda
/dev/sda2  partition   -     -                07   31G   /dev/sda
/dev/sda3  partition   -     -                27   530M  /dev/sda
/dev/sda   device      -     -                -    32G   -

Step 18

Rename the target image file.

# mv /var/lib/libvirt/images/windows-target.qcow2 /var/lib/libvirt/images/windows.qcow2

All done!

You can also modify format specific options for an existing image without having to create a target disk image. Or alternatively expand into a target image that uses a format compatible with other hypervisors, such as RAW, VMDK, VDI, VHD, VHDX or QED.

Monitoring storage devices with smartmontools on Debian or Ubuntu

The acronym SMART stands for Self-Monitoring, Analysis and Reporting Technology and is a monitoring system built into most modern storage devices. The package smartmontools includes the utilities smartctl and smartd, which process SMART data to ‘provide advanced warning of disk degradation and failure‘.

Step 1

Start by configuring nullmailer to receive status updates from your system.

Step 2

Install smartmontools and update its drive database to the latest version.

$ sudo apt-get install --yes smartmontools smart-notifier && sudo update-smart-drivedb

Step 3

Continue by obtaining relevant information about available storage devices.

$ sudo smartctl --scan

Depending on the type of disk, you should see a block of information similar to the following.

/dev/sda -d scsi # /dev/sda, SCSI device

Step 4

Enable SMART support for and display detailed information about the device.

$ sudo smartctl -iHs on /dev/sda

Ideally, information about the device to be monitored would be found in the drive database.

=== START OF INFORMATION SECTION ===
Device is: In smartctl database [for details use: -P show]

The device should report a successful self-assessment test.

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

Please note: the drive database does not extend to NVMe devices. SMART support for NVMe devices is curently limited to a subset of features.

Step 5

Verify the SMART capabilities of the device.

$ sudo smartctl -c /dev/sda

The following output confirms that the device /dev/sda has both short and extended self-test capabilites.

=== START OF READ SMART DATA SECTION ===
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 (  85) minutes.

The output provides estimates for the duration of short and extended (long) self-test routines.

If the device is capable of self-tests

Use the following command to run a short self-test.

$ sudo smartctl -t short /dev/sda

Use the following command to run a long self-test.

$ sudo smartctl -t long /dev/sda

Display a list with the results of recent self-tests in reverse chronological order.

$ sudo smartctl -l selftest /dev/sda

All tests should have completed without error.

Step 6

Edit the default configuration /etc/smartd.conf and comment out any DEVICESCAN options, thus preventing smartd from attempting to search for attached devices indiscriminately.

On Debian 12, you can use the following command to comment out the DEVICESCAN option in the default configuration file.

$ sudo sed -i 's/DEVICESCAN -d removable -n standby -m root -M exec/#DEVICESCAN -d removable -n standby -m root -M exec/' /etc/smartd.conf

Step 7

Example configuration for smartd and SATA devices

For the device /dev/sda, the following configuration for monitoring the device with smartd would have to be added to /etc/smartd.conf.

/dev/sda -H -l error -l selftest -S on -s (L/../.././06|S/../.././18) -m root -M test

-H display the health status as reported by the device

-l error show the increase in the number of SMART errors since last check

-l selftest show the increase in the number of failed tests in the SMART Self-Test Log

-S on enable Attribute Autosave on startup

-s (L/../.././06|S/../.././18) schedule a long self-test between 06:00 and 07:00 daily and a short self-test between 18:00 and 19:00 daily

-m root local user root receives warning by email

-M test send a test email on startup

Example configuration for smartd and NVMe devices

Current versions of smartmontools offer experimental support for NVMe devices. In practice this means that only a limited, but still useful, feature set is available.

For the device /dev/nvme0, the following configuration for monitoring the device with smartd would have to be added to the end of /etc/smartd.conf.

/dev/nvme0 -H -l error -m root -M test

-H display the health status as reported by the device

-l error show the increase in the number of SMART errors since last check

-m root local user root receives warning by email

-M test send a test email on startup

Install and configure SSH on Debian or Ubuntu

SSH is a protocol that enables secure connections over unsecured networks. It supports the use of asymmetric encryption for user authentication. Private keys are kept locally, while public keys are stored on the remote machine.

The following configuration disables root logins on the remote machine. Only users belonging to the group ssh-users may establish a connection. Access to the remote machine is tied to the local user’s private key.

In this example, the name of the remote machine is debian-server, which has the address 192.168.1.10 on the network. sid is a user on debian-server, whereas bookworm is a user on the local machine. Choose an encryption passphrase to secure the private key that you will generate in Step 5.

On the remote machine

Step 1

Install the secure shell server with the following command:

$ sudo apt install --yes openssh-server

Step 2

If you are using ufw as a host-based firewall

Configure ufw to allow connections to the secure shell server.

$ sudo ufw limit ssh

If you are using firewalld as a host-based firewall

Configure firewalld to allow connections to the secure shell server.

$ sudo -- bash -c 'firewall-cmd --zone=public --add-service=ssh --permanent && firewall-cmd --reload && firewall-cmd --info-zone=public'

Step 3

Restrict access to the remote machine to members of a specific group. Start by creating the group ssh-users.

$ sudo addgroup --system ssh-users

Add the user sid to the group ssh-users.

$ sudo adduser sid ssh-users

On the local machine

Step 4

Install the secure shell client with the following command.

$ sudo apt install openssh-client

Step 5

Generate a new key pair for the local user bookworm:

$ ssh-keygen -t ed25519 -o -a 100

Save the key pair to the directory /home/bookworm/.ssh/. Choose a name that facilitates easy identification.

Enter file in which to save the key (/home/bookworm/.ssh/id_ed25519): ~/.ssh/id_ed25519-debian-server

The use of an appropriate passphrase to secure the private key is mandatory.

Step 6

Create the file ~/.ssh/config to configure the secure shell client.

$ nano ~/.ssh/config

Add the follwing minimal entry for the host debian-server.

Host debian-server
   Hostname 192.168.1.10
   IdentitiesOnly yes

Step 7

Deploy the public key with the following command.

$ ssh-copy-id -i ~/.ssh/id_ed25519-debian-server.pub sid@debian-server

Step 8

Log into the remote machine.

$ ssh -i ~/.ssh/id_ed25519-debian-server sid@debian-server

When prompted to confirm the authenticity of the host debian-server, type yes and press [Enter].

The authenticity of host 'debian-server (192.168.1.10)' can't be established.
ED25519 key fingerprint is SHA256:C9RxLLVbvFwVJc0L4JHzcuHQSaPHJZe/GrRDvqy6rAG.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? 

In the next step, enter the passphrase for your private key.

Enter passphrase for key '/home/bookworm/.ssh/id_ed25519-debian-server':

Step 9

On the remote machine, download a file to harden the ssh server. You are encouraged to inspect its contents.

$ sudo -- bash -c 'wget -P /etc/ssh/sshd_config.d/ --show-progress https://edafe.de/debian/sshd_config.conf'

Activate the modifications on the remote machine.

$ sudo systemctl restart ssh.service

Step 9

On the local machine, open a new terminal window and run the following command.

$ ssh -i ~/.ssh/id_ed25519-debian-server sid@debian-server

In the next step, enter the passphrase for your private key.

Enter passphrase for key '/home/bookworm/.ssh/id_ed25519-debian-server':

Display the active configuration for the remote ssh server and verify its settings, paying particular attention to options for maxauthtries, permitrootlogin and passwordauthentication.

$ sudo sshd -T

All done!

For more in-depth information, please see stribika’s post-Snowden advice on hardening OpenSSH server installations.

The book SSH The Secure Shell by Daniel Barrett, Richard Silverman and Robert Byrnes is still useful today and has information on other clever stuff you can do with SSH.

Install OneDrive Client for Linux on Debian or Ubuntu

The OneDrive Client for Linux connects your Debian or Ubuntu system to Microsoft’s OneDrive Personal, OneDrive for Business, OneDrive for Office365, Sharepoint and other such deployments.

Step 1

Install the OneDrive Client from the Debian or Ubuntu repository.

$ sudo -- bash -c 'apt update && apt install --yes onedrive'

Step 2

Begin to connect the client to your OneDrive account.

$ onedrive --synchronize

You will be presented with a message similar to the following:

Configuring Global Azure AD endpoints
Authorize this app visiting:

https://login.microsoftonline.com/common/oauth2/v2.0/authorize?client_id&scope=Files.ReadWrite%20Files.ReadWrite.all%20Sites.Read.All%Sites.ReadWrite.All%20offline_accessresponse_type=code&redirect_uri=https://login.microsoftonline.com/common/oauth2/nativeclient

Enter the response uri:

In the above dialog, copy or [Ctrl + Click] the URI beginning with https://login.microsoftonline.com/.

In a web browser

Use the URI from the previous step to sign into your Microsoft account. You will be redirected to a response URI displaying a blank page. Copy the response URI from the address field of your browser.

In the terminal

Paste the response URI into the terminal. On successful authentication, the OneDrive Client will connect to your Microsoft account and begin to download your data.

Initializing the Synchronization Engine …
Syncing changes from OneDrive …
Creating local directory:
Downloading file … done.
Uploading differences of ~/OneDrive
Uploading new items of ~/OneDrive

Step 3

After downloading your data to ~/OneDrive, validate the configuration of the client.

$ onedrive --display-config

If required, you may change the default configuration.

Step 4

Enable OneDrive Client for the local user bookworm.

$ sudo -- bash -c 'systemctl enable onedrive@bookworm.service && systemctl start onedrive@bookworm.service && systemctl status onedrive@bookworm.service'

All done!

Install and configure nullmailer using Fastmail as a smarthost

If you want to receive status updates from your Debian or Ubuntu system, you need to employ the help of a mail tansfer agent (MTA). nullmailer is a relay-only forwarding MTA that can be used as an alternative to more complex MTAs such as Exim, Sendmail or Postfix.

nullmailer can be configured to use Fastmail as a smarthost and hence ensure the deliverability of your messages. In principle, these instructions should also be applicable to service providers other than Fastmail.

In the following example configuration, debian is the hostname, bookworm the local username and linus.torvalds@fastmail.com the Fastmail username.

Step 1

Log into your Fastmail account and set up a new app password for SMTP authentication.

Step 2

Create the new directory /etc/nullmailer and the file /etc/nullmailer/adminaddr.

$ sudo mkdir /etc/nullmailer && sudo nano /etc/nullmailer/adminaddr

Your Fastmail username is the only entry in /etc/nullmailer/adminaddr.

linus.torvalds@fastmail.com

Step 3

Install the required packages.

$ sudo apt-get install --yes nullmailer mailutils

Step 4

Perform the initial configuration using debconf. Reconfigure nullmailer at any time after the initial installation using the following comand.

$ sudo dpkg-reconfigure nullmailer

Setting the mail name

Set the system mail name. If you are setting up on a home network, you should use home.arpa as the domain name.

Configuring nullmailer

Mailname of your system:

debian.home.arpa

Ok

Configuring the smarthost

Set the Fastmail server as the smarthost. Use the app password you set in Step 1.

Configuring nullmailer

Smarthosts:

smtp.fastmail.com smtp --port=587 --auth-login --starttls --user=linus.torvalds@fastmail.com --pass=password

Ok

Step 5

Test your configuration with the following command.

$ echo "Test mail from nullmailer on debian.home.arpa to the local root user and forwarded on to Fastmail" | mail -s "Test nullmailer" root

Check your Inbox, Linus!

Install Syncthing for continuous file synchronisation on Debian or Ubuntu

Syncthing is an open source tool that synchronises data across multiple devices. It transfers your files peer-to-peer, without the requirement to upload your information to the cloud. Packages are available for Android, Windows, macOS and Linux (including Synology DSM).

The usefulness of this project cannot be overstated.

Running the Syncthing stable channel

Syncthing is included in the Debian and Ubuntu repositories, respectively. These instructions are targeting the latest release of the Syncthing stable channel. It is therefore necessary to add the Syncthing repository to your list of APT sources.

In the following example, bookworm is the local username.

Step 1

Add the Syncthing release key for validation of packages downloaded from the Syncthing repository.

$ sudo curl -o /usr/share/keyrings/syncthing-archive-keyring.gpg https://syncthing.net/release-key.gpg

Step 2

Add the Syncthing repository.

$ echo "deb [signed-by=/usr/share/keyrings/syncthing-archive-keyring.gpg] https://apt.syncthing.net/ syncthing stable" | sudo tee /etc/apt/sources.list.d/syncthing.list

Step 3

Install Syncthing on your system.

$ sudo -- bash -c 'apt update && apt install --yes syncthing apt-transport-https'

Step 4

Enable Syncthing for the local user bookworm.

$ sudo -- bash -c 'systemctl enable syncthing@bookworm.service && systemctl start syncthing@bookworm.service && systemctl status syncthing@bookworm.service'

Step 5

You may need to edit your firewall settings to open ports for incoming and outgoing traffic.

If you are using ufw as a host-based firewall

Configure ufw to allow connections to Syncthing.

$ sudo ufw limit syncthing

If you are using firewalld as a host-based firewall

Configure firewalld to allow connections to Syncthing.

$ sudo -- bash -c 'firewall-cmd --zone=public --add-service=syncthing --permanent && firewall-cmd --reload && firewall-cmd --info-zone=public'

Step 6

Access the Syncthing configuration page by using your browser to navigate to the following address:

http://localhost:8384

Step 7

Complete your setup by referring to the Syncthing documentation.

GNOME shell tutorial: desktop workflow explained


Playing this video requires sharing information with Google. Read the privacy policy

“Before we get started, let me say this upfront: GNOME shell is not a traditional desktop and if you try to use it as one, you will not be very efficient.”

AJ Reissig

What’s your favourite desktop and why?

In response to Voice of the Masses

My favourite Desktop is Unity because it is not MATE. This has been bugging me for quite some time.
Like almost everyone else on the planet, I was unhappy when in 2011 Canonical declared Unity Ubuntu’s new default desktop. After years of using GNOME 2, I just thought that Unity felt a bit awkward. But I stuck with it, mainly for a perceived lack of alternatives and my wish to avoid PPAs if at all possible.
Fast-forward a few years and, thanks to the excellent Martin Wimpress, I hear of MATE Desktop Environment almost every other podcast I listen to. With the release of Ubuntu 15.10, MATE is finally elevated to official flavour status and I was sure to be making the switch away from Unity.
I ended up using MATE for about one day before going back to Unity. It was quite an uncomfortable thing to have to admit, but there was a problem: After years of using Unity, I just thought that MATE felt a bit awkward…

ubuntu-mate.org

Click to copy