Shrink and optimise an existing QCOW2 image

A virtual disk image is a block device in a file. There are a number of different disk image formats to choose from when setting up a virtual machine. QEMU Copy On Write version 2 (QCOW2) is the default virtual disk image format for the Quick Emulator (QEMU). Features such as thin provisioning, snapshots and compression make QCOW2 one of the most versatile virtual disk formats available.

In the following example, the overall objective is to shrink and optimise an existing image for random read and write operations. You can also grow disk images using the same approach.

These instructions specifically use Debian 12 with a GNOME desktop as the host, but they should also be applicable to other Linux distributions such as Ubuntu or Linux Mint. The virtual machine in this case is a Windows 10 guest using the NTFS file system.

Before you begin

Shut down the virtual machine and delete all existing snapshots from the image file.

Never modify images currently in use by a running virtual machine.

Step 1

On the host, install the necessary tools for working with virtual disk images.

$ sudo apt-get install --yes libguestfs-tools gnome-disk-utility

Step 2

Only root can access the host directory /var/lib/libvirt/images. Use the following command to obtain the necessary privileges.

$ sudo su

Step 3

Continue by creating a directory in which to keep your virtual machine backups.

# mkdir /var/lib/libvirt/backups

Step 4

Now create a backup of the virtual machine with the name windows by copying its QCOW2 image file to the backups directory.

# cp /var/lib/libvirt/images/windows.qcow2 /var/lib/libvirt/backups/windows-backup.qcow2

Step 5

Sparsify the image file to convert any free space within the disk image to free space on the host.

# virt-sparsify --in-place /var/lib/libvirt/images/windows.qcow2

Step 6

Rename the sparsified image file.

# mv /var/lib/libvirt/images/windows.qcow2 /var/lib/libvirt/images/windows-sparsified.qcow2

Step 7

Check the disk size of the sparsified image file. The disk size should be smaller than the virtual size. In this particular case, the disk size is 26.7 GiB and the virtual size 64 GiB.

# qemu-img info /var/lib/libvirt/images/windows-sparsified.qcow2
image: /var/lib/libvirt/images/windows-sparsified.qcow2
file format: qcow2
virtual size: 64 GiB (68719476736 bytes)
disk size: 26.7 GiB
cluster_size: 2097152
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false

Step 8

Determine which partition to resize by obtaining more detailed information about the contents of the sparsified disk image.

# virt-filesystems --long -h --all -a /var/lib/libvirt/images/windows-sparsified.qcow2

On the virtual device /dev/sda, the size of the partition /dev/sda2 is 63G. It appears to offer the greatest scope for resizing, as the overall disk size in Step 7 is only 26.7 GiB in total.

Name       Type        VFS   Label            MBR  Size  Parent
/dev/sda1  filesystem  ntfs  System Reserved  -    50M   -
/dev/sda2  filesystem  ntfs  -                -    63G   -
/dev/sda3  filesystem  ntfs  -                -    530M  -
/dev/sda1  partition   -     -                07   50M   /dev/sda
/dev/sda2  partition   -     -                07   63G   /dev/sda
/dev/sda3  partition   -     -                27   530M  /dev/sda
/dev/sda   device      -     -                -    64G   -

Step 9

Load the network block device (NBD) kernel module.

# modprobe nbd max_part=8

Step 10

Connect the sparsified image.

# qemu-nbd --connect=/dev/nbd9 /var/lib/libvirt/images/windows-sparsified.qcow2

Step 11

The partition /dev/sda2 listed in Step 8 is equivalent to /dev/nbd9p2 connected as a network block device. Use GNOME Disks to shrink /dev/nbd9p2 to its Minimal Size.

Use a graphical utility to minimise the risk of introducing errors.

Select the correct partition and from the pop-up menu choose the option to resize.
Select Minimal Size and resize the partition.

Step 12

Disconnect the resized image.

# qemu-nbd -d /dev/nbd9

Step 13

Unload the NBD kernel module.

# modprobe -r nbd

Step 14

Create a target image larger than the resized source image. In this example, the size of the target image is 32G and its format QCOW2 with full preallocation and a cluster size of 2M.

# qemu-img create -f qcow2 -o preallocation=full -o cluster_size=2M /var/lib/libvirt/images/windows-target.qcow2 32G

Step 15

Copy the source image to the target image and specify the partition to expand in the process.

# virt-resize --expand /dev/sda2 /var/lib/libvirt/images/windows-sparsified.qcow2 /var/lib/libvirt/images/windows-target.qcow2

Step 16

Confirm the actual disk size of the target image.

# qemu-img info /var/lib/libvirt/images/windows-target.qcow2
image: /var/lib/libvirt/images/windows-target.qcow2
file format: qcow2
virtual size: 32 GiB (34359738368 bytes)
disk size: 32 GiB
cluster_size: 2097152
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false

Step 17

Obtain detailed information about the contents of the target disk image.

# virt-filesystems --long -h --all -a /var/lib/libvirt/images/windows-target.qcow2

The partition /dev/sda2 of the virtual device /dev/sda is now 31G in size. The overall disk size in Step 16 is now only 32 GiB in total.

Name       Type        VFS   Label            MBR  Size  Parent
/dev/sda1  filesystem  ntfs  System Reserved  -    50M   -
/dev/sda2  filesystem  ntfs  -                -    31G   -
/dev/sda3  filesystem  ntfs  -                -    530M  -
/dev/sda1  partition   -     -                07   50M   /dev/sda
/dev/sda2  partition   -     -                07   31G   /dev/sda
/dev/sda3  partition   -     -                27   530M  /dev/sda
/dev/sda   device      -     -                -    32G   -

Step 18

Rename the target image file.

# mv /var/lib/libvirt/images/windows-target.qcow2 /var/lib/libvirt/images/windows.qcow2

All done!

You can also modify format specific options for an existing image without having to create a target disk image. Or alternatively expand into a target image that uses a format compatible with other hypervisors, such as RAW, VMDK, VDI, VHD, VHDX or QED.

A hacker’s perspective: social media account takeover prevention guide

“If you watched the SEC Twitter account hack that moved markets yesterday and wondered how to prevent account takeover for your personal, business, or high profile social media account, here’s an Account Takeover Prevention Guide for you and/or your organization.” @racheltobac neatly summarises the steps you should take to prevent the hijacking of your online accounts.

www.linkedin.com

Monitoring storage devices with smartmontools on Debian or Ubuntu

The acronym SMART stands for Self-Monitoring, Analysis and Reporting Technology and is a monitoring system built into most modern storage devices. The package smartmontools includes the utilities smartctl and smartd, which process SMART data to ‘provide advanced warning of disk degradation and failure‘.

Step 1

Start by configuring nullmailer to receive status updates from your system.

Step 2

Install smartmontools and update its drive database to the latest version.

$ sudo apt-get install --yes smartmontools smart-notifier && sudo update-smart-drivedb

Step 3

Continue by obtaining relevant information about available storage devices.

$ sudo smartctl --scan

Depending on the type of disk, you should see a block of information similar to the following.

/dev/sda -d scsi # /dev/sda, SCSI device

Step 4

Enable SMART support for and display detailed information about the device.

$ sudo smartctl -iHs on /dev/sda

Ideally, information about the device to be monitored would be found in the drive database.

=== START OF INFORMATION SECTION ===
Device is: In smartctl database [for details use: -P show]

The device should report a successful self-assessment test.

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

Please note: the drive database does not extend to NVMe devices. SMART support for NVMe devices is curently limited to a subset of features.

Step 5

Verify the SMART capabilities of the device.

$ sudo smartctl -c /dev/sda

The following output confirms that the device /dev/sda has both short and extended self-test capabilites.

=== START OF READ SMART DATA SECTION ===
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 (  85) minutes.

The output provides estimates for the duration of short and extended (long) self-test routines.

If the device is capable of self-tests

Use the following command to run a short self-test.

$ sudo smartctl -t short /dev/sda

Use the following command to run a long self-test.

$ sudo smartctl -t long /dev/sda

Display a list with the results of recent self-tests in reverse chronological order.

$ sudo smartctl -l selftest /dev/sda

All tests should have completed without error.

Step 6

Edit the default configuration /etc/smartd.conf and comment out any DEVICESCAN options, thus preventing smartd from attempting to search for attached devices indiscriminately.

On Debian 12, you can use the following command to comment out the DEVICESCAN option in the default configuration file.

$ sudo sed -i 's/DEVICESCAN -d removable -n standby -m root -M exec/#DEVICESCAN -d removable -n standby -m root -M exec/' /etc/smartd.conf

Step 7

Example configuration for smartd and SATA devices

For the device /dev/sda, the following configuration for monitoring the device with smartd would have to be added to /etc/smartd.conf.

/dev/sda -H -l error -l selftest -S on -s (L/../.././06|S/../.././18) -m root -M test

-H display the health status as reported by the device

-l error show the increase in the number of SMART errors since last check

-l selftest show the increase in the number of failed tests in the SMART Self-Test Log

-S on enable Attribute Autosave on startup

-s (L/../.././06|S/../.././18) schedule a long self-test between 06:00 and 07:00 daily and a short self-test between 18:00 and 19:00 daily

-m root local user root receives warning by email

-M test send a test email on startup

Example configuration for smartd and NVMe devices

Current versions of smartmontools offer experimental support for NVMe devices. In practice this means that only a limited, but still useful, feature set is available.

For the device /dev/nvme0, the following configuration for monitoring the device with smartd would have to be added to the end of /etc/smartd.conf.

/dev/nvme0 -H -l error -m root -M test

-H display the health status as reported by the device

-l error show the increase in the number of SMART errors since last check

-m root local user root receives warning by email

-M test send a test email on startup

Letter from Birmingham jail

“Moreover, I am cognizant of the interrelatedness of all communities and states. I cannot sit idly by in Atlanta and not be concerned about what happens in Birmingham. Injustice anywhere is a threat to justice everywhere. We are caught in an inescapable network of mutuality, tied in a single garment of destiny. Whatever affects one directly, affects all indirectly.”

Dr. Martin Luther King, writing from a prison cell in 1963

letterfromjail.com

With thanks to @Snowshadow.

Descartes was wrong: ‘a person is a person through other persons’

“Nothing simply is itself, outside the matrix of relationships in which it appears. Instead, being is an act or event that must happen in the space between the self and the world.” Abeba Birhane does not simply regard human beings as either self-contained or self-sufficient. But can relational and autonomous accounts of the self be reconciled?

aeon.co

With thanks to @chinicuil.

European Comission gives EU-US data transfers third round at CJEU

“This third attempt to pass largely the same unlawful decision also raises questions as to the larger role of the European Commission being the guardian of the EU treaties. Instead of upholding the ‘rule of law’ the Commission simply passes an invalid decision over and over again, despite clear rulings by the CJEU.” By agreeing the Data Privacy Framework with the US, the European Commission likely prioritised diplomatic and business interests over the rights of Europeans.

noyb.eu

Nichts zu verbergen? Ein moderner Mythos und 12 Argumente dagegen

“Aber der Satz und seine Verbreitung schaden viel mehr der Gesellschaft und anderen Menschen, als jenen, die ihn aussprechen. Deshalb sind mir die sozial orientierten Antworten darauf am liebsten: Weil man Menschen damit stigmatisiert, sie unsolidarisch behandelt, ihre Diskriminierungserfahrungen negiert, und weil es Demokratie und Widerstand untergräbt.” @reticuleena legt offen, wie wir unsere Vertrauenswürdigkeit aufs Spiel setzen.

www.kuketz-blog.de

Install and configure SSH on Debian or Ubuntu

SSH is a protocol that enables secure connections over unsecured networks. It supports the use of asymmetric encryption for user authentication. Private keys are kept locally, while public keys are stored on the remote machine.

The following configuration disables root logins on the remote machine. Only users belonging to the group ssh-users may establish a connection. Access to the remote machine is tied to the local user’s private key.

In this example, the name of the remote machine is debian-server, which has the address 192.168.1.10 on the network. sid is a user on debian-server, whereas bookworm is a user on the local machine. Choose an encryption passphrase to secure the private key that you will generate in Step 5.

On the remote machine

Step 1

Install the secure shell server with the following command:

$ sudo apt install --yes openssh-server

Step 2

If you are using ufw as a host-based firewall

Configure ufw to allow connections to the secure shell server.

$ sudo ufw limit ssh

If you are using firewalld as a host-based firewall

Configure firewalld to allow connections to the secure shell server.

$ sudo -- bash -c 'firewall-cmd --zone=public --add-service=ssh --permanent && firewall-cmd --reload && firewall-cmd --info-zone=public'

Step 3

Restrict access to the remote machine to members of a specific group. Start by creating the group ssh-users.

$ sudo addgroup --system ssh-users

Add the user sid to the group ssh-users.

$ sudo adduser sid ssh-users

On the local machine

Step 4

Install the secure shell client with the following command.

$ sudo apt install openssh-client

Step 5

Generate a new key pair for the local user bookworm:

$ ssh-keygen -t ed25519 -o -a 100

Save the key pair to the directory /home/bookworm/.ssh/. Choose a name that facilitates easy identification.

Enter file in which to save the key (/home/bookworm/.ssh/id_ed25519): ~/.ssh/id_ed25519-debian-server

The use of an appropriate passphrase to secure the private key is mandatory.

Step 6

Create the file ~/.ssh/config to configure the secure shell client.

$ nano ~/.ssh/config

Add the follwing minimal entry for the host debian-server.

Host debian-server
   Hostname 192.168.1.10
   IdentitiesOnly yes

Step 7

Deploy the public key with the following command.

$ ssh-copy-id -i ~/.ssh/id_ed25519-debian-server.pub sid@debian-server

Step 8

Log into the remote machine.

$ ssh -i ~/.ssh/id_ed25519-debian-server sid@debian-server

When prompted to confirm the authenticity of the host debian-server, type yes and press [Enter].

The authenticity of host 'debian-server (192.168.1.10)' can't be established.
ED25519 key fingerprint is SHA256:C9RxLLVbvFwVJc0L4JHzcuHQSaPHJZe/GrRDvqy6rAG.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? 

In the next step, enter the passphrase for your private key.

Enter passphrase for key '/home/bookworm/.ssh/id_ed25519-debian-server':

Step 9

On the remote machine, download a file to harden the ssh server. You are encouraged to inspect its contents.

$ sudo -- bash -c 'wget -P /etc/ssh/sshd_config.d/ --show-progress https://edafe.de/debian/sshd_config.conf'

Activate the modifications on the remote machine.

$ sudo systemctl restart ssh.service

Step 9

On the local machine, open a new terminal window and run the following command.

$ ssh -i ~/.ssh/id_ed25519-debian-server sid@debian-server

In the next step, enter the passphrase for your private key.

Enter passphrase for key '/home/bookworm/.ssh/id_ed25519-debian-server':

Display the active configuration for the remote ssh server and verify its settings, paying particular attention to options for maxauthtries, permitrootlogin and passwordauthentication.

$ sudo sshd -T

All done!

For more in-depth information, please see stribika’s post-Snowden advice on hardening OpenSSH server installations.

The book SSH The Secure Shell by Daniel Barrett, Richard Silverman and Robert Byrnes is still useful today and has information on other clever stuff you can do with SSH.

Click to copy