Tag: debian

Shrink, optimise and expand an existing QCOW2 image

A virtual disk image is a block device in a file. There are a number of different disk image formats to choose from when setting up a virtual machine. QEMU Copy On Write version 2 (QCOW2) is the default virtual disk image format for the Quick Emulator (QEMU). Features such as thin provisioning, snapshots and compression make QCOW2 one of the most versatile virtual disk formats available.

These instructions specifically target Debian 12 with a GNOME desktop as the host, but they should also be applicable to other Linux distributions such as Ubuntu or Linux Mint. The guest in this particular example is a Windows 11 virtual machine that has run out of space.

The overall objective is to shrink and optimise the 64 GiB disk image for random read and write operations before expanding it to a desired size of 128 GiB.

With thanks to Fam Zheng.

Before you begin

Shut down the virtual machine and delete all existing snapshots from the image file.

Never modify images currently in use by a running virtual machine.

Step 1

On the host, install the necessary tools for working with virtual disk images.

$ sudo apt-get install --yes libguestfs-tools gnome-disk-utility

Step 2

Only root can access the host directory /var/lib/libvirt/images. Use the following command to obtain the necessary privileges.

$ sudo su

Step 3

Continue by creating a directory in which to keep your virtual machine backups.

# mkdir /var/lib/libvirt/backups

Step 4

Now create a backup of the virtual machine with the name windows by copying its QCOW2 image file to the backups directory.

# cp /var/lib/libvirt/images/windows.qcow2 /var/lib/libvirt/backups/windows-backup.qcow2

Step 5

Sparsify the image file to convert any free space within the disk image to free space on the host.

# virt-sparsify --in-place /var/lib/libvirt/images/windows.qcow2

Step 6

Rename the sparsified image file.

# mv /var/lib/libvirt/images/windows.qcow2 /var/lib/libvirt/images/windows-sparsified.qcow2

Step 7

Check the disk size of the sparsified image file.

# qemu-img info /var/lib/libvirt/images/windows-sparsified.qcow2

The disk size should be smaller than the virtual size. In this particular case, the disk size is 33.7 GiB and the virtual size 64 GiB.

image: /var/lib/libvirt/images/windows-sparsified.qcow2
file format: qcow2
virtual size: 64 GiB (68719476736 bytes)
disk size: 33.7 GiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: true
    refcount bits: 16
    corrupt: false
    extended l2: false

Step 8

Determine which partition to resize by obtaining more detailed information about the contents of the sparsified disk image.

# virt-filesystems --long -h --all -a /var/lib/libvirt/images/windows-sparsified.qcow2

On the virtual device /dev/sda, the partition /dev/sda3 is equivalent to the Local Disk (C:) of the Windows 11 virtual machine.

Name       Type        VFS   Label  MBR  Size  Parent
/dev/sda1  filesystem  vfat  -      -    96M   -
/dev/sda3  filesystem  ntfs  -      -    63G   -
/dev/sda4  filesystem  ntfs  -      -    768M  -
/dev/sda1  partition   -     -      -    100M  /dev/sda
/dev/sda2  partition   -     -      -    16M   /dev/sda
/dev/sda3  partition   -     -      -    63G   /dev/sda
/dev/sda4  partition   -     -      -    768M  /dev/sda
/dev/sda   device      -     -      -    64G   -

Step 9

Load the network block device (NBD) kernel module.

# modprobe nbd max_part=8

Step 10

Connect the sparsified image.

# qemu-nbd --connect=/dev/nbd9 /var/lib/libvirt/images/windows-sparsified.qcow2

Step 11

The partition /dev/sda3 listed in Step 8 is equivalent to /dev/nbd9p3 connected as a network block device. Use GNOME Disks to shrink /dev/nbd9p3 to its Minimal Size.

Use a graphical utility to minimise the risk of introducing errors.

This image has an empty alt attribute; its file name is disks-options-825x586.png
Select the correct partition and from the pop-up menu, choose the option Resize…
This image has an empty alt attribute; its file name is resize-volume.png
Select Minimal Size and resize the partition.

Step 12

Disconnect the resized image.

# qemu-nbd -d /dev/nbd9

Step 13

Unload the NBD kernel module.

# modprobe -r nbd

Step 14

Create a target image larger than the resized source image. In this example, the size of the target image is 128G and its format QCOW2 with full preallocation and a cluster size of 2M.

# qemu-img create -f qcow2 -o preallocation=full -o cluster_size=2M /var/lib/libvirt/images/windows-target.qcow2 128G

Step 15

Copy the source image to the target image. Specify the correct partition which to expand in the process.

# virt-resize --expand /dev/sda3 /var/lib/libvirt/images/windows-sparsified.qcow2 /var/lib/libvirt/images/windows-target.qcow2

Step 16

Confirm the size of the target image.

# qemu-img info /var/lib/libvirt/images/windows-target.qcow2

The overall disk size is now 128 GiB in total.

image: /var/lib/libvirt/images/windows-target.qcow2
file format: qcow2
virtual size: 128 GiB (137438953472 bytes)
disk size: 128 GiB
cluster_size: 2097152
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false

Step 17

Obtain more detailed information about the contents of the target disk image.

# virt-filesystems --long -h --all -a /var/lib/libvirt/images/windows-target.qcow2

The partition /dev/sda3 of the virtual device /dev/sda is now 127G in size.

Name       Type        VFS   Label  MBR  Size  Parent
/dev/sda1  filesystem  vfat  -      -    96M   -
/dev/sda3  filesystem  ntfs  -      -    127G  -
/dev/sda4  filesystem  ntfs  -      -    768M  -
/dev/sda1  partition   -     -      -    100M  /dev/sda
/dev/sda2  partition   -     -      -    16M   /dev/sda
/dev/sda3  partition   -     -      -    127G  /dev/sda
/dev/sda4  partition   -     -      -    768M  /dev/sda
/dev/sda   device      -     -      -    128G  -

Step 18

Rename the target image file.

# mv /var/lib/libvirt/images/windows-target.qcow2 /var/lib/libvirt/images/windows.qcow2

All done!

You can also modify format specific options for an existing image without having to create a target disk image. Or alternatively expand into a target image that uses a format compatible with other hypervisors, such as RAW, VMDK, VDI, VHD, VHDX or QED.

How to install Espanso on Debian 12 bookworm from source

Unmet dependencies continue to prevent available Espanso packages from installing on my machine running Debian 12. Given that I depend on Espanso to expand all kinds of text and wanted to upgrade to the latest version for Wayland, I updated the following instructions and installed from source.

Following these instructions, Espanso 2.2.1 for Wayland will be installed on your system and enabled for the current user.

Compiling Espanso from source code

Side-step any dependency problems by compliling Espanso from source and moving the binary into place.

Step 1

Install the required C/C++ compiler and some additional tools.

$ sudo apt-get install --yes build-essential curl git wl-clipboard libxkbcommon-dev libdbus-1-dev libwxgtk3.*-dev libssl-dev

Step 2

Install the required Rust compiler, which is managed by the rustup tool.

$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh&&source ~/.bashrc

Press [Enter] to proceed with the installation.

Current installation options:

1) Proceed with instalation (default)
2) Customize installation
3) Cancel installation
>

Install cargo-make, which is required during the build process.

$ cargo install --force cargo-make --version 0.34.0

Step 3

Get the source code by cloning the Espanso repository to the local directory ~/.local/src/espanso.

$ git clone --progress https://github.com/federico-terzi/espanso ~/.local/src/espanso

Step 4

Compile the Espanso binary as a Wayland-only build in release mode.

$ cargo make --cwd ~/.local/src/espanso --profile release --env NO_X11=true build-binary

Move the resulting binary to the /usr/local/bin directory.

$ sudo mv ~/.local/src/espanso/target/release/espanso /usr/local/bin/

Step 5

Give Espanso the permissions required for operation.

$ sudo setcap "cap_dac_override+p" $(which espanso)

Check to see if the Espanso binary was installed successfuly.

$ espanso --version

Step 6

Register Espanso as a systemd service.

$ espanso service register

Launch Espanso.

$ espanso start && espanso status

Step 7

GNOME desktop only: remove the conflicting default shortcut for activating the window menu.

Settings > Keyboard > Keyboard Shortcuts > View and Customize Shortcuts > Windows > Activate the window menu > [Backspace]

Use [Alt + Space] to open Espanso’s Search bar.

All Done!

Experimental support for Wayland

Espanso has some known limitations under Wayland. Most notably, “there is currently no support for App-specific configurations”.

Big ‘Thank you’ to Federico Terzi for creating Espanso!

Monitoring storage devices with smartmontools on Debian or Ubuntu

The acronym SMART stands for Self-Monitoring, Analysis and Reporting Technology and is a monitoring system built into most modern storage devices. The package smartmontools includes the utilities smartctl and smartd, which process SMART data to ‘provide advanced warning of disk degradation and failure‘.

Step 1

Start by configuring nullmailer to receive status updates from your system.

Step 2

Install smartmontools and update its drive database to the latest version.

$ sudo apt-get install --yes smartmontools smart-notifier && sudo update-smart-drivedb

Step 3

Continue by obtaining relevant information about available storage devices.

$ sudo smartctl --scan

Depending on the type of disk, you should see a block of information similar to the following.

/dev/sda -d scsi # /dev/sda, SCSI device

Step 4

Enable SMART support for and display detailed information about the device.

$ sudo smartctl -iHs on /dev/sda

Ideally, information about the device to be monitored would be found in the drive database.

=== START OF INFORMATION SECTION ===
Device is: In smartctl database [for details use: -P show]

The device should report a successful self-assessment test.

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

Please note: the drive database does not extend to NVMe devices. SMART support for NVMe devices is curently limited to a subset of features.

Step 5

Verify the SMART capabilities of the device.

$ sudo smartctl -c /dev/sda

The following output confirms that the device /dev/sda has both short and extended self-test capabilites.

=== START OF READ SMART DATA SECTION ===
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 (  85) minutes.

The output provides estimates for the duration of short and extended (long) self-test routines.

If the device is capable of self-tests

Use the following command to run a short self-test.

$ sudo smartctl -t short /dev/sda

Use the following command to run a long self-test.

$ sudo smartctl -t long /dev/sda

Display a list with the results of recent self-tests in reverse chronological order.

$ sudo smartctl -l selftest /dev/sda

All tests should have completed without error.

Step 6

Edit the default configuration /etc/smartd.conf and comment out any DEVICESCAN options, thus preventing smartd from attempting to search for attached devices indiscriminately.

On Debian 12, you can use the following command to comment out the DEVICESCAN option in the default configuration file.

$ sudo sed -i 's/DEVICESCAN -d removable -n standby -m root -M exec/#DEVICESCAN -d removable -n standby -m root -M exec/' /etc/smartd.conf

Step 7

Example configuration for smartd and SATA devices

For the device /dev/sda, the following configuration for monitoring the device with smartd would have to be added to /etc/smartd.conf.

/dev/sda -H -l error -l selftest -S on -s (L/../.././06|S/../.././18) -m root -M test

-H display the health status as reported by the device

-l error show the increase in the number of SMART errors since last check

-l selftest show the increase in the number of failed tests in the SMART Self-Test Log

-S on enable Attribute Autosave on startup

-s (L/../.././06|S/../.././18) schedule a long self-test between 06:00 and 07:00 daily and a short self-test between 18:00 and 19:00 daily

-m root local user root receives warning by email

-M test send a test email on startup

Example configuration for smartd and NVMe devices

Current versions of smartmontools offer experimental support for NVMe devices. In practice this means that only a limited, but still useful, feature set is available.

For the device /dev/nvme0, the following configuration for monitoring the device with smartd would have to be added to the end of /etc/smartd.conf.

/dev/nvme0 -H -l error -m root -M test

-H display the health status as reported by the device

-l error show the increase in the number of SMART errors since last check

-m root local user root receives warning by email

-M test send a test email on startup

Install and configure SSH on Debian or Ubuntu

SSH is a protocol that enables secure connections over unsecured networks. It supports the use of asymmetric encryption for user authentication. Private keys are kept locally, while public keys are stored on the remote machine.

The following configuration disables root logins on the remote machine. Only users belonging to the group ssh-users may establish a connection. Access to the remote machine is tied to the local user’s private key.

In this example, the name of the remote machine is debian-server, which has the address 192.168.1.10 on the network. sid is a user on debian-server, whereas bookworm is a user on the local machine. Choose an encryption passphrase to secure the private key that you will generate in Step 5.

On the remote machine

Step 1

Install the secure shell server with the following command:

$ sudo apt install --yes openssh-server

Step 2

If you are using ufw as a host-based firewall

Configure ufw to allow connections to the secure shell server.

$ sudo ufw limit ssh

If you are using firewalld as a host-based firewall

Configure firewalld to allow connections to the secure shell server.

$ sudo -- bash -c 'firewall-cmd --zone=public --add-service=ssh --permanent && firewall-cmd --reload && firewall-cmd --info-zone=public'

Step 3

Restrict access to the remote machine to members of a specific group. Start by creating the group ssh-users.

$ sudo addgroup --system ssh-users

Add the user sid to the group ssh-users.

$ sudo adduser sid ssh-users

On the local machine

Step 4

Install the secure shell client with the following command.

$ sudo apt install openssh-client

Step 5

Generate a new key pair for the local user bookworm:

$ cd ~/.ssh && ssh-keygen -t ed25519 -o -a 100

Save the key pair to the directory /home/bookworm/.ssh/. Choose a name that facilitates easy identification.

Enter file in which to save the key (/home/bookworm/.ssh/id_ed25519): id_ed25519-debian-server

The use of an appropriate passphrase to secure the private key is mandatory.

Step 6

Create the file ~/.ssh/config to configure the secure shell client.

$ nano ~/.ssh/config

Add the follwing minimal entry for the host debian-server.

Host debian-server
   Hostname 192.168.1.10
   IdentityFile ~/.ssh/id_ed25519-debian-server
   IdentitiesOnly yes

Step 7

Deploy the public key with the following command.

$ ssh-copy-id -i ~/.ssh/id_ed25519-debian-server.pub sid@debian-server

When prompted to confirm the authenticity of the host debian-server, type yes and press [Enter].

The authenticity of host 'debian-server (192.168.1.10)' can't be established.
ED25519 key fingerprint is SHA256:C9RxLLVbvFwVJc0L4JHzcuHQSaPHJZe/GrRDvqy6rAG.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

Step 8

Log into the remote machine.

$ ssh -i ~/.ssh/id_ed25519-debian-server sid@debian-server

In the next step, enter the passphrase for your private key.

Enter passphrase for key '/home/bookworm/.ssh/id_ed25519-debian-server':

Step 9

On the remote machine, download a file to harden the ssh server. You are encouraged to inspect its contents.

$ sudo -- bash -c 'wget -P /etc/ssh/sshd_config.d/ --show-progress https://edafe.de/debian/sshd_config.conf'

Activate the modifications on the remote machine.

$ sudo systemctl restart ssh.service

Step 9

On the local machine, open a new terminal window and run the following command.

$ ssh -i ~/.ssh/id_ed25519-debian-server sid@debian-server

In the next step, enter the passphrase for your private key.

Enter passphrase for key '/home/bookworm/.ssh/id_ed25519-debian-server':

Display the active configuration for the remote ssh server and verify its settings, paying particular attention to options for maxauthtries, permitrootlogin and passwordauthentication.

$ sudo sshd -T

All done!

For more in-depth information, please see stribika’s post-Snowden advice on hardening OpenSSH server installations.

The book SSH The Secure Shell by Daniel Barrett, Richard Silverman and Robert Byrnes is still useful today and has information on other clever stuff you can do with SSH.

Install OneDrive Client for Linux on Debian or Ubuntu

The OneDrive Client for Linux connects your Debian or Ubuntu system to Microsoft’s OneDrive Personal, OneDrive for Business, OneDrive for Office365, Sharepoint and other such deployments.

Step 1

Install the OneDrive Client from the Debian or Ubuntu repository.

$ sudo -- bash -c 'apt update && apt install --yes onedrive'

Step 2

Begin to connect the client to your OneDrive account.

$ onedrive --synchronize

You will be presented with a message similar to the following:

Configuring Global Azure AD endpoints
Authorize this app visiting:

https://login.microsoftonline.com/common/oauth2/v2.0/authorize?client_id&scope=Files.ReadWrite%20Files.ReadWrite.all%20Sites.Read.All%Sites.ReadWrite.All%20offline_accessresponse_type=code&redirect_uri=https://login.microsoftonline.com/common/oauth2/nativeclient

Enter the response uri:

In the above dialog, copy or [Ctrl + Click] the URI beginning with https://login.microsoftonline.com/.

In a web browser

Use the URI from the previous step to sign into your Microsoft account. You will be redirected to a response URI displaying a blank page. Copy the response URI from the address field of your browser.

In the terminal

Paste the response URI into the terminal. On successful authentication, the OneDrive Client will connect to your Microsoft account and begin to download your data.

Initializing the Synchronization Engine …
Syncing changes from OneDrive …
Creating local directory:
Downloading file … done.
Uploading differences of ~/OneDrive
Uploading new items of ~/OneDrive

Step 3

After downloading your data to ~/OneDrive, validate the configuration of the client.

$ onedrive --display-config

If required, you may change the default configuration.

Step 4

Enable OneDrive Client for the local user bookworm.

$ sudo -- bash -c 'systemctl enable onedrive@bookworm.service && systemctl start onedrive@bookworm.service && systemctl status onedrive@bookworm.service'

All done!

Install and configure nullmailer using Fastmail as a smarthost

If you want to receive status updates from your Debian or Ubuntu system, you need to employ the help of a mail tansfer agent (MTA). nullmailer is a relay-only forwarding MTA that can be used as an alternative to more complex MTAs such as Exim, Sendmail or Postfix.

nullmailer can be configured to use Fastmail as a smarthost and hence ensure the deliverability of your messages. In principle, these instructions should also be applicable to service providers other than Fastmail.

In the following example configuration, debian is the hostname, bookworm the local username and linus.torvalds@fastmail.com the Fastmail username.

Step 1

Log into your Fastmail account and set up a new app password for SMTP authentication.

Step 2

Create the new directory /etc/nullmailer and the file /etc/nullmailer/adminaddr.

$ sudo mkdir /etc/nullmailer && sudo nano /etc/nullmailer/adminaddr

Your Fastmail username is the only entry in /etc/nullmailer/adminaddr.

linus.torvalds@fastmail.com

Step 3

Install the required packages.

$ sudo apt-get install --yes nullmailer mailutils

Step 4

Perform the initial configuration using debconf. Reconfigure nullmailer at any time after the initial installation using the following comand.

$ sudo dpkg-reconfigure nullmailer

Setting the mail name

Set the system mail name. If you are setting up on a home network, you should use home.arpa as the domain name.

Configuring nullmailer

Mailname of your system:

debian.home.arpa

Ok

Configuring the smarthost

Set the Fastmail server as the smarthost. Use the app password you set in Step 1.

Configuring nullmailer

Smarthosts:

smtp.fastmail.com smtp --port=587 --auth-login --starttls --user=linus.torvalds@fastmail.com --pass=password

Ok

Step 5

Test your configuration with the following command.

$ echo "Test mail from nullmailer on debian.home.arpa to the local root user and forwarded on to Fastmail" | mail -s "Test nullmailer" root

Check your Inbox, Linus!

Install Syncthing for continuous file synchronisation on Debian or Ubuntu

Syncthing is an open source tool that synchronises data across multiple devices. It transfers your files peer-to-peer, without the requirement to upload your information to the cloud. Packages are available for Android, Windows, macOS and Linux (including Synology DSM).

The usefulness of this project cannot be overstated.

Running the Syncthing stable channel

Syncthing is included in the Debian and Ubuntu repositories, respectively. These instructions are targeting the latest release of the Syncthing stable channel. It is therefore necessary to add the Syncthing repository to your list of APT sources.

In the following example, bookworm is the local username.

Step 1

Add the Syncthing release key for validation of packages downloaded from the Syncthing repository.

$ sudo curl -o /usr/share/keyrings/syncthing-archive-keyring.gpg https://syncthing.net/release-key.gpg

Step 2

Add the Syncthing repository.

$ echo "deb [signed-by=/usr/share/keyrings/syncthing-archive-keyring.gpg] https://apt.syncthing.net/ syncthing stable" | sudo tee /etc/apt/sources.list.d/syncthing.list

Step 3

Install Syncthing on your system.

$ sudo -- bash -c 'apt update && apt install --yes syncthing apt-transport-https'

Step 4

Enable Syncthing for the local user bookworm.

$ sudo -- bash -c 'systemctl enable syncthing@bookworm.service && systemctl start syncthing@bookworm.service && systemctl status syncthing@bookworm.service'

Step 5

You may need to edit your firewall settings to open ports for incoming and outgoing traffic.

If you are using ufw as a host-based firewall

Configure ufw to allow connections to Syncthing.

$ sudo ufw limit syncthing

If you are using firewalld as a host-based firewall

Configure firewalld to allow connections to Syncthing.

$ sudo -- bash -c 'firewall-cmd --zone=public --add-service=syncthing --permanent && firewall-cmd --reload && firewall-cmd --info-zone=public'

Step 6

Access the Syncthing configuration page by using your browser to navigate to the following address:

http://localhost:8384

Step 7

Complete your setup by referring to the Syncthing documentation.

Install Cockpit on Debian 12 bookworm

Cockpit is a web-based management tool for Linux systems. It aims to simplify management tasks while maintaining compatibility with other administration tools.

Step 1

Cockpit requires the use of the firewalld service to be able to make changes to your firewall rules.

If you are using ufw as a host-based firewall

Remove ufw before replacing it with firewalld.

$ sudo apt-get remove --purge --yes ufw

Install firewalld as a host-based firewall

Install firewalld and maintain ssh access as well as enabling cockpit to receive incoming connections.

$ sudo -- bash -c 'apt-get install --show-progress --yes firewalld && systemctl enable --now firewalld.service && firewall-cmd --zone=public --add-service=ssh --permanent && firewall-cmd --zone=public --add-service=cockpit --permanent && firewall-cmd --reload && firewall-cmd --info-zone=public'

Step 2

Proceed to install Cockpit and selected add-on applications.

$ sudo apt-get install --show-progress --yes cockpit cockpit-machines cockpit-pcp nullmailer ssh tuned-utils

Step 3

If you selected a wireless interface as the primary network interface for use during installation of the Debian system, you will need to re-establish connection to the wireless network after rebooting.

Enable the Network Manager for the primary network interface and reboot your system.

$ sudo sed -i 's/managed=false/managed=true/' /etc/NetworkManager/NetworkManager.conf&&sudo sed -i '/# The primary network interface/,$d' /etc/network/interfaces && sudo reboot

Step 4

By default, the Cockpit web console listens on port 9090 for connections. If you want to make changes from the default, use the following command to edit /etc/systemd/system/cockpit.socket.d/override.conf.

$ sudo systemctl edit cockpit.socket

The example below changes the web console port from 9090 to 9091 and restricts access to the localhost.

### Editing /etc/systemd/system/cockpit.socket.d/override.conf
### Anything between here and the comment below will become the new contents of the file

[Socket]
ListenStream=
ListenStream=127.0.0.1:9091

### Lines below this comment will be discarded

Use the following command for your changes to take effect.

$ sudo -- bash -c 'systemctl daemon-reload && systemctl restart cockpit.socket && systemctl status cockpit.socket'

If you installed Cockpit on the local machine and changed the listening port to 9091, you can now access the Cockpit web console on https://localhost:9091.

Click to copy