Chapter 9. System tips

Table of Contents

9.1. The screen program
9.1.1. The use scenario for screen(1)
9.1.2. Key bindings for the screen command
9.2. Data recording and presentation
9.2.1. The log daemon
9.2.2. Log analyzer
9.2.3. Recording the shell activities cleanly
9.2.4. Customized display of text data
9.2.5. Customized display of time and date
9.2.6. Colorized shell echo
9.2.7. Colorized commands
9.2.8. Recording the editor activities for complex repeats
9.2.9. Recording the graphic image of an X application
9.2.10. Recording changes in configuration files
9.3. Monitoring, controlling, and starting program activities
9.3.1. Timing a process
9.3.2. The scheduling priority
9.3.3. The ps command
9.3.4. The top command
9.3.5. Listing files opened by a process
9.3.6. Tracing program activities
9.3.7. Identification of processes using files or sockets
9.3.8. Repeating a command with a constant interval
9.3.9. Repeating a command looping over files
9.3.10. Starting a program from GUI
9.3.11. Customizing program to be started
9.3.12. Killing a process
9.3.13. Scheduling tasks once
9.3.14. Scheduling tasks regularly
9.3.15. Alt-SysRq key
9.4. System maintenance tips
9.4.1. Who is on the system?
9.4.2. Warning everyone
9.4.3. Hardware identification
9.4.4. Hardware configuration
9.4.5. System and hardware time
9.4.6. The terminal configuration
9.4.7. The sound infrastructure
9.4.8. Disabling the screen saver
9.4.9. Disabling beep sounds
9.4.10. Memory usage
9.4.11. System security and integrity check
9.5. Data storage tips
9.5.1. Disk space usage
9.5.2. Disk partition configuration
9.5.3. Accessing partition using UUID
9.5.4. LVM2
9.5.5. Filesystem configuration
9.5.6. Filesystem creation and integrity check
9.5.7. Optimization of filesystem by mount options
9.5.8. Optimization of filesystem via superblock
9.5.9. Optimization of hard disk
9.5.10. Optimization of solid state drive
9.5.11. Using SMART to predict hard disk failure
9.5.12. Specify temporary storage directory via $TMPDIR
9.5.13. Expansion of usable storage space via LVM
9.5.14. Expansion of usable storage space by mounting another partition
9.5.15. Expansion of usable storage space by bind-mounting another directory
9.5.16. Expansion of usable storage space by overlay-mounting another directory
9.5.17. Expansion of usable storage space using symlink
9.6. The disk image
9.6.1. Making the disk image file
9.6.2. Writing directly to the disk
9.6.3. Mounting the disk image file
9.6.4. Cleaning a disk image file
9.6.5. Making the empty disk image file
9.6.6. Making the ISO9660 image file
9.6.7. Writing directly to the CD/DVD-R/RW
9.6.8. Mounting the ISO9660 image file
9.7. The binary data
9.7.1. Viewing and editing binary data
9.7.2. Manipulating files without mounting disk
9.7.3. Data redundancy
9.7.4. Data file recovery and forensic analysis
9.7.5. Splitting a large file into small files
9.7.6. Clearing file contents
9.7.7. Dummy files
9.7.8. Erasing an entire hard disk
9.7.9. Erasing unused area of an hard disk
9.7.10. Undeleting deleted but still open files
9.7.11. Searching all hardlinks
9.7.12. Invisible disk space consumption
9.8. Data encryption tips
9.8.1. Removable disk encryption with dm-crypt/LUKS
9.8.2. Encrypted swap partition with dm-crypt
9.8.3. Mounting encrypted disk with dm-crypt/LUKS
9.8.4. Automatically encrypting files with eCryptfs
9.8.5. Automatically mounting eCryptfs
9.9. The kernel
9.9.1. Linux kernel 2.6/3.x
9.9.2. Kernel parameters
9.9.3. Kernel headers
9.9.4. Compiling the kernel and related modules
9.9.5. Compiling the kernel source: Debian Kernel Team recommendation
9.9.6. Hardware drivers and firmware
9.10. Virtualized system
9.10.1. Virtualization tools
9.10.2. Virtualization work flow
9.10.3. Mounting the virtual disk image file
9.10.4. Chroot system
9.10.5. Multiple desktop systems

Here, I describe basic tips to configure and manage systems, mostly from the console.

screen(1) is a very useful tool for people to access remote sites via unreliable or intermittent connections since it supports interrupted network connections.

The simple use of script(1) (see Section 1.4.9, “Recording the shell activities”) to record shell activity produces a file with control characters. This can be avoided by using col(1) as the following.

$ script
Script started, file is typescript

Do whatever … and press Ctrl-D to exit script.

$ col -bx <typescript >cleanedfile
$ vim cleanedfile

If you don't have script (for example, during the boot process in the initramfs), you can use the following instead.

$ sh -i 2>&1 | tee typescript
[Tip] Tip

Some x-terminal-emulator such as gnome-terminal can record. You may wish to extend line buffer for scrollback.

[Tip] Tip

You may use screen(1) with "^A H" (see Section 9.1.2, “Key bindings for the screen command”) to perform recording of console.

[Tip] Tip

You may use emacs(1) with "M-x shell", "M-x eshell", or "M-x term" to perform recording of console. You may later use "C-x C-w" to write the buffer to a file.

Although pager tools such as more(1) and less(1) (see Section 1.4.5, “The pager”) and custom tools for highlighting and formatting (see Section 11.1.8, “Highlighting and formatting plain text data”) can display text data nicely, general purpose editors (see Section 1.4.6, “The text editor”) are most versatile and customizable.

[Tip] Tip

For vim(1) and its pager mode alias view(1), ":set hls" enables highlighted search.

Shell echo to most modern terminals can be colorized using ANSI escape code (see "/usr/share/doc/xterm/ctlseqs.txt.gz").

For example, try the following

$ RED=$(printf "\x1b[31m")
$ NORMAL=$(printf "\x1b[0m")
$ REVERSE=$(printf "\x1b[7m")

You can record the editor activities for complex repeats.

For Vim, as follows.

  • "qa": start recording typed characters into named register "a".

  • … editor activities

  • "q": end recording typed characters.

  • "@a": execute the contents of register "a".

For Emacs, as follows.

  • "C-x (": start defining a keyboard macro.

  • … editor activities

  • "C-x )": end defining a keyboard macro.

  • "C-x e": execute a keyboard macro.

Program activities can be monitored and controlled using specialized tools.

[Tip] Tip

The procps packages provide very basics of monitoring, controlling, and starting program activities. You should learn all of them.

There are several ways to repeat a command looping over files matching some condition, e.g. matching glob pattern "*.ext".

for x in *.ext; do if [ -f "$x"]; then command "$x" ; fi; done
  • find(1) and xargs(1) combination:

find . -type f -maxdepth 1 -name '*.ext' -print0 | xargs -0 -n 1 command
  • find(1) with "-exec" option with a command:

find . -type f -maxdepth 1 -name '*.ext' -exec command '{}' \;
  • find(1) with "-exec" option with a short shell script:

find . -type f -maxdepth 1 -name '*.ext' -exec sh -c "command '{}' && echo 'successful'" \;

The above examples are written to ensure proper handling of funny file names such as ones containing spaces. See Section 10.1.5, “Idioms for the selection of files” for more advance uses of find(1).

For the command-line interface (CLI), the first program with the matching name found in the directories specified in the $PATH environment variable is executed. See Section 1.5.3, “The "$PATH" variable”.

For the graphical user interface (GUI) compliant to the standards, the *.desktop files in the /usr/share/applications/ directory provide necessary attributes for the GUI menu display of each program. See Section 7.2.2, “ menu”.

For example, the chromium.desktop file defines attributes for the "Chromium Web Browser" such as "Name" for the program name, "Exec" for the program execution path and arguments, "Icon" for the icon used, etc. (see the Desktop Entry Specification) as follows:

[Desktop Entry]
Name=Chromium Web Browser
GenericName=Web Browser
Comment=Access the Internet
Comment[fr]=Explorer le Web
Exec=/usr/bin/chromium %U

This is an oversimplified description. The *.desktop files are scanned as follows.

The desktop environment sets $XDG_DATA_HOME and $XDG_DATA_DIR environment variables. For example, under the GNOME 3:

  • $XDG_DATA_HOME is unset. (The default value of $HOME/.local/share is used.)

  • $XDG_DATA_DIRS is set to /usr/share/gnome:/usr/local/share/:/usr/share/.

So the base directories (see XDG Base Directory Specification) and the applications directories are as follows.

  • $HOME/.local/share/$HOME/.local/share/applications/

  • /usr/share/gnome//usr/share/gnome/applications/

  • /usr/local/share//usr/local/share/applications/

  • /usr/share//usr/share/applications/

The *.desktop files are scanned in these applications directories in this order.

[Tip] Tip

A user custom GUI menu entry can be created by adding a *.desktop file in the $HOME/.local/share/applications/ directory.

[Tip] Tip

Similarly, if a *.desktop file is created in the autostart directory under these base directories, the specified program in the *.desktop file is executed automatically when the desktop environment is started. See Desktop Application Autostart Specification.

[Tip] Tip

Similarly, if a *.desktop file is created in the $HOME/Desktop directory and the Desktop environment is configured to support the desktop icon launcher feature, the specified program in it is executed upon clicking the icon. Please note that the actual name of the $HOME/Desktop directory is locale dependent. See xdg-user-dirs-update(1).

Some programs start another program automatically. Here are check points for customizing this process.

[Tip] Tip

update-mime(8) updates the "/etc/mailcap" file using "/etc/mailcap.order" file (see mailcap.order(5)).

[Tip] Tip

The debianutils package provides sensible-browser(1), sensible-editor(1), and sensible-pager(1) which make sensible decisions on which editor, pager, and web browser to call, respectively. I recommend you to read these shell scripts.

[Tip] Tip

In order to run a console application such as mutt under X as your preferred application, you should create an X application as following and set "/usr/local/bin/mutt-term" as your preferred application to be started as described.

# cat /usr/local/bin/mutt-term <<EOF
gnome-terminal -e "mutt \$@"
chmod 755 /usr/local/bin/mutt-term

Use cron(8) to schedule tasks regularly. See crontab(1) and crontab(5).

You can schedule to run processes as a normal user, e.g. foo by creating a crontab(5) file as "/var/spool/cron/crontabs/foo" with "crontab -e" command.

Here is an example of a crontab(5) file.

# use /bin/sh to run commands, no matter what /etc/passwd says
# mail any output to paul, no matter whose crontab this is
# Min Hour DayOfMonth Month DayOfWeek command (Day... are OR'ed)
# run at 00:05, every day
5  0  *  * *   $HOME/bin/daily.job >> $HOME/tmp/out 2>&1
# run at 14:15 on the first of every month -- output mailed to paul
15 14 1  * *   $HOME/bin/monthly
# run at 22:00 on weekdays(1-5), annoy Joe. % for newline, last % for cc:
0 22 *   * 1-5 mail -s "It's 10pm" joe%Joe,%%Where are your kids?%.%%
23 */2 1 2 *   echo "run 23 minutes after 0am, 2am, 4am ..., on Feb 1"
5  4 *   * sun echo "run at 04:05 every Sunday"
# run at 03:40 on the first Monday of each month
40 3 1-7 * *   [ "$(date +%a)" == "Mon" ] && command -args
[Tip] Tip

For the system not running continuously, install the anacron package to schedule periodic commands at the specified intervals as closely as machine-uptime permits. See anacron(8) and anacrontab(5).

[Tip] Tip

For scheduled system maintenance scripts, you can run them periodically from root account by placing such scripts in "/etc/cron.hourly/", "/etc/cron.daily/", "/etc/cron.weekly/", or "/etc/cron.monthly/". Execution timings of these scripts can be customized by "/etc/crontab" and "/etc/anacrontab".

Insurance against system malfunction is provided by the kernel compile option "Magic SysRq key" (SAK key) which is now the default for the Debian kernel. Pressing Alt-SysRq followed by one of the following keys does the magic of rescuing control of the system.

[Tip] Tip

Read the signal(7), kill(1), and sync(1) manpages to understand the description above.

The combination of "Alt-SysRq s", "Alt-SysRq u", and "Alt-SysRq r" is good for getting out of really bad situations and gaining usable keyboard access without stopping the system.

See "/usr/share/doc/linux-doc-3.*/Documentation/sysrq.txt.gz".

[Caution] Caution

The Alt-SysRq feature may be considered a security risk by allowing users access to root-privileged functions. Placing "echo 0 >/proc/sys/kernel/sysrq" in "/etc/rc.local" or "kernel.sysrq = 0" in "/etc/sysctl.conf" disables the Alt-SysRq feature.

[Tip] Tip

From SSH terminal etc., you can use the Alt-SysRq feature by writing to the "/proc/sysrq-trigger". For example, "echo s > /proc/sysrq-trigger; echo u > /proc/sysrq-trigger" from the root shell prompt syncs and umounts all mounted filesystems.

The following sets system and hardware time to MM/DD hh:mm, CCYY.

# date MMDDhhmmCCYY
# hwclock --utc --systohc
# hwclock --show

Times are normally displayed in the local time on the Debian system but the hardware and system time usually use UTC(GMT).

If the hardware (BIOS) time is set to UTC, change the setting to "UTC=yes" in the "/etc/default/rcS".

The following reconfigure the timezone used by the Debian system.

# dpkg-reconfigure tzdata

If you wish to update system time via network, consider to use the NTP service with the packages such as ntp, ntpdate, and chrony.

[Tip] Tip

Under systemd, use systemd-timesyncd for the network time synchronization instead. See systemd-timesyncd(8).

See the following.

[Tip] Tip

ntptrace(8) in the ntp package can trace a chain of NTP servers back to the primary source.

Device drivers for sound cards for current Linux are provided by Advanced Linux Sound Architecture (ALSA). ALSA provides emulation mode for previous Open Sound System (OSS) for compatibility.

[Tip] Tip

Use "cat /dev/urandom > /dev/audio" or speaker-test(1) to test speaker (^C to stop).

[Tip] Tip

If you can not get sound, your speaker may be connected to a muted output. Modern sound system has many outputs. alsamixer(1) in the alsa-utils package is useful to configure volume and mute settings.

Application softwares may be configured not only to access sound devices directly but also to access them via some standardized sound server system.

There is usually a common sound engine for each popular desktop environment. Each sound engine used by the application can choose to connect to different sound servers.

Poor system maintenance may expose your system to external exploitation.

For system security and integrity check, you should start with the following.

Here is a simple script to check for typical world writable incorrect file permissions.

# find / -perm 777 -a \! -type s -a \! -type l -a \! \( -type d -a -perm 1777 \)
[Caution] Caution

Since the debsums package uses MD5 checksums stored locally, it can not be fully trusted as the system security audit tool against malicious attacks.

Booting your system with Linux live CDs or debian-installer CDs in rescue mode makes it easy for you to reconfigure data storage on your boot device.

For disk partition configuration, although fdisk(8) has been considered standard, parted(8) deserves some attention. "Disk partitioning data", "partition table", "partition map", and "disk label" are all synonyms.

Most PCs use the classic Master Boot Record (MBR) scheme to hold disk partitioning data in the first sector, i.e., LBA sector 0 (512 bytes).

[Note] Note

Some new PCs with Extensible Firmware Interface (EFI), including Intel-based Macs, use GUID Partition Table (GPT) scheme to hold disk partitioning data not in the first sector.

Although fdisk(8) has been standard for the disk partitioning tool, parted(8) is replacing it.

[Caution] Caution

Although parted(8) claims to create and to resize filesystem too, it is safer to do such things using best maintained specialized tools such as mkfs(8) (mkfs.msdos(8), mkfs.ext2(8), mkfs.ext3(8), mkfs.ext4(8), …) and resize2fs(8).

[Note] Note

In order to switch between GPT and MBR, you need to erase first few blocks of disk contents directly (see Section 9.7.6, “Clearing file contents”) and use "parted /dev/sdx mklabel gpt" or "parted /dev/sdx mklabel msdos" to set it. Please note "msdos" is use here for MBR.

LVM2 is a logical volume manager for the Linux kernel. With LVM2, disk partitions can be created on logical volumes instead of the physical harddisks.

LVM requires the following.

  • device-mapper support in the Linux kernel (default for Debian kernels)

  • the userspace device-mapper support library (libdevmapper* package)

  • the userspace LVM2 tools (lvm2 package)

Please start learning LVM2 from the following manpages.

  • lvm(8): Basics of LVM2 mechanism (list of all LVM2 commands)

  • lvm.conf(5): Configuration file for LVM2

  • lvs(8): Report information about logical volumes

  • vgs(8): Report information about volume groups

  • pvs(8): Report information about physical volumes

For ext4 filesystem, the e2fsprogs package provides the following.

  • mkfs.ext4(8) to create new ext4 filesystem

  • fsck.ext4(8) to check and to repair existing ext4 filesystem

  • tune2fs(8) to configure superblock of ext4 filesystem

  • debugfs(8) to debug ext4 filesystem interactively. (It has undel command to recover deleted files.)

The mkfs(8) and fsck(8) commands are provided by the e2fsprogs package as front-ends to various filesystem dependent programs (mkfs.fstype and fsck.fstype). For ext4 filesystem, they are mkfs.ext4(8) and fsck.ext4(8) (they are symlinked to mke2fs(8) and e2fsck(8)).

Similar commands are available for each filesystem supported by Linux.

[Tip] Tip

Ext4 filesystem is the default filesystem for the Linux system and strongly recommended to use it unless you have some specific reasons not to.

[Tip] Tip

Btrfs filesystem is available in Linux kernel 3.2 (Debian wheezy). It is expected to be the next default filesystem after the ext4 filesystem.

[Warning] Warning

You should not use the Btrfs filesystem for your critical data yet before it acquires the live kernel space fsck(8) feature and the boot loader support.

[Tip] Tip

Some tools allow access to filesystem without Linux kernel support (see Section 9.7.2, “Manipulating files without mounting disk”).

The basic static filesystem configuration is given by "/etc/fstab". For example,

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
UUID=709cbe4c-80c1-56db-8ab1-dbce3146d2f7 / ext4 noatime,errors=remount-ro 0 1
UUID=817bae6b-45d2-5aca-4d2a-1267ab46ac23 none swap sw  0       0
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto 0       0
[Tip] Tip

UUID (see Section 9.5.3, “Accessing partition using UUID”) may be used to identify a block device instead of normal block device names such as "/dev/sda1", "/dev/sda2", …

Performance and characteristics of a filesystem can be optimized by mount options used on it (see fstab(5) and mount(8)). Notable ones are the following.

  • "defaults" option implies default options: "rw,suid,dev,exec,auto,nouser,async". (general)

  • "noatime" or "relatime" option is very effective for speeding up the read access. (general)

  • "user" option allows an ordinary user to mount the filesystem. This option implies "noexec,nosuid,nodev" option combination. (general, used for CDs or usb storage devices)

  • "noexec,nodev,nosuid" option combination is used to enhance security. (general)

  • "noauto" option limits mounting by explicit operation only. (general)

  • "data=journal" option for ext3fs can enhance data integrity against power failure with some loss of write speed.

[Tip] Tip

You need to provide kernel boot parameter (see Section 3.1.2, “Stage 2: the boot loader”), e.g. "rootflags=data=journal" to deploy a non-default journaling mode for the root filesystem. For lenny, the default jounaling mode is "rootflags=data=ordered". For squeeze, it is "rootflags=data=writeback".

Performance and disk wear of the solid state drive (SSD) can be optimized as follows.

  • Use the latest Linux kernel. (>= 3.2)

  • Reduce disk writes for read disk accesses.

    • Set "noatime" or "relatime" mount option in /etc/fstab.

  • Enable the TRIM command.

    • Set "discard" mount option in /etc/fstab for the ext4 filesystem, swap partition, Btrfs, etc. See fstab(5).

    • Set "discard" option in /etc/lvm/lvm.conf for LVM. See lvm.conf(5).

    • Set "discard" option in /etc/crypttab for dm-crypt. See crypttab(5).

  • Enable the SSD optimized disk space allocation scheme.

    • Set "ssd" mount option in /etc/fstab for the Btrfs.

  • Make system flush data to the disk every 10 minutes for laptop PCs.

    • Set "commit=600" mount option in /etc/fstab. See fstab(5).

    • Set pm-utils to use laptop-mode even under AC operation. See Debian BTS #659260.

[Warning] Warning

Changing flushing interval from normal 5 seconds to 10 minutes makes your data vulnerable to the power failure.

You can monitor and log your hard disk which is compliant to SMART with the smartd(8) daemon.

  1. Enable SMART feature in BIOS.

  2. Install the smartmontools package.

  3. Identify your hard disk drives by listing them with df(1).

    • Let's assume a hard disk drive to be monitored as "/dev/hda".

  4. Check the output of "smartctl -a /dev/hda" to see if SMART feature is actually enabled.

    • If not, enable it by "smartctl -s on -a /dev/hda".

  5. Enable smartd(8) daemon to run by the following.

    • uncomment "start_smartd=yes" in the "/etc/default/smartmontools" file.

    • restart the smartd(8) daemon by "sudo /etc/init.d/smartmontools restart".

[Tip] Tip

The smartd(8) daemon can be customized with the /etc/smartd.conf file including how to be notified of warnings.

For partitions created on Logical Volume Manager (LVM) (Linux feature) at install time, they can be resized easily by concatenating extents onto them or truncating extents from them over multiple storage devices without major system reconfiguration.

If you have usable space in another partition (e.g., "/path/to/empty" and "/path/to/work"), you can create a directory in it and stack that on to an old directory (e.g., "/path/to/old") where you need space using the OverlayFS for Linux kernel 3.18 or newer (Debian Stretch 9.0 or newer).

$ sudo mount -t overlay overlay \

Here, "/path/to/empty" and "/path/to/work" should be on the RW-enabled partition to write on "/path/to/old".

Here, we discuss manipulations of the disk image.

The disk image file, "disk.img", of an unmounted device, e.g., the second SCSI or serial ATA drive "/dev/sdb", can be made using cp(1) or dd(1) by the following.

# cp /dev/sdb disk.img
# dd if=/dev/sdb of=disk.img

The disk image of the traditional PC's master boot record (MBR) (see Section 9.5.2, “Disk partition configuration”) which reside on the first sector on the primary IDE disk can be made by using dd(1) by the following.

# dd if=/dev/hda of=mbr.img bs=512 count=1
# dd if=/dev/hda of=mbr-nopart.img bs=446 count=1
# dd if=/dev/hda of=mbr-part.img skip=446 bs=1 count=66
  • "mbr.img": The MBR with the partition table

  • "mbr-nopart.img": The MBR without the partition table

  • "mbr-part.img": The partition table of the MBR only

If you have an SCSI or serial ATA device as the boot disk, substitute "/dev/hda" with "/dev/sda".

If you are making an image of a disk partition of the original disk, substitute "/dev/hda" with "/dev/hda1" etc.

The disk image "partition.img" containing a single partition image can be mounted and unmounted by using the loop device as follows.

# losetup -v -f partition.img
Loop device is /dev/loop0
# mkdir -p /mnt/loop0
# mount -t auto /dev/loop0 /mnt/loop0
# umount /dev/loop0
# losetup -d /dev/loop0

This can be simplified as follows.

# mkdir -p /mnt/loop0
# mount -t auto -o loop partition.img /mnt/loop0
# umount partition.img

Each partition of the disk image "disk.img" containing multiple partitions can be mounted by using the loop device. Since the loop device does not manage partitions by default, we need to reset it as follows.

# modinfo -p loop # verify kernel capability
max_part:Maximum number of partitions per loop device
max_loop:Maximum number of loop devices
# losetup -a # verify nothing using the loop device
# rmmod loop
# modprobe loop max_part=16

Now, the loop device can manage up to 16 partitions.

# losetup -v -f disk.img
Loop device is /dev/loop0
# fdisk -l /dev/loop0

Disk /dev/loop0: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x452b6464

      Device Boot      Start         End      Blocks   Id  System
/dev/loop0p1               1         600     4819468+  83  Linux
/dev/loop0p2             601         652      417690   83  Linux
# mkdir -p /mnt/loop0p1
# mount -t ext4 /dev/loop0p1 /mnt/loop0p1
# mkdir -p /mnt/loop0p2
# mount -t ext4 /dev/loop0p2 /mnt/loop0p2
# umount /dev/loop0p1
# umount /dev/loop0p2
# losetup -d /dev/loop0

Alternatively, similar effects can be done by using the device mapper devices created by kpartx(8) from the kpartx package as follows.

# kpartx -a -v disk.img
# mkdir -p /mnt/loop0p2
# mount -t ext4 /dev/mapper/loop0p2 /mnt/loop0p2
# umount /dev/mapper/loop0p2
# kpartx -d /mnt/loop0
[Note] Note

You can mount a single partition of such disk image with loop device using offset to skip MBR etc., too. But this is more error prone.

The empty disk image "disk.img" which can grow up to 5GiB can be made using dd(1) as follows.

$ dd bs=1 count=0 if=/dev/zero of=disk.img seek=5G

You can create an ext4 filesystem on this disk image "disk.img" using the loop device as follows.

# losetup -f -v disk.img
Loop device is /dev/loop1
# mkfs.ext4 /dev/loop1
# losetup -d /dev/loop1
$ du  --apparent-size -h disk.img
5.0G  disk.img
$ du -h disk.img
83M disk.img

For "disk.img", its file size is 5.0 GiB and its actual disk usage is mere 83MiB. This discrepancy is possible since ext4 can hold sparse file.

[Tip] Tip

The actual disk usage of sparse file grows with data which are written to it.

Using similar operation on devices created by the loop device or the device mapper devices as Section 9.6.3, “Mounting the disk image file”, you can partition this disk image "disk.img" using parted(8) or fdisk(8), and can create filesystem on it using mkfs.ext4(8), mkswap(8), etc.

The ISO9660 image file, "cd.iso", from the source directory tree at "source_directory" can be made using genisoimage(1) provided by cdrkit by the following.

#  genisoimage -r -J -T -V volume_id -o cd.iso source_directory

Similarly, the bootable ISO9660 image file, "cdboot.iso", can be made from debian-installer like directory tree at "source_directory" by the following.

#  genisoimage -r -o cdboot.iso -V volume_id \
   -b isolinux/isolinux.bin -c isolinux/ \
   -no-emul-boot -boot-load-size 4 -boot-info-table source_directory

Here Isolinux boot loader (see Section 3.1.2, “Stage 2: the boot loader”) is used for booting.

You can calculate the md5sum value and make the ISO9660 image directly from the CD-ROM device as follows.

$ isoinfo -d -i /dev/cdrom
CD-ROM is in ISO 9660 format
Logical block size is: 2048
Volume size is: 23150592
# dd if=/dev/cdrom bs=2048 count=23150592 conv=notrunc,noerror | md5sum
# dd if=/dev/cdrom bs=2048 count=23150592 conv=notrunc,noerror > cd.iso
[Warning] Warning

You must carefully avoid ISO9660 filesystem read ahead bug of Linux as above to get the right result.

Here, we discuss direct manipulations of the binary data on storage media.

With physical access to your PC, anyone can easily gain root privilege and access all the files on your PC (see Section 4.7.4, “Securing the root password”). This means that login password system can not secure your private and sensitive data against possible theft of your PC. You must deploy data encryption technology to do it. Although GNU privacy guard (see Section 10.3, “Data security infrastructure”) can encrypt files, it takes some user efforts.

dm-crypt and eCryptfs facilitates automatic data encryption natively via Linux kernel modules with minimal user efforts.

Dm-crypt is a cryptographic filesystem using device-mapper. Device-mapper maps one block device to another.

eCryptfs is another cryptographic filesystem using stacked filesystem. Stacked filesystem stacks itself on top of an existing directory of a mounted filesystem.

[Caution] Caution

Data encryption costs CPU time etc. Please weigh its benefits and costs.

[Note] Note

Entire Debian system can be installed on a encrypted disk by the debian-installer (lenny or newer) using dm-crypt/LUKS and initramfs.

[Tip] Tip

See Section 10.3, “Data security infrastructure” for user space encryption utility: GNU Privacy Guard.

You can encrypt contents of removable mass devices, e.g. USB memory stick on "/dev/sdx", using dm-crypt/LUKS. You simply format it as the following.

# badblocks -c 1024 -s -w -t random -v /dev/sdx
# fdisk /dev/sdx
... "n" "p" "1" "return" "return" "w"
# cryptsetup luksFormat /dev/sdx1
# cryptsetup open --type luks /dev/sdx1 sdx1
# ls -l /dev/mapper/
total 0
crw-rw---- 1 root root  10, 60 2008-10-04 18:44 control
brw-rw---- 1 root disk 254,  0 2008-10-04 23:55 sdx1
# mkfs.vfat /dev/mapper/sdx1
# cryptsetup luksClose sdx1

Then, it can be mounted just like normal one on to "/media/<disk_label>", except for asking password (see Section 10.1.7, “Removable storage device”) under modern desktop environment, such as GNOME using gnome-mount(1). The difference is that every data written to it is encrypted. You may alternatively format media in different filesystem, e.g., ext4 with "mkfs.ext4 /dev/mapper/sdx1".

[Note] Note

If you are really paranoid for the security of data, you may need to overwrite multiple times (the "badblocks" command in the above example). This operation is very time consuming though.

Let's assume that your original "/etc/fstab" contains the following.

/dev/sda7 swap sw 0 0

You can enable encrypted swap partition using dm-crypt by as the following.

# aptitude install cryptsetup
# swapoff -a
# echo "cswap /dev/sda7 /dev/urandom swap" >> /etc/crypttab
# perl -i -p -e "s/\/dev\/sda7/\/dev\/mapper\/cswap/" /etc/fstab
# /etc/init.d/cryptdisks restart
# swapon -a

You can encrypt files written under "~/Private/" automatically using eCryptfs and the ecryptfs-utils package.

  • Run ecryptfs-setup-private(1) and set up "~/Private/" by following prompts.

  • Activate "~/Private/" by running ecryptfs-mount-private(1).

  • Move sensitive data files to "~/Private/" and make symlinks as needed.

    • Candidates: "~/.fetchmailrc", "~/.ssh/identity", "~/.ssh/id_rsa", "~/.ssh/id_dsa" and other files with "go-rwx"

  • Move sensitive data directories to a subdirectory in "~/Private/" and make symlinks as needed.

    • Candidates: "~/.gnupg" and other directories with "go-rwx"

  • Create symlink from "~/Desktop/Private/" to "~/Private/" for easier desktop operations.

  • Deactivate "~/Private/" by running ecryptfs-umount-private(1).

  • Activate "~/Private/" by issuing "ecryptfs-mount-private" as you need encrypted data.

[Tip] Tip

Since eCryptfs selectively encrypt only the sensitive files, its system cost is much less than using dm-crypt on the entire root or "/home" device. It does not require any special on-disk storage allocation effort but cannot keep all filesystem metadata confidential.

If you use your login password for wrapping encryption keys, you can automate mounting eCryptfs via PAM (Pluggable Authentication Modules).

Insert the following line just before "" in "/etc/pam.d/common-auth".

auth required unwrap

Insert the following line just at the last line in "/etc/pam.d/common-session".

session optional unwrap

Insert the following line at first active line in "/etc/pam.d/common-password".

password required

This is quite convenient.

[Warning] Warning

Configuration errors of PAM may lock you out of your own system. See Chapter 4, Authentication.

[Caution] Caution

If you use your login password for wrapping encryption keys, your encrypted data are as secure as your user login password (see Section 4.3, “Good password”). Unless you are careful to set up a strong password, your data is at risk when someone runs password cracking software after stealing your laptop (see Section 4.7.4, “Securing the root password”).

Debian distributes modularized Linux kernel as packages for supported architectures.

There are few notable features on Linux kernel 2.6/3.x compared to 2.4.

The version bump from Linux 2.6.39 to Linux 3.0 is not about major technological changes but about the 20th anniversary.

Many Linux features are configurable via kernel parameters as follows.

See "kernel-parameters.txt(.gz)" and other related documents in the Linux kernel documentation ("/usr/share/doc/linux-doc-3.*/Documentation/filesystems/*") provided by the linux-doc-3.* package.

Debian has its own method of compiling the kernel and related modules.

If you use initrd in Section 3.1.2, “Stage 2: the boot loader”, make sure to read the related information in initramfs-tools(8), update-initramfs(8), mkinitramfs(8) and initramfs.conf(5).

[Warning] Warning

Do not put symlinks to the directories in the source tree (e.g. "/usr/src/linux*") from "/usr/include/linux" and "/usr/include/asm" when compiling the Linux kernel source. (Some outdated documents suggest this.)

[Note] Note

When compiling the latest Linux kernel on the Debian stable system, the use of backported latest tools from the Debian unstable may be needed.

[Note] Note

The dynamic kernel module support (DKMS) is a new distribution independent framework designed to allow individual kernel modules to be upgraded without changing the whole kernel. This is used for the maintenance of out-of-tree modules. This also makes it very easy to rebuild modules as you upgrade kernels.

The hardware driver is the code running on the target system. Most hardware drivers are available as free software now and are included in the normal Debian kernel packages in the main area.

  • GPU driver

    • Intel GPU driver (main)

    • AMD/ATI GPU driver (main)

    • NVIDIA GPU driver (main for nouveau driver, and non-free for binary-only drivers supported by the vendor.)

  • Softmodem driver

    • martian-modem and sl-modem-dkms packages (non-free)

The firmware is the code or data loaded on the device (e.g., CPU microcode, rendering code running on GPU, or FPGA / CPLD data, …). Some firmware packages are available as free software but many firmware packages are not available as free software since they contain sourceless binary data.

  • firmware-linux-free (main)

  • firmware-linux-nonfree (non-free)

  • firmware-linux-* (non-free)

  • *-firmware (non-free)

  • intel-microcode (non-free)

  • amd64-microcode (non-free)

Please note that non-free and contrib packages are not part of the Debian system. The access configuration to enable and to disable the non-free and contrib areas is described in Section 2.1.4, “Debian archive basics”. You should be aware of negatives associated with the use of the non-free and contrib packages as described in Section 2.1.5, “Debian is 100% free software”.

Use of virtualized system enables us to run multiple instances of system simultaneously on a single hardware.

There are several system virtualization and emulation related packages in Debian beyond simple chroot. Some packages also help you to setup such system.

Table 9.26. List of virtualization tools

package popcon size description
schroot V:7, I:10 2708 specialized tool for executing Debian binary packages in chroot
sbuild V:1, I:4 286 tool for building Debian binary packages from Debian sources
pbuilder V:2, I:16 966 personal package builder for Debian packages
debootstrap V:6, I:63 298 bootstrap a basic Debian system (written in sh)
cdebootstrap V:0, I:3 116 bootstrap a Debian system (written in C)
virt-manager V:10, I:42 2298 Virtual Machine Manager: desktop application for managing virtual machines
libvirt-clients V:43, I:62 1167 programs for the libvirt library
bochs V:0, I:1 7194 Bochs: IA-32 PC emulator
qemu I:34 94 QEMU: fast generic processor emulator
qemu-system I:21 95 QEMU: full system emulation binaries
qemu-user V:0, I:13 89671 QEMU: user mode emulation binaries
qemu-utils V:11, I:107 6083 QEMU: utilities
qemu-kvm V:10, I:61 107 KVM: full virtualization on x86 hardware with the hardware-assisted virtualization
virtualbox V:12, I:16 106495 VirtualBox: x86 virtualization solution on i386 and amd64
xen-tools V:0, I:4 727 tools to manage debian XEN virtual server
wine V:19, I:82 192 Wine: Windows API Implementation (standard suite)
dosbox V:2, I:18 2742 DOSBox: x86 emulator with Tandy/Herc/CGA/EGA/VGA/SVGA graphics, sound and DOS
dosemu V:0, I:2 4891 DOSEMU: The Linux DOS Emulator
vzctl V:0, I:1 1112 OpenVZ server virtualization solution - control tools
vzquota V:0, I:1 236 OpenVZ server virtualization solution - quota tools
lxc V:10, I:15 18761 Linux containers user space tools

See Wikipedia article Comparison of platform virtual machines for detail comparison of different platform virtualization solutions.

[Note] Note

Some functionalities described here are only available in squeeze or later.

[Note] Note

Default Debian kernels support KVM since lenny.

Typical work flow for virtualization involves several steps.

For the raw disk image file, see Section 9.6, “The disk image”.

For other virtual disk image files, you can use qemu-nbd(8) to export them using network block device protocol and mount them using the nbd kernel module.

qemu-nbd(8) supports disk formats supported by QEMU: QEMU supports following disk formats: raw, qcow2, qcow, vmdk, vdi, bochs, cow (user-mode Linux copy-on-write), parallels, dmg, cloop, vpc, vvfat (virtual VFAT), and host_device.

The network block device can support partitions in the same way as the loop device (see Section 9.6.3, “Mounting the disk image file”). You can mount the first partition of "disk.img" as follows.

# modprobe nbd max_part=16
# qemu-nbd -v -c /dev/nbd0 disk.img
# mkdir /mnt/part1
# mount /dev/nbd0p1 /mnt/part1
[Tip] Tip

You may export only the first partition of "disk.img" using "-P 1" option to qemu-nbd(8).

chroot(8) offers most basic way to run different instances of the GNU/Linux environment on a single system simultaneously without rebooting.

[Caution] Caution

Examples below assumes both parent system and chroot system share the same CPU architecture.

You can learn how to setup and use chroot(8) by running pbuilder(8) program under script(1) as follows.

$ sudo mkdir /sid-root
$ sudo pbuilder --create --no-targz --debug --buildplace /sid-root

You see how debootstrap(8) or cdebootstrap(1) populate system data for sid environment under "/sid-root".

[Tip] Tip

These debootstrap(8) or cdebootstrap(1) are used to install Debian by the Debian Installer. These can also be used to install Debian to a system without using a Debian install disk, but instead from another GNU/Linux distribution.

$ sudo pbuilder --login --no-targz  --debug --buildplace /sid-root

You see how a system shell running under sid environment is created as the following.

  1. Copy local configuration ("/etc/hosts", "/etc/hostname", "/etc/resolv.conf")

  2. Mount "/proc" filesystem

  3. Mount "/dev/pts" filesystem

  4. Create "/usr/sbin/policy-rc.d" which always exits with 101

  5. Run "chroot /sid-root bin/bash -c 'exec -a -bash bin/bash'"

[Note] Note

Some programs under chroot may require access to more files from the parent system to function than pbuilder provides. For example, "/sys", "/etc/passwd", "/etc/group", "/var/run/utmp", "/var/log/wtmp", etc. may need to be bind-mounted or copied.

[Note] Note

The "/usr/sbin/policy-rc.d" file prevents daemon programs to be started automatically on the Debian system. See "/usr/share/doc/sysv-rc/README.policy-rc.d.gz".

[Tip] Tip

The original purpose of the specialized chroot package, pbuilder is to construct a chroot system and builds a package inside the chroot. It is an ideal system to use to check that a package's build-dependencies are correct, and to be sure that unnecessary and wrong build dependencies do not exist in the resulting package.

[Tip] Tip

Similar schroot package may give you an idea to run i386 chroot system under amd64 parent system.

I recommend you to use QEMU or VirtualBox on a Debian stable system to run multiple desktop systems safely using virtualization. These enable you to run desktop applications of Debian unstable and testing without usual risks associated with them.

Since pure QEMU is very slow, it is recommended to accelerate it with KVM when the host system support it.

The virtual disk image "virtdisk.qcow2" containing a Debian system for QEMU can be created using debian-installer: Small CDs as follows.

$ wget
$ qemu-img create -f qcow2 virtdisk.qcow2 5G
$ qemu -hda virtdisk.qcow2 -cdrom debian-503-amd64-netinst.iso -boot d -m 256

See more tips at Debian wiki: QEMU.

VirtualBox comes with Qt GUI tools and quite intuitive. Its GUI and command line tools are explained in VirtualBox User Manual and VirtualBox User Manual (PDF).

[Tip] Tip

Running other GNU/Linux distributions such as Ubuntu and Fedora under virtualization is a great way to learn configuration tips. Other proprietary OSs may be run nicely under this GNU/Linux virtualization, too.