Thanks for all your error reports, I didn't forget it. I'll cleanup my guide soon. Thanks again!

Exercise 20. Filesystems: modifying and creating filesystems, tune2fs, mkfs

Let me introduce you to filesystem-related terminology:

  1. Filesystem — a means to organize data expected to be retained after a program terminates by providing procedures to store, retrieve and update data, as well as manage the available space on the device(s) which contain it.
  2. Inode — an Index Node, is a structure which store all the information about a file system object (file, directory, etc.) except data content and file name.
  3. Block — a minimum chunk of disk space which can be allocated. It usually defaults to 4096 bytes, or 4 Kilobytes.
  4. Journal — a structure which allows the filesystem track of what was written and when. This allows to quickly understand what was not properly written in case of power outage or similar problem.

Next, let me give you an outline explaining in broad strokes how a filesystem works. To access file by name Linux kernel:

  1. Finds file name in directory containing the file.
  2. Gets file Inode number.
  3. Finds Inode by its number in Inodes area.
  4. Reads locations of blocks containing the data from this Inode.
  5. Using this locations reads file from this blocks in data area.

Now, every file system has many options associated with it. These options may be viewed and changed via tune2fs program. Here is an annotated output of tune2fs -l /dev/sda8:

user1@vm1:~$ sudo tune2fs -l /dev/sda8
tune2fs 1.41.12 (17-May-2010)
# Just a label. Can be anything or nothing.
Filesystem volume name:   <none>
# Here should be last mount point.
Last mounted on:          <not available>
# Unique file system identifier.
Filesystem UUID:          869db6b4-aea0-4a25-8bd2-f0b53dd7a88e
# A special number at known place, which defines FS type. Try this:
# sudo dd if=/dev/sda8 of=/dev/stdout bs=1 skip=1080 count=2 | hexdump -C
Filesystem magic number:  0xEF53
# FS version
Filesystem revision #:    1 (dynamic)
# Enabled FS features
Filesystem features:
    # It is a journaling file system. A journaling file system is a file system that
    # keeps track of the changes that will be made in a journal (usually a circular log in
    # a dedicated area of the file system) before committing them to the main file system.
    # In the event of a system crash or power failure, such file systems are quicker to bring
    # back online and less likely to become corrupted.
    # http://en.wikipedia.org/wiki/Journaling_file_system
    has_journal
    # Has extended attributes like extended ACL.
    ext_attr
    # Reserves space for system information, which allows FS resizing.
    resize_inode
    # Use indexes to speed up look-ups in large directories.
    dir_index
    # Store file type information in directory entries.
    filetype
    # Means that fsck needs to be run.
    needs_recovery
    # Fewer superblock backup copies, saves space on large FS.
    sparse_super
    # Can contain files > 2GB. Kernel sets this up automatically on >2GB file creation.
    large_file
# Which hash to use in dir_index.
Filesystem flags:         signed_directory_hash
# Which options to use when mounting.
Default mount options:    (none)
# Need to run fsck?
Filesystem state:         clean
# What to do on error? Continue, remount read-only or panic?
Errors behavior:          Continue
# Which OS uses this FS
Filesystem OS type:       Linux
# Number of inodes total. Inode means "Index Node" and is a structure which
# store all the information about a file system object (file, directory, etc.) execpt
# data content and file name. The implication of this is that you can not have more files
# than you have Inodes.
# This is Inode structure which describes what information which is stored in Inode:
#/* Inode structure as it appears on an Inode table. */
#struct dinode
#{ ushort  di_mode;     /* mode and type of file   */
#  short   di_nlink;    /* number of links to file */
#  ushort  di_uid;      /* owner's user id         */
#  ushort  di_gid;      /* owner's group id        */
#  off_t   di_size;     /* number of bytes in file */
#  char    di_addr[39]; /* disk block addresses    */
#  char    di_gen;      /* file generation number  */
#  time_t  di_atime;    /* time last accessed      */
#  time_t  di_mtime;    /* time last modified      */
#  time_t  di_ctime;    /* time created            */
#};
# This is also a good explanation:
# http://support.tux4u.nl/dokuwiki/lib/exe/fetch.php?media=linux:disk_bestandssysteem:inode.pdf
Inode count:              62464
# How many inodes are currently free.
Free inodes:              62452
# First block on device. Because each partition is represented as a separate device,
# this is set to 0.
First block:              0
# This is the Inode number of the first Inode in the file system.
First inode:              11
# Inode size in bytes. This is sometimes increased by default in newer Linux distributions,
# to permit storage of extended attributes in file, for example nanosecond timestamps.
Inode size:               256
# Requred space for additional Inode fields.
Required extra isize:     28
# Desired space for additional Inode fields. Does not matter because this size is
# requred anyway.
Desired extra isize:      28
# An invisible inode which stores file system journal.
Journal inode:            8
# Number of blocks total. A block is a minimum chunk of disk space which can be allocated.
# You can calculate partition size in GB with the following formula:
# Block count * Block size
# ------------------------
#         1024^3
Block count:              249856
# How many blocks are reserved for superuser only. Ordinary users cannot use this
# reserved space. This is done for system remain operational in case some rogue
# program decides to fill up all available disk space.
Reserved block count:     12492
# How many blocks are currently free.
Free blocks:              241518
# An algorithm used for directory index, dir_index.
Default directory hash:   half_md4
# As far as I can tell this is a seed value for dir_index hashing algorithm.
Directory Hash Seed:      d02c6099-bd06-4d29-a3d7-779df2aa2410
# Journal backup options.
Journal backup:           inode blocks                          #
# Block size, in bytes. 4096 bytes means 4 KB here.
Block size:               4096
# Not implemented in ext3 FS. It is a feature which would allow to write several small files
# in one block to conserve space.
Fragment size:            4096
# Reserved space so Group Descriptor Table may grow in the future.
Reserved GDT blocks:      60
# Number of blocks per block group. Block groups contain a redundant copy of crucial filesystem
# control information (superblock and the FS descriptors) and also contains a
# part of the filesystem (a block bitmap, an Inode bitmap, a piece of the Inode table,
# and data blocks). The structure of a block group is represented in this table:
#,---------+---------+---------+---------+---------+---------,
#| Super   | FS      | Block   | Inode   | Inode   | Data    |
#| block   | desc.   | bitmap  | bitmap  | table   | blocks  |
#`---------+---------+---------+---------+---------+---------'
# http://tldp.org/HOWTO/Filesystems-HOWTO-6.html
Blocks per group:         32768
# Number of fragments per group. Because there are no fragments in ext3 FS this is
# equal to Blocks per group.
Fragments per group:      32768
# Number of inodes per group.
Inodes per group:         7808
# The Inode blocks per group. Inode blocks an index into a teable that describe
# all file properties except the file name. It has references to the data blocks
# that contain the actual contents of the file.
# http://www.porcupine.org/forensics/forensic-discovery/chapter3.html
Inode blocks per group:   488
# FS Creation time.
Filesystem created:       Mon Jul  2 06:16:24 2012
# Last FS mount time.
Last mount time:          Mon Jul  2 06:57:21 2012
# Last FS write time.
Last write time:          Mon Jul  2 06:57:21 2012
# Number of times this FS was mounted.
Mount count:              6
# Number of counts before automatic checking. FS will be automatically checked if
# filesystem was mounted this number of times, or if check interval has come.
Maximum mount count:      34
# Last fsck run time
Last checked:             Mon Jul  2 06:16:24 2012
# Next FS check interval. FS will be checked if this interval has come,
# or if Maximum mount count is reached.
Check interval:           15552000 (6 months)
# Next FS check interval in human readable format.
Next check after:         Sat Dec 29 05:16:24 2012
# User ID for user which is able to user reserved space.
# It is root user (superuser) by default.
Reserved blocks uid:      0 (user root)
# Group ID for group which is able to user reserved space.
# It is root group by default.
Reserved blocks gid:      0 (group root)

Scary, huh? In practice you will find that only several parameters from this description are actually useful, which are:

  1. Reserved block count.
  2. Maximum mount count.
  3. Check interval.

Usually you would not want to modify other parameters, they are sane by default. Here is a list of commands for working with file systems:

  1. mkfs.ext3 — create an ext3 file system. If this command is executed on a device with existing file system, this file system will be destroyed, so proceed with caution.
  2. mkfs.ext4 — create an ext4 file system. This is actually the same program, try sudo find /sbin -samefile sbin/mkfs.ext3.
  3. tune2fs — print out and change file system parameters.

Now you will learn how to create new filesystem and modify its parameters.

Do this

1: sudo -s
2: umount /tmp
3: blkid | grep /dev/sda8
4: mkfs.ext3 /dev/sda8
5: blkid | grep /dev/sda8
6: blkid | grep /dev/sda8 >> /etc/fstab
7: vim /etc/fstab

Now you must replace UUID in line for /tmp:

# /tmp was on /dev/sda8 during installation
UUID=869db6b4-aea0-4a25-8bd2-f0b53dd7a88e /tmp            ext3    defaults        0       2

With one that you appended to the end of file:

/dev/sda8: UUID="53eed507-18e8-4f71-9003-bcea8c4fd2dd" TYPE="ext3" SEC_TYPE="ext2"

Because UUID are unique by definition yours will differ from mine. After replacing this UUID, write file, quit and continue to type:

 8: mount /tmp
 9: tune2fs -c 2 /dev/sda8
10: unmount /tmp
11: fsck /tmp
12: for ((i=1;i<=4;i++)); do mount /tmp ; umount /tmp ; cat /var/log/messages | tail -n 4 ; done
13: fsck /tmp
14: mount -a

What you should see

user1@vm1:~$ sudo -s
root@vm1:/home/user1# umount /tmp
root@vm1:/home/user1# blkid | grep /dev/sda8
/dev/sda8: UUID="869db6b4-aea0-4a25-8bd2-f0b53dd7a88e" TYPE="ext3" SEC_TYPE="ext2"
root@vm1:/home/user1# mkfs.ext3 /dev/sda8
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
62464 inodes, 249856 blocks
12492 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=255852544
8 block groups
32768 blocks per group, 32768 fragments per group
7808 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376
 
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done
 
This filesystem will be automatically checked every 28 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
root@vm1:/home/user1# blkid | grep /dev/sda8
/dev/sda8: UUID="53eed507-18e8-4f71-9003-bcea8c4fd2dd" TYPE="ext3" SEC_TYPE="ext2"
root@vm1:/home/user1# blkid | grep /dev/sda8 >> /etc/fstab
root@vm1:/home/user1# vim /etc/fstab
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
# / was on /dev/vda5 during installation
UUID=128559db-a2e0-4983-91ad-d4f43f27da49 /               ext3    errors=re
# /home was on /dev/vda10 during installation
UUID=32852d29-ddee-4a8d-9b1e-f46569a6b897 /home           ext3    defaults
# /tmp was on /dev/sda8 during installation
UUID=869db6b4-aea0-4a25-8bd2-f0b53dd7a88e /tmp            ext3    defaults
# /usr was on /dev/vda9 during installation
UUID=0221be16-496b-4277-b131-2371ce097b44 /usr            ext3    defaults
# /var was on /dev/vda8 during installation
UUID=2db00f94-3605-4229-8813-0ee23ad8634e /var            ext3    defaults
# swap was on /dev/vda6 during installation
UUID=3a936af2-2c04-466d-b98d-09eacc5d104c none            swap    sw
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto     0       0
/dev/sda8: UUID="53eed507-18e8-4f71-9003-bcea8c4fd2dd" TYPE="ext3" SEC_TYPE
                                                         22,1          Bot
 
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
# / was on /dev/vda5 during installation
UUID=128559db-a2e0-4983-91ad-d4f43f27da49 /               ext3    errors=re
# /home was on /dev/vda10 during installation
UUID=32852d29-ddee-4a8d-9b1e-f46569a6b897 /home           ext3    defaults
# /tmp was on /dev/sda8 during installation
UUID=53eed507-18e8-4f71-9003-bcea8c4fd2dd /tmp            ext3    defaults
# /usr was on /dev/vda9 during installation
UUID=0221be16-496b-4277-b131-2371ce097b44 /usr            ext3    defaults
# /var was on /dev/vda8 during installation
UUID=2db00f94-3605-4229-8813-0ee23ad8634e /var            ext3    defaults
 
# swap was on /dev/vda6 during installation
UUID=3a936af2-2c04-466d-b98d-09eacc5d104c none            swap    sw
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto     0       0
"/etc/fstab" 22L, 1277C written
root@vm1:/home/user1# mount /tmp
root@vm1:/home/user1# tune2fs -c 2 /dev/sda8
tune2fs 1.41.12 (17-May-2010)
Setting maximal mount count to 2
root@vm1:/home/user1# unmount /tmp
root@vm1:/home/user1# fsck /tmp
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
/dev/sda8: clean, 11/62464 files, 8337/249856 blocks (check in 2 mounts)
root@vm1:/home/user1# for ((i=1;i<=4;i++)); do mount /tmp ; umount /tmp ; cat /var/log/messages | tail -n 4 ; done
Jul  2 12:11:43 vm1 kernel: [21080.920658] EXT3-fs: mounted filesystem with ordered data mode.
Jul  2 12:11:58 vm1 kernel: [21096.363787] kjournald starting.  Commit interval 5 seconds
Jul  2 12:11:58 vm1 kernel: [21096.364167] EXT3 FS on sda8, internal journal
Jul  2 12:11:58 vm1 kernel: [21096.364171] EXT3-fs: mounted filesystem with ordered data mode.
Jul  2 12:11:58 vm1 kernel: [21096.364171] EXT3-fs: mounted filesystem with ordered data mode.
Jul  2 12:11:58 vm1 kernel: [21096.381372] kjournald starting.  Commit interval 5 seconds
Jul  2 12:11:58 vm1 kernel: [21096.381539] EXT3 FS on sda8, internal journal
Jul  2 12:11:58 vm1 kernel: [21096.381542] EXT3-fs: mounted filesystem with ordered data mode.
Jul  2 12:11:58 vm1 kernel: [21096.396152] kjournald starting.  Commit interval 5 seconds
Jul  2 12:11:58 vm1 kernel: [21096.396158] EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
Jul  2 12:11:58 vm1 kernel: [21096.396344] EXT3 FS on sda8, internal journal
Jul  2 12:11:58 vm1 kernel: [21096.396348] EXT3-fs: mounted filesystem with ordered data mode.
Jul  2 12:11:58 vm1 kernel: [21096.412434] kjournald starting.  Commit interval 5 seconds
Jul  2 12:11:58 vm1 kernel: [21096.412441] EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
Jul  2 12:11:58 vm1 kernel: [21096.412610] EXT3 FS on sda8, internal journal
Jul  2 12:11:58 vm1 kernel: [21096.412612] EXT3-fs: mounted filesystem with ordered data mode.
root@vm1:/home/user1# fsck /tmp
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
/dev/sda8 has been mounted 4 times without being checked, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sda8: 11/62464 files (0.0% non-contiguous), 8337/249856 blocks
root@vm1:/home/user1# mount -a
root@vm1:/home/user1#

Explanation

  1. Executes root (superuser) shell.
  2. Unmounts /tmp, reading its location from /etc/fstab.
  3. Prints out /dev/sda8 UUID, /dev/sda8 being the file system which mounts on /tmp.
  4. Makes a new file system on /dev/sda8.
  5. Prints out /dev/sda8 UUID one again, notice how it changed because you created a new file system.
  6. Appends this UUID to /etc/fstab.
  7. Opens /etc/fstab for editing.
  8. Mounts newly created file system. This is actually a check if you had replaced UUID correctly, if not there will be an error message.
  9. Sets checking /dev/sda8 every 2 mounts.
  10. Unmounts /dev/sda8.
  11. Checks /dev/sda8.
  12. Mounts, unmounts /dev/sda8 and shows you last 4 lines from /var/log/messages four times in a row. Notice how starting from 3-rd mount system notifies you that e2fsck needs to be run. If you reboot your system, it would run e2fsck for you.
  13. Checks /dev/sda8. fsck determines file system type and calls e2fsck for you automatically.
  14. Mounts all file systems. If there is no error, you successfully finished this exercise.

Extra credit

  1. Read man mkfs, man mkfs.ext3, man tune2fs.
  2. Read tune2fs -l listing at top of the page and read magic numbers for all your file systems.
  3. Calculate size of your file systems by hand using formula provided in block description of tune2fs -l listing.
  4. Read this presentation and do what is shown it in: http://mcgrewsecurity.com/training/extx.pdf

Discussion

Navigation

Learn Linux The Hard Way