Exercise 20. Filesystems: modifying and creating filesystems, tune2fs, mkfs
Let me introduce you to filesystem-related terminology:
- Filesystem — a means to organize data expected to be retained after a program terminates by providing procedures to store, retrieve and update data, as well as manage the available space on the device(s) which contain it.
- Inode — an Index Node, is a structure which store all the information about a file system object (file, directory, etc.) except data content and file name.
- Block — a minimum chunk of disk space which can be allocated. It usually defaults to 4096 bytes, or 4 Kilobytes.
- Journal — a structure which allows the filesystem track of what was written and when. This allows to quickly understand what was not properly written in case of power outage or similar problem.
Next, let me give you an outline explaining in broad strokes how a filesystem works. To access file by name Linux kernel:
- Finds file name in directory containing the file.
- Gets file Inode number.
- Finds Inode by its number in Inodes area.
- Reads locations of blocks containing the data from this Inode.
- Using this locations reads file from this blocks in data area.
Now, every file system has many options associated with it. These options may be viewed and changed via tune2fs program. Here is an annotated output of tune2fs -l /dev/sda8:
user1@vm1:~$ sudo tune2fs -l /dev/sda8 tune2fs 1.41.12 (17-May-2010) # Just a label. Can be anything or nothing. Filesystem volume name: <none> # Here should be last mount point. Last mounted on: <not available> # Unique file system identifier. Filesystem UUID: 869db6b4-aea0-4a25-8bd2-f0b53dd7a88e # A special number at known place, which defines FS type. Try this: # sudo dd if=/dev/sda8 of=/dev/stdout bs=1 skip=1080 count=2 | hexdump -C Filesystem magic number: 0xEF53 # FS version Filesystem revision #: 1 (dynamic) # Enabled FS features Filesystem features: # It is a journaling file system. A journaling file system is a file system that # keeps track of the changes that will be made in a journal (usually a circular log in # a dedicated area of the file system) before committing them to the main file system. # In the event of a system crash or power failure, such file systems are quicker to bring # back online and less likely to become corrupted. # http://en.wikipedia.org/wiki/Journaling_file_system has_journal # Has extended attributes like extended ACL. ext_attr # Reserves space for system information, which allows FS resizing. resize_inode # Use indexes to speed up look-ups in large directories. dir_index # Store file type information in directory entries. filetype # Means that fsck needs to be run. needs_recovery # Fewer superblock backup copies, saves space on large FS. sparse_super # Can contain files > 2GB. Kernel sets this up automatically on >2GB file creation. large_file # Which hash to use in dir_index. Filesystem flags: signed_directory_hash # Which options to use when mounting. Default mount options: (none) # Need to run fsck? Filesystem state: clean # What to do on error? Continue, remount read-only or panic? Errors behavior: Continue # Which OS uses this FS Filesystem OS type: Linux # Number of inodes total. Inode means "Index Node" and is a structure which # store all the information about a file system object (file, directory, etc.) execpt # data content and file name. The implication of this is that you can not have more files # than you have Inodes. # This is Inode structure which describes what information which is stored in Inode: #/* Inode structure as it appears on an Inode table. */ #struct dinode #{ ushort di_mode; /* mode and type of file */ # short di_nlink; /* number of links to file */ # ushort di_uid; /* owner's user id */ # ushort di_gid; /* owner's group id */ # off_t di_size; /* number of bytes in file */ # char di_addr[39]; /* disk block addresses */ # char di_gen; /* file generation number */ # time_t di_atime; /* time last accessed */ # time_t di_mtime; /* time last modified */ # time_t di_ctime; /* time created */ #}; # This is also a good explanation: # http://support.tux4u.nl/dokuwiki/lib/exe/fetch.php?media=linux:disk_bestandssysteem:inode.pdf Inode count: 62464 # How many inodes are currently free. Free inodes: 62452 # First block on device. Because each partition is represented as a separate device, # this is set to 0. First block: 0 # This is the Inode number of the first Inode in the file system. First inode: 11 # Inode size in bytes. This is sometimes increased by default in newer Linux distributions, # to permit storage of extended attributes in file, for example nanosecond timestamps. Inode size: 256 # Requred space for additional Inode fields. Required extra isize: 28 # Desired space for additional Inode fields. Does not matter because this size is # requred anyway. Desired extra isize: 28 # An invisible inode which stores file system journal. Journal inode: 8 # Number of blocks total. A block is a minimum chunk of disk space which can be allocated. # You can calculate partition size in GB with the following formula: # Block count * Block size # ------------------------ # 1024^3 Block count: 249856 # How many blocks are reserved for superuser only. Ordinary users cannot use this # reserved space. This is done for system remain operational in case some rogue # program decides to fill up all available disk space. Reserved block count: 12492 # How many blocks are currently free. Free blocks: 241518 # An algorithm used for directory index, dir_index. Default directory hash: half_md4 # As far as I can tell this is a seed value for dir_index hashing algorithm. Directory Hash Seed: d02c6099-bd06-4d29-a3d7-779df2aa2410 # Journal backup options. Journal backup: inode blocks # # Block size, in bytes. 4096 bytes means 4 KB here. Block size: 4096 # Not implemented in ext3 FS. It is a feature which would allow to write several small files # in one block to conserve space. Fragment size: 4096 # Reserved space so Group Descriptor Table may grow in the future. Reserved GDT blocks: 60 # Number of blocks per block group. Block groups contain a redundant copy of crucial filesystem # control information (superblock and the FS descriptors) and also contains a # part of the filesystem (a block bitmap, an Inode bitmap, a piece of the Inode table, # and data blocks). The structure of a block group is represented in this table: #,---------+---------+---------+---------+---------+---------, #| Super | FS | Block | Inode | Inode | Data | #| block | desc. | bitmap | bitmap | table | blocks | #`---------+---------+---------+---------+---------+---------' # http://tldp.org/HOWTO/Filesystems-HOWTO-6.html Blocks per group: 32768 # Number of fragments per group. Because there are no fragments in ext3 FS this is # equal to Blocks per group. Fragments per group: 32768 # Number of inodes per group. Inodes per group: 7808 # The Inode blocks per group. Inode blocks an index into a teable that describe # all file properties except the file name. It has references to the data blocks # that contain the actual contents of the file. # http://www.porcupine.org/forensics/forensic-discovery/chapter3.html Inode blocks per group: 488 # FS Creation time. Filesystem created: Mon Jul 2 06:16:24 2012 # Last FS mount time. Last mount time: Mon Jul 2 06:57:21 2012 # Last FS write time. Last write time: Mon Jul 2 06:57:21 2012 # Number of times this FS was mounted. Mount count: 6 # Number of counts before automatic checking. FS will be automatically checked if # filesystem was mounted this number of times, or if check interval has come. Maximum mount count: 34 # Last fsck run time Last checked: Mon Jul 2 06:16:24 2012 # Next FS check interval. FS will be checked if this interval has come, # or if Maximum mount count is reached. Check interval: 15552000 (6 months) # Next FS check interval in human readable format. Next check after: Sat Dec 29 05:16:24 2012 # User ID for user which is able to user reserved space. # It is root user (superuser) by default. Reserved blocks uid: 0 (user root) # Group ID for group which is able to user reserved space. # It is root group by default. Reserved blocks gid: 0 (group root)
Scary, huh? In practice you will find that only several parameters from this description are actually useful, which are:
- Reserved block count.
- Maximum mount count.
- Check interval.
Usually you would not want to modify other parameters, they are sane by default. Here is a list of commands for working with file systems:
- mkfs.ext3 — create an ext3 file system. If this command is executed on a device with existing file system, this file system will be destroyed, so proceed with caution.
- mkfs.ext4 — create an ext4 file system. This is actually the same program, try sudo find /sbin -samefile sbin/mkfs.ext3.
- tune2fs — print out and change file system parameters.
Now you will learn how to create new filesystem and modify its parameters.
Do this
1: sudo -s 2: umount /tmp 3: blkid | grep /dev/sda8 4: mkfs.ext3 /dev/sda8 5: blkid | grep /dev/sda8 6: blkid | grep /dev/sda8 >> /etc/fstab 7: vim /etc/fstab
Now you must replace UUID in line for /tmp:
# /tmp was on /dev/sda8 during installation UUID=869db6b4-aea0-4a25-8bd2-f0b53dd7a88e /tmp ext3 defaults 0 2
With one that you appended to the end of file:
/dev/sda8: UUID="53eed507-18e8-4f71-9003-bcea8c4fd2dd" TYPE="ext3" SEC_TYPE="ext2"
Because UUID are unique by definition yours will differ from mine. After replacing this UUID, write file, quit and continue to type:
8: mount /tmp 9: tune2fs -c 2 /dev/sda8 10: unmount /tmp 11: fsck /tmp 12: for ((i=1;i<=4;i++)); do mount /tmp ; umount /tmp ; cat /var/log/messages | tail -n 4 ; done 13: fsck /tmp 14: mount -a
What you should see
user1@vm1:~$ sudo -s root@vm1:/home/user1# umount /tmp root@vm1:/home/user1# blkid | grep /dev/sda8 /dev/sda8: UUID="869db6b4-aea0-4a25-8bd2-f0b53dd7a88e" TYPE="ext3" SEC_TYPE="ext2" root@vm1:/home/user1# mkfs.ext3 /dev/sda8 mke2fs 1.41.12 (17-May-2010) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=0 blocks, Stripe width=0 blocks 62464 inodes, 249856 blocks 12492 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=255852544 8 block groups 32768 blocks per group, 32768 fragments per group 7808 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376 Writing inode tables: done Creating journal (4096 blocks): done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 28 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override. root@vm1:/home/user1# blkid | grep /dev/sda8 /dev/sda8: UUID="53eed507-18e8-4f71-9003-bcea8c4fd2dd" TYPE="ext3" SEC_TYPE="ext2" root@vm1:/home/user1# blkid | grep /dev/sda8 >> /etc/fstab root@vm1:/home/user1# vim /etc/fstab # that works even if disks are added and removed. See fstab(5). # # <file system> <mount point> <type> <options> <dump> <pass> proc /proc proc defaults 0 0 # / was on /dev/vda5 during installation UUID=128559db-a2e0-4983-91ad-d4f43f27da49 / ext3 errors=re # /home was on /dev/vda10 during installation UUID=32852d29-ddee-4a8d-9b1e-f46569a6b897 /home ext3 defaults # /tmp was on /dev/sda8 during installation UUID=869db6b4-aea0-4a25-8bd2-f0b53dd7a88e /tmp ext3 defaults # /usr was on /dev/vda9 during installation UUID=0221be16-496b-4277-b131-2371ce097b44 /usr ext3 defaults # /var was on /dev/vda8 during installation UUID=2db00f94-3605-4229-8813-0ee23ad8634e /var ext3 defaults # swap was on /dev/vda6 during installation UUID=3a936af2-2c04-466d-b98d-09eacc5d104c none swap sw /dev/scd0 /media/cdrom0 udf,iso9660 user,noauto 0 0 /dev/sda8: UUID="53eed507-18e8-4f71-9003-bcea8c4fd2dd" TYPE="ext3" SEC_TYPE 22,1 Bot # # <file system> <mount point> <type> <options> <dump> <pass> proc /proc proc defaults 0 0 # / was on /dev/vda5 during installation UUID=128559db-a2e0-4983-91ad-d4f43f27da49 / ext3 errors=re # /home was on /dev/vda10 during installation UUID=32852d29-ddee-4a8d-9b1e-f46569a6b897 /home ext3 defaults # /tmp was on /dev/sda8 during installation UUID=53eed507-18e8-4f71-9003-bcea8c4fd2dd /tmp ext3 defaults # /usr was on /dev/vda9 during installation UUID=0221be16-496b-4277-b131-2371ce097b44 /usr ext3 defaults # /var was on /dev/vda8 during installation UUID=2db00f94-3605-4229-8813-0ee23ad8634e /var ext3 defaults # swap was on /dev/vda6 during installation UUID=3a936af2-2c04-466d-b98d-09eacc5d104c none swap sw /dev/scd0 /media/cdrom0 udf,iso9660 user,noauto 0 0 "/etc/fstab" 22L, 1277C written root@vm1:/home/user1# mount /tmp root@vm1:/home/user1# tune2fs -c 2 /dev/sda8 tune2fs 1.41.12 (17-May-2010) Setting maximal mount count to 2 root@vm1:/home/user1# unmount /tmp root@vm1:/home/user1# fsck /tmp fsck from util-linux-ng 2.17.2 e2fsck 1.41.12 (17-May-2010) /dev/sda8: clean, 11/62464 files, 8337/249856 blocks (check in 2 mounts) root@vm1:/home/user1# for ((i=1;i<=4;i++)); do mount /tmp ; umount /tmp ; cat /var/log/messages | tail -n 4 ; done Jul 2 12:11:43 vm1 kernel: [21080.920658] EXT3-fs: mounted filesystem with ordered data mode. Jul 2 12:11:58 vm1 kernel: [21096.363787] kjournald starting. Commit interval 5 seconds Jul 2 12:11:58 vm1 kernel: [21096.364167] EXT3 FS on sda8, internal journal Jul 2 12:11:58 vm1 kernel: [21096.364171] EXT3-fs: mounted filesystem with ordered data mode. Jul 2 12:11:58 vm1 kernel: [21096.364171] EXT3-fs: mounted filesystem with ordered data mode. Jul 2 12:11:58 vm1 kernel: [21096.381372] kjournald starting. Commit interval 5 seconds Jul 2 12:11:58 vm1 kernel: [21096.381539] EXT3 FS on sda8, internal journal Jul 2 12:11:58 vm1 kernel: [21096.381542] EXT3-fs: mounted filesystem with ordered data mode. Jul 2 12:11:58 vm1 kernel: [21096.396152] kjournald starting. Commit interval 5 seconds Jul 2 12:11:58 vm1 kernel: [21096.396158] EXT3-fs warning: maximal mount count reached, running e2fsck is recommended Jul 2 12:11:58 vm1 kernel: [21096.396344] EXT3 FS on sda8, internal journal Jul 2 12:11:58 vm1 kernel: [21096.396348] EXT3-fs: mounted filesystem with ordered data mode. Jul 2 12:11:58 vm1 kernel: [21096.412434] kjournald starting. Commit interval 5 seconds Jul 2 12:11:58 vm1 kernel: [21096.412441] EXT3-fs warning: maximal mount count reached, running e2fsck is recommended Jul 2 12:11:58 vm1 kernel: [21096.412610] EXT3 FS on sda8, internal journal Jul 2 12:11:58 vm1 kernel: [21096.412612] EXT3-fs: mounted filesystem with ordered data mode. root@vm1:/home/user1# fsck /tmp fsck from util-linux-ng 2.17.2 e2fsck 1.41.12 (17-May-2010) /dev/sda8 has been mounted 4 times without being checked, check forced. Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/sda8: 11/62464 files (0.0% non-contiguous), 8337/249856 blocks root@vm1:/home/user1# mount -a root@vm1:/home/user1#
Explanation
- Executes root (superuser) shell.
- Unmounts /tmp, reading its location from /etc/fstab.
- Prints out /dev/sda8 UUID, /dev/sda8 being the file system which mounts on /tmp.
- Makes a new file system on /dev/sda8.
- Prints out /dev/sda8 UUID one again, notice how it changed because you created a new file system.
- Appends this UUID to /etc/fstab.
- Opens /etc/fstab for editing.
- Mounts newly created file system. This is actually a check if you had replaced UUID correctly, if not there will be an error message.
- Sets checking /dev/sda8 every 2 mounts.
- Unmounts /dev/sda8.
- Checks /dev/sda8.
- Mounts, unmounts /dev/sda8 and shows you last 4 lines from /var/log/messages four times in a row. Notice how starting from 3-rd mount system notifies you that e2fsck needs to be run. If you reboot your system, it would run e2fsck for you.
- Checks /dev/sda8. fsck determines file system type and calls e2fsck for you automatically.
- Mounts all file systems. If there is no error, you successfully finished this exercise.
Extra credit
- Read man mkfs, man mkfs.ext3, man tune2fs.
- Read tune2fs -l listing at top of the page and read magic numbers for all your file systems.
- Calculate size of your file systems by hand using formula provided in block description of tune2fs -l listing.
- Read this presentation and do what is shown it in: http://mcgrewsecurity.com/training/extx.pdf