Study guide
5.4 Advanced Text Filters: sort, uniq, wc, tr, split, less & Checksums (103.2)
LPIC-1 Linux Administrator — Exam 101 101-500
Track your progress as you read
Finished sections get a check mark.
1 Exam Overview & Study Strategy
1.1 LPIC-1 101-500 Exam Structure & Blueprint 1.2 Command Line & Shell Fundamentals for Exam Success 1.3 Fill-in-the-Blank Strategies & Common Traps
2 Topic 101: System Architecture
2.1 Hardware Discovery, Sysfs, Procfs & Kernel Modules (101.1) 2.2 Udev, Dbus & Device Management (101.1) 2.3 BIOS vs. UEFI & System Boot Process (101.2) 2.4 Kernel Initialization, Initramfs & System Logs (101.2) 2.5 Systemd Boot Targets & Unit Management (101.3) 2.6 SysVinit Runlevels, Telinit & System Shutdown (101.3)
3 Topic 102: Linux Installation & Boot Management
3.1 Hard Disk Layout, Partition Schemes & Swap Design (102.1) 3.2 LVM & Standard Mount Point Allocation (102.1) 3.3 GRUB 2 Configuration & Troubleshooting (102.2) 3.4 Legacy GRUB & Alternative Boot Loaders (102.2) 3.5 Managing Shared Libraries & ldconfig (102.3) 3.6 Linux as a Virtualization Guest & Cloud Instances (102.6)
4 Topic 102: Package Management Systems
4.1 Debian Low-Level Package Management with dpkg (102.4) 4.2 High-Level Debian Package Management: apt, apt-get & apt-cache (102.4) 4.3 RPM Package Management with rpm (102.5) 4.4 YUM, DNF & Zypper Package Management (102.5) 4.5 Package Repositories & Integrity Verification (102.4/102.5) 4.6 Universal Linux Packages: Snap, Flatpak & AppImage (102.4/102.5)
5 Topic 103: Command Line & Text Processing
5.1 Shell Environment, Environment Variables & Aliases (103.1) 5.2 Command History, Quoting & Expansion (103.1) 5.3 Basic Text Filters: cat, head, tail, cut, paste, nl, od (103.2) 5.4 Advanced Text Filters: sort, uniq, wc, tr, split, less & Checksums (103.2) 5.5 Stream Editing with sed (103.2) 5.6 Basic File Management: cp, mv, rm, touch, mkdir, rmdir (103.3) 5.7 Archiving and Compression: tar, gzip, bzip2, xz, cpio & dd (103.3)
6 Topic 103: Streams, Processes, Regex & Vi
6.1 Standard Streams, Redirection & File Descriptors (103.4) 6.2 Pipelines, tee & xargs (103.4) 6.3 Process Monitoring: ps, top, free, watch, pgrep, pstree & uptime (103.5) 6.4 Signals, kill, Job Control, nohup, screen & tmux (103.5) 6.5 Process Priorities: nice & renice (103.6) 6.6 Regular Expressions & grep, egrep, fgrep (103.7) 6.7 vi / vim Modes, plus nano, Emacs & EDITOR Awareness (103.8)
7 Topic 104: Devices, Partitions & Filesystems
7.1 Block Devices, fdisk & gdisk (104.1) 7.2 GNU Parted & Partition Table Management (104.1) 7.3 Creating Filesystems: mkfs, ext4, XFS, VFAT/exFAT & Btrfs (104.1) 7.4 Filesystem Integrity: fsck, e2fsck, xfs_repair, xfs_fsr & xfs_db (104.2) 7.5 Monitoring Disk Usage: df, du & tune2fs (104.2) 7.6 Managing Swap Space: mkswap, swapon & swapoff (104.1/104.2)
8 Topic 104: Mounts, Permissions, Links & FHS
8.1 Mounting and Unmounting Filesystems: mount & umount (104.3) 8.2 Persistent Mounts: /etc/fstab, UUIDs & systemd mount units (104.3) 8.3 Standard File Permissions: chmod, chown, chgrp & umask (104.5) 8.4 Special Permissions: SUID, SGID & Sticky Bit (104.5) 8.5 Hard Links vs. Symbolic Links: ln & Inodes (104.6) 8.6 Filesystem Hierarchy Standard (FHS) & File Location (104.7)
5.4 Advanced Text Filters: sort, uniq, wc, tr, split, less & Checksums (103.2)
Key Takeaways Explain This Topic AI
sort arranges lines lexicographically by default; use -n for numeric, -r for reverse, -t for field delimiters, and -k for specific column keys.
uniq filters or reports duplicate lines and REQUIRES sorted input; -c displays counts, -d displays duplicates only, and -u displays unique lines only.
wc counts lines (-l), words (-w), characters (-m), and bytes (-c); split divides files into line (-l) or byte (-b) sized chunks; and expand and unexpand convert between TAB characters and spaces.
tr translates, deletes (-d), or squeezes (-s) characters strictly from stdin; it does NOT accept filenames as direct command arguments.
Objective 103.2 also names less (backward-scrolling pager), the md5sum/sha256sum/sha512sum digest tools (verify with -c against a published checksum file), and zcat/bzcat/xzcat for reading .gz/.bz2/.xz files without decompressing them to disk.
Last updated: August 2026
5.4 Advanced Text Filters: sort, uniq, wc, tr, split, expand (103.2)
Quick Summary: Analyzing logs and processing system data streams requires advanced pipeline filtering. The LPIC-1 exam tests multi-key sorting with sort , duplicate suppression and frequency analysis with uniq (which strictly requires pre-sorted input), word and line metrics with wc , character translation and deletion with tr , file chunking with split , and tab whitespace expansion with expand and unexpand .
1. sort : Sorting Text Streams
The sort command rearranges lines of text from files or standard input into a sorted sequence. By default, sort evaluates lines lexicographically according to the system locale (ASCII collation where numbers precede uppercase letters, which precede lowercase letters).
Essential sort Command-Line Flags
- Flag Long Option Function & Operation
- n --numeric-sort Compares strings according to their numeric numerical value (e.g., 2 comes before 10 )
- r --reverse Inverts the sort order (outputs descending order)
- t '<delim>' --field-separator Sets the field delimiter character (default is whitespace)
- k <pos1>[,pos2] --key Specifies key definition: sorts on column pos1 through pos2
- u --unique Outputs only unique lines (purges duplicate sorting keys)
- f --ignore-case Folds lowercase characters to uppercase (case-insensitive sorting)
- h --human-numeric-sort Compares human-readable numbers with suffixes (e.g., 2K , 5M , 1G )
- M --month-sort Sorts by 3-letter month abbreviations ( JAN < FEB < MAR ... < DEC )
- V --version-sort Natural sorting of version numbers (e.g., v1.2 < v1.10 )
- o <file> --output=<file> Writes result directly to <file> (safe to overwrite input file directly)
- Multi-Column Key Specification ( -k )
The -k option takes a starting column position and an optional ending column position, along with individual column modifier flags: -k start [ .char ] [ flags ] [ , end [ .char ] [ flags ] ] \text{-k } \text{start}[\text{.char}][\text{flags}][,\text{end}[\text{.char}][\text{flags}]] -k start [ .char ] [ flags ] [ , end [ .char ] [ flags ]] # Sorting /etc/passwd numerically by UID (field 3), using ':' delimiter:
- $ sort -t: -k3,3n /etc/passwd
- root:x:0:0:root:/root:/bin/bash
- daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
- bin:x:2:2:bin:/bin:/usr/sbin/nologin
- student:x:1000:1000:student:/home/student:/bin/bash
# Multi-key sort: sort first by department (field 2 alphabetically), then by salary (field 3 numerically descending):
$ sort -t, -k2,2 -k3,3nr employees.csv
# Sorting directly in-place without shell redirection data corruption:
$ sort -n -o numbers.txt numbers.txt
⚠️ LPIC-1 Trap: Running sort file.txt > file.txt with standard shell redirection empties the file before sort can read it because the shell truncates the file during tokenization. Always use sort -o file.txt file.txt for in-place sorting.
2. uniq : Duplicate Line Filtering & Reporting
The uniq filter compares adjacent lines and removes or reports duplicate occurrences. CRITICAL PIPELINE RULE FOR uniq:
Unsorted Stream: Sorted Stream (sort): Deduplicated (uniq):
- ┌──────────┐ ┌──────────┐ ┌──────────┐
- │ apple │ │ apple │ │ apple │
- │ banana │ ==> uniq fails! │ apple │ ==> uniq succeeds! │ banana │
- │ apple │ (Non-adjacent) │ banana │ │ orange │
- │ orange │ │ orange │ └──────────┘
- └──────────┘ └──────────┘
The Fundamental Rule of uniq : uniq detects duplicates ONLY among adjacent, consecutive lines . To filter or count all duplicate entries across an entire file, you MUST pass the input through sort first: sort input.txt | uniq .
Essential uniq Command-Line Flags
- Flag Long Option Function & Description
- c --count Prefixes each line with the number of times it occurred in the stream
- d --repeated Prints only duplicate lines (lines appearing 2 or more times)
- u --unique Prints only unique lines (lines appearing strictly once )
- i --ignore-case Ignores case differences when comparing lines
- f <N> --skip-fields=N Skips the first N whitespace-separated fields before comparing
- s <N> --skip-chars=N Skips the first N characters before comparing
-w <N> --check-chars=N Compares at most N characters per line # Counting frequency of failed login attempts by IP address:
- $ awk '{print $1}' /var/log/auth_failures.log | sort | uniq -c | sort -nr
- 42 192.168.1.105
- 15 10.0.0.50
- 3 172.16.4.12
# Extracting only duplicate usernames from a raw list:
$ sort raw_users.txt | uniq -d
# Extracting entries that occur exactly once (strictly unique):
$ sort raw_users.txt | uniq -u
3. wc : Word, Line, Character and Byte Counting
The wc (word count) utility analyzes text files or standard input streams to compute line, word, byte, and character metrics. # Default wc invocation prints: [Lines] [Words] [Bytes] [Filename]
- $ wc /etc/passwd
- 45 85 2450 /etc/passwd
- Essential wc Command-Line Flags
- Flag Long Option Metric Counted & Exam Focus
- l --lines Counts the number of newline characters ( \n )
- w --words Counts words (strings delimited by whitespace: space, tab, newline)
- c --bytes Counts the number of bytes
- m --chars Counts the number of characters (differs from -c in multi-byte UTF-8 files)
-L --max-line-length Displays the display width (length) of the longest line in the file # Counting total active user accounts on the system:
- $ cut -d: -f1 /etc/passwd | wc -l
- 45
# Inspecting byte size vs UTF-8 character count:
- $ printf "München" | wc -c
- 8
- $ printf "München" | wc -m
- 7
- 4. tr : Character Translation and Deletion
The tr (translate) utility transforms, compresses, or deletes individual characters from standard input and writes the result to standard output.
⚠️ LPIC-1 Trap: tr NEVER accepts file paths as command arguments. Running tr a-z A-Z file.txt results in a fatal syntax error. tr operates strictly on standard input via shell redirection or pipelines ( tr 'a-z' 'A-Z' < file.txt or cat file.txt | tr ... ).
Essential tr Syntax & Flags tr [options] SET1 [SET2] \text{tr [options] SET1 [SET2]} tr [options] SET1 [SET2]
- Flag Long Option Function & Description
- d --delete Deletes all characters matching SET1 from the input stream
- s --squeeze-repeats Squeezes repeated consecutive occurrences of characters in SET1 into a single instance
- c / -C --complement Uses the complement of SET1 (all characters except those in SET1 )
- t --truncate-set1 Truncates SET1 to the length of SET2 before translating
- POSIX Character Classes for tr
- [:lower:] : All lowercase letters ( a-z )
- [:upper:] : All uppercase letters ( A-Z )
- [:digit:] : All numeric digits ( 0-9 )
- [:alpha:] : All alphabetic characters ( [:lower:] + [:upper:] )
- [:alnum:] : All alphanumeric characters ( [:alpha:] + [:digit:] )
- [:space:] : All horizontal and vertical whitespace (spaces, tabs, newlines, carriage returns)
[:blank:] : Horizontal whitespace (spaces and tabs only) # Converting lowercase text to uppercase:
- $ echo "linux professional institute" | tr '[:lower:]' '[:upper:]'
- LINUX PROFESSIONAL INSTITUTE
# Stripping DOS carriage returns (\r) from a text file:
$ tr -d '\r' < dos_file.txt > unix_file.txt
# Squeezing multiple consecutive spaces into a single space:
- $ echo "This has too many spaces" | tr -s ' '
- This has too many spaces
# Generating a frequency list of unique words in a document:
- $ tr -cs '[:alnum:]' '\n' < document.txt | tr '[:upper:]' '[:lower:]' | sort | uniq -c | sort -nr
- 5. split : Splitting Files into Chunks
The split utility partitions a large file into smaller, fixed-size segments. By default, output chunks are named with prefix x followed by alphabetical suffixes ( xaa , xab , xac ...).
Essential split Command-Line Flags
- Flag Description & Syntax Example
- l Splits by number of lines (default 1000 lines) split -l 500 access.log log_chunk_
- b Splits by byte size ( K , M , G multipliers) split -b 100M backup.tar.gz part_
-d Uses numeric suffixes ( 00 , 01 , 02 ) instead of letters split -b 50M -d backup.iso iso_chunk.
-a <N> Specifies suffix length (default 2 digits/letters) split -d -a 3 data.csv chunk_ # Splitting a 500MB tar archive into 100MB pieces with numeric extensions:
- $ split -b 100M -d database.tar.gz db_part_
- # Generates: db_part_00, db_part_01, db_part_02, db_part_03, db_part_04
# Reassembling split pieces into the original archive:
- $ cat db_part_* > database_restored.tar.gz
- 6. expand and unexpand : Tab and Space Normalization
expand : Converts TAB characters in files or stdin into spaces.
-t <tabstop> (or -<tabstop> ): Sets tab stop width (default is 8 spaces). Example: expand -t 4 code.py > code_spaces.py .
--initial : Converts only leading tabs at the beginning of lines.
unexpand : Converts spaces back into TAB characters.
-a ( --all ): Converts all sequences of spaces to tabs where possible (by default, unexpand only converts leading whitespace).
-t <tabstop> : Sets tab stops.
7. Paging Long Output with less
less is named in the 103.2 Terms and Utilities list, and it is the pager the exam assumes you use when a filter's output overflows one screen. Unlike more , less can scroll backwards and does not read the entire file before displaying the first page — which is why it opens a multi-gigabyte log instantly.
- Keystroke Action
- Space / f Forward one screen
- b Back one screen
- g / G Jump to first / last line
- /pattern Search forward for a pattern
- ?pattern Search backward
- n / N Next / previous match
- F Follow mode — behaves like tail -f until Ctrl+C
- q Quit # Page through a filter pipeline
- $ sort -k3 -n /etc/passwd | less
- # Keep ANSI colours readable and allow line-by-line control
- $ less -R /var/log/syslog
- # Search from the start without wrapping the terminal
- $ less -S wide_report.csv
Exam Rule: less is the pager man itself uses. The mnemonic LPI likes is that "less is more than more " — less is the superset with backward scrolling.
8. Integrity Checksums: md5sum , sha256sum and sha512sum
The 103.2 Terms and Utilities list includes three digest utilities: md5sum , sha256sum and sha512sum . They are filters in exactly the same sense as wc — they read a stream (or named files) and emit a fixed-size summary line — and the exam uses them to test whether a downloaded ISO or package payload is byte-identical to the publisher's original.
All three share one output format: <digest> <filename> , separated by two spaces .
Utility Digest length Status
md5sum 128-bit (32 hex characters) Cryptographically broken — collision-prone; use for accidental-corruption checks only
sha256sum 256-bit (64 hex characters) Current mainstream standard for distribution images
sha512sum 512-bit (128 hex characters) Strongest of the three; often faster than SHA-256 on 64-bit CPUs # Generate a digest for one file
- $ sha256sum debian-12.5.0-amd64-netinst.iso
- a4c8f3...c91d debian-12.5.0-amd64-netinst.iso
- # Read from standard input (note the '-' filename in the output)
- $ echo -n "linux" | md5sum
- 70f8c327d3f6bbfd... -
- # Record digests for a whole directory
- $ sha512sum /srv/release/*.tar.gz > SHA512SUMS
- # Verify against a published list: -c reads the digest file and re-computes each entry
- $ sha256sum -c SHA256SUMS
- debian-12.5.0-amd64-netinst.iso: OK
- broken-download.iso: FAILED
- sha256sum: WARNING: 1 computed checksum did NOT match
Verification flags shared by all three tools:
-c ( --check ): Read digests from a file and verify each listed target. This is the flag exam items ask for.
--quiet : With -c , print only the failures.
--ignore-missing : With -c , do not fail on files listed but absent.
-b / -t : Binary or text read mode (identical behaviour on Linux; the distinction exists for portability).
Exam Rule: A digest proves integrity , not authenticity . Anyone who can replace the ISO can replace the SHA256SUMS file next to it. Authenticity requires a GPG signature over the digest file — which is why distributions ship both SHA256SUMS and SHA256SUMS.gpg .
9. Reading Compressed Text Without Decompressing: zcat , bzcat , xzcat
Log files rotate into compressed archives, but you still need to filter them. The 103.2 objective lists three decompressing readers that write plain text to standard output, leaving the compressed file on disk untouched:
- Command Reads Equivalent to
- zcat .gz (and legacy .Z ) gunzip -c
- bzcat .bz2 bunzip2 -c
- xzcat .xz unxz -c # Count error lines inside a rotated, gzip-compressed log
- $ zcat /var/log/syslog.2.gz | grep -c ' ERROR '
- # Chain a bzip2 archive straight into a field extractor
- $ bzcat /var/log/audit/audit.log.1.bz2 | cut -d' ' -f1,3 | sort | uniq -c
- # Feed an xz-compressed dump into a pager
- $ xzcat backup.sql.xz | less
Exam Rule: The z* family extends to other filters as well — zgrep , zless , zdiff , and their bz / xz counterparts. None of these tools modify or delete the compressed source file; they only stream its decompressed contents.