uniq

uniq [OPTION]... [INPUT [OUTPUT]]

Report or omit repeated lines.

Options

--all-repeated=<delimit-method>, -D <delimit-method>

print all duplicate lines. Delimiting is done with blank lines. [default: none]

--group=<group-method>

show all items, separating groups with an empty line. [default: separate]

--check-chars=<N>, -w <N>

compare no more than N characters in lines

--count, -c

prefix lines by the number of occurrences

--ignore-case, -i

ignore differences in case when comparing

--repeated, -d

only print duplicate lines

--skip-chars=<N>, -s <N>

avoid comparing the first N characters

--skip-fields=<N>, -f <N>

avoid comparing the first N fields

--unique, -u

only print unique lines

--zero-terminated, -z

end lines with 0 byte, not newline

Filter adjacent matching lines from INPUT (or standard input), writing to OUTPUT (or standard output).

Note: uniq does not detect repeated lines unless they are adjacent. You may want to sort the input first, or use sort -u without uniq.

Examples

Display each line once:

sort {{path/to/file}} | uniq

Display only unique lines:

sort {{path/to/file}} | uniq {{[-u|--unique]}}

Display only duplicate lines:

sort {{path/to/file}} | uniq {{[-d|--repeated]}}

Display number of occurrences of each line along with that line:

sort {{path/to/file}} | uniq {{[-c|--count]}}

Display number of occurrences of each line, sorted by the most frequent:

sort {{path/to/file}} | uniq {{[-c|--count]}} | sort {{[-nr|--numeric-sort --reverse]}}

Compare only the first 10 characters on each line for uniqueness:

sort {{path/to/file}} | uniq {{[-w|--check-chars]}} 10

Compare text after the first 5 characters on each line for uniqueness:

sort {{path/to/file}} | uniq {{[-s|--skip-chars]}} 5

The examples are provided by the tldr-pages project under the CC BY 4.0 License.

Please note that, as uutils is a work in progress, some examples might fail.