Remove Duplicate Lines
Remove duplicate lines from a list, keeping each unique line once — with optional alphabetical sorting, whitespace trimming and empty-line removal.
Loading calculator…
What this tool does
Remove duplicate lines from a list, keeping each unique line once — with optional alphabetical sorting, whitespace trimming and empty-line removal. Comparison is exact and case-sensitive — 'Apple' and 'apple' are different lines — after the optional trim step. A trailing newline in the input counts as a final empty line, which the default settings remove.
How to use the Remove Duplicate Lines
- Enter or select lines (one per line).
- Enter or select sort result.
- Enter or select trim each line.
- Enter or select empty lines.
- Read the calculated result; change any measurement to compare alternatives.
Formula
Lines are split on newlines, optionally trimmed, deduplicated with a Set keyed on the exact line text (first occurrence wins), optionally emptied and sorted. Lines removed = input lines − output lines.
- text
- Lines (one per line)
- sort
- Sort result
- trim
- Trim each line
- empty
- Empty lines
Comparison is exact and case-sensitive — 'Apple' and 'apple' are different lines — after the optional trim step. A trailing newline in the input counts as a final empty line, which the default settings remove.
Worked example
For remove duplicate lines, the following measurements illustrate the exact method: Lines (one per line): apple banana apple orange banana; Sort result: none; Trim each line: yes; Empty lines: remove.
Inputs
- Lines (one per line)apple banana apple orange banana
- Sort resultKeep first-seen order
- Trim each lineTrim leading/trailing spaces
- Empty linesRemove empty lines
Result
- Unique linesapple banana orange
- Lines before5
- Lines after3
- Duplicates removed2
Results explained
- Unique lines
- Unique lines from the formula above. Comparison is exact and case-sensitive — 'Apple' and 'apple' are different lines — after the optional trim step. A trailing newline in the input counts as a final empty line, which the default settings remove.
- Lines before
- Lines before from the formula above. Comparison is exact and case-sensitive — 'Apple' and 'apple' are different lines — after the optional trim step. A trailing newline in the input counts as a final empty line, which the default settings remove.
- Lines after
- Lines after from the formula above. Comparison is exact and case-sensitive — 'Apple' and 'apple' are different lines — after the optional trim step. A trailing newline in the input counts as a final empty line, which the default settings remove.
- Duplicates removed
- Duplicates removed from the formula above. Comparison is exact and case-sensitive — 'Apple' and 'apple' are different lines — after the optional trim step. A trailing newline in the input counts as a final empty line, which the default settings remove.
Frequently asked questions
Two lines are duplicates when their text matches exactly (after trimming, if trimming is on). The first occurrence is kept in first-seen order mode; sorting, when selected, is applied after deduplication.
Deduplication is case-sensitive, like most list and programming tools, because capitalisation is often meaningful (product codes, names, tags). Normalise case in a case converter first if you need case-insensitive deduplication.
By default they are removed before deduplication. Choose 'Keep one empty line' if blank separators matter — multiple consecutive blanks still collapse to a single empty line because they are duplicates of each other.
Yes — choose Sort A → Z or Z → A. Sorting uses locale-aware alphabetical order, so numbers sort as text ('10' before '2'); pad numbers with leading zeros if you need numeric order.
No. The lines are processed in your browser with a Set; your list is never uploaded or stored — safe for emails, SKUs and private lists.