Remove Duplicate Lines
Remove repeated lines, or show only the ones that repeated.
Your list is processed in this page and never uploaded.
Merging two mailing lists, cleaning a column pasted out of a spreadsheet, tidying a log before reading it: the job is always the same, and doing it in a spreadsheet means a helper column and a formula nobody remembers.
More about Remove Duplicate Lines
The comparison is the part worth getting right. Every line is normalised to NFC before it is compared, which matters more for Vietnamese than for most languages. The same word can be typed with a precomposed `ế` or with an `e` followed by two combining marks; the two look identical on screen, compare as different strings, and both survive a deduplication that skips the normalisation step.
Whitespace and case are optional in the comparison and independent of each other, so a list where some lines were indented and others were not can be deduplicated without first cleaning it. Whether to keep the first or the last occurrence is also a choice: for a log where later lines are more recent, keeping the last one is usually what you want.
The inverted mode answers the opposite question. Instead of removing the duplicates it shows only them, which is how you find out what was repeated rather than just getting a clean list.
- Normalises to NFC before comparing, so identical-looking lines really match
- Optional case-insensitive matching
- Optional trimming, so indentation does not hide a duplicate
- Keep the first or the last occurrence
- Inverted mode showing only the lines that were repeated
- Reports how many lines were removed and how many remain
How to use it
- 1Paste your lines into the input.
- 2Choose whether to match case and whether to ignore surrounding spaces.
- 3Choose to keep the first or the last of each repeat.
- 4Copy the result, or switch to inverted mode to see what was duplicated.
Questions
Does it handle Vietnamese text correctly?
Yes. Lines are normalised to NFC before comparison, so a word typed with a precomposed letter matches the same word typed with combining marks.
Does it keep the original order?
Yes: the lines that survive stay in the order they arrived and nothing is sorted. Use the sort tool if you want that too.
What is the difference between keeping first and last?
Which copy of a repeated line survives. For logs where later entries are more recent, keeping the last is usually what you want.
What does inverted mode do?
It shows only the lines that appeared more than once, each shown a single time. It answers what was repeated rather than giving a clean list.
Are blank lines treated as duplicates?
Yes, repeated blank lines collapse to one like any other repeat. Use the text cleaner if you want them removed entirely.