What this tool does
Lists collect repeats. A mailing list picks up the same address twice, a keyword export repeats a phrase, and a log file prints the same message again and again. This tool takes a list with one item per line and deletes the repeated lines, so each line appears only once. It works in your browser, so the list stays on your device.
If you want to know how long the cleaned list is, the word counter and the character counter can measure it.
How duplicates are found
Two lines are duplicates when they are the same after the options you chose are applied. With no options ticked, the lines must match exactly: same letters, same capitals, same spaces. "Apple" and "apple" are different, and so are "apple" and "apple " with a space at the end.
Ignore case treats capital and small letters as the same. The line that stays is shown as it was written, so you do not lose the original spelling.
Trim whitespace on each line removes spaces and tabs at the start and end of every line before comparing. The lines in the result are trimmed too. This fixes the most common problem with pasted lists, a space at the end of a line that you cannot see.
Ignore empty lines removes blank lines, including lines that only contain spaces or tabs. Without it, empty lines are treated like any other line, so a list with many blank lines ends up with just one.
Which copy is kept
By default the first occurrence stays and later copies are removed. Lines keep their original order, so the list looks like your list, only shorter.
If you prefer the last occurrence, choose Last occurrence in the Keep menu. The kept line then sits where the last copy was. This is useful when later lines are newer and you want to keep the latest entry.
Sorting the result
Sorting is optional and happens after the duplicates are removed. Choose A to Z or Z to A to put the result in alphabetical order, or leave it on Keep original order. The sort follows English alphabet rules, so accented letters sit next to their plain versions, and a small and a capital letter of the same letter stay together. Numbers in text are compared as text, so "10" comes before "2". If you need numeric order, pad the numbers with zeros first.
Show only duplicated lines
This option turns the tool around. Instead of deleting repeats, it lists every line that appears more than once, one time each. It answers the question "what is repeated in my list?", which is handy for finding double entries in a customer list or a repeated keyword in a spreadsheet column. Combine it with Ignore case or Trim whitespace to catch repeats that differ only in capitals or spacing.
Reading the statistics
Three numbers sit above the result.
- Lines in is the number of lines you pasted.
- Lines out is the number of lines in the result.
- Duplicates removed is the number of extra copies found. If a line appears four times, three copies count as removed.
When you tick Ignore empty lines, a fourth number shows how many empty lines were removed. In the normal mode, lines in minus duplicates removed minus empty lines removed equals lines out.
Line endings
Text from Windows uses a two-character line break, while text from Mac and Linux uses a single one. The tool understands both, even when they are mixed in one text. The result uses the style of the first line break it finds, and a line break at the end of your text is kept.
Size limit
You can paste up to 1,000,000 characters and up to 100,000 lines. If your text is larger, the tool keeps the first part, shows a message, and carries on, so the page never freezes. For a longer list, clean it in parts. Remember that duplicates in different parts will not be found, so run the combined result through the tool once more.
Tips
- Paste one item per line. If your items are separated by commas, put each on its own line first.
- Tick Trim whitespace and Ignore case when cleaning email addresses or URLs, then check the result by eye before you use it.
- Use Use result as input to apply a second pass, for example to sort a list you have just cleaned.
- Keep a copy of the original. The tool does not store anything, so refreshing the page clears your list.
Privacy
The list is processed in your browser. It is not uploaded, saved or tracked, and there is no sign-up. Like every page on this site, this one may show ads from third-party vendors once ads are switched on, and those vendors can use cookies. Read the privacy policy for details.