DevToolsmith

Remove Duplicate Lines

Paste a list and get every line once, in the original order. Ignore case or surrounding spaces, keep the first or the last occurrence, and see how many duplicates were removed. Works on large lists and nothing is uploaded.

Runs 100% in your browser — nothing is uploaded

Lines
Tip: drop a file here52 chars · 52 bytes · 9 lines
Unique lines
31 chars · 31 bytes · 6 lines

Removed 3 duplicate lines.

When it’s useful

  • Merging email lists, keyword lists or tag lists from several sources.
  • Cleaning up log excerpts, stack traces or error messages that repeat.
  • Getting the distinct values of a column copied from a spreadsheet or a SQL result.
  • Removing repeated entries from .gitignore, requirements.txt or a hosts file.

Removing duplicate lines in code

Shell
# Keep the original order (first occurrence wins)
awk '!seen[$0]++' list.txt

# Ignoring case
awk '!seen[tolower($0)]++' list.txt

# Sorted output; uniq only removes *adjacent* duplicates, so sort first
sort -u list.txt
Python
lines = open("list.txt").read().splitlines()

# dict keeps insertion order (Python 3.7+)
unique = list(dict.fromkeys(lines))

# Ignoring case and surrounding spaces, keeping the first spelling
seen, unique = set(), []
for line in lines:
    key = line.strip().lower()
    if key not in seen:
        seen.add(key)
        unique.append(line)
JavaScript
const unique = [...new Set(text.split('\n'))].join('\n');
SQL
SELECT DISTINCT email FROM subscribers;

-- Find which values are duplicated, and how often
SELECT email, COUNT(*) FROM subscribers GROUP BY email HAVING COUNT(*) > 1;

Need the list sorted as well? Use sort lines with “Remove duplicates”. To see what changed between two versions of a list, use the diff checker.

Frequently asked questions

Does removing duplicates change the order of the lines?

No. Lines stay in their original order; only repeats are dropped. That’s the main difference from sort -u on the command line, which sorts the lines as a side effect. Choose “Last occurrence” to keep the last copy of each line in its position instead of the first.

Why are some lines that look the same not removed?

They differ in something you can’t see: a trailing space, a tab, a no-break space or a zero-width character copied from a web page. Trailing and leading spaces are ignored by default. For the invisible characters, clean the list with the remove whitespace tool first; it lists exactly which ones are in the text.

Are empty lines treated as duplicates?

No, blank lines are kept so paragraphs and groups in your text stay separated. Tick “Remove empty lines” to drop them all.

How many lines can it handle?

Hundreds of thousands. Each line is checked against a hash set, so the work grows linearly with the number of lines, and nothing is uploaded, so the only limit is your browser’s memory.