Remove Duplicate Lines
Remove duplicate lines, or comma, tab or space separated items, keeping the first or last copy, with case, trimming, blank-line and sort options. How it works
Use the Remove Duplicate Lines
6 lines in, 3 kept, 3 duplicates removed
Line breaks (Windows CRLF, old Mac CR, Unix LF) are all read as line breaks and the result uses LF. One line break at the very end is not a blank line. With Split on set, each piece between the separators counts as a line, and the result lists the pieces one per line. Limits: 200,000 lines and 5,000,000 characters.
About the Remove Duplicate Lines
The list apple, banana, Apple, cherry, banana, " cherry" comes back as apple, banana, cherry: 6 lines in, 3 kept, 3 duplicates removed. To remove duplicate lines the tool ignores case and trims spaces at both ends of each line by default, and keeps the first copy of each line in its original place; each of these can be changed.
The sample list under each setting
| Setting | Kept lines | Summary |
|---|---|---|
| Defaults: case ignored, trimmed, keep first | apple, banana, cherry | 6 lines in, 3 kept, 3 duplicates removed |
| Case sensitive | apple, banana, Apple, cherry | 6 lines in, 4 kept, 2 duplicates removed |
| Trim whitespace off | apple, banana, cherry, " cherry" | 6 lines in, 4 kept, 2 duplicates removed |
| Keep last | Apple, banana, cherry | 6 lines in, 3 kept, 3 duplicates removed |
| Sort Z-A | cherry, banana, apple | 6 lines in, 3 kept, 3 duplicates removed |
Sample: apple, banana, Apple, cherry, banana and " cherry" (two leading spaces), one per line. Quotes mark a line with spaces at either end.
Blank lines
| Remove blank lines | Kept lines | Summary |
|---|---|---|
| Off | a, "", b | 5 lines in, 3 kept, 2 duplicates removed |
| On | a, b | 5 lines in, 2 kept, 1 duplicate removed, 2 blank lines removed |
List: a, a blank line, b, a blank line, a. "" is a blank line.
Frequently asked questions
- Is the duplicate check case-sensitive?
- Not by default: Apple and apple count as the same line, and the first one written is kept. Tick Case sensitive to keep both; the sample list then keeps 4 lines instead of 3.
- Does removing duplicates change the order of my list?
- No. Each line stays where its first copy was. Keep last keeps the last copy instead, so the sample gives Apple, banana, cherry. The sort options order the result by character code, which puts capitals first: B, Z, a, b.
- Can it remove duplicates from a comma-separated list?
- Yes. Set Split on to comma, tab, space, semicolon or pipe and each piece between the separators counts as a line; with Trim whitespace on, the space after each comma goes too. apple, banana, apple, cherry split on commas gives apple, banana, cherry: 4 lines in, 3 kept, 1 duplicate removed. The result lists one item per line, and a run of spaces counts as one separator.
- Why are two lines that look the same both kept?
- They differ by a character you cannot see. Trim whitespace removes spaces, tabs, no-break spaces and zero-width characters at the ends of a line, so apple with a zero-width space in front of it counts as apple. Inside a line nothing is removed: apple with a zero-width space between the two p letters, a no-break space or a double space stays a different line. Paste the line into Text to ASCII, which marks a zero-width character as (invisible) and a no-break space as (other space), or into Unicode Lookup to name it.
- Does it handle Windows line endings?
- Yes. CRLF and a lone CR count as line breaks, and the result uses LF. a, b, a with Windows line endings keeps a and b even with Trim whitespace off: 3 lines in, 2 kept, 1 duplicate removed.
- How long a list can it take?
- Up to 200,000 lines or 5,000,000 characters; a longer list gets a message asking you to split it. A list over 20,000 lines or 400,000 characters updates a quarter of a second after you stop typing, so typing stays smooth.
Related Tools
View all toolsWord Counter
Paste or type text to count words, characters with and without spaces, sentences, paragraphs and lines, plus reading time at 200 wpm and speaking time at 130.
Case Converter
Convert text to UPPER CASE, lower case, Title Case, Sentence case, camelCase, PascalCase, snake_case, kebab-case or CONSTANT_CASE in one click.
Reverse String
Reverse text by character, word order, letters in each word or line order. Emoji, flags and accented letters stay whole instead of breaking apart.
Text Replacement
Find and replace text in a pasted block, as plain text or regex, with a case option. Replace All gives the result and the number of replacements made.
Lorem Ipsum Generator
Generate lorem ipsum placeholder text as paragraphs, sentences or words, up to 100 paragraphs or 5,000 words, with optional HTML paragraph tags.
Random Text Generator
Generate 1 to 100 sentences or paragraphs picked from a fixed list of 20 well-known sayings. Useful as filler text and writing prompts.