Sort and Remove Duplicate Lines on Mac

Why this is harder than it looks

Sorting a list and deleting repeated lines are two of the most common text chores. On a Mac they are also two of the ones the built-in tools handle poorly. TextEdit does not offer a command for either, so people end up pasting into a spreadsheet or dropping to Terminal.

Both jobs have a hidden decision in them: what should happen to the original order, and what counts as a duplicate. The methods below make different choices, so it is worth knowing which one you are using.

Option 1: Terminal

Two commands cover most cases. Replace list.txt with your file name.

sort -u list.txt > cleaned.txt

sort -u sorts the lines and keeps one copy of each. Because it sorts, your original order is gone.

awk '!seen[$0]++' list.txt > cleaned.txt

The awk command keeps the first occurrence of every line and leaves the order as it was. On macOS it also collapses repeated blank lines into a single blank line, since blank lines are identical to each other.

Always write to a new file, as above. Redirecting output back onto the file you are reading empties it.

Option 2: Built-in scripts in notePad++++

notePad++++ ships with ready-made scripts for these jobs, and shows a preview before it changes anything.

1. Open the file, or paste your list into a new tab.
2. Open the script window.
3. Choose Deduplicate Lines (Keep Order), Sort Lines A-Z, or Sort Lines Z-A.
4. Run it on the current file. To limit it to part of the text, select one block of text, right-click, and choose Run Current Script on Selection. The script needs a single, non-empty selection.
5. Review the diff preview, then apply the result, or cancel to leave the file alone.

What each script actually does

• Deduplicate Lines (Keep Order) compares lines exactly, including capitals and spaces. It keeps the first occurrence of each line and leaves the order unchanged. Blank lines count as duplicates of each other, so one is kept.
• Sort Lines A-Z / Z-A ignores capital letters and accents when comparing, and understands numbers inside text, so file2 comes before file10. Blank lines sort to the top.

These behaviors are worth knowing. A plain byte-order sort puts file10 before file2 and every capital letter before every lowercase one, which is rarely what you want for a list a person will read.

Clean first, then deduplicate

The most common reason deduplication seems to fail is invisible: two lines look identical but one ends with a trailing space. Because comparison is exact, they are different lines.

A reliable order for messy input is:

1. Trim Trailing Whitespace
2. Remove Empty Lines, if you do not need blank lines
3. To Lowercase, if capitals should not matter
4. Deduplicate Lines (Keep Order)
5. Sort Lines A-Z, if you want a sorted result

If you run this sequence often, chain the steps into a Script Pipeline and run it in one go.

Which method to use