Convert File Encoding to UTF-8 on Mac

Why the text is garbled

Text files store characters as bytes, and different encodings map the same bytes to different characters. Modern files are almost always UTF-8. Older files from Chinese, Japanese, and Korean systems often are not: they use GBK or GB18030, Big5, Shift-JIS, or a Korean code page. Open one of those as UTF-8 and you get random symbols or question marks.

The fix has two steps: read the file using its real encoding, then, if you want to, save a copy as UTF-8.

Which encoding is it

If you know where the file came from, this table usually settles it:

If you are not sure, try the likely candidates and see which one produces readable text.

Option 1: iconv in Terminal

macOS includes iconv. Give it the source encoding and the target, and write to a new file:

iconv -f GBK -t UTF-8 input.txt > output.txt
iconv -f SHIFT_JIS -t UTF-8 input.txt > output.txt
iconv -f EUC-KR -t UTF-8 input.txt > output.txt

Each of these was tested on macOS with a short sample and produced the expected characters. iconv -l lists every encoding name your Mac accepts.

To convert a whole folder of files into a new utf8 folder:

mkdir utf8
for f in *.txt; do iconv -f GBK -t UTF-8 "$f" > "utf8/$f"; done

Always write to a new file. Redirecting onto the input file would empty it.

Option 2: The encoding menu in notePad++++

1. Open the file. notePad++++ tries to detect the encoding when it opens, and can recognize GB18030, GBK, Big5, Shift-JIS, and Korean text among others.
2. Look at the encoding shown in the status bar. If the text is garbled, click it to open the encoding menu.
3. Choose Re-read using specified encoding and pick the correct one, for example GBK or Shift-JIS. The text is reloaded using that encoding. You must save or discard unsaved changes first, because the current text is replaced.
4. When the text reads correctly, open the encoding menu again, choose Convert to specified encoding and save, and pick UTF-8. Confirm the prompt.

The menu lists UTF-8, UTF-16 LE and BE, Latin-1, Win-1252, GBK, GB18030, Big5, Shift-JIS, and Win-949.

Two things worth knowing

• Detection is only a guess. Even command-line detection can be wrong. In a quick test, file -I reported a short GBK sample as iso-8859-1. Always confirm by reading the result.
• Fix the display before you save. Re-reading is reversible because it does not change the file. Saving in a different encoding does. If you save while the text is still garbled, you make the garbage permanent.

Which method to use