CSV Encoding and UTF-8: Why Your Characters Turn to Gibberish

When a CSV shows José as José or naïve as naïve, the file is fine. The encoding used to read it is wrong. CSV files store text as bytes, and if the reader guesses the wrong character encoding, accented letters, emoji, and symbols turn into gibberish. The fix is almost always to save and open the file as UTF-8, the encoding every modern tool agrees on.

That garbled text has a name: mojibake. Here is why it happens and how to get your characters back without losing data.

What character encoding means (bytes vs characters)

A CSV file does not actually store letters. It stores numbers. Every character you see, from a plain A to a curly quote to a heart emoji, is written to disk as one or more bytes. A character encoding is the lookup table that maps those bytes back to the characters you meant.

The oldest table is ASCII, which covers 128 characters: the English alphabet, digits, and basic punctuation. ASCII has no slot for é, ñ, €, or 京. To handle those, older systems used single-byte tables like Windows-1252 (common in Excel on Western machines) or Latin-1. Each of these reuses the same byte values for different characters, so the byte 0xE9 might mean é in one table and something else entirely in another.

Here is the core problem. A byte does not carry a label saying which table it came from. If a file was written with one encoding and read with another, the reader pulls the wrong character out of its own table. That mismatch is what produces mojibake. The bytes never changed. The interpretation did.

Why UTF-8 became the safe default for CSV

UTF-8 solved the “which table” problem by having one table for every character in the world. It can encode all of Unicode, which covers roughly 150,000 characters across every writing system, plus emoji and symbols. There is no ambiguity about what a byte sequence means, because there is only one standard to check against.

UTF-8 is also backward compatible with ASCII. The first 128 characters map to the exact same single byte they always did, so a plain-English CSV with no accents is valid UTF-8 automatically. Characters outside ASCII use two to four bytes. That é becomes the two-byte sequence 0xC3 0xA9, which is why a Windows-1252 reader shows it as two separate characters, é.

For CSV specifically, UTF-8 is the right default because CSV has no built-in place to declare its encoding. There is no header line, no metadata field, nothing inside the file that says “read me as UTF-8.” So the whole ecosystem standardized on one encoding by convention. If you save as UTF-8 and the other person reads as UTF-8, the text survives. If you understand what a CSV actually is at the byte level, encoding stops being mysterious. Our guide on what a CSV file really contains covers the format itself.

The BOM problem: why Excel needs ‘CSV UTF-8’

Open a clean UTF-8 CSV in Excel and you may still see é instead of é. This is Excel’s most infamous encoding trap. On many versions, especially on Windows, Excel does not assume UTF-8 when you double-click a .csv file. It falls back to the system’s legacy code page, usually Windows-1252, and mangles every accented character.

The workaround is the BOM, or byte order mark. A BOM is a short, invisible sequence of bytes (0xEF 0xBB 0xBF for UTF-8) placed at the very start of the file. It acts as a signal that says “this file is UTF-8.” When Excel sees a UTF-8 BOM, it reads the file correctly. When the BOM is missing, Excel guesses, and it usually guesses wrong.

This is exactly what Excel’s “CSV UTF-8 (Comma delimited)” save option does. It writes your data as UTF-8 and prepends the BOM so Excel itself can read it back. If you are producing CSVs that non-technical colleagues will open in Excel, save with the BOM. If you are feeding the file to a database importer, a Python script, or a web app, skip the BOM, because some strict parsers treat those three leading bytes as part of your first field name and you end up with a column called id instead of id.

How accents, emoji, and special characters get corrupted

The corruption pattern tells you what went wrong. Once you learn to read it, you can diagnose a broken CSV in seconds.

One character becomes two or three. When é, ü, or ñ each show up as a pair like é, ü, or ñ, a UTF-8 file was read as Windows-1252 or Latin-1. The multi-byte UTF-8 sequence got split, and each byte was rendered as its own single-byte character.

Characters become a black diamond with a question mark, or a plain ?. This usually means the reverse: a character that has no slot in the target encoding was replaced during a lossy conversion. Once a character is replaced with ?, the original is gone. This is the dangerous case, because you cannot recover it by re-reading.

Emoji and CJK text vanish or scramble. Emoji sit high in the Unicode range and use four bytes in UTF-8. Any single-byte encoding will shred them completely. The same goes for Chinese, Japanese, Korean, Arabic, and other non-Latin scripts. If your special characters in the CSV are emoji or non-Latin text, UTF-8 is not optional, it is the only encoding that works.

Curly quotes, em spaces, non-breaking spaces, and currency symbols like € and £ are the quiet troublemakers. They look like plain text but live outside ASCII, so they break the moment an encoding mismatch happens even when the rest of the file looks fine.

Detecting a file’s real encoding before you open it

Since a CSV carries no encoding label, detection is about inspecting the bytes, not trusting the file extension. A few reliable methods:

On macOS or Linux, run file -I yourfile.csv in a terminal. It reports a best guess like charset=utf-8 or charset=iso-8859-1. It is not perfect, but it is a fast first read.

In Python, the charset-normalizer or chardet library will analyze the byte distribution and return a probable encoding with a confidence score. This is the practical choice when you process many files and cannot open each one by hand.

In a text editor like VS Code or Notepad++, the current encoding is shown in the status bar, and both let you reopen the same file with a different encoding without changing it on disk. That reopen-as feature is the fastest way to test a theory. If reopening as UTF-8 fixes the accents, you have your answer.

Watch for the BOM as a clue too. If the first three bytes are EF BB BF, the file is UTF-8 with a BOM and almost certainly meant to be read that way. Encoding detection is fundamentally a guess, so treat any tool’s answer as a strong hint, not gospel, and confirm by eye on a row you know has accented text.

Fixing an already-broken CSV without data loss

The safest fix is to re-read the original file with the correct encoding rather than trying to find-and-replace the mojibake. Find-and-replace can work for one or two known characters, but it is fragile and easy to get wrong across a large file.

The reliable path is a round trip through UTF-8. In Python it is three lines: open the file with the encoding it was actually written in, then write it back out as UTF-8.

with open("broken.csv", encoding="windows-1252") as f:
    data = f.read()
with open("fixed.csv", "w", encoding="utf-8") as f:
    f.write(data)

The key is naming the source encoding correctly (windows-1252, latin-1, whatever detection suggested). Get it right and every é collapses back into é.

One hard rule: keep the original file untouched until you have confirmed the fix. If a lossy conversion already replaced characters with ? before the file reached you, no amount of re-reading brings them back, and you will need to regenerate the export from the source system. Always fix into a new filename so a wrong guess costs you nothing.

If the file also looks broken because columns are split in the wrong places, that is a separate issue from encoding. That is usually a delimiter mismatch, which our guide on how CSV delimiters work walks through.

Keeping encoding intact when you convert a CSV

The moment you convert a CSV into another format, you get a second chance to break the encoding. Every conversion step reads bytes in and writes bytes out, and each side has its own encoding assumption.

The rule is to declare UTF-8 explicitly on both ends. When you export from Excel or Google Sheets, choose the UTF-8 option rather than the plain “CSV” default, which may fall back to a legacy code page. When you import into a database, set the connection or import charset to UTF-8 instead of letting it default. When you transform a CSV into another readable format, use a tool that preserves UTF-8 rather than one that silently re-encodes.

That is where a clean converter helps. Our CSV to Markdown tool keeps text as UTF-8 end to end, so accents and symbols survive the conversion into a Markdown table you can paste anywhere. If you are weighing when a spreadsheet is the better home for your data, our CSV versus Excel breakdown covers the tradeoffs.

FAQ

Why does my CSV show weird characters like é instead of é? The file is UTF-8 but your program is reading it as Windows-1252 or Latin-1. The single é is stored as two bytes in UTF-8, and the wrong reader shows those two bytes as two separate characters. Reopen the file as UTF-8 and the accents come back.

What is the difference between CSV and CSV UTF-8 in Excel? “CSV UTF-8” saves your data as UTF-8 and adds a byte order mark (BOM) so Excel reads it back correctly. Plain “CSV” often uses your system’s legacy code page, which corrupts accented and non-English characters. Use “CSV UTF-8” whenever your data contains anything beyond basic English.

Do I need a BOM in my UTF-8 CSV file? It depends on who opens it. Add the BOM if the file is meant for Excel, which needs that signal to detect UTF-8. Skip the BOM for databases, scripts, and web apps, since strict parsers may treat the three BOM bytes as part of your first field.

Can I recover a CSV where characters already turned into question marks? Usually no. A literal ? means the character was replaced during a lossy conversion, and the original value is gone from the file. Your only real fix is to re-export the data from the source system with UTF-8 set from the start.