CSV is not split on commas: quoting and newlines from RFC 4180

The moment a field is quoted, split stops working: quotes hold delimiters and newlines, and a quote inside a field is doubled. A thirty-line state machine beats any regex.

CSV is just commas, right. That belief dies on the first quoted field. a,"b,c" splits into three fields and the correct answer is two.

The rules are short

RFC 4180 leaves four things to handle:

  1. A field may be wrapped in quotes, and inside, delimiters, newlines and quotes may appear
  2. A quote inside a field is written twice, so "" means "
  3. The line separator is CRLF, though LF is common in the wild and both must be accepted
  4. A trailing newline must not add an extra empty record

The state machine

One boolean for whether we are inside quotes, plus a buffer. The subtle part is that a quote inside quotes depends on the next character:

function parseCsv(input, delimiter) {
  const rows = [];
  let row = [];
  let field = '';
  let quoted = false;
  for (let i = 0; i < input.length; i += 1) {
    const ch = input[i];
    if (quoted) {
      if (ch === '\u0022') {
        if (input[i + 1] === '\u0022') { field += ch; i += 1; }
        else quoted = false;
      } else field += ch;
    } else if (ch === '\u0022') {
      quoted = true;
    } else if (ch === delimiter) {
      row.push(field);
      field = '';
    } else if (ch === '\n' || ch === '\r') {
      if (ch === '\r' && input[i + 1] === '\n') i += 1;
      row.push(field);
      rows.push(row);
      row = [];
      field = '';
    } else field += ch;
  }
  if (field !== '' || row.length) { row.push(field); rows.push(row); }
  return rows;
}

The final if implements rule four. When the file ends with a newline, both field and row are empty, so nothing is appended. Without that line every ordinary file gains a phantom empty row, and the next program in the chain writes an empty record into a database.

Four traps

  • The BOM. Files exported from Excel often start with \uFEFF, which glues itself onto the first column name so that id cannot be found. Strip it before parsing.
  • CRLF and LF mixed. Seeing both in one file is not rare. Treating both as a line ending per character handles it.
  • Short rows not padded. With a,b followed by a row containing only 1, the second row has one column. Padding with empty strings beats rendering a misaligned table.
  • Over-quoting on output. Quote only when needed: a delimiter, a quote or a newline inside. Otherwise a,b becomes "a","b", legal and unpleasant to read.

The choice on this site

In the CSV tool parsing runs the state machine above, writing uses minimal quoting, and delimiter sniffing takes the first line and counts candidate delimiters: comma, tab, semicolon, pipe. Sniffing inspects only the first line and strips quoted runs first, or a field like "a;b,c" would skew the count.

Do not parse CSV with a regex, and do not split on commas. A boolean and a buffer are easier to debug than anything clever.

← Back to all posts

Comments

…