Opened 3 years ago

#5254 new defect

Correct handling of csv files when reading and writing results

Reported by: jsten Owned by: Christian Andersson
Priority: critical Milestone:
Component: PyFMI Version: trunk
Keywords: Cc:

Description

We don't handle csv files correctly in result writing and reading. For example, we might not escape the data correctly if the data contains the csv delimiter. The problem lies in ResultCSVTextual and ResultHandlerCSV.
Although there is no standard for csv files, there is an RFC which is an effort to unify the different implementations, I think it would be good if we adopted that rfc as well. Also wikipedia have a pretty good summary on how to handle csv files:

RFC 4180 formalized CSV. It defines the MIME type "text/csv", and CSV files that follow its rules should be very widely portable. Among its requirements:

  • MS-DOS-style lines that end with (CR/LF) characters (optional for the last line).
  • An optional header record (there is no sure way to detect whether it is present, so care is required when importing).
  • Each record "should" contain the same number of comma-separated fields.
  • Any field may be quoted (with double quotes).
  • Fields containing a line-break, double-quote, and/or commas should be quoted. (If they are not, the file will likely be impossible to process correctly).
  • A (double) quote character in a field must be represented by two (double) quote characters.

In the end what we should do is to use a proper library for reading and writing csv files. Then someone has to deal with all the mess and we can simply point to the library if someone asks for limitations and information on how we interpret csv files. Even better is that Python has one out of the box! https://docs.python.org/2/library/csv.html

Change History (0)

Note: See TracTickets for help on using tickets.