You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "zhenyue-xu (via GitHub)" <gi...@apache.org> on 2025/08/04 09:36:35 UTC

[PR] [Improve][Csv] support configurable CSV delimiter in file connector [seatunnel]

zhenyue-xu opened a new pull request, #9660:
URL: https://github.com/apache/seatunnel/pull/9660

   ### Purpose of this pull request
   
   This pull request restores configurable CSV delimiter support in the file connector. Previously, in PR #9066, the CSV delimiter configuration was removed with the suggestion to "use text format if you want custom delimiter." However, this approach has significant limitations:
   
   1. **Text format cannot handle standard CSV features** as defined in [RFC 4180](https://datatracker.ietf.org/doc/html/rfc4180#page-2):
      - Fields enclosed in double quotes
      - Escaped quotes (`""` for literal `"`)
      - Multi-line fields within quotes
      - Delimiters within quoted fields
   
   2. **Text format only supports simple delimited data** without any CSV-specific parsing logic
   
   This PR adds back the `csv_field_delimiter` configuration option to properly support CSV files with custom delimiters (semicolon, tab, pipe, etc.) while maintaining full CSV standard compliance.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes, this PR introduces a new configuration option for CSV file reading:
   
   **New configuration:**
   - `csv_field_delimiter`: Configurable CSV field delimiter (default: `,`)
   
   
   This change is backward compatible - existing configurations without the `csv_field_delimiter` option will continue to use comma as the default delimiter.
   
   ### How was this patch tested?
   
   The patch was tested with:
   
   1. **Unit tests** for the new configuration option parsing
   2. **Integration tests** with various CSV formats:
      - Default comma delimiter (backward compatibility)
      - Semicolon delimiter with quoted fields
      - Tab delimiter with escaped quotes
      - Pipe delimiter with multi-line fields
   
   3. **Manual testing** with real CSV files:
      ```
      "1";"b
      a";"10"
      "2";"b";"100"
      ```
   
   问题是 Markdown 的 CSV 语法高亮在处理多行字段时会出现渲染问题。使用普通的代码块(不指定语言)可以避免这个问题。
   
   All tests confirmed that the CSV format properly handles standard CSV features while respecting the custom delimiter, unlike the text format which would incorrectly parse the above example.
   
   ### Check list
   
   * [ ] If any new Jar binary package adding in your PR, please add License Notice according
     [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
   * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If you are contributing the connector code, please check that the following files are updated:
     1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
     2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
     3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
     4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
     5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Csv] support configurable CSV delimiter in file connector [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on PR #9660:
URL: https://github.com/apache/seatunnel/pull/9660#issuecomment-3155036242

   waiting test case passes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Csv] support configurable CSV delimiter in file connector [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9660:
URL: https://github.com/apache/seatunnel/pull/9660#discussion_r2252996851


##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/test/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/CsvReadStrategyTest.java:
##########
@@ -71,6 +71,36 @@ public void testReadCsv() throws Exception {
         Assertions.assertEquals(100, testCollector.getRows().get(1).getField(2));
     }
 
+    @Test
+    public void testReadComplexCsv() throws Exception {
+        URL resource = CsvReadStrategyTest.class.getResource("/test-csv.csv");
+        String path = Paths.get(resource.toURI()).toString();
+        CsvReadStrategy csvReadStrategy = new CsvReadStrategy();
+        LocalConf localConf = new LocalConf(FS_DEFAULT_NAME_DEFAULT);
+        csvReadStrategy.init(localConf);
+        csvReadStrategy.getFileNamesByPath(path);
+        System.setProperty("csv_field_delimiter", ";");
+        csvReadStrategy.setPluginConfig(ConfigFactory.systemProperties());
+        csvReadStrategy.setCatalogTable(
+                CatalogTableUtil.getCatalogTable(
+                        "test",
+                        new SeaTunnelRowType(
+                                new String[] {"id", "name", "age"},
+                                new SeaTunnelDataType[] {
+                                    BasicType.INT_TYPE, BasicType.STRING_TYPE, BasicType.INT_TYPE
+                                })));
+        TestCollector testCollector = new TestCollector();
+        csvReadStrategy.read(path, "", testCollector);
+
+        Assertions.assertEquals(2, testCollector.getRows().size());
+        Assertions.assertEquals(1, testCollector.getRows().get(0).getField(0));
+        Assertions.assertEquals("b\r\na", testCollector.getRows().get(0).getField(1));

Review Comment:
   should be `\n` in linux?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Csv] support configurable CSV delimiter in file connector [seatunnel]

Posted by "zhenyue-xu (via GitHub)" <gi...@apache.org>.
zhenyue-xu commented on PR #9660:
URL: https://github.com/apache/seatunnel/pull/9660#issuecomment-3150230657

   > I think the delimiter of CSV better not to be configurable, as the name of CSV means(Comma-Separated Values).
   
   You make a very valid point. RFC 4180 itself acknowledges "considerable differences among implementations" and advises implementors to "be liberal in what you accept from others" when processing CSV files. Major data processing tools like Spark and DataX and etc support configurable CSV delimiters to handle this reality. As evidenced by Apache Commons CSV's predefined formats, different database systems use various delimiters - MySQL uses tabs, Informix uses pipes, while Oracle and PostgreSQL offer both comma and tab variants. The standard's own flexibility principle supports accommodating these widespread industry practices rather than enforcing rigid comma-only interpretation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Csv] support configurable CSV delimiter in file connector [seatunnel]

Posted by "zhenyue-xu (via GitHub)" <gi...@apache.org>.
zhenyue-xu commented on code in PR #9660:
URL: https://github.com/apache/seatunnel/pull/9660#discussion_r2253106708


##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/FileBaseSinkOptions.java:
##########
@@ -108,6 +108,13 @@ public class FileBaseSinkOptions extends FileBaseOptions {
                     .withDescription(
                             "The separator between columns in a row of data. Only needed by `text` and `csv` file format");
 
+    public static final Option<String> CSV_FIELD_DELIMITER =
+            Options.key("csv_field_delimiter")
+                    .stringType()
+                    .defaultValue(",")
+                    .withDescription(
+                            "The separator between columns in a row of data. Only needed by `csv` file format");

Review Comment:
   fixed 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Csv] support configurable CSV delimiter in file connector [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9660:
URL: https://github.com/apache/seatunnel/pull/9660#discussion_r2253000334


##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/FileBaseSinkOptions.java:
##########
@@ -108,6 +108,13 @@ public class FileBaseSinkOptions extends FileBaseOptions {
                     .withDescription(
                             "The separator between columns in a row of data. Only needed by `text` and `csv` file format");
 
+    public static final Option<String> CSV_FIELD_DELIMITER =
+            Options.key("csv_field_delimiter")
+                    .stringType()
+                    .defaultValue(",")
+                    .withDescription(
+                            "The separator between columns in a row of data. Only needed by `csv` file format");

Review Comment:
   why not reuse the `field_delimiter`? we can set different default value by ourself when format is text or csv.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Csv] support configurable CSV delimiter in file connector [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on PR #9660:
URL: https://github.com/apache/seatunnel/pull/9660#issuecomment-3153112260

   cc @corgy-w 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Csv] support configurable CSV delimiter in file connector [seatunnel]

Posted by "corgy-w (via GitHub)" <gi...@apache.org>.
corgy-w merged PR #9660:
URL: https://github.com/apache/seatunnel/pull/9660


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Csv] support configurable CSV delimiter in file connector [seatunnel]

Posted by "zhenyue-xu (via GitHub)" <gi...@apache.org>.
zhenyue-xu commented on code in PR #9660:
URL: https://github.com/apache/seatunnel/pull/9660#discussion_r2253106277


##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/test/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/CsvReadStrategyTest.java:
##########
@@ -71,6 +71,36 @@ public void testReadCsv() throws Exception {
         Assertions.assertEquals(100, testCollector.getRows().get(1).getField(2));
     }
 
+    @Test
+    public void testReadComplexCsv() throws Exception {
+        URL resource = CsvReadStrategyTest.class.getResource("/test-csv.csv");
+        String path = Paths.get(resource.toURI()).toString();
+        CsvReadStrategy csvReadStrategy = new CsvReadStrategy();
+        LocalConf localConf = new LocalConf(FS_DEFAULT_NAME_DEFAULT);
+        csvReadStrategy.init(localConf);
+        csvReadStrategy.getFileNamesByPath(path);
+        System.setProperty("csv_field_delimiter", ";");
+        csvReadStrategy.setPluginConfig(ConfigFactory.systemProperties());
+        csvReadStrategy.setCatalogTable(
+                CatalogTableUtil.getCatalogTable(
+                        "test",
+                        new SeaTunnelRowType(
+                                new String[] {"id", "name", "age"},
+                                new SeaTunnelDataType[] {
+                                    BasicType.INT_TYPE, BasicType.STRING_TYPE, BasicType.INT_TYPE
+                                })));
+        TestCollector testCollector = new TestCollector();
+        csvReadStrategy.read(path, "", testCollector);
+
+        Assertions.assertEquals(2, testCollector.getRows().size());
+        Assertions.assertEquals(1, testCollector.getRows().get(0).getField(0));
+        Assertions.assertEquals("b\r\na", testCollector.getRows().get(0).getField(1));

Review Comment:
   Fixed! Changed to use `System.lineSeparator()` for cross-platform compatibility.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Csv] support configurable CSV delimiter in file connector [seatunnel]

Posted by "litiliu (via GitHub)" <gi...@apache.org>.
litiliu commented on PR #9660:
URL: https://github.com/apache/seatunnel/pull/9660#issuecomment-3149906541

   I think the delimiter of CSV better not to be configurable, as the name of CSV means(Comma-Separated Values).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Csv] support configurable CSV delimiter in file connector [seatunnel]

Posted by "litiliu (via GitHub)" <gi...@apache.org>.
litiliu commented on PR #9660:
URL: https://github.com/apache/seatunnel/pull/9660#issuecomment-3149895135

   There is a PR https://github.com/apache/seatunnel/pull/8919, should slove the issue you described, would you mind using it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Csv] support configurable CSV delimiter in file connector [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on PR #9660:
URL: https://github.com/apache/seatunnel/pull/9660#issuecomment-3153098670

   > This PR adds back the csv_field_delimiter configuration option to properly support CSV files with custom delimiters (semicolon, tab, pipe, etc.) while maintaining full CSV standard compliance.
   
   make sense to me.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org