You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/02/14 22:33:37 UTC

[GitHub] [arrow] sanjibansg opened a new pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

sanjibansg opened a new pull request #12426:
URL: https://github.com/apache/arrow/pull/12426


   This PR modifies the WriteOptions for CSV to allow changing the default delimiter.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] lidavidm commented on pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#issuecomment-1047856224


   @sanjibansg the reason why it's failing is that there's two assertions, one on like 1890, one on line 1897, and only the latter got updated


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] sanjibansg commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r811397787



##########
File path: python/pyarrow/tests/test_csv.py
##########
@@ -326,7 +326,7 @@ def test_write_options():
     opts = cls()
 
     check_options_class(
-        cls, include_header=[True, False])
+        cls, include_header=[True, False], delimiter=[",", "\t", "|"])

Review comment:
       made the change.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] sanjibansg commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r808781144



##########
File path: cpp/src/arrow/csv/options.h
##########
@@ -194,6 +194,9 @@ struct ARROW_EXPORT WriteOptions {
   /// This number can impact performance.
   int32_t batch_size = 1024;
 
+  /// Field delimiter
+  char delimiter = ',';

Review comment:
       I am not sure about its implementation, can you please help me with it.
   
   cc: @westonpace @lidavidm 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] sanjibansg commented on pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#issuecomment-1047828587


   > Odd. I kicked the build, let's see if it persists. If it still fails maybe we can rebase and try again.
   
   @lidavidm Rebased the branch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] cyb70289 commented on pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
cyb70289 commented on pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#issuecomment-1047373300


   Appveyor test failure looks related to this PR, needs be fixed.
   https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/42659260/job/qk6x3kld8ffabyuu#L2758


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] sanjibansg commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r811597957



##########
File path: cpp/src/arrow/csv/writer_test.cc
##########
@@ -189,6 +198,12 @@ std::vector<WriterTestParams> GenerateTestCases() {
             /*expected_output*/ "", expected_status_no_quotes_with_structural(error_val)};
   };
 
+  // Schema/expected message for delimiter test
+  auto schema_custom_delimiter = schema({field("a", int64()), field("b", int64())});
+  auto batch_custom_delimiter = R"([{"a": 42, "b": -12}])";
+  auto expected_output_delimiter_tabs = "42\t-12\n";
+  auto expected_output_delimiter_pipe = "42|-12\n";

Review comment:
       Removed duplicate case.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] cyb70289 commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
cyb70289 commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r808784252



##########
File path: cpp/src/arrow/csv/options.h
##########
@@ -194,6 +194,9 @@ struct ARROW_EXPORT WriteOptions {
   /// This number can impact performance.
   int32_t batch_size = 1024;
 
+  /// Field delimiter
+  char delimiter = ',';

Review comment:
       No worry, I will upload the change above your PR.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] sanjibansg commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r808784784



##########
File path: cpp/src/arrow/csv/options.h
##########
@@ -194,6 +194,9 @@ struct ARROW_EXPORT WriteOptions {
   /// This number can impact performance.
   int32_t batch_size = 1024;
 
+  /// Field delimiter
+  char delimiter = ',';

Review comment:
       Thanks!
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] cyb70289 commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
cyb70289 commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r811530024



##########
File path: cpp/src/arrow/csv/writer_test.cc
##########
@@ -171,6 +173,13 @@ std::vector<WriterTestParams> GenerateTestCases() {
         "style is \"None\". See RFC4180. Invalid value: ",
         value);
   };
+
+  auto expected_status_illegal_delimiter = [](const char value) {
+    return Status::Invalid(
+        "WriteOptions: delimiter cannot be \\r or \\n or \" or EOL. Invalid value: ",
+        value);
+  };

Review comment:
       Better to move it under line 205 together with delimiter tests.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] sanjibansg edited a comment on pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
sanjibansg edited a comment on pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#issuecomment-1047511108


   > Appveyor test failure looks related to this PR, needs be fixed. https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/42659260/job/qk6x3kld8ffabyuu#L2758
   
   I checked the test logs. It shows an older piece of code which is failing now.
   https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/42659260/job/qk6x3kld8ffabyuu#L2745
   
   Here, the error log shows this 
   `assert t == read_csv(buf, read_options=read_options)`
   while the [updated code](https://github.com/apache/arrow/pull/12426/files#diff-d09f577a8ebe28cba37f0339c75f9eba3bedd1dade08e10521b345667447294cR1897) has this
    ```
           assert t == read_csv(buf, read_options=read_options,
                                parse_options=parse_options)
   ```
   
   
   
   > Correction (Updated on 24/02/2022)
   > Missed the parse_options in Line 1890, which actually caused the error, we need parse_options along with read_options for custom delimiter.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] sanjibansg commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r812177099



##########
File path: cpp/src/arrow/csv/options.cc
##########
@@ -73,6 +73,12 @@ Status ReadOptions::Validate() const {
 WriteOptions WriteOptions::Defaults() { return WriteOptions(); }
 
 Status WriteOptions::Validate() const {
+  if (ARROW_PREDICT_FALSE(delimiter == '\n' || delimiter == '\r' || delimiter == '"' ||
+                          std::string(1, delimiter) == eol)) {

Review comment:
       Made the change.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] cyb70289 commented on pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
cyb70289 commented on pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#issuecomment-1049419247


   "R / rhub/debian-gcc-devel:latest" job failure is not related.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] westonpace commented on pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
westonpace commented on pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#issuecomment-1049184109


   The header is still using commas instead of the delimiter (see line 524 of writer.cc).  This is causing the write_csv.py test to fail.  A more interesting question is why the Linux CI jobs are passing but I'm pretty sure that is unrelated to your PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] sanjibansg commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r811598053



##########
File path: cpp/src/arrow/csv/writer_test.cc
##########
@@ -171,6 +173,13 @@ std::vector<WriterTestParams> GenerateTestCases() {
         "style is \"None\". See RFC4180. Invalid value: ",
         value);
   };
+
+  auto expected_status_illegal_delimiter = [](const char value) {
+    return Status::Invalid(
+        "WriteOptions: delimiter cannot be \\r or \\n or \" or EOL. Invalid value: ",
+        value);
+  };

Review comment:
       Made the change.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] cyb70289 commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
cyb70289 commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r809660827



##########
File path: cpp/src/arrow/csv/options.cc
##########
@@ -73,6 +73,9 @@ Status ReadOptions::Validate() const {
 WriteOptions WriteOptions::Defaults() { return WriteOptions(); }
 
 Status WriteOptions::Validate() const {
+  if (ARROW_PREDICT_FALSE(delimiter == '\n' || delimiter == '\r' || delimiter == '"')) {
+    return Status::Invalid("WriteOptions: delimiter cannot be \\r or \\n");

Review comment:
       We support customized eol strings excepts `\n, \r, \r\n`, such as `_EOL_` as line separator in csv writer [1].
   Is it useful in practice? Our csv parser doesn't suppot it, AFAIK, @pitrou 
   
   Pandas.read_csv can specify a **single char** eol breaker by `lineterminator` parameter [2], works as expected per my test. But looks our eol string is too much?
   
   Pandas.to_csv does supports specifying line terminator as a string.
   
   [1] https://github.com/apache/arrow/blob/master/cpp/src/arrow/csv/writer_test.cc#L116
   [2] https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] cyb70289 commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
cyb70289 commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r809660827



##########
File path: cpp/src/arrow/csv/options.cc
##########
@@ -73,6 +73,9 @@ Status ReadOptions::Validate() const {
 WriteOptions WriteOptions::Defaults() { return WriteOptions(); }
 
 Status WriteOptions::Validate() const {
+  if (ARROW_PREDICT_FALSE(delimiter == '\n' || delimiter == '\r' || delimiter == '"')) {
+    return Status::Invalid("WriteOptions: delimiter cannot be \\r or \\n");

Review comment:
       We support customized eol strings excepts `\n, \r, \r\n`, such as `_EOL_` as line separator in csv writer [1].
   Is it useful in practice? Our csv parser doesn't suppot it, AFAIK, @pitrou 
   
   Pandas.read_csv can specify a **single char** eol breaker by `lineterminator` parameter [2], works as expected per my test. But looks our eol string is too much?
   
   [1] https://github.com/apache/arrow/blob/master/cpp/src/arrow/csv/writer_test.cc#L116
   [2] https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] ursabot edited a comment on pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#issuecomment-1049424643


   Benchmark runs are scheduled for baseline = effed6bb7c226cf4895b7e6804d70f71b861c06a and contender = 7c7abadd79d9f7ce530c8caae7c36e3c39425e2d. 7c7abadd79d9f7ce530c8caae7c36e3c39425e2d is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/50b303138295475ab1c14e8bf7cb1a9e...995e6800e7ee4fbeacf4b80dae203b37/)
   [Finished :arrow_down:0.17% :arrow_up:0.0%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/f2ac675effe74982b79e69f2f871c4da...6bc7a0c00c3b42c68ff6279ffff16971/)
   [Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/663f651f47214f848af46179e99c5e7d...4286263496f14c379424497ce8e716c4/)
   [Finished :arrow_down:0.34% :arrow_up:0.09%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/c39d62c564cc474db564e713d7780af6...9b6d807883584af295d78ddc8b3b4df7/)
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
   test-mac-arm: Supported benchmark langs: C++, Python, R
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] sanjibansg commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r812177458



##########
File path: cpp/src/arrow/csv/writer.cc
##########
@@ -234,7 +235,7 @@ class UnquotedColumnPopulator : public ColumnPopulator {
     while (offset < buffer_size) {
       // error happened or remaining bytes to check
       const char c = static_cast<char>(data[offset]);
-      if (c == '\n' || c == '\r' || c == ',' || c == '"') {
+      if (c == '\n' || c == '\r' || c == ',' || c == '"' || c == delimiter) {

Review comment:
       Made the change, removed comma from check.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] lidavidm commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r809022015



##########
File path: cpp/src/arrow/csv/options.cc
##########
@@ -73,6 +73,9 @@ Status ReadOptions::Validate() const {
 WriteOptions WriteOptions::Defaults() { return WriteOptions(); }
 
 Status WriteOptions::Validate() const {
+  if (ARROW_PREDICT_FALSE(delimiter == '\n' || delimiter == '\r' || delimiter == '"')) {
+    return Status::Invalid("WriteOptions: delimiter cannot be \\r or \\n");

Review comment:
       The error message should be updated as well.
   
   Actually, we should check delimiter against `eol` too right?

##########
File path: cpp/src/arrow/csv/writer_test.cc
##########
@@ -233,7 +241,16 @@ std::vector<WriterTestParams> GenerateTestCases() {
       reject_structural_params({"0123456789", nullptr, "abcde,", nullptr}, "abcde,"),
       reject_structural_params({"0123456789", nullptr, "abcdef,", nullptr}, "abcdef,"),
       reject_structural_params({nullptr, nullptr, ",0123456789", "abcde"}, ",0123456789"),
-      reject_structural_params({"0123456", nullptr, "7\\\"89", ",abcdef"}, "7\"89")};
+      reject_structural_params({"0123456", nullptr, "7\\\"89", ",abcdef"}, "7\"89"),
+      // exercise custom delimiter
+      {schema_custom_delimiter, batch_custom_delimiter,
+       DefaultTestOptions(/*include_header=*/false, /*null_string=*/"",
+                          QuotingStyle::Needed, "\n", /*delimiter=*/'\t'),
+       expected_output_delimiter_tabs},
+      {schema_custom_delimiter, batch_custom_delimiter,
+       DefaultTestOptions(/*include_header=*/false, /*null_string=*/"",
+                          QuotingStyle::Needed, "\n", /*delimiter=*/'|'),
+       expected_output_delimiter_pipe}};

Review comment:
       Can we add a couple cases with invalid parameters to make sure they're properly rejected? See how reject_structural_params sets things up.

##########
File path: python/pyarrow/tests/test_csv.py
##########
@@ -326,7 +326,7 @@ def test_write_options():
     opts = cls()
 
     check_options_class(
-        cls, include_header=[True, False])
+        cls, include_header=[True, False], delimiter=[",", "\t", "|"])

Review comment:
       Can we add a case to https://github.com/apache/arrow/blob/db2401c53f63643b06cab4c683cc086b915ec5c4/python/pyarrow/tests/test_csv.py#L1875-L1880 ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] cyb70289 commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
cyb70289 commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r808618899



##########
File path: cpp/src/arrow/csv/options.cc
##########
@@ -73,6 +73,9 @@ Status ReadOptions::Validate() const {
 WriteOptions WriteOptions::Defaults() { return WriteOptions(); }
 
 Status WriteOptions::Validate() const {
+  if (ARROW_PREDICT_FALSE(delimiter == '\n' || delimiter == '\r')) {

Review comment:
       Also exclude `"`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] sanjibansg commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r810858569



##########
File path: cpp/src/arrow/csv/options.cc
##########
@@ -73,6 +73,9 @@ Status ReadOptions::Validate() const {
 WriteOptions WriteOptions::Defaults() { return WriteOptions(); }
 
 Status WriteOptions::Validate() const {
+  if (ARROW_PREDICT_FALSE(delimiter == '\n' || delimiter == '\r' || delimiter == '"')) {
+    return Status::Invalid("WriteOptions: delimiter cannot be \\r or \\n");

Review comment:
       Made the change for checking with eol as well.

##########
File path: cpp/src/arrow/csv/options.cc
##########
@@ -73,6 +73,9 @@ Status ReadOptions::Validate() const {
 WriteOptions WriteOptions::Defaults() { return WriteOptions(); }
 
 Status WriteOptions::Validate() const {
+  if (ARROW_PREDICT_FALSE(delimiter == '\n' || delimiter == '\r' || delimiter == '"')) {
+    return Status::Invalid("WriteOptions: delimiter cannot be \\r or \\n");

Review comment:
       Made the change for checking with `eol` as well.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] sanjibansg commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r811418379



##########
File path: cpp/src/arrow/csv/writer_test.cc
##########
@@ -233,7 +241,16 @@ std::vector<WriterTestParams> GenerateTestCases() {
       reject_structural_params({"0123456789", nullptr, "abcde,", nullptr}, "abcde,"),
       reject_structural_params({"0123456789", nullptr, "abcdef,", nullptr}, "abcdef,"),
       reject_structural_params({nullptr, nullptr, ",0123456789", "abcde"}, ",0123456789"),
-      reject_structural_params({"0123456", nullptr, "7\\\"89", ",abcdef"}, "7\"89")};
+      reject_structural_params({"0123456", nullptr, "7\\\"89", ",abcdef"}, "7\"89"),
+      // exercise custom delimiter
+      {schema_custom_delimiter, batch_custom_delimiter,
+       DefaultTestOptions(/*include_header=*/false, /*null_string=*/"",
+                          QuotingStyle::Needed, "\n", /*delimiter=*/'\t'),
+       expected_output_delimiter_tabs},
+      {schema_custom_delimiter, batch_custom_delimiter,
+       DefaultTestOptions(/*include_header=*/false, /*null_string=*/"",
+                          QuotingStyle::Needed, "\n", /*delimiter=*/'|'),
+       expected_output_delimiter_pipe}};

Review comment:
       Made the changes.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] lidavidm commented on pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#issuecomment-1047777097


   Odd. I kicked the build, let's see if it persists. If it still fails maybe we can rebase and try again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] sanjibansg commented on pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#issuecomment-1049207693


   > The header is still using commas instead of the delimiter (see line 524 of writer.cc). This is causing the write_csv.py test to fail. A more interesting question is why the Linux CI jobs are passing but I'm pretty sure that is unrelated to your PR.
   
   Yes, sorry I missed that, and previously the parse_options was missing too. Corrected that. Thanks for help!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] sanjibansg commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r807745246



##########
File path: cpp/src/arrow/csv/writer.cc
##########
@@ -406,7 +405,7 @@ class CSVWriterImpl : public ipc::RecordBatchWriter {
     std::vector<std::unique_ptr<ColumnPopulator>> populators(schema->num_fields());
     for (int col = 0; col < schema->num_fields(); col++) {
       const std::string& end_chars =
-          col < schema->num_fields() - 1 ? kStrComma : options.eol;
+          col < schema->num_fields() - 1 ? std::string(1,options.delimiter) : options.eol;

Review comment:
       Yes, right, that's better.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] sanjibansg commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r807744751



##########
File path: python/pyarrow/_csv.pyx
##########
@@ -1025,17 +1025,21 @@ cdef class WriteOptions(_Weakrefable):
     batch_size : int, optional (default 1024)
         How many rows to process together when converting and writing
         CSV data
+    delimiter : char, optional (default `'`)

Review comment:
       Yes, sorry I missed it. Corrected that now.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] cyb70289 commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
cyb70289 commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r809672163



##########
File path: cpp/src/arrow/csv/options.h
##########
@@ -194,6 +194,9 @@ struct ARROW_EXPORT WriteOptions {
   /// This number can impact performance.
   int32_t batch_size = 1024;
 
+  /// Field delimiter
+  char delimiter = ',';

Review comment:
       [to-sanjiban.diff.gz](https://github.com/apache/arrow/files/8094142/to-sanjiban.diff.gz)
   @sanjibansg , you can apply above diff to your code. Please note this is a quick fix, needs adding test cases, or refinement maybe, and may have format issue.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] sanjibansg commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r810858995



##########
File path: cpp/src/arrow/csv/options.h
##########
@@ -194,6 +194,9 @@ struct ARROW_EXPORT WriteOptions {
   /// This number can impact performance.
   int32_t batch_size = 1024;
 
+  /// Field delimiter
+  char delimiter = ',';

Review comment:
       okay thanks applying this diff. Thanks a lot @cyb70289!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] ursabot commented on pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
ursabot commented on pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#issuecomment-1049424643


   Benchmark runs are scheduled for baseline = effed6bb7c226cf4895b7e6804d70f71b861c06a and contender = 7c7abadd79d9f7ce530c8caae7c36e3c39425e2d. 7c7abadd79d9f7ce530c8caae7c36e3c39425e2d is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Scheduled] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/50b303138295475ab1c14e8bf7cb1a9e...995e6800e7ee4fbeacf4b80dae203b37/)
   [Scheduled] [test-mac-arm](https://conbench.ursa.dev/compare/runs/f2ac675effe74982b79e69f2f871c4da...6bc7a0c00c3b42c68ff6279ffff16971/)
   [Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/663f651f47214f848af46179e99c5e7d...4286263496f14c379424497ce8e716c4/)
   [Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/c39d62c564cc474db564e713d7780af6...9b6d807883584af295d78ddc8b3b4df7/)
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
   test-mac-arm: Supported benchmark langs: C++, Python, R
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] sanjibansg commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r808780581



##########
File path: cpp/src/arrow/csv/options.cc
##########
@@ -73,6 +73,9 @@ Status ReadOptions::Validate() const {
 WriteOptions WriteOptions::Defaults() { return WriteOptions(); }
 
 Status WriteOptions::Validate() const {
+  if (ARROW_PREDICT_FALSE(delimiter == '\n' || delimiter == '\r')) {

Review comment:
       Made the change.

##########
File path: cpp/src/arrow/csv/writer_test.cc
##########
@@ -233,7 +241,15 @@ std::vector<WriterTestParams> GenerateTestCases() {
       reject_structural_params({"0123456789", nullptr, "abcde,", nullptr}, "abcde,"),
       reject_structural_params({"0123456789", nullptr, "abcdef,", nullptr}, "abcdef,"),
       reject_structural_params({nullptr, nullptr, ",0123456789", "abcde"}, ",0123456789"),
-      reject_structural_params({"0123456", nullptr, "7\\\"89", ",abcdef"}, "7\"89")};
+      reject_structural_params({"0123456", nullptr, "7\\\"89", ",abcdef"}, "7\"89"),
+      {schema_custom_delimiter, batch_custom_delimiter,

Review comment:
       Made the change.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] cyb70289 commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
cyb70289 commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r811529389



##########
File path: cpp/src/arrow/csv/writer_test.cc
##########
@@ -189,6 +198,12 @@ std::vector<WriterTestParams> GenerateTestCases() {
             /*expected_output*/ "", expected_status_no_quotes_with_structural(error_val)};
   };
 
+  // Schema/expected message for delimiter test
+  auto schema_custom_delimiter = schema({field("a", int64()), field("b", int64())});
+  auto batch_custom_delimiter = R"([{"a": 42, "b": -12}])";
+  auto expected_output_delimiter_tabs = "42\t-12\n";
+  auto expected_output_delimiter_pipe = "42|-12\n";

Review comment:
       Looks duplicated, one test should be enough.

##########
File path: cpp/src/arrow/csv/writer_test.cc
##########
@@ -60,13 +60,15 @@ void PrintTo(const WriterTestParams& p, std::ostream* os) {
 WriteOptions DefaultTestOptions(bool include_header = false,
                                 const std::string& null_string = "",
                                 QuotingStyle quoting_style = QuotingStyle::Needed,
-                                const std::string& eol = "\n") {
+                                const std::string& eol = "\n",
+                                const char& delimiter = ',') {

Review comment:
       `const char&` -> `char`

##########
File path: cpp/src/arrow/csv/writer_test.cc
##########
@@ -171,6 +173,13 @@ std::vector<WriterTestParams> GenerateTestCases() {
         "style is \"None\". See RFC4180. Invalid value: ",
         value);
   };
+
+  auto expected_status_illegal_delimiter = [](const char value) {
+    return Status::Invalid(
+        "WriteOptions: delimiter cannot be \\r or \\n or \" or EOL. Invalid value: ",
+        value);
+  };

Review comment:
       Move it under line 205 in delimiter test section.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] ursabot edited a comment on pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#issuecomment-1049424643


   Benchmark runs are scheduled for baseline = effed6bb7c226cf4895b7e6804d70f71b861c06a and contender = 7c7abadd79d9f7ce530c8caae7c36e3c39425e2d. 7c7abadd79d9f7ce530c8caae7c36e3c39425e2d is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/50b303138295475ab1c14e8bf7cb1a9e...995e6800e7ee4fbeacf4b80dae203b37/)
   [Scheduled] [test-mac-arm](https://conbench.ursa.dev/compare/runs/f2ac675effe74982b79e69f2f871c4da...6bc7a0c00c3b42c68ff6279ffff16971/)
   [Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/663f651f47214f848af46179e99c5e7d...4286263496f14c379424497ce8e716c4/)
   [Finished :arrow_down:0.34% :arrow_up:0.09%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/c39d62c564cc474db564e713d7780af6...9b6d807883584af295d78ddc8b3b4df7/)
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
   test-mac-arm: Supported benchmark langs: C++, Python, R
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] sanjibansg commented on pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#issuecomment-1047511108


   > Appveyor test failure looks related to this PR, needs be fixed. https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/42659260/job/qk6x3kld8ffabyuu#L2758
   
   I checked the test logs. It shows an older piece of code which is failing now.
   https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/42659260/job/qk6x3kld8ffabyuu#L2745
   
   Here, the error log shows this 
   `assert t == read_csv(buf, read_options=read_options)`
   while the [updated code](https://github.com/apache/arrow/pull/12426/files#diff-d09f577a8ebe28cba37f0339c75f9eba3bedd1dade08e10521b345667447294cR1897) has this
    ```
           assert t == read_csv(buf, read_options=read_options,
                                parse_options=parse_options)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] sanjibansg commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r812177777



##########
File path: python/pyarrow/includes/libarrow.pxd
##########
@@ -1651,7 +1651,7 @@ cdef extern from "arrow/python/csv.h" namespace "arrow::py::csv":
 cdef extern from "arrow/csv/api.h" namespace "arrow::csv" nogil:
 
     cdef cppclass CCSVParseOptions" arrow::csv::ParseOptions":
-        unsigned char delimiter
+        char delimiter

Review comment:
       Changed to unsigned char for delimiter.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] github-actions[bot] commented on pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#issuecomment-1039640235






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] cyb70289 commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
cyb70289 commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r808625398



##########
File path: cpp/src/arrow/csv/options.h
##########
@@ -194,6 +194,9 @@ struct ARROW_EXPORT WriteOptions {
   /// This number can impact performance.
   int32_t batch_size = 1024;
 
+  /// Field delimiter
+  char delimiter = ',';

Review comment:
       Shall we treat the customized delimiter as non structural chars in non-quoted string checking [1]? Currently, it's hard coded to check `,` and `\r`, `\n`, `"`.
   
   https://github.com/apache/arrow/blob/master/cpp/src/arrow/csv/writer.cc#L213




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] sanjibansg commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r808781144



##########
File path: cpp/src/arrow/csv/options.h
##########
@@ -194,6 +194,9 @@ struct ARROW_EXPORT WriteOptions {
   /// This number can impact performance.
   int32_t batch_size = 1024;
 
+  /// Field delimiter
+  char delimiter = ',';

Review comment:
       I am not sure about its implementation, could you please help me with it.
   
   cc: @westonpace @lidavidm 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] sanjibansg commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r808781872



##########
File path: python/pyarrow/includes/libarrow.pxd
##########
@@ -1713,6 +1713,7 @@ cdef extern from "arrow/csv/api.h" namespace "arrow::csv" nogil:
     cdef cppclass CCSVWriteOptions" arrow::csv::WriteOptions":
         c_bool include_header
         int32_t batch_size
+        unsigned char delimiter

Review comment:
       Made the change.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] lidavidm commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r806807912



##########
File path: cpp/src/arrow/csv/writer.cc
##########
@@ -406,7 +405,7 @@ class CSVWriterImpl : public ipc::RecordBatchWriter {
     std::vector<std::unique_ptr<ColumnPopulator>> populators(schema->num_fields());
     for (int col = 0; col < schema->num_fields(); col++) {
       const std::string& end_chars =
-          col < schema->num_fields() - 1 ? kStrComma : options.eol;
+          col < schema->num_fields() - 1 ? std::string(1,options.delimiter) : options.eol;

Review comment:
       While the compiler will likely do this for us, we should probably hoist the string construction outside of the loop.

##########
File path: python/pyarrow/_csv.pyx
##########
@@ -1025,17 +1025,21 @@ cdef class WriteOptions(_Weakrefable):
     batch_size : int, optional (default 1024)
         How many rows to process together when converting and writing
         CSV data
+    delimiter : char, optional (default `'`)

Review comment:
       See https://github.com/apache/arrow/blob/36a76914cd42606be7ef847d500af67c64ccef26/python/pyarrow/_csv.pyx#L278 for instance

##########
File path: python/pyarrow/_csv.pyx
##########
@@ -1025,17 +1025,21 @@ cdef class WriteOptions(_Weakrefable):
     batch_size : int, optional (default 1024)
         How many rows to process together when converting and writing
         CSV data
+    delimiter : char, optional (default `'`)

Review comment:
       nit, but I think Python would use `"'"`, and char is not a type in Python (it should be str). And in any case, is the delimiter not `,`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] lidavidm commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r806807912



##########
File path: cpp/src/arrow/csv/writer.cc
##########
@@ -406,7 +405,7 @@ class CSVWriterImpl : public ipc::RecordBatchWriter {
     std::vector<std::unique_ptr<ColumnPopulator>> populators(schema->num_fields());
     for (int col = 0; col < schema->num_fields(); col++) {
       const std::string& end_chars =
-          col < schema->num_fields() - 1 ? kStrComma : options.eol;
+          col < schema->num_fields() - 1 ? std::string(1,options.delimiter) : options.eol;

Review comment:
       While the compiler will likely do this for us, we should probably hoist the string construction outside of the loop.

##########
File path: python/pyarrow/_csv.pyx
##########
@@ -1025,17 +1025,21 @@ cdef class WriteOptions(_Weakrefable):
     batch_size : int, optional (default 1024)
         How many rows to process together when converting and writing
         CSV data
+    delimiter : char, optional (default `'`)

Review comment:
       See https://github.com/apache/arrow/blob/36a76914cd42606be7ef847d500af67c64ccef26/python/pyarrow/_csv.pyx#L278 for instance

##########
File path: python/pyarrow/_csv.pyx
##########
@@ -1025,17 +1025,21 @@ cdef class WriteOptions(_Weakrefable):
     batch_size : int, optional (default 1024)
         How many rows to process together when converting and writing
         CSV data
+    delimiter : char, optional (default `'`)

Review comment:
       nit, but I think Python would use `"'"`, and char is not a type in Python (it should be str). And in any case, is the delimiter not `,`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] pitrou commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r812037255



##########
File path: cpp/src/arrow/csv/options.cc
##########
@@ -73,6 +73,12 @@ Status ReadOptions::Validate() const {
 WriteOptions WriteOptions::Defaults() { return WriteOptions(); }
 
 Status WriteOptions::Validate() const {
+  if (ARROW_PREDICT_FALSE(delimiter == '\n' || delimiter == '\r' || delimiter == '"' ||
+                          std::string(1, delimiter) == eol)) {

Review comment:
       You should look for `delimiter` in `eol` instead, so for example `eol.find(delimiter) != std::string::npos`.

##########
File path: cpp/src/arrow/csv/writer.cc
##########
@@ -234,7 +235,7 @@ class UnquotedColumnPopulator : public ColumnPopulator {
     while (offset < buffer_size) {
       // error happened or remaining bytes to check
       const char c = static_cast<char>(data[offset]);
-      if (c == '\n' || c == '\r' || c == ',' || c == '"') {
+      if (c == '\n' || c == '\r' || c == ',' || c == '"' || c == delimiter) {

Review comment:
       The comma can be removed here:
   ```suggestion
         if (c == '\n' || c == '\r' || c == '"' || c == delimiter) {
   ```

##########
File path: python/pyarrow/includes/libarrow.pxd
##########
@@ -1713,6 +1713,7 @@ cdef extern from "arrow/csv/api.h" namespace "arrow::csv" nogil:
     cdef cppclass CCSVWriteOptions" arrow::csv::WriteOptions":
         c_bool include_header
         int32_t batch_size
+        char delimiter

Review comment:
       Here as well.

##########
File path: python/pyarrow/includes/libarrow.pxd
##########
@@ -1651,7 +1651,7 @@ cdef extern from "arrow/python/csv.h" namespace "arrow::py::csv":
 cdef extern from "arrow/csv/api.h" namespace "arrow::csv" nogil:
 
     cdef cppclass CCSVParseOptions" arrow::csv::ParseOptions":
-        unsigned char delimiter
+        char delimiter

Review comment:
       Character numbers in Python are unsigned, so IMHO you should keep this unsigned.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] github-actions[bot] commented on pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#issuecomment-1039640235






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] cyb70289 commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
cyb70289 commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r808621398



##########
File path: cpp/src/arrow/csv/writer_test.cc
##########
@@ -233,7 +241,15 @@ std::vector<WriterTestParams> GenerateTestCases() {
       reject_structural_params({"0123456789", nullptr, "abcde,", nullptr}, "abcde,"),
       reject_structural_params({"0123456789", nullptr, "abcdef,", nullptr}, "abcdef,"),
       reject_structural_params({nullptr, nullptr, ",0123456789", "abcde"}, ",0123456789"),
-      reject_structural_params({"0123456", nullptr, "7\\\"89", ",abcdef"}, "7\"89")};
+      reject_structural_params({"0123456", nullptr, "7\\\"89", ",abcdef"}, "7\"89"),
+      {schema_custom_delimiter, batch_custom_delimiter,

Review comment:
       Please insert a comment, e.g., `// exercise custom delimiter`, to separate from simd tests above.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] sanjibansg commented on pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#issuecomment-1041312350


   > Thanks for doing this. We should also add unit tests (see https://github.com/apache/arrow/blob/master/cpp/src/arrow/csv/writer_test.cc and https://github.com/apache/arrow/blob/master/python/pyarrow/tests/test_csv.py) and you'll want to reformat the code here (see https://arrow.apache.org/docs/developers/cpp/development.html#code-style-linting-and-ci)
   
   Adding the tests now,


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] cyb70289 commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
cyb70289 commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r808622146



##########
File path: python/pyarrow/includes/libarrow.pxd
##########
@@ -1713,6 +1713,7 @@ cdef extern from "arrow/csv/api.h" namespace "arrow::csv" nogil:
     cdef cppclass CCSVWriteOptions" arrow::csv::WriteOptions":
         c_bool include_header
         int32_t batch_size
+        unsigned char delimiter

Review comment:
       `unsigned char` or `char`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] sanjibansg commented on a change in pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
sanjibansg commented on a change in pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#discussion_r811597854



##########
File path: cpp/src/arrow/csv/writer_test.cc
##########
@@ -60,13 +60,15 @@ void PrintTo(const WriterTestParams& p, std::ostream* os) {
 WriteOptions DefaultTestOptions(bool include_header = false,
                                 const std::string& null_string = "",
                                 QuotingStyle quoting_style = QuotingStyle::Needed,
-                                const std::string& eol = "\n") {
+                                const std::string& eol = "\n",
+                                const char& delimiter = ',') {

Review comment:
       Corrected that.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] cyb70289 closed pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
cyb70289 closed pull request #12426:
URL: https://github.com/apache/arrow/pull/12426


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] ursabot edited a comment on pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#issuecomment-1049424643


   Benchmark runs are scheduled for baseline = effed6bb7c226cf4895b7e6804d70f71b861c06a and contender = 7c7abadd79d9f7ce530c8caae7c36e3c39425e2d. 7c7abadd79d9f7ce530c8caae7c36e3c39425e2d is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/50b303138295475ab1c14e8bf7cb1a9e...995e6800e7ee4fbeacf4b80dae203b37/)
   [Scheduled] [test-mac-arm](https://conbench.ursa.dev/compare/runs/f2ac675effe74982b79e69f2f871c4da...6bc7a0c00c3b42c68ff6279ffff16971/)
   [Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/663f651f47214f848af46179e99c5e7d...4286263496f14c379424497ce8e716c4/)
   [Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/c39d62c564cc474db564e713d7780af6...9b6d807883584af295d78ddc8b3b4df7/)
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
   test-mac-arm: Supported benchmark langs: C++, Python, R
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] ursabot edited a comment on pull request #12426: ARROW-15672: [C++] Enable CSV writer to control the field delimiter

Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12426:
URL: https://github.com/apache/arrow/pull/12426#issuecomment-1049424643


   Benchmark runs are scheduled for baseline = effed6bb7c226cf4895b7e6804d70f71b861c06a and contender = 7c7abadd79d9f7ce530c8caae7c36e3c39425e2d. 7c7abadd79d9f7ce530c8caae7c36e3c39425e2d is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/50b303138295475ab1c14e8bf7cb1a9e...995e6800e7ee4fbeacf4b80dae203b37/)
   [Finished :arrow_down:0.17% :arrow_up:0.0%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/f2ac675effe74982b79e69f2f871c4da...6bc7a0c00c3b42c68ff6279ffff16971/)
   [Finished :arrow_down:0.0% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/663f651f47214f848af46179e99c5e7d...4286263496f14c379424497ce8e716c4/)
   [Finished :arrow_down:0.34% :arrow_up:0.09%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/c39d62c564cc474db564e713d7780af6...9b6d807883584af295d78ddc8b3b4df7/)
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
   test-mac-arm: Supported benchmark langs: C++, Python, R
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org