You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "rootvector2 (via GitHub)" <gi...@apache.org> on 2026/06/17 11:09:49 UTC

[PR] clear delimiter buffer before each peek in isDelimiter [commons-csv]

rootvector2 opened a new pull request, #611:
URL: https://github.com/apache/commons-csv/pull/611

   `isDelimiter` peeks the next characters into the reused `delimiterBuf` look-ahead buffer without clearing it, so a non-matching peek earlier in the same token leaves stale chars in the trailing positions and a truncated multi-character delimiter at EOF false-matches. With delimiter `[|]`, input `x[a][|` is split into `x[a]` and an empty field, because the earlier `[a]` peek left `]` in the buffer and the trailing `[|` (only two of the three delimiter chars present before EOF) matches against that stale `]`. CSV-324 cleared `delimiterBuf` once per `nextToken`, but `isDelimiter` peeks repeatedly within a token, so the reset has to happen before the peek, the same way `isEscapeDelimiter` already does it; the once-per-token clear is then redundant and dropped.
   
   Found while auditing the multi-character delimiter look-ahead after CSV-324.
   
   - [x] Read the [contribution guidelines](CONTRIBUTING.md) for this project.
   - [ ] Read the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) if you use Artificial Intelligence (AI).
   - [ ] I used AI to create any part of, or all of, this pull request. Which AI tool was used to create this pull request, and to what extent did it contribute?
   - [x] Run a successful build using the default [Maven](https://maven.apache.org/) goal with `mvn`; that's `mvn` on the command line by itself.
   - [x] Write unit tests that match behavioral changes, where the tests fail if the changes to the runtime are not applied. This may not always be possible, but it is a best practice.
   - [x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
   - [x] Each commit in the pull request should have a meaningful subject line and body.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@commons.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Clear delimiter buffer before each peek in isDelimiter [commons-csv]

Posted by "garydgregory (via GitHub)" <gi...@apache.org>.
garydgregory commented on code in PR #611:
URL: https://github.com/apache/commons-csv/pull/611#discussion_r3427889023


##########
src/test/java/org/apache/commons/csv/CSVParserTest.java:
##########
@@ -1696,6 +1696,21 @@ void testPartialMultiCharacterDelimiterAtEOF() throws IOException {
         }
     }
 
+    /**
+     * A truncated multi-character delimiter at EOF must not be completed from the look-ahead buffer left dirty by an
+     * earlier non-matching peek in the same token.
+     */
+    @Test
+    void testPartialMultiCharacterDelimiterAtEOFAfterMismatch() throws IOException {
+        final CSVFormat format = CSVFormat.DEFAULT.builder().setDelimiter("[|]").get();
+        // The "[a]" peek leaves ']' in the look-ahead buffer; the trailing "[|" must not match "[|]".
+        try (CSVParser parser = format.parse(new StringReader("x[a][|"))) {

Review Comment:
   @rootvector2 
   
   Thank you for your update.
   
   In the future, please make magic strings like `"x[a][|"` a local variable such that you never have to check that the assertion below is checking for a possibly different value.
   



##########
src/test/java/org/apache/commons/csv/LexerTest.java:
##########
@@ -433,6 +433,19 @@ void testPartialMultiCharacterDelimiterAtEOF() throws IOException {
         }
     }
 
+    /**
+     * A truncated multi-character delimiter at EOF must not be accepted by reusing the look-ahead buffer left dirty by an
+     * earlier non-matching peek in the same token (CSV-324 only cleared the buffer once per token).
+     */
+    @Test
+    void testPartialMultiCharacterDelimiterAtEOFAfterMismatch() throws IOException {
+        final CSVFormat format = CSVFormat.DEFAULT.builder().setDelimiter("[|]").get();
+        // The "[a]" peek leaves ']' in the look-ahead buffer; the trailing "[|" must not match "[|]".
+        try (Lexer lexer = createLexer("x[a][|", format)) {

Review Comment:
   @rootvector2 
   
   Thank you for your update.
   
   In the future, please make magic strings like `"x[a][|"` a local variable such that you never have to check that the assertion below is checking for a possibly different value.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@commons.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Clear delimiter buffer before each peek in isDelimiter [commons-csv]

Posted by "garydgregory (via GitHub)" <gi...@apache.org>.
garydgregory merged PR #611:
URL: https://github.com/apache/commons-csv/pull/611


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@commons.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Clear delimiter buffer before each peek in isDelimiter [commons-csv]

Posted by "rootvector2 (via GitHub)" <gi...@apache.org>.
rootvector2 commented on PR #611:
URL: https://github.com/apache/commons-csv/pull/611#issuecomment-4730308678

   makes sense, will pull magic strings like that into locals going forward. thanks for the merge.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@commons.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Clear delimiter buffer before each peek in isDelimiter [commons-csv]

Posted by "rootvector2 (via GitHub)" <gi...@apache.org>.
rootvector2 commented on PR #611:
URL: https://github.com/apache/commons-csv/pull/611#issuecomment-4729497338

   Added `testPartialMultiCharacterDelimiterAtEOFAfterMismatch` in `CSVParserTest` that drives this through the public parsing API. Parsing `x[a][|` with delimiter `[|]`: without the fix it splits into `x[a]` and an empty field, with the fix it stays a single field `x[a][|`. The test fails on `master` and passes with the patch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@commons.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Clear delimiter buffer before each peek in isDelimiter [commons-csv]

Posted by "garydgregory (via GitHub)" <gi...@apache.org>.
garydgregory commented on PR #611:
URL: https://github.com/apache/commons-csv/pull/611#issuecomment-4729855313

   @rootvector2 
   Please see my comments for future PRs. Merged and thank you! 🚀 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@commons.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org