You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "baeminbo (via GitHub)" <gi...@apache.org> on 2024/08/20 06:53:48 UTC
[I] [Bug]: TextIO.read() with non-default delimiter doesn't split at the right position [beam]
baeminbo opened a new issue, #32251:
URL: https://github.com/apache/beam/issues/32251
### What happened?
The pipeline [1] splits a text file with the delimiter "ABC". For the input text "ABABCD", the expected result is ["AB", "D"]. But, the actual pipeline result is ["ABABCD", "D"]. See [2] for the result with DirectRunner.
I guess the delimiter match at [TextSource](https://github.com/apache/beam/blob/v2.58.1/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextSource.java#L448-L458) has the root cause. It processes the input text "ABABCD" as follows, so fails to match the delimiter "ABC" in the input text.
```
"A": "A" == delimiter[0] (= "A"), set delPosn to 1
"B": "B" == delimiter[1] (= "B"), set delPosn to 2
"A": "A" != delimiter[2] (= "C"), set delPosn to 0 <-- This is wrong. delPosn should be 1 as "A" matches delimiter[0]
"B": "B" != delimiter[0] (= "A"), set delPosn to 0
"C": "C" != delimiter[0] (= "A"), set delPosn to 0
"D": "D" != delimiter[0] (= "A"), set delPosn to 0
```
I think this is something like a regex match problem (e.g. delimiter "ABCABCABD" and input text "...ABCABCABC...". It may need to have multiple `delPosn`s for partial matches).
[1]
```
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.io.TextIO;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.ParDo;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class TextReadJob {
private static final Logger LOG = LoggerFactory.getLogger(TextReadJob.class);
private static final String INPUT_PATH = "short.csv"; // content: "ABABCD"
private static final byte[] DELIMITER = "ABC".getBytes(StandardCharsets.UTF_8);
public static void main(String[] args) throws IOException {
PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
Pipeline pipeline = Pipeline.create(options);
pipeline.apply(TextIO.read().from(INPUT_PATH).withDelimiter(DELIMITER)).apply(ParDo.of(new DoFn<String, Void>() {
@ProcessElement
public void processElement(@Element String input) {
LOG.info("input: <{}>", input);
}
}));
pipeline.run();
}
}
```
[2]
```
ug 19, 2024 11:30:43 PM org.apache.beam.sdk.io.FileBasedSource getEstimatedSizeBytes
INFO: Filepattern short.csv matched 1 files with total size 6
Aug 19, 2024 11:30:43 PM org.apache.beam.sdk.io.FileBasedSource split
INFO: Splitting filepattern short.csv into bundles of size 0 took 1 ms and produced 1 files and 6 bundles
Aug 19, 2024 11:30:43 PM baeminbo.TextReadJob$1 processElement
INFO: input: <ABABCD>
Aug 19, 2024 11:30:43 PM baeminbo.TextReadJob$1 processElement
INFO: input: <D>
```
### Issue Priority
Priority: 1 (data loss / total loss of function)
### Issue Components
- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: TextIO.read() with non-default delimiter doesn't split at the right position [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto commented on issue #32251:
URL: https://github.com/apache/beam/issues/32251#issuecomment-2325139615
> I'm afraid that #32298 cannot fix this issue with a delimiter including a "repeated pattern".
>
> For example, the delimiter is "ABABC" and the text is "ABABABCD". The expected result is ["AB", "D"]. But, the actual result with the fix #32298 is ["ABABABCD", "D"].
>
> The fix [considers](https://github.com/apache/beam/pull/32298/files#diff-638a856265a3f2e12f33c436603b99e8151a27072a07b05101fac77b15878129R459-R460) that the index 5 char "A" matches the index 0 char "A" in the delimiter.
>
> ```
> In the fix:
> 01234567
> ABABABCD
> ^
> ABABC
> ```
>
> But, it must match the index 2 char "A" in the delimiter.
>
> ```
> Right match:
> 01234567
> ABABABCD
> ^
> ABABC
> ```
I will take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: TextIO.read() with non-default delimiter doesn't split at the right position [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto commented on issue #32251:
URL: https://github.com/apache/beam/issues/32251#issuecomment-2303927729
.take-issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: TextIO.read() with non-default delimiter doesn't split at the right position [beam]
Posted by "baeminbo (via GitHub)" <gi...@apache.org>.
baeminbo commented on issue #32251:
URL: https://github.com/apache/beam/issues/32251#issuecomment-2323647986
I'm afraid that #32298 cannot fix this issue with a delimiter including a "repeated pattern".
For example, the delimiter is "ABABC" and the text is "ABABABCD". The expected result is ["AB", "D"]. But, the actual result with the fix #32298 is ["ABABABCD", "D"].
The fix [considers](https://github.com/apache/beam/pull/32298/files#diff-638a856265a3f2e12f33c436603b99e8151a27072a07b05101fac77b15878129R459-R460) that the index 5 char "A" matches the index 0 char "A" in the delimiter.
```
In the fix:
01234567
ABABABCD
^
ABABC
```
But, it must match the index 2 char "A" in the delimiter.
```
Right match:
01234567
ABABABCD
^
ABABC
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: TextIO.read() with non-default delimiter doesn't split at the right position [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #32251: [Bug]: TextIO.read() with non-default delimiter doesn't split at the right position
URL: https://github.com/apache/beam/issues/32251
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org