You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "arkadioz (via GitHub)" <gi...@apache.org> on 2024/05/14 22:32:05 UTC

[I] [Bug]: TextIO.read().withEmptyMatchTreatment(EmptyMatchTreatment.ALLOW) still fails if no file is found [beam]

arkadioz opened a new issue, #31296:
URL: https://github.com/apache/beam/issues/31296

   ### What happened?
   
   According to the documentation https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/io/TextIO.html
   
   We can configure the behaviour of the read() method to also allow no matches or empty matches, so I configured a pipeline step like this:
   
   ```
   PCollection<KV<String, Incident>> incidents = pipeline.apply("READ CSV", TextIO.read().withEmptyMatchTreatment(EmptyMatchTreatment.ALLOW).from("gs://bucket/incidents.csv"))
                   .apply("Convert to bean and output KV Collection", ParDo.of(new IncidentCsvToBeanFunction()));
   // The IncidentCsvToBeanFunction is just mapping the csv content to a java class.
   ```
   but it still fails to start the pipeline even with this configuration when the incidents.csv is not present at the bucket, but when I use
   a wildcard (\*) for example incidents*.csv or *.csv it works even if the incidents.csv does not exists, but according to the documentation and what I understand is that with the .withEmptyMatchTreatment(EmptyMatchTreatment.ALLOW) it should work with just incidents.csv and no wildcards even if the csv is not present, so I consider it a bug unless I misunderstood... The error im getting on google cloud dataflow logs is:
   ```
   "[The preflight pipeline validation failed for job 2024-05-14_15_25_49-jobnumbers. To bypass the validation, use the Dataflow service option with the value enable_preflight_validation=false. Learn more at https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline#validation] NOT_FOUND: Unable to find object incidents.csv in bucket bucket_name.
   ```
   
   What I expect is the same behaviour when I use the wildcard, just continue the pipeline and return an empty PCollection, even when the csv is not present in the bucket (does not exists).
   
   ### Issue Priority
   
   Priority: 2 (default / most bugs should be filed as P2)
   
   ### Issue Components
   
   - [ ] Component: Python SDK
   - [X] Component: Java SDK
   - [ ] Component: Go SDK
   - [ ] Component: Typescript SDK
   - [ ] Component: IO connector
   - [ ] Component: Beam YAML
   - [ ] Component: Beam examples
   - [ ] Component: Beam playground
   - [ ] Component: Beam katas
   - [ ] Component: Website
   - [ ] Component: Spark Runner
   - [ ] Component: Flink Runner
   - [ ] Component: Samza Runner
   - [ ] Component: Twister2 Runner
   - [ ] Component: Hazelcast Jet Runner
   - [X] Component: Google Cloud Dataflow Runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: TextIO.read().withEmptyMatchTreatment(EmptyMatchTreatment.ALLOW) still fails if no file is found [beam]

Posted by "arkadioz (via GitHub)" <gi...@apache.org>.
arkadioz commented on issue #31296:
URL: https://github.com/apache/beam/issues/31296#issuecomment-2111271034

   Sorry I think the problem is the dataflow preflight service, I tried what this link in the error message says: https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline?hl=es-419#validation
   
   using: `--dataflowServiceOptions=enable_preflight_validation=false`
   
   and it allowed the pipeline to start and work as expected, but I believe this is bad because what if I still want the preflight service on? the preflight service should also be able to recognize about the .withEmptyMatchTreatment(EmptyMatchTreatment.ALLOW) and allow the start of the pipeline even if it detects separately that the csv does not exists in the bucket


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: TextIO.read().withEmptyMatchTreatment(EmptyMatchTreatment.ALLOW) still fails if no file is found [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #31296: [Bug]: TextIO.read().withEmptyMatchTreatment(EmptyMatchTreatment.ALLOW) still fails if no file is found
URL: https://github.com/apache/beam/issues/31296


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: TextIO.read().withEmptyMatchTreatment(EmptyMatchTreatment.ALLOW) still fails if no file is found [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #31296:
URL: https://github.com/apache/beam/issues/31296#issuecomment-3693956340

   This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: TextIO.read().withEmptyMatchTreatment(EmptyMatchTreatment.ALLOW) still fails if no file is found [beam]

Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #31296:
URL: https://github.com/apache/beam/issues/31296#issuecomment-2111488845

   This looks like a bug in the service. Can you open a Google cloud support case?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: TextIO.read().withEmptyMatchTreatment(EmptyMatchTreatment.ALLOW) still fails if no file is found [beam]

Posted by "arkadioz (via GitHub)" <gi...@apache.org>.
arkadioz commented on issue #31296:
URL: https://github.com/apache/beam/issues/31296#issuecomment-2111526040

   @liferoad Sure could do it, can you please share the link to open one? Have not open one before unless you mean the google cloud [community forum](https://www.googlecloudcommunity.com/gc/Google-Cloud/ct-p/google-cloud)?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: TextIO.read().withEmptyMatchTreatment(EmptyMatchTreatment.ALLOW) still fails if no file is found [beam]

Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #31296:
URL: https://github.com/apache/beam/issues/31296#issuecomment-2112901825

   Please check this: https://cloud.google.com/dataflow/docs/support/getting-support#file-bugs-or-feature-requests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: TextIO.read().withEmptyMatchTreatment(EmptyMatchTreatment.ALLOW) still fails if no file is found [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #31296:
URL: https://github.com/apache/beam/issues/31296#issuecomment-3581175569

   This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org