You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@beam.apache.org by "damnMeddlingKid (via GitHub)" <gi...@apache.org> on 2024/01/12 00:59:06 UTC

[I] [Feature Request]: Remove dependency of KafkaIO withDynamicRead on beam_fn_api [beam]

damnMeddlingKid opened a new issue, #29998:
URL: https://github.com/apache/beam/issues/29998

   ### What happened?
   
   The withDynamicRead method was gated with the `beam_fn_api` experimental flag in https://issues.apache.org/jira/browse/BEAM-11946 && https://github.com/apache/beam/pull/14168. 
   
   From reading the ticket it sounds like the intention was to ensure that we read from Kafka with an SDF when using x-lang features.
   
   Im wondering if this is still necessary today because reading from Kafka via SDF is the default behaviour and there are a few exceptional cases where it [falls back to an unbounded source](https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L1423-L1433) .
   
   heres a link to a previous PR that removed `beam_fn_api` from the rest of KafkaIO but kept it for dynamic reads https://github.com/apache/beam/pull/14419
   
   cc: @kennknowles you might have some context on if this is still required ?
   
   
   
   ### Issue Priority
   
   Priority: 3 (minor)
   
   ### Issue Components
   
   - [ ] Component: Python SDK
   - [X] Component: Java SDK
   - [ ] Component: Go SDK
   - [ ] Component: Typescript SDK
   - [ ] Component: IO connector
   - [ ] Component: Beam YAML
   - [ ] Component: Beam examples
   - [ ] Component: Beam playground
   - [ ] Component: Beam katas
   - [ ] Component: Website
   - [ ] Component: Spark Runner
   - [ ] Component: Flink Runner
   - [ ] Component: Samza Runner
   - [ ] Component: Twister2 Runner
   - [ ] Component: Hazelcast Jet Runner
   - [ ] Component: Google Cloud Dataflow Runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature Request]: Remove dependency of KafkaIO withDynamicRead on beam_fn_api [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.

Abacn closed issue #29998: [Feature Request]: Remove dependency of KafkaIO withDynamicRead on beam_fn_api 
URL: https://github.com/apache/beam/issues/29998


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature Request]: Remove dependency of KafkaIO withDynamicRead on beam_fn_api [beam]

Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.

johnjcasey commented on issue #29998:
URL: https://github.com/apache/beam/issues/29998#issuecomment-2438347750

   I believe we still need this, because we cannot fall back to the legacy implementation if the user needs dynamic reads


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature Request]: Remove dependency of KafkaIO withDynamicRead on beam_fn_api [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] commented on issue #29998:
URL: https://github.com/apache/beam/issues/29998#issuecomment-4188841342

   This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature Request]: Remove dependency of KafkaIO withDynamicRead on beam_fn_api [beam]

Posted by "kennknowles (via GitHub)" <gi...@apache.org>.

kennknowles commented on issue #29998:
URL: https://github.com/apache/beam/issues/29998#issuecomment-4192966761

   This should still be handled somehow, if it hasn't been addressed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature Request]: Remove dependency of KafkaIO withDynamicRead on beam_fn_api [beam]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.

goutamadwant commented on issue #29998:
URL: https://github.com/apache/beam/issues/29998#issuecomment-5263404894

   .take-issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature Request]: Remove dependency of KafkaIO withDynamicRead on beam_fn_api [beam]

Posted by "kennknowles (via GitHub)" <gi...@apache.org>.

kennknowles commented on issue #29998:
URL: https://github.com/apache/beam/issues/29998#issuecomment-2455152087

   The way this should work is that the transform should expand for dynamic reads and the runner can say yes/no to whether it can execute it. We should definitely eliminate the `beam_fn_api` experiment entirely. It is really old and was meant for running stuff in a nonstandard way before it was really ready for production. TBH we probably never needed it IMO.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org