You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "EilidhMayne (via GitHub)" <gi...@apache.org> on 2023/06/19 09:25:21 UTC
[GitHub] [beam] EilidhMayne opened a new issue, #27164: [Feature Request]: WriteToParquet schema as side input
EilidhMayne opened a new issue, #27164:
URL: https://github.com/apache/beam/issues/27164
### What would you like to happen?
We have a need to pass a dynamic schema to the WriteToParquet PTransform but it will not accept a side input in the schema arg. Would this be possible to implement? My thought is that a similar approach to the WriteToBigQuery PTransform could be used.
### Issue Priority
Priority: 3 (nice-to-have improvement)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [X] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohan472000 commented on issue #27164: [Feature Request]: WriteToParquet schema as side input
Posted by "rohan472000 (via GitHub)" <gi...@apache.org>.
rohan472000 commented on issue #27164:
URL: https://github.com/apache/beam/issues/27164#issuecomment-1609075541
Yes same here ...I also want to pass a dynamic bucket location other than a static one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohan472000 commented on issue #27164: [Feature Request]: WriteToParquet schema as side input
Posted by "rohan472000 (via GitHub)" <gi...@apache.org>.
rohan472000 commented on issue #27164:
URL: https://github.com/apache/beam/issues/27164#issuecomment-1608107014
@EilidhMayne look into issue #27201 , is it the same that are you asking for?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] EilidhMayne commented on issue #27164: [Feature Request]: WriteToParquet schema as side input
Posted by "EilidhMayne (via GitHub)" <gi...@apache.org>.
EilidhMayne commented on issue #27164:
URL: https://github.com/apache/beam/issues/27164#issuecomment-1609038417
> look into issue #27201 , is it the same that are you asking for?
@rohan472000, No we're looking for a way to pass a dynamic pyarrow schema into the WriteToParquet PTransform. Currently it will only take a pyarrow object which makes it very difficult to implement in the way it was intended to be used. An ideal solution would be to allow side inputs for the schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: WriteToParquet schema as side input [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #27164:
URL: https://github.com/apache/beam/issues/27164#issuecomment-5970000042
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org