You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/05/04 09:44:55 UTC
[GitHub] [arrow] AlenkaF opened a new pull request, #13062: ARROW-16420: [Python] pq.write_to_dataset always ignores partitioning
AlenkaF opened a new pull request, #13062:
URL: https://github.com/apache/arrow/pull/13062
Remove the lines that unconditionally set `partitioning` and `file_visitor` in `pq.write_to_dataset` to None. This is a leftover from https://github.com/apache/arrow/pull/12811 where additional `pq.write_dataset` keywords were exposed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #13062: ARROW-16420: [Python] pq.write_to_dataset always ignores partitioning
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on PR #13062:
URL: https://github.com/apache/arrow/pull/13062#issuecomment-1130205125
@AlenkaF do you can to do that here, or in a follow-up PR? (either way is fine)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13062: ARROW-16420: [Python] pq.write_to_dataset always ignores partitioning
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13062:
URL: https://github.com/apache/arrow/pull/13062#issuecomment-1117122226
https://issues.apache.org/jira/browse/ARROW-16420
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] AlenkaF commented on pull request #13062: ARROW-16420: [Python] pq.write_to_dataset always ignores partitioning
Posted by GitBox <gi...@apache.org>.
AlenkaF commented on PR #13062:
URL: https://github.com/apache/arrow/pull/13062#issuecomment-1118235181
I added a test that checks for `partitioning` and `file_visitor` being correctly passed in `pq.write_to_dataset`.
While writing the test I bumped into another error. If the `basename_template` is specified as a keyword in `pq.write_to_dataset` (not being None) the code missed the check for `existing_data_behavior` and so the call to `ds.write_dataset` errored due to `existing_data_behavior` being None and not a string. I decided to add a correction here as this is also my leftover, but from https://github.com/apache/arrow/pull/12838. I could do a separate PR if there will be any opinion in favour of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #13062: ARROW-16420: [Python] pq.write_to_dataset always ignores partitioning
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #13062:
URL: https://github.com/apache/arrow/pull/13062#issuecomment-1132287248
Benchmark runs are scheduled for baseline = 1cdedc4cbf0709ce440d69242afd47474a7148c7 and contender = 0a0d7fea9550b48706cf678036eaa8ddb1ef2be8. 0a0d7fea9550b48706cf678036eaa8ddb1ef2be8 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/691d1239b5254172a54ac4f55ad5bc49...b22d25d344b546a3adb36c047efaaec6/)
[Failed :arrow_down:0.51% :arrow_up:0.0%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/b880f981da07466fa6835f84ba17d7ac...d4a9464a43a54bbfafa6c2b869c8d5f7/)
[Failed :arrow_down:0.0% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/cc76aa92fc9c4fe5ac0713318aa958db...4dfbd27c569747f2aa4ce77186d4904f/)
[Finished :arrow_down:0.2% :arrow_up:0.04%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/49278e72c18b40849421fa70024b3387...66a0c16db77f4f7985978f06ad8d86c3/)
Buildkite builds:
[Finished] [`0a0d7fea` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/796)
[Failed] [`0a0d7fea` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/793)
[Failed] [`0a0d7fea` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/783)
[Finished] [`0a0d7fea` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/799)
[Finished] [`1cdedc4c` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/795)
[Failed] [`1cdedc4c` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/792)
[Failed] [`1cdedc4c` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/782)
[Finished] [`1cdedc4c` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/798)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche closed pull request #13062: ARROW-16420: [Python] pq.write_to_dataset always ignores partitioning
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche closed pull request #13062: ARROW-16420: [Python] pq.write_to_dataset always ignores partitioning
URL: https://github.com/apache/arrow/pull/13062
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] AlenkaF commented on pull request #13062: ARROW-16420: [Python] pq.write_to_dataset always ignores partitioning
Posted by GitBox <gi...@apache.org>.
AlenkaF commented on PR #13062:
URL: https://github.com/apache/arrow/pull/13062#issuecomment-1130224083
Sorry, am a bit distracted by other issues.
Let's do a follow-up so this PR can get closed. Will create a JIRA for it today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #13062: ARROW-16420: [Python] pq.write_to_dataset always ignores partitioning
Posted by GitBox <gi...@apache.org>.
lidavidm commented on PR #13062:
URL: https://github.com/apache/arrow/pull/13062#issuecomment-1117234872
Thanks!
Is it possible to add tests for these?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13062: ARROW-16420: [Python] pq.write_to_dataset always ignores partitioning
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13062:
URL: https://github.com/apache/arrow/pull/13062#issuecomment-1117122246
:warning: Ticket **has not been started in JIRA**, please click 'Start Progress'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] AlenkaF commented on pull request #13062: ARROW-16420: [Python] pq.write_to_dataset always ignores partitioning
Posted by GitBox <gi...@apache.org>.
AlenkaF commented on PR #13062:
URL: https://github.com/apache/arrow/pull/13062#issuecomment-1122156993
Yes, that makes sense. Will do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] AlenkaF commented on pull request #13062: ARROW-16420: [Python] pq.write_to_dataset always ignores partitioning
Posted by GitBox <gi...@apache.org>.
AlenkaF commented on PR #13062:
URL: https://github.com/apache/arrow/pull/13062#issuecomment-1130387551
Created a JIRA for the follow-up:
https://issues.apache.org/jira/browse/ARROW-16610
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org