You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/05/13 21:14:23 UTC
[GitHub] [beam] brills opened a new pull request #11699: Update the range for pyarrow to qualify pyarrow 0.17.x
brills opened a new pull request #11699:
URL: https://github.com/apache/beam/pull/11699
R: @aaltay
R: @theneuralbit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11699: Update the range for pyarrow to qualify pyarrow 0.17.x
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11699:
URL: https://github.com/apache/beam/pull/11699#issuecomment-628880849
Run PythonDocker PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11699: Update the range for pyarrow to qualify pyarrow 0.17.x
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11699:
URL: https://github.com/apache/beam/pull/11699#issuecomment-628275313
Bunch of parquet tests are failing. This is probably related to a chance in pyarrow.
/cc @chamikaramj
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11699: Update the range for pyarrow to qualify pyarrow 0.17.x
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11699:
URL: https://github.com/apache/beam/pull/11699#issuecomment-628312418
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay merged pull request #11699: Update the range for pyarrow to qualify pyarrow 0.17.x
Posted by GitBox <gi...@apache.org>.
aaltay merged pull request #11699:
URL: https://github.com/apache/beam/pull/11699
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay edited a comment on pull request #11699: Update the range for pyarrow to qualify pyarrow 0.17.x
Posted by GitBox <gi...@apache.org>.
aaltay edited a comment on pull request #11699:
URL: https://github.com/apache/beam/pull/11699#issuecomment-628275313
Bunch of parquet tests are failing. This is probably related to a chance in pyarrow. (Test link: https://builds.apache.org/job/beam_PreCommit_Python_Commit/12847/)
/cc @chamikaramj
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #11699: Update the range for pyarrow to qualify pyarrow 0.17.x
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #11699:
URL: https://github.com/apache/beam/pull/11699#discussion_r424788208
##########
File path: sdks/python/apache_beam/io/parquetio.py
##########
@@ -567,5 +567,6 @@ def _flush_buffer(self):
size = 0
for x in arrays:
for b in x.buffers():
- size = size + b.size
+ if b is not None:
+ size = size + b.size
Review comment:
Thanks for tracking this down!
I wonder what changed in 0.17 to reveal this. Maybe the parquet writer (which we use to generate test data) wasn't eliding the null buffer before?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11699: Update the range for pyarrow to qualify pyarrow 0.17.x
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11699:
URL: https://github.com/apache/beam/pull/11699#issuecomment-628252175
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11699: Update the range for pyarrow to qualify pyarrow 0.17.x
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11699:
URL: https://github.com/apache/beam/pull/11699#issuecomment-628311774
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11699: Update the range for pyarrow to qualify pyarrow 0.17.x
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11699:
URL: https://github.com/apache/beam/pull/11699#issuecomment-628252281
I can merge after tests pass.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11699: Update the range for pyarrow to qualify pyarrow 0.17.x
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11699:
URL: https://github.com/apache/beam/pull/11699#issuecomment-628311886
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brills commented on a change in pull request #11699: Update the range for pyarrow to qualify pyarrow 0.17.x
Posted by GitBox <gi...@apache.org>.
brills commented on a change in pull request #11699:
URL: https://github.com/apache/beam/pull/11699#discussion_r424855320
##########
File path: sdks/python/apache_beam/io/parquetio.py
##########
@@ -567,5 +567,6 @@ def _flush_buffer(self):
size = 0
for x in arrays:
for b in x.buffers():
- size = size + b.size
+ if b is not None:
+ size = size + b.size
Review comment:
Maybe the array factory function (pa.array()) is smarter (`x` there was created just several lines ago, using that factory function)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11699: Update the range for pyarrow to qualify pyarrow 0.17.x
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11699:
URL: https://github.com/apache/beam/pull/11699#issuecomment-628777431
Run PythonDocker PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brills commented on pull request #11699: Update the range for pyarrow to qualify pyarrow 0.17.x
Posted by GitBox <gi...@apache.org>.
brills commented on pull request #11699:
URL: https://github.com/apache/beam/pull/11699#issuecomment-628768038
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] brills commented on pull request #11699: Update the range for pyarrow to qualify pyarrow 0.17.x
Posted by GitBox <gi...@apache.org>.
brills commented on pull request #11699:
URL: https://github.com/apache/beam/pull/11699#issuecomment-628293443
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11699: Update the range for pyarrow to qualify pyarrow 0.17.x
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11699:
URL: https://github.com/apache/beam/pull/11699#issuecomment-628770474
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org