You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/01/20 06:23:31 UTC
[GitHub] [incubator-doris] HangyuanLiu opened a new pull request #2812: Fix
parquet arrow read batch bug
HangyuanLiu opened a new pull request #2812: Fix parquet arrow read batch bug
URL: https://github.com/apache/incubator-doris/pull/2812
Fix parquet arrow read batch bug
#2811
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] HangyuanLiu commented on issue #2812: Fix parquet
arrow read batch bug
Posted by GitBox <gi...@apache.org>.
HangyuanLiu commented on issue #2812: Fix parquet arrow read batch bug
URL: https://github.com/apache/incubator-doris/pull/2812#issuecomment-576143120
The original code was to determine the number of rows in the batch based on the number of rows in the parquet RowGroup.
It could be a batch as many as a rowgroup in last arrow lib?
But now it's a batch take 65535 lines , so when parquet row greater than 65535，be may crash
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on issue #2812: Fix parquet
arrow read batch bug
Posted by GitBox <gi...@apache.org>.
chaoyli commented on issue #2812: Fix parquet arrow read batch bug
URL: https://github.com/apache/incubator-doris/pull/2812#issuecomment-576141567
I have a question, why it will be crashed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2812:
Fix parquet arrow read batch bug
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2812: Fix parquet arrow read batch bug
URL: https://github.com/apache/incubator-doris/pull/2812#discussion_r368575295
##########
File path: be/src/exec/parquet_reader.h
##########
@@ -103,6 +103,7 @@ class ParquetReaderWrap {
int _rows_of_group; // rows in a group.
int _current_line_of_group;
+ int _current_line_of_batch;
Review comment:
can you give some comments for these two variables?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay merged pull request #2812: Fix parquet arrow
read batch bug
Posted by GitBox <gi...@apache.org>.
imay merged pull request #2812: Fix parquet arrow read batch bug
URL: https://github.com/apache/incubator-doris/pull/2812
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org