You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/10/20 14:06:30 UTC
[GitHub] [iceberg] hzxiongyinke opened a new pull request #3328: Hive handler estimate file size
hzxiongyinke opened a new pull request #3328:
URL: https://github.com/apache/iceberg/pull/3328
### Hive query on Iceberg tables
MapRedTask will get full file inputSummary to setNumberOfReducers .
when Iceberg table is big, getContentSummary operator will put a lot of pressure on the namenode.
so we need estimate the file total size in HiveIcebergStorageHandler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #3328: Hive handler estimate file size
Posted by GitBox <gi...@apache.org>.
pvary commented on pull request #3328:
URL: https://github.com/apache/iceberg/pull/3328#issuecomment-949396398
@hzxiongyinke: I have seen several serious issues with 2.1.1.
We try the `InputEstimator` way, and it was not helping. We had to set the basic stats and had to move forward with https://issues.apache.org/jira/browse/HIVE-24928
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hzxiongyinke commented on pull request #3328: Hive handler estimate file size
Posted by GitBox <gi...@apache.org>.
hzxiongyinke commented on pull request #3328:
URL: https://github.com/apache/iceberg/pull/3328#issuecomment-949412896
ok,I will it ,Thanks !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #3328: Hive handler estimate file size
Posted by GitBox <gi...@apache.org>.
pvary commented on pull request #3328:
URL: https://github.com/apache/iceberg/pull/3328#issuecomment-949533586
Oh... and if implementing the Estimator helps with older Hive releases, then I am not against merging it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #3328: Hive handler estimate file size
Posted by GitBox <gi...@apache.org>.
pvary commented on pull request #3328:
URL: https://github.com/apache/iceberg/pull/3328#issuecomment-949532847
Sorry for the many short comments (not too much time today), but let me summarise:
- We should double check everything which I am stating here for Hive 2.3.8/9, and 3.1.2 - We were working on Hive 4.0.0 when considered stats.
- We found that the estimation causes issues for tables with plenty of files. If we turn off `hive.stats.estimate` then we do not end up listing the directories recursively, so that could fix the planning performance issue
- We found that if the `hive.stats.autogather` is true, then the statistics are collected, but there was problem with the `rowDataSize`. We fixed this in HIVE-24928 using the Iceberg table statistics where we propagate the Iceberg statistics to use as Hive statistics.
- For automatic Column statistics we need: HIVE-25276
- We still has to consider other engines writing these tables, and we have to invalidate column statistics, if other engine is update the table. For this we created this change: HIVE-25286
I hope this finally helps 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary edited a comment on pull request #3328: Hive handler estimate file size
Posted by GitBox <gi...@apache.org>.
pvary edited a comment on pull request #3328:
URL: https://github.com/apache/iceberg/pull/3328#issuecomment-949396398
@hzxiongyinke: I have seen several serious issues with 2.1.1 unrelated to this, so you still might have other issues, even if this was fixed. Could you try 2.3.8/9 instead?
We tried the `InputEstimator` way, and it was not helping. We had to set the basic stats and had to move forward with https://issues.apache.org/jira/browse/HIVE-24928
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hzxiongyinke commented on pull request #3328: Hive handler estimate file size
Posted by GitBox <gi...@apache.org>.
hzxiongyinke commented on pull request #3328:
URL: https://github.com/apache/iceberg/pull/3328#issuecomment-950465993
I will use the information you gave to explore, and then evaluate whether this pr is necessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #3328: Hive handler estimate file size
Posted by GitBox <gi...@apache.org>.
pvary commented on pull request #3328:
URL: https://github.com/apache/iceberg/pull/3328#issuecomment-949525086
Also you can try to turn off the estimation by setting `set hive.stats.estimate=false` as a workaround in the meantime
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #3328: Hive handler estimate file size
Posted by GitBox <gi...@apache.org>.
pvary commented on pull request #3328:
URL: https://github.com/apache/iceberg/pull/3328#issuecomment-948522723
I am not sure about the Hive 2.3, and 3.1, but we did this in Hive upstream to get stats:
- https://issues.apache.org/jira/browse/HIVE-24928
- https://issues.apache.org/jira/browse/HIVE-25276
- https://issues.apache.org/jira/browse/HIVE-25286
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hzxiongyinke commented on pull request #3328: Hive handler estimate file size
Posted by GitBox <gi...@apache.org>.
hzxiongyinke commented on pull request #3328:
URL: https://github.com/apache/iceberg/pull/3328#issuecomment-949378517
We used hive-2.1.1 for iceberg-0.10.0 , have a problem with getcontantSummary for quey iceberg table data location,which put a lot of pressure on Namenode. I'm not sure if I should do getcontantSummary on the iceberg table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org