You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Rajesh Balamohan (Jira)" <ji...@apache.org> on 2020/12/16 11:18:00 UTC
[jira] [Created] (HIVE-24546) Avoid unwanted cloud storage call
during dynamic partition load
Rajesh Balamohan created HIVE-24546:
---------------------------------------
Summary: Avoid unwanted cloud storage call during dynamic partition load
Key: HIVE-24546
URL: https://issues.apache.org/jira/browse/HIVE-24546
Project: Hive
Issue Type: Improvement
Reporter: Rajesh Balamohan
{code:java}
private void createDpDirCheckSrc(final Path dpStagingPath, final Path dpFinalPath) throws IOException {
if (!fs.exists(dpStagingPath) && !fs.exists(dpFinalPath)) {
fs.mkdirs(dpStagingPath);
// move task will create dp final path
if (reporter != null) {
reporter.incrCounter(counterGroup, Operator.HIVE_COUNTER_CREATED_DYNAMIC_PARTITIONS, 1);
}
}
}
{code}
{noformat}
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:370)
at org.apache.hadoop.fs.s3a.S3AFileSystem.listObjects(S3AFileSystem.java:1960)
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3164)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3031)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2899)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1723)
at org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:4157)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createDpDir(FileSinkOperator.java:948)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.updateDPCounters(FileSinkOperator.java:916)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:849)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:814)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createNewPaths(FileSinkOperator.java:1200)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:1324)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:1036)
at org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator.process(VectorFileSinkOperator.java:111)
at org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:969)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)