You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2022/10/06 06:14:24 UTC
[GitHub] [hive] atsaonerk opened a new pull request, #3644: HIVE-26437: Dump unpartitioned tables in parallel.
atsaonerk opened a new pull request, #3644:
URL: https://github.com/apache/hive/pull/3644
Currently partitions of table is dump in parallel manner. But if table is not partitioned, it is dumped serially.
This change introduces parallelism at table level as well. A single thread pool which is currently being used
for partition level is also used for table level. The table level dump task is added to same thread pool.
The degree of parallelism depends upon config parameter REPL_PARTITIONS_DUMP_PARALLELISM whose defaul value is 100.
The new ExportService is introduced with this change which would be responsible for exporting table and partitions during repl dump. The ExportService is initialized and configured with thread pools by HiveServer2 service.
A new Hiveconfig variable ie "REPL_TABLE_DUMP_PARALLELISM is introduced to define the number of threads which would be created in thread pool.
The ExportService which is created as singleton instance would be used by ReplDumpTask.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pudidic commented on pull request #3644: HIVE-26437: Dump unpartitioned tables in parallel.
Posted by GitBox <gi...@apache.org>.
pudidic commented on PR #3644:
URL: https://github.com/apache/hive/pull/3644#issuecomment-1286547794
+1 Looks good to me. I tested on my laptop and the failing test passed. Please re-trigger the CI system to make sure it works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] atsaonerk commented on pull request #3644: HIVE-26437: Dump unpartitioned tables in parallel.
Posted by GitBox <gi...@apache.org>.
atsaonerk commented on PR #3644:
URL: https://github.com/apache/hive/pull/3644#issuecomment-1298419541
<meta charset="utf-8"><b style="font-weight:normal;" id="docs-internal-guid-329ca8fe-7fff-05d0-74b7-0602c56ca25c"><ol style="margin-top:0;margin-bottom:0;padding-inline-start:48px;"><li dir="ltr" style="list-style-type:decimal;font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;" aria-level="1"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;" role="presentation"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">Jmh Performance Benchmark Test </span><span style="font-size:10pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre
;white-space:pre-wrap;">(itests/hive-jmh/src/main/java/org/apache/hive/benchmark/ql/exec/TableAndPartitionExportBench.java)</span></p></li></ol><p dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align: justify;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:10pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">The test runs in two modes, one is parallel and other is serial. The parallel mode makes use of ExportService and creates 100 threads to execute a total of 500 tasks. The serial mode does not use ExportService and runs in single threaded mode to execute 500 tasks. The comparison is made between both runs. The total 5 iterations of each mode is run by the test and time of run is measured in milliseconds. The average time required for 5 iterations is then output as a benchmark for the operations.</span></p><br />
<br /><p dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align: justify;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:10pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">Results:-</span></p><p dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align: justify;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:10pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"></span><span style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">Results of jmh performance benchmark test indicates an
improvement in TableExport operation. If the table is dumped in parallel instead of serial , the operation completes much faster. Below is the result seen when 500 tableexport operations are done both in serial and parallel manner.</span></p><p dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align: justify;background-color:#ffffff;margin-top:0pt;margin-bottom:0pt;padding:0pt 0pt 12pt 0pt;"><span style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">Result "org.apache.hive.benchmark.ql.exec.TableAndPartitionExportBench.BaseBench.parallel":</span></p><p dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align: justify;background-color:#ffffff;margin-top:0pt;margin-bottom:0pt;padding:0pt 0pt 12pt 0pt;"><span style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight
:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">N = 5</span></p><p dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align: justify;background-color:#ffffff;margin-top:0pt;margin-bottom:0pt;padding:0pt 0pt 12pt 0pt;"><span style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">mean = 640.862 ?(99.9%) 113.354 ms/op</span></p><p dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align: justify;background-color:#ffffff;margin-top:0pt;margin-bottom:0pt;padding:0pt 0pt 12pt 0pt;"><span style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">Result "org.apache.hive.benchma
rk.ql.exec.TableAndPartitionExportBench.BaseBench.serial":</span></p><p dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align: justify;background-color:#ffffff;margin-top:0pt;margin-bottom:0pt;padding:0pt 0pt 12pt 0pt;"><span style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">N = 5</span></p><p dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align: justify;background-color:#ffffff;margin-top:0pt;margin-bottom:0pt;padding:0pt 0pt 12pt 0pt;"><span style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">mean = 51697.322 ?(99.9%) 322.747 ms/op</span></p><p dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align: justify;background-c
olor:#ffffff;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">Benchmark </span><span style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"><span class="Apple-tab-span" style="white-space:pre;"> </span></span><span style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"><span class="Apple-tab-span" style="white-space:pre;"> </span></span><span style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:700;font-style:norma
l;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"><span class="Apple-tab-span" style="white-space:pre;"> </span></span><span style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"><span class="Apple-tab-span" style="white-space:pre;"> </span></span><span style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"><span class="Apple-tab-span" style="white-space:pre;"> </span></span><span style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"> M
ode Cnt Score Error Units</span></p><p dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align: justify;background-color:#ffffff;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">TableAndPartitionExportBench.BaseBench.parallel ss 5 640.862 ? 113.354 ms/op</span></p><p dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align: justify;background-color:#ffffff;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">TableAndPartitionExportBench.BaseBench.serial ss 5 51697.322 ?
322.747 ms/op</span></p><p dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align: justify;background-color:#ffffff;margin-top:0pt;margin-bottom:0pt;"> </p><p dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align: justify;background-color:#ffffff;margin-top:0pt;margin-bottom:0pt;"> </p><ol style="margin-top:0;margin-bottom:0;padding-inline-start:48px;" start="2"><li dir="ltr" style="list-style-type:decimal;font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;" aria-level="1"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;" role="presentation"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">End-End Performance benchm
ark number </span></p></li></ol><p dir="ltr" style="line-height:1.38;margin-left: 36pt;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"> </span></p><p dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align: justify;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:10pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">A database is created with 1k managed acid tables which are all unpartitioned tables.</span></p><p dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align: justify;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:10pt;font-family:Arial;color:#000000;background-color:transp
arent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">The config parameter REPL_TABLE_DUMP_PARALLELISM value of 100 is set before the replication dump command is executed. It is found that replication dump takes 9 sec to complete table metadata dump with new Export service. </span></p><p dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align: justify;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:10pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">When export service is not used it is found that the same number of tables took around 27 sec to complete the entire dump process. Hence it is seen that there is 3x improvement in performance of replication dump command execution.</span></p><br /><div dir="ltr" style="margin-lef
t:36pt;" align="left">
No. of Tables | REPL_TABLE_DUMP_PARALLELISM | Export Service used | Time taken for REPL DUMP
-- | -- | -- | --
1000 | 100 | Yes | 9 sec
1000 | 100 | No | 27 sec
</div><br /></b>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3644: HIVE-26437: Dump unpartitioned tables in parallel.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3644:
URL: https://github.com/apache/hive/pull/3644#issuecomment-1269645392
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3644)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=BUG) [10 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3644&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3644&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3644&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=CODE_SMELL) [77 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3644&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3644&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] atsaonerk commented on a diff in pull request #3644: HIVE-26437: Dump unpartitioned tables in parallel.
Posted by GitBox <gi...@apache.org>.
atsaonerk commented on code in PR #3644:
URL: https://github.com/apache/hive/pull/3644#discussion_r1000183331
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java:
##########
@@ -978,6 +981,22 @@ private Long incrementalDump(Path dumpRoot, DumpMetaData dmd, Path cmRoot, Hive
LOG.debug(te.getMessage());
}
}
+
+ if (exportService != null && exportService.isExportServiceRunning()) {
Review Comment:
Done
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java:
##########
@@ -952,6 +954,7 @@ private Long incrementalDump(Path dumpRoot, DumpMetaData dmd, Path cmRoot, Hive
}
}
for(String matchedDbName : Utils.matchesDb(hiveDb, work.dbNameOrPattern)) {
+ ExportService exportService = new ExportService(conf);
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3644: HIVE-26437: Dump unpartitioned tables in parallel.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3644:
URL: https://github.com/apache/hive/pull/3644#issuecomment-1286660394
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3644)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=BUG) [6 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3644&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3644&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3644&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=CODE_SMELL) [22 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3644&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3644&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pudidic merged pull request #3644: HIVE-26437: Dump unpartitioned tables in parallel.
Posted by GitBox <gi...@apache.org>.
pudidic merged PR #3644:
URL: https://github.com/apache/hive/pull/3644
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] atsaonerk commented on a diff in pull request #3644: HIVE-26437: Dump unpartitioned tables in parallel.
Posted by GitBox <gi...@apache.org>.
atsaonerk commented on code in PR #3644:
URL: https://github.com/apache/hive/pull/3644#discussion_r1000183663
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java:
##########
@@ -1254,6 +1273,7 @@ Long bootStrapDump(Path dumpRoot, DumpMetaData dmd, Path cmRoot, Hive hiveDb)
// Get the counter to store the snapshots created & deleted at source.
replSnapshotCount = new SnapshotUtils.ReplSnapshotCount();
}
+ ExportService exportService = new ExportService(conf);
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3644: HIVE-26437: Dump unpartitioned tables in parallel.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3644:
URL: https://github.com/apache/hive/pull/3644#issuecomment-1277235317
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3644)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=BUG) [6 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3644&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3644&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3644&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=CODE_SMELL) [23 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3644&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3644&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3644: HIVE-26437: Dump unpartitioned tables in parallel.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3644:
URL: https://github.com/apache/hive/pull/3644#issuecomment-1277720917
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3644)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=BUG) [6 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3644&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3644&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3644&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=CODE_SMELL) [22 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3644&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3644&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] rbalamohan commented on pull request #3644: HIVE-26437: Dump unpartitioned tables in parallel.
Posted by GitBox <gi...@apache.org>.
rbalamohan commented on PR #3644:
URL: https://github.com/apache/hive/pull/3644#issuecomment-1306653148
LGTM. +1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pudidic commented on a diff in pull request #3644: HIVE-26437: Dump unpartitioned tables in parallel.
Posted by GitBox <gi...@apache.org>.
pudidic commented on code in PR #3644:
URL: https://github.com/apache/hive/pull/3644#discussion_r1000168804
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java:
##########
@@ -1254,6 +1273,7 @@ Long bootStrapDump(Path dumpRoot, DumpMetaData dmd, Path cmRoot, Hive hiveDb)
// Get the counter to store the snapshots created & deleted at source.
replSnapshotCount = new SnapshotUtils.ReplSnapshotCount();
}
+ ExportService exportService = new ExportService(conf);
Review Comment:
Please move the ExportService object creation higher than a database level.
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java:
##########
@@ -1290,6 +1310,21 @@ Long bootStrapDump(Path dumpRoot, DumpMetaData dmd, Path cmRoot, Hive hiveDb)
}
}
+ if (exportService != null && exportService.isExportServiceRunning()) {
Review Comment:
Please move the ExportService object creation higher than a database level.
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java:
##########
@@ -952,6 +954,7 @@ private Long incrementalDump(Path dumpRoot, DumpMetaData dmd, Path cmRoot, Hive
}
}
for(String matchedDbName : Utils.matchesDb(hiveDb, work.dbNameOrPattern)) {
+ ExportService exportService = new ExportService(conf);
Review Comment:
Please move the ExportService object creation higher than a database level.
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java:
##########
@@ -978,6 +981,22 @@ private Long incrementalDump(Path dumpRoot, DumpMetaData dmd, Path cmRoot, Hive
LOG.debug(te.getMessage());
}
}
+
+ if (exportService != null && exportService.isExportServiceRunning()) {
Review Comment:
Please move the ExportService object creation higher than a database level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3644: HIVE-26437: Dump unpartitioned tables in parallel.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3644:
URL: https://github.com/apache/hive/pull/3644#issuecomment-1285050984
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3644)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=BUG) [6 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3644&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3644&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3644&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=CODE_SMELL) [23 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3644&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3644&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pudidic commented on pull request #3644: HIVE-26437: Dump unpartitioned tables in parallel.
Posted by GitBox <gi...@apache.org>.
pudidic commented on PR #3644:
URL: https://github.com/apache/hive/pull/3644#issuecomment-1288395487
+1. LGTM. The failing test passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3644: HIVE-26437: Dump unpartitioned tables in parallel.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3644:
URL: https://github.com/apache/hive/pull/3644#issuecomment-1273343443
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3644)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=BUG) [13 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3644&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3644&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3644&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=CODE_SMELL) [83 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3644&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3644&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3644&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org