You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@phoenix.apache.org by GitBox <gi...@apache.org> on 2022/05/09 14:34:59 UTC
[GitHub] [phoenix-connectors] jichen20210919 opened a new pull request, #79: PHOENIX-6698 hive-connector will take long time to generate splits fo…
jichen20210919 opened a new pull request, #79:
URL: https://github.com/apache/phoenix-connectors/pull/79
This patch enables PhoenixInputFormat to generate splits in parallel, it introduce two parameters to control the degree of parallelism.
1.'hive.phoenix.split.parallel.threshold' is used to contrl if split should be generated in parallel.it will generate splits in serial for following condition:
(1) hive.phoenix.split.parallel.threshold<0, it will generate split in serial.
(2) number of scans in query plan is less than the value setting.
in other conditions, it will generate split in parallel.
2. hive.phoenix.split.parallel.level
is used to control the number of work threads for the splits.(2*cpu cores by default).
A unit test is created for unit test, the test case will compare the time cost of generating split for phoenix table with 128 regions.
the output shows that: parallel method is 6x faster than serial method, and it will be better for tables with more regions
```
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/jichen/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.10.0/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/jichen/.m2/repository/org/slf4j/slf4j-log4j12/1.7.30/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Formatting using clusterid: testClusterID
generate testing table with 128 splits
get split in serial requires:12843 ms
get split in parallel requires:2728 ms
```
in production environment, we have tested the time cost for table with 2048 regions, it reduces time cost from nearly 30 mins to 2 mins with default configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] stoty commented on pull request #79: PHOENIX-6698 hive-connector will take long time to generate splits fo…
Posted by GitBox <gi...@apache.org>.
stoty commented on PR #79:
URL: https://github.com/apache/phoenix-connectors/pull/79#issuecomment-1137615034
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 56s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 12s | master passed |
| +1 :green_heart: | compile | 1m 9s | master passed |
| +1 :green_heart: | checkstyle | 0m 19s | master passed |
| +1 :green_heart: | javadoc | 0m 26s | master passed |
| +0 :ok: | spotbugs | 2m 7s | phoenix-hive-base in master has 46 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 28s | the patch passed |
| +1 :green_heart: | compile | 1m 12s | the patch passed |
| +1 :green_heart: | javac | 1m 12s | the patch passed |
| -1 :x: | checkstyle | 0m 19s | phoenix-hive-base: The patch generated 43 new + 130 unchanged - 32 fixed = 173 total (was 162) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 26s | the patch passed |
| +1 :green_heart: | spotbugs | 2m 0s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 55m 34s | phoenix-hive-base in the patch failed. |
| -1 :x: | asflicense | 0m 16s | The patch generated 2 ASF License warnings. |
| | | 127m 3s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | phoenix.hive.HiveTezIT |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/10/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix-connectors/pull/79 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile |
| uname | Linux a59573105523 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-connectors-personality.sh |
| git revision | master / 9ed127f |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/10/artifact/yetus-general-check/output/diff-checkstyle-phoenix-hive-base.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/10/artifact/yetus-general-check/output/patch-unit-phoenix-hive-base.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/10/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/10/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 1671 (vs. ulimit of 30000) |
| modules | C: phoenix-hive-base U: phoenix-hive-base |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/10/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] jichen20210919 commented on a diff in pull request #79: PHOENIX-6698 hive-connector will take long time to generate splits fo…
Posted by GitBox <gi...@apache.org>.
jichen20210919 commented on code in PR #79:
URL: https://github.com/apache/phoenix-connectors/pull/79#discussion_r890693820
##########
phoenix-hive-base/src/test/java/org/apache/phoenix/hive/HivePhoenixInputFormatTest.java:
##########
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hive;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.Locale;
+import java.util.Properties;
+
+import javax.annotation.concurrent.NotThreadSafe;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.mapred.InputSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.phoenix.end2end.ParallelStatsDisabledIT;
+import org.apache.phoenix.end2end.ParallelStatsDisabledTest;
+import org.apache.phoenix.hive.constants.PhoenixStorageHandlerConstants;
+import org.apache.phoenix.hive.mapreduce.PhoenixInputFormat;
+import org.apache.phoenix.mapreduce.PhoenixRecordWritable;
+import org.apache.phoenix.schema.TableAlreadyExistsException;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+
+/**
+ * Test class for Hive PhoenixInputFormat
+ */
+@NotThreadSafe
+@Category(ParallelStatsDisabledTest.class)
+public class HivePhoenixInputFormatTest extends ParallelStatsDisabledIT {
Review Comment:
Q:Do the existing Phoenix-hive tests activate your new property and implicitly validate that it is functional?
A:yes, will implicitly validate its functionality using default settings of new properties.
Q:I think we have some test classes but do we create multi-region Phoenix tables in those tests (or those with enough data to have multiple guideposts)?
A:have discussed with chrajeshbabu and update Test Class based on ParallelStatsEnabledIT instead of ParallelStatsDisabledIT, and create tables with multi-regions and multiple guideposts .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] stoty commented on pull request #79: PHOENIX-6698 hive-connector will take long time to generate splits fo…
Posted by GitBox <gi...@apache.org>.
stoty commented on PR #79:
URL: https://github.com/apache/phoenix-connectors/pull/79#issuecomment-1138175383
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 55s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 31s | master passed |
| +1 :green_heart: | compile | 1m 12s | master passed |
| +1 :green_heart: | checkstyle | 0m 19s | master passed |
| +1 :green_heart: | javadoc | 0m 25s | master passed |
| +0 :ok: | spotbugs | 2m 9s | phoenix-hive-base in master has 46 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 34s | the patch passed |
| +1 :green_heart: | compile | 1m 12s | the patch passed |
| +1 :green_heart: | javac | 1m 12s | the patch passed |
| -1 :x: | checkstyle | 0m 19s | phoenix-hive-base: The patch generated 42 new + 130 unchanged - 32 fixed = 172 total (was 162) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 26s | the patch passed |
| +1 :green_heart: | spotbugs | 1m 56s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 56m 9s | phoenix-hive-base in the patch failed. |
| -1 :x: | asflicense | 0m 16s | The patch generated 2 ASF License warnings. |
| | | 130m 8s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | phoenix.hive.HiveTezIT |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/12/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix-connectors/pull/79 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile |
| uname | Linux 2b96720d70c4 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-connectors-personality.sh |
| git revision | master / 9ed127f |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/12/artifact/yetus-general-check/output/diff-checkstyle-phoenix-hive-base.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/12/artifact/yetus-general-check/output/patch-unit-phoenix-hive-base.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/12/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/12/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 1585 (vs. ulimit of 30000) |
| modules | C: phoenix-hive-base U: phoenix-hive-base |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/12/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] stoty commented on pull request #79: PHOENIX-6698 hive-connector will take long time to generate splits fo…
Posted by GitBox <gi...@apache.org>.
stoty commented on PR #79:
URL: https://github.com/apache/phoenix-connectors/pull/79#issuecomment-1121330486
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 5m 36s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 33s | master passed |
| +1 :green_heart: | compile | 1m 5s | master passed |
| +1 :green_heart: | checkstyle | 0m 19s | master passed |
| +1 :green_heart: | javadoc | 0m 26s | master passed |
| +0 :ok: | spotbugs | 2m 2s | phoenix-hive-base in master has 46 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 19m 27s | the patch passed |
| +1 :green_heart: | compile | 1m 7s | the patch passed |
| +1 :green_heart: | javac | 1m 7s | the patch passed |
| -1 :x: | checkstyle | 0m 19s | phoenix-hive-base: The patch generated 61 new + 57 unchanged - 27 fixed = 118 total (was 84) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 27s | the patch passed |
| +1 :green_heart: | spotbugs | 1m 55s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 56m 0s | phoenix-hive-base in the patch failed. |
| -1 :x: | asflicense | 0m 19s | The patch generated 2 ASF License warnings. |
| | | 124m 18s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | phoenix.hive.HiveTezIT |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix-connectors/pull/79 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile |
| uname | Linux b463efd6004b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-connectors-personality.sh |
| git revision | master / 9ed127f |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/1/artifact/yetus-general-check/output/diff-checkstyle-phoenix-hive-base.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/1/artifact/yetus-general-check/output/patch-unit-phoenix-hive-base.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/1/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/1/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 1826 (vs. ulimit of 30000) |
| modules | C: phoenix-hive-base U: phoenix-hive-base |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/1/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] stoty commented on pull request #79: PHOENIX-6698 hive-connector will take long time to generate splits fo…
Posted by GitBox <gi...@apache.org>.
stoty commented on PR #79:
URL: https://github.com/apache/phoenix-connectors/pull/79#issuecomment-1136750700
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 58s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 24s | master passed |
| +1 :green_heart: | compile | 1m 10s | master passed |
| +1 :green_heart: | checkstyle | 0m 19s | master passed |
| +1 :green_heart: | javadoc | 0m 27s | master passed |
| +0 :ok: | spotbugs | 2m 10s | phoenix-hive-base in master has 46 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 12s | the patch passed |
| +1 :green_heart: | compile | 1m 10s | the patch passed |
| +1 :green_heart: | javac | 1m 10s | the patch passed |
| -1 :x: | checkstyle | 0m 20s | phoenix-hive-base: The patch generated 45 new + 130 unchanged - 32 fixed = 175 total (was 162) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 26s | the patch passed |
| +1 :green_heart: | spotbugs | 2m 0s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 56m 3s | phoenix-hive-base in the patch failed. |
| -1 :x: | asflicense | 0m 15s | The patch generated 2 ASF License warnings. |
| | | 127m 36s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | phoenix.hive.HiveTezIT |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/8/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix-connectors/pull/79 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile |
| uname | Linux 0f8752719cde 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-connectors-personality.sh |
| git revision | master / 9ed127f |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/8/artifact/yetus-general-check/output/diff-checkstyle-phoenix-hive-base.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/8/artifact/yetus-general-check/output/patch-unit-phoenix-hive-base.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/8/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/8/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 1671 (vs. ulimit of 30000) |
| modules | C: phoenix-hive-base U: phoenix-hive-base |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/8/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] stoty commented on pull request #79: PHOENIX-6698 hive-connector will take long time to generate splits fo…
Posted by GitBox <gi...@apache.org>.
stoty commented on PR #79:
URL: https://github.com/apache/phoenix-connectors/pull/79#issuecomment-1129600452
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 6m 25s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 24s | master passed |
| +1 :green_heart: | compile | 1m 9s | master passed |
| +1 :green_heart: | checkstyle | 0m 17s | master passed |
| +1 :green_heart: | javadoc | 0m 25s | master passed |
| +0 :ok: | spotbugs | 2m 12s | phoenix-hive-base in master has 46 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 10s | the patch passed |
| +1 :green_heart: | compile | 1m 11s | the patch passed |
| +1 :green_heart: | javac | 1m 11s | the patch passed |
| -1 :x: | checkstyle | 0m 17s | phoenix-hive-base: The patch generated 39 new + 52 unchanged - 32 fixed = 91 total (was 84) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 26s | the patch passed |
| +1 :green_heart: | spotbugs | 2m 1s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 58m 55s | phoenix-hive-base in the patch failed. |
| -1 :x: | asflicense | 0m 12s | The patch generated 2 ASF License warnings. |
| | | 135m 43s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/4/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix-connectors/pull/79 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile |
| uname | Linux 5bb88adec8bd 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-connectors-personality.sh |
| git revision | master / 9ed127f |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/4/artifact/yetus-general-check/output/diff-checkstyle-phoenix-hive-base.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/4/artifact/yetus-general-check/output/patch-unit-phoenix-hive-base.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/4/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/4/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 1691 (vs. ulimit of 30000) |
| modules | C: phoenix-hive-base U: phoenix-hive-base |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/4/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] stoty commented on pull request #79: PHOENIX-6698 hive-connector will take long time to generate splits fo…
Posted by GitBox <gi...@apache.org>.
stoty commented on PR #79:
URL: https://github.com/apache/phoenix-connectors/pull/79#issuecomment-1129909266
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 57s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 51s | master passed |
| +1 :green_heart: | compile | 1m 10s | master passed |
| +1 :green_heart: | checkstyle | 0m 17s | master passed |
| +1 :green_heart: | javadoc | 0m 25s | master passed |
| +0 :ok: | spotbugs | 2m 6s | phoenix-hive-base in master has 46 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 23s | the patch passed |
| +1 :green_heart: | compile | 1m 9s | the patch passed |
| +1 :green_heart: | javac | 1m 9s | the patch passed |
| -1 :x: | checkstyle | 0m 17s | phoenix-hive-base: The patch generated 34 new + 52 unchanged - 32 fixed = 86 total (was 84) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 25s | the patch passed |
| +1 :green_heart: | spotbugs | 1m 58s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 67m 20s | phoenix-hive-base in the patch failed. |
| -1 :x: | asflicense | 0m 12s | The patch generated 2 ASF License warnings. |
| | | 139m 8s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/6/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix-connectors/pull/79 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile |
| uname | Linux 28532a923c04 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-connectors-personality.sh |
| git revision | master / 9ed127f |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/6/artifact/yetus-general-check/output/diff-checkstyle-phoenix-hive-base.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/6/artifact/yetus-general-check/output/patch-unit-phoenix-hive-base.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/6/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/6/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 1686 (vs. ulimit of 30000) |
| modules | C: phoenix-hive-base U: phoenix-hive-base |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/6/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] stoty commented on pull request #79: PHOENIX-6698 hive-connector will take long time to generate splits fo…
Posted by GitBox <gi...@apache.org>.
stoty commented on PR #79:
URL: https://github.com/apache/phoenix-connectors/pull/79#issuecomment-1123632464
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 26s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 13s | master passed |
| +1 :green_heart: | compile | 1m 5s | master passed |
| +1 :green_heart: | checkstyle | 0m 18s | master passed |
| +1 :green_heart: | javadoc | 0m 26s | master passed |
| +0 :ok: | spotbugs | 2m 2s | phoenix-hive-base in master has 46 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 19m 10s | the patch passed |
| +1 :green_heart: | compile | 1m 5s | the patch passed |
| +1 :green_heart: | javac | 1m 5s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 18s | phoenix-hive-base: The patch generated 0 new + 52 unchanged - 32 fixed = 52 total (was 84) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 27s | the patch passed |
| +1 :green_heart: | spotbugs | 1m 52s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 41m 21s | phoenix-hive-base in the patch failed. |
| -1 :x: | asflicense | 0m 14s | The patch generated 2 ASF License warnings. |
| | | 103m 42s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/3/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix-connectors/pull/79 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile |
| uname | Linux 32e79c454808 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-connectors-personality.sh |
| git revision | master / 9ed127f |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/3/artifact/yetus-general-check/output/patch-unit-phoenix-hive-base.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/3/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/3/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 1667 (vs. ulimit of 30000) |
| modules | C: phoenix-hive-base U: phoenix-hive-base |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/3/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] joshelser commented on a diff in pull request #79: PHOENIX-6698 hive-connector will take long time to generate splits fo…
Posted by GitBox <gi...@apache.org>.
joshelser commented on code in PR #79:
URL: https://github.com/apache/phoenix-connectors/pull/79#discussion_r882833463
##########
phoenix-hive-base/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixInputFormat.java:
##########
@@ -119,74 +124,144 @@ public InputSplit[] getSplits(JobConf jobConf, int numSplits) throws IOException
}
private List<InputSplit> generateSplits(final JobConf jobConf, final QueryPlan qplan,
- final List<KeyRange> splits, String query) throws
- IOException {
- if (qplan == null){
+ final List<KeyRange> splits, final String query)
+ throws IOException {
+
+ if (qplan == null) {
throw new NullPointerException();
- }if (splits == null){
+ }
+ if (splits == null) {
throw new NullPointerException();
}
final List<InputSplit> psplits = new ArrayList<>(splits.size());
- Path[] tablePaths = FileInputFormat.getInputPaths(ShimLoader.getHadoopShims()
- .newJobContext(new Job(jobConf)));
- boolean splitByStats = jobConf.getBoolean(PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
+ final Path[] tablePaths = FileInputFormat.getInputPaths(
+ ShimLoader.getHadoopShims().newJobContext(new Job(jobConf)));
+ final boolean splitByStats = jobConf.getBoolean(
+ PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
false);
-
+ final int parallelThreshold = jobConf.getInt(
+ PhoenixStorageHandlerConstants.PHOENIX_MINIMUM_PARALLEL_SCANS_THRESHOLD,
+ PhoenixStorageHandlerConstants.DEFAULT_PHOENIX_MINIMUM_PARALLEL_SCANS_THRESHOLD);
setScanCacheSize(jobConf);
+ try (org.apache.hadoop.hbase.client.Connection connection = ConnectionFactory
+ .createConnection(PhoenixConnectionUtil.getConfiguration(jobConf))) {
+ final RegionLocator regionLocator = connection.getRegionLocator(TableName.valueOf(
+ qplan.getTableRef().getTable().getPhysicalName().toString()));
+ final int scanSize = qplan.getScans().size();
+ if (useParallelInputGeneration(parallelThreshold, scanSize)) {
+ final int parallism = jobConf.getInt(
Review Comment:
```suggestion
final int parallelism = jobConf.getInt(
```
##########
phoenix-hive-base/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixInputFormat.java:
##########
@@ -119,74 +124,144 @@ public InputSplit[] getSplits(JobConf jobConf, int numSplits) throws IOException
}
private List<InputSplit> generateSplits(final JobConf jobConf, final QueryPlan qplan,
- final List<KeyRange> splits, String query) throws
- IOException {
- if (qplan == null){
+ final List<KeyRange> splits, final String query)
+ throws IOException {
+
+ if (qplan == null) {
throw new NullPointerException();
- }if (splits == null){
+ }
+ if (splits == null) {
throw new NullPointerException();
}
final List<InputSplit> psplits = new ArrayList<>(splits.size());
- Path[] tablePaths = FileInputFormat.getInputPaths(ShimLoader.getHadoopShims()
- .newJobContext(new Job(jobConf)));
- boolean splitByStats = jobConf.getBoolean(PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
+ final Path[] tablePaths = FileInputFormat.getInputPaths(
+ ShimLoader.getHadoopShims().newJobContext(new Job(jobConf)));
+ final boolean splitByStats = jobConf.getBoolean(
+ PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
false);
-
+ final int parallelThreshold = jobConf.getInt(
+ PhoenixStorageHandlerConstants.PHOENIX_MINIMUM_PARALLEL_SCANS_THRESHOLD,
+ PhoenixStorageHandlerConstants.DEFAULT_PHOENIX_MINIMUM_PARALLEL_SCANS_THRESHOLD);
setScanCacheSize(jobConf);
+ try (org.apache.hadoop.hbase.client.Connection connection = ConnectionFactory
+ .createConnection(PhoenixConnectionUtil.getConfiguration(jobConf))) {
+ final RegionLocator regionLocator = connection.getRegionLocator(TableName.valueOf(
+ qplan.getTableRef().getTable().getPhysicalName().toString()));
+ final int scanSize = qplan.getScans().size();
+ if (useParallelInputGeneration(parallelThreshold, scanSize)) {
+ final int parallism = jobConf.getInt(
+ PhoenixStorageHandlerConstants.PHOENIX_INPUTSPLIT_GENERATION_THREAD_COUNT,
+ PhoenixStorageHandlerConstants
+ .DEFAULT_PHOENIX_INPUTSPLIT_GENERATION_THREAD_COUNT);
+ ExecutorService executorService = Executors.newFixedThreadPool(parallism);
+ LOG.info("Generate Input Splits in Parallel with {} threads", parallism);
- // Adding Localization
- try (org.apache.hadoop.hbase.client.Connection connection = ConnectionFactory.createConnection(PhoenixConnectionUtil.getConfiguration(jobConf))) {
- RegionLocator regionLocator = connection.getRegionLocator(TableName.valueOf(qplan
- .getTableRef().getTable().getPhysicalName().toString()));
-
- for (List<Scan> scans : qplan.getScans()) {
- PhoenixInputSplit inputSplit;
-
- HRegionLocation location = regionLocator.getRegionLocation(scans.get(0).getStartRow()
- , false);
- long regionSize = CompatUtil.getSize(regionLocator, connection.getAdmin(), location);
- String regionLocation = PhoenixStorageHandlerUtil.getRegionLocation(location, LOG);
-
- if (splitByStats) {
- for (Scan aScan : scans) {
- if (LOG.isDebugEnabled()) {
- LOG.debug("Split for scan : " + aScan + "with scanAttribute : " + aScan
- .getAttributesMap() + " [scanCache, cacheBlock, scanBatch] : [" +
- aScan.getCaching() + ", " + aScan.getCacheBlocks() + ", " + aScan
- .getBatch() + "] and regionLocation : " + regionLocation);
- }
+ List<Future<List<InputSplit>>> tasks = new ArrayList<>();
- inputSplit = new PhoenixInputSplit(new ArrayList<>(Arrays.asList(aScan)), tablePaths[0],
- regionLocation, regionSize);
- inputSplit.setQuery(query);
- psplits.add(inputSplit);
+ try {
+ for (final List<Scan> scans : qplan.getScans()) {
+ Future<List<InputSplit>> task = executorService.submit(
+ new Callable<List<InputSplit>>() {
+ @Override public List<InputSplit> call() throws Exception {
+ return generateSplitsInternal(query, scans, splitByStats,
+ connection, regionLocator, tablePaths);
+ }
+ });
+ tasks.add(task);
+ }
+ for (Future<List<InputSplit>> task : tasks) {
+ psplits.addAll(task.get());
+ }
+ } catch (ExecutionException | InterruptedException exception) {
+ throw new IOException("Failed to Generate Input Splits in Parallel, reason:",
+ exception);
Review Comment:
Good to unwrap the ExecutionException and throw back the real exception. It may already be an IOException which you can throw with a cast, rather than rewrapping in another IOException.
##########
phoenix-hive-base/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixInputFormat.java:
##########
@@ -119,74 +124,144 @@ public InputSplit[] getSplits(JobConf jobConf, int numSplits) throws IOException
}
private List<InputSplit> generateSplits(final JobConf jobConf, final QueryPlan qplan,
- final List<KeyRange> splits, String query) throws
- IOException {
- if (qplan == null){
+ final List<KeyRange> splits, final String query)
+ throws IOException {
+
+ if (qplan == null) {
throw new NullPointerException();
- }if (splits == null){
+ }
+ if (splits == null) {
throw new NullPointerException();
}
final List<InputSplit> psplits = new ArrayList<>(splits.size());
- Path[] tablePaths = FileInputFormat.getInputPaths(ShimLoader.getHadoopShims()
- .newJobContext(new Job(jobConf)));
- boolean splitByStats = jobConf.getBoolean(PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
+ final Path[] tablePaths = FileInputFormat.getInputPaths(
+ ShimLoader.getHadoopShims().newJobContext(new Job(jobConf)));
+ final boolean splitByStats = jobConf.getBoolean(
+ PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
false);
-
+ final int parallelThreshold = jobConf.getInt(
+ PhoenixStorageHandlerConstants.PHOENIX_MINIMUM_PARALLEL_SCANS_THRESHOLD,
+ PhoenixStorageHandlerConstants.DEFAULT_PHOENIX_MINIMUM_PARALLEL_SCANS_THRESHOLD);
setScanCacheSize(jobConf);
+ try (org.apache.hadoop.hbase.client.Connection connection = ConnectionFactory
+ .createConnection(PhoenixConnectionUtil.getConfiguration(jobConf))) {
+ final RegionLocator regionLocator = connection.getRegionLocator(TableName.valueOf(
+ qplan.getTableRef().getTable().getPhysicalName().toString()));
+ final int scanSize = qplan.getScans().size();
+ if (useParallelInputGeneration(parallelThreshold, scanSize)) {
+ final int parallism = jobConf.getInt(
+ PhoenixStorageHandlerConstants.PHOENIX_INPUTSPLIT_GENERATION_THREAD_COUNT,
+ PhoenixStorageHandlerConstants
+ .DEFAULT_PHOENIX_INPUTSPLIT_GENERATION_THREAD_COUNT);
+ ExecutorService executorService = Executors.newFixedThreadPool(parallism);
+ LOG.info("Generate Input Splits in Parallel with {} threads", parallism);
Review Comment:
```suggestion
LOG.info("Generating Input Splits in Parallel with {} threads", parallism);
```
##########
phoenix-hive-base/src/test/java/org/apache/phoenix/hive/HivePhoenixInputFormatTest.java:
##########
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hive;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.Locale;
+import java.util.Properties;
+
+import javax.annotation.concurrent.NotThreadSafe;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.mapred.InputSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.phoenix.end2end.ParallelStatsDisabledIT;
+import org.apache.phoenix.end2end.ParallelStatsDisabledTest;
+import org.apache.phoenix.hive.constants.PhoenixStorageHandlerConstants;
+import org.apache.phoenix.hive.mapreduce.PhoenixInputFormat;
+import org.apache.phoenix.mapreduce.PhoenixRecordWritable;
+import org.apache.phoenix.schema.TableAlreadyExistsException;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+
+/**
+ * Test class for Hive PhoenixInputFormat
+ */
+@NotThreadSafe
+@Category(ParallelStatsDisabledTest.class)
+public class HivePhoenixInputFormatTest extends ParallelStatsDisabledIT {
+ private static final Logger LOG = LoggerFactory.getLogger(HivePhoenixInputFormatTest.class);
+ private static final String TABLE_NAME = "HivePhoenixInputFormatTest".toUpperCase(Locale.ROOT);
+ private static final String DDL = "CREATE TABLE " + TABLE_NAME
+ + " (V1 varchar NOT NULL PRIMARY KEY, V2 integer)";
+ private static final int SPLITS = 256;
+
+ // This test will create phoenix table with 128 splits and compare performance of
+ // serial split-generation method and parallel split-generation method.
+ @Test
+ public void testGetSplitsSerialOrParallel() throws IOException, SQLException {
+ PhoenixInputFormat<PhoenixRecordWritable> inputFormat =
+ new PhoenixInputFormat<PhoenixRecordWritable>();
+ long start;
+ long end;
+ // create table with N splits
+ System.out.println(
+ String.format("generate testing table with %s splits", String.valueOf(SPLITS)));
+ setupTestTable();
+ // setup configuration required for PhoenixInputFormat
+ Configuration conf = getUtility().getConfiguration();
+ JobConf jobConf = new JobConf(conf);
+ configureTestInput(jobConf);
+ inputFormat.getSplits(jobConf, SPLITS);
+ InputSplit[] inputSplitsSerial;
+ // test get splits in serial
+ start = System.currentTimeMillis();
+ jobConf.set(PhoenixStorageHandlerConstants.PHOENIX_MINIMUM_PARALLEL_SCANS_THRESHOLD, "0");
+ inputSplitsSerial = inputFormat.getSplits(jobConf, SPLITS);
+ end = System.currentTimeMillis();
+ long durationInSerial = end - start;
+ System.out.println(String.format("get split in serial requires:%s ms",
+ String.valueOf(durationInSerial)));
+
+ // test get splits in parallel
+ start = System.currentTimeMillis();
+ jobConf.set(PhoenixStorageHandlerConstants.PHOENIX_MINIMUM_PARALLEL_SCANS_THRESHOLD, "1");
+ jobConf.set(PhoenixStorageHandlerConstants.PHOENIX_INPUTSPLIT_GENERATION_THREAD_COUNT,"24");
+ InputSplit[] inputSplitsParallel = inputFormat.getSplits(jobConf, SPLITS);
+ end = System.currentTimeMillis();
+ long durationInParallel = end - start;
+
+ System.out.println(String.format("get split in parallel requires:%s ms",
+ String.valueOf(durationInParallel)));
+
+ // Test if performance of parallel method is better than serial method
+ Assert.assertTrue(durationInParallel < durationInSerial);
Review Comment:
This will result in flaky tests as the environments which will run this test are guaranteed to not be deterministic. Unit tests should be about functional correctness, not performance.
##########
phoenix-hive-base/src/test/java/org/apache/phoenix/hive/HivePhoenixInputFormatTest.java:
##########
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hive;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.Locale;
+import java.util.Properties;
+
+import javax.annotation.concurrent.NotThreadSafe;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.mapred.InputSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.phoenix.end2end.ParallelStatsDisabledIT;
+import org.apache.phoenix.end2end.ParallelStatsDisabledTest;
+import org.apache.phoenix.hive.constants.PhoenixStorageHandlerConstants;
+import org.apache.phoenix.hive.mapreduce.PhoenixInputFormat;
+import org.apache.phoenix.mapreduce.PhoenixRecordWritable;
+import org.apache.phoenix.schema.TableAlreadyExistsException;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+
+/**
+ * Test class for Hive PhoenixInputFormat
+ */
+@NotThreadSafe
+@Category(ParallelStatsDisabledTest.class)
+public class HivePhoenixInputFormatTest extends ParallelStatsDisabledIT {
Review Comment:
Do the existing Phoenix-hive tests activate your new property and implicitly validate that it is functional? I think we have some test classes but do we create multi-region Phoenix tables in those tests (or those with enough data to have multiple guideposts)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] chrajeshbabu commented on a diff in pull request #79: PHOENIX-6698 hive-connector will take long time to generate splits fo…
Posted by GitBox <gi...@apache.org>.
chrajeshbabu commented on code in PR #79:
URL: https://github.com/apache/phoenix-connectors/pull/79#discussion_r872323611
##########
phoenix-hive-base/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixInputFormat.java:
##########
@@ -121,73 +126,192 @@ public InputSplit[] getSplits(JobConf jobConf, int numSplits) throws IOException
private List<InputSplit> generateSplits(final JobConf jobConf, final QueryPlan qplan,
final List<KeyRange> splits, String query) throws
IOException {
- if (qplan == null){
+ if (qplan == null) {
throw new NullPointerException();
- }if (splits == null){
+ }
+ if (splits == null) {
throw new NullPointerException();
}
final List<InputSplit> psplits = new ArrayList<>(splits.size());
- Path[] tablePaths = FileInputFormat.getInputPaths(ShimLoader.getHadoopShims()
- .newJobContext(new Job(jobConf)));
- boolean splitByStats = jobConf.getBoolean(PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
+ final Path[] tablePaths = FileInputFormat.getInputPaths(
+ ShimLoader.getHadoopShims().newJobContext(new Job(jobConf)));
+ final boolean splitByStats = jobConf.getBoolean(
+ PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
false);
-
+ final int parallelThreshould = jobConf.getInt(
+ "hive.phoenix.split.parallel.threshold",
+ 32);
setScanCacheSize(jobConf);
+ if (
+ (parallelThreshould <= 0)
Review Comment:
Remove unnecessary parentheses.
##########
phoenix-hive-base/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixInputFormat.java:
##########
@@ -121,73 +126,192 @@ public InputSplit[] getSplits(JobConf jobConf, int numSplits) throws IOException
private List<InputSplit> generateSplits(final JobConf jobConf, final QueryPlan qplan,
final List<KeyRange> splits, String query) throws
IOException {
- if (qplan == null){
+ if (qplan == null) {
throw new NullPointerException();
- }if (splits == null){
+ }
+ if (splits == null) {
throw new NullPointerException();
}
final List<InputSplit> psplits = new ArrayList<>(splits.size());
- Path[] tablePaths = FileInputFormat.getInputPaths(ShimLoader.getHadoopShims()
- .newJobContext(new Job(jobConf)));
- boolean splitByStats = jobConf.getBoolean(PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
+ final Path[] tablePaths = FileInputFormat.getInputPaths(
+ ShimLoader.getHadoopShims().newJobContext(new Job(jobConf)));
+ final boolean splitByStats = jobConf.getBoolean(
+ PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
false);
-
+ final int parallelThreshould = jobConf.getInt(
+ "hive.phoenix.split.parallel.threshold",
+ 32);
setScanCacheSize(jobConf);
+ if (
+ (parallelThreshould <= 0)
+ ||
+ (qplan.getScans().size() < parallelThreshould)
+ ) {
+ LOG.info("generate splits in serial");
Review Comment:
Make it clear that Generate Input Splits in serial.
##########
phoenix-hive-base/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixInputFormat.java:
##########
@@ -121,73 +126,192 @@ public InputSplit[] getSplits(JobConf jobConf, int numSplits) throws IOException
private List<InputSplit> generateSplits(final JobConf jobConf, final QueryPlan qplan,
final List<KeyRange> splits, String query) throws
IOException {
- if (qplan == null){
+ if (qplan == null) {
throw new NullPointerException();
- }if (splits == null){
+ }
+ if (splits == null) {
throw new NullPointerException();
}
final List<InputSplit> psplits = new ArrayList<>(splits.size());
- Path[] tablePaths = FileInputFormat.getInputPaths(ShimLoader.getHadoopShims()
- .newJobContext(new Job(jobConf)));
- boolean splitByStats = jobConf.getBoolean(PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
+ final Path[] tablePaths = FileInputFormat.getInputPaths(
+ ShimLoader.getHadoopShims().newJobContext(new Job(jobConf)));
+ final boolean splitByStats = jobConf.getBoolean(
+ PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
false);
-
+ final int parallelThreshould = jobConf.getInt(
+ "hive.phoenix.split.parallel.threshold",
+ 32);
setScanCacheSize(jobConf);
+ if (
+ (parallelThreshould <= 0)
+ ||
+ (qplan.getScans().size() < parallelThreshould)
+ ) {
+ LOG.info("generate splits in serial");
+ for (final List<Scan> scans : qplan.getScans()) {
+ psplits.addAll(
+ generateSplitsInternal(
+ jobConf,
+ qplan,
+ splits,
+ query,
+ scans,
+ splitByStats,
+ tablePaths)
+ );
+ }
+ } else {
+ final int parallism = jobConf.getInt(
+ "hive.phoenix.split.parallel.level",
+ Runtime.getRuntime().availableProcessors() * 2);
+ ExecutorService executorService = Executors.newFixedThreadPool(
+ parallism);
+ LOG.info("generate splits in parallel with {} threads", parallism);
- // Adding Localization
- try (org.apache.hadoop.hbase.client.Connection connection = ConnectionFactory.createConnection(PhoenixConnectionUtil.getConfiguration(jobConf))) {
- RegionLocator regionLocator = connection.getRegionLocator(TableName.valueOf(qplan
- .getTableRef().getTable().getPhysicalName().toString()));
+ List<Future<List<InputSplit>>> tasks = new ArrayList<>();
- for (List<Scan> scans : qplan.getScans()) {
+ try {
+ for (final List<Scan> scans : qplan.getScans()) {
+ Future<List<InputSplit>> task = executorService.submit(
+ new Callable<List<InputSplit>>() {
+ @Override
+ public List<InputSplit> call() throws Exception {
+ return generateSplitsInternal(jobConf,
+ qplan,
+ splits,
+ query,
+ scans,
+ splitByStats,
+ tablePaths);
+ }
+ });
+ tasks.add(task);
+ }
+ for (Future<List<InputSplit>> task : tasks) {
+ psplits.addAll(task.get());
+ }
+ } catch (ExecutionException | InterruptedException exception) {
+ throw new IOException("failed to get splits,reason:",
+ exception);
+ } finally {
+ executorService.shutdown();
+ }
+ }
+ return psplits;
+ }
+ /**
+ * This method is used to generate splits for each scan list.
+ * @param jobConf MapReduce Job Configuration
+ * @param qplan phoenix query plan
+ * @param splits phoenix table splits
+ * @param query phoenix query statement
+ * @param scans scan list slice of query plan
+ * @param splitByStats split by stat enabled
+ * @param tablePaths table paths
+ * @return List of Input Splits
+ * @throws IOException if function fails
+ */
+ private List<InputSplit> generateSplitsInternal(final JobConf jobConf,
+ final QueryPlan qplan,
+ final List<KeyRange> splits,
+ final String query,
+ final List<Scan> scans,
+ final boolean splitByStats,
+ final Path[] tablePaths) throws IOException {
+
+ final List<InputSplit> psplits = new ArrayList<>(scans.size());
+ try (org.apache.hadoop.hbase.client.Connection connection =
+ ConnectionFactory.createConnection(
+ PhoenixConnectionUtil.getConfiguration(jobConf))) {
+ RegionLocator regionLocator =
+ connection.getRegionLocator(TableName.valueOf(
+ qplan.getTableRef().getTable()
+ .getPhysicalName().toString()));
PhoenixInputSplit inputSplit;
- HRegionLocation location = regionLocator.getRegionLocation(scans.get(0).getStartRow()
- , false);
- long regionSize = CompatUtil.getSize(regionLocator, connection.getAdmin(), location);
- String regionLocation = PhoenixStorageHandlerUtil.getRegionLocation(location, LOG);
+ HRegionLocation location = regionLocator.getRegionLocation(
+ scans.get(0).getStartRow(),
+ false);
+ long regionSize = CompatUtil.getSize(regionLocator,
+ connection.getAdmin(),
+ location);
+ String regionLocation =
+ PhoenixStorageHandlerUtil.getRegionLocation(location,
+ LOG);
if (splitByStats) {
for (Scan aScan : scans) {
if (LOG.isDebugEnabled()) {
- LOG.debug("Split for scan : " + aScan + "with scanAttribute : " + aScan
- .getAttributesMap() + " [scanCache, cacheBlock, scanBatch] : [" +
- aScan.getCaching() + ", " + aScan.getCacheBlocks() + ", " + aScan
- .getBatch() + "] and regionLocation : " + regionLocation);
+ LOG.debug("Split for scan : "
Review Comment:
Format the code properly.
##########
phoenix-hive-base/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixInputFormat.java:
##########
@@ -121,73 +126,192 @@ public InputSplit[] getSplits(JobConf jobConf, int numSplits) throws IOException
private List<InputSplit> generateSplits(final JobConf jobConf, final QueryPlan qplan,
final List<KeyRange> splits, String query) throws
IOException {
- if (qplan == null){
+ if (qplan == null) {
throw new NullPointerException();
- }if (splits == null){
+ }
+ if (splits == null) {
throw new NullPointerException();
}
final List<InputSplit> psplits = new ArrayList<>(splits.size());
- Path[] tablePaths = FileInputFormat.getInputPaths(ShimLoader.getHadoopShims()
- .newJobContext(new Job(jobConf)));
- boolean splitByStats = jobConf.getBoolean(PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
+ final Path[] tablePaths = FileInputFormat.getInputPaths(
+ ShimLoader.getHadoopShims().newJobContext(new Job(jobConf)));
+ final boolean splitByStats = jobConf.getBoolean(
+ PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
false);
-
+ final int parallelThreshould = jobConf.getInt(
+ "hive.phoenix.split.parallel.threshold",
+ 32);
setScanCacheSize(jobConf);
+ if (
+ (parallelThreshould <= 0)
+ ||
+ (qplan.getScans().size() < parallelThreshould)
+ ) {
+ LOG.info("generate splits in serial");
+ for (final List<Scan> scans : qplan.getScans()) {
+ psplits.addAll(
+ generateSplitsInternal(
+ jobConf,
+ qplan,
+ splits,
+ query,
+ scans,
+ splitByStats,
+ tablePaths)
+ );
+ }
+ } else {
+ final int parallism = jobConf.getInt(
+ "hive.phoenix.split.parallel.level",
+ Runtime.getRuntime().availableProcessors() * 2);
+ ExecutorService executorService = Executors.newFixedThreadPool(
+ parallism);
+ LOG.info("generate splits in parallel with {} threads", parallism);
- // Adding Localization
- try (org.apache.hadoop.hbase.client.Connection connection = ConnectionFactory.createConnection(PhoenixConnectionUtil.getConfiguration(jobConf))) {
- RegionLocator regionLocator = connection.getRegionLocator(TableName.valueOf(qplan
- .getTableRef().getTable().getPhysicalName().toString()));
+ List<Future<List<InputSplit>>> tasks = new ArrayList<>();
- for (List<Scan> scans : qplan.getScans()) {
+ try {
+ for (final List<Scan> scans : qplan.getScans()) {
+ Future<List<InputSplit>> task = executorService.submit(
+ new Callable<List<InputSplit>>() {
+ @Override
+ public List<InputSplit> call() throws Exception {
+ return generateSplitsInternal(jobConf,
+ qplan,
+ splits,
+ query,
+ scans,
+ splitByStats,
+ tablePaths);
+ }
+ });
+ tasks.add(task);
+ }
+ for (Future<List<InputSplit>> task : tasks) {
+ psplits.addAll(task.get());
+ }
+ } catch (ExecutionException | InterruptedException exception) {
+ throw new IOException("failed to get splits,reason:",
Review Comment:
Log message can be improved.
##########
phoenix-hive-base/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixInputFormat.java:
##########
@@ -121,73 +126,192 @@ public InputSplit[] getSplits(JobConf jobConf, int numSplits) throws IOException
private List<InputSplit> generateSplits(final JobConf jobConf, final QueryPlan qplan,
final List<KeyRange> splits, String query) throws
IOException {
- if (qplan == null){
+ if (qplan == null) {
throw new NullPointerException();
- }if (splits == null){
+ }
+ if (splits == null) {
throw new NullPointerException();
}
final List<InputSplit> psplits = new ArrayList<>(splits.size());
- Path[] tablePaths = FileInputFormat.getInputPaths(ShimLoader.getHadoopShims()
- .newJobContext(new Job(jobConf)));
- boolean splitByStats = jobConf.getBoolean(PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
+ final Path[] tablePaths = FileInputFormat.getInputPaths(
+ ShimLoader.getHadoopShims().newJobContext(new Job(jobConf)));
+ final boolean splitByStats = jobConf.getBoolean(
+ PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
false);
-
+ final int parallelThreshould = jobConf.getInt(
+ "hive.phoenix.split.parallel.threshold",
+ 32);
setScanCacheSize(jobConf);
+ if (
+ (parallelThreshould <= 0)
+ ||
+ (qplan.getScans().size() < parallelThreshould)
+ ) {
+ LOG.info("generate splits in serial");
+ for (final List<Scan> scans : qplan.getScans()) {
+ psplits.addAll(
+ generateSplitsInternal(
Review Comment:
Reduce the number of lines used to call the method.
##########
phoenix-hive-base/src/test/java/org/apache/phoenix/hive/HivePhoenixInputFormatTest.java:
##########
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hive;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.mapred.InputSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.phoenix.end2end.ParallelStatsDisabledIT;
+import org.apache.phoenix.end2end.ParallelStatsDisabledTest;
+import org.apache.phoenix.hive.constants.PhoenixStorageHandlerConstants;
+import org.apache.phoenix.hive.mapreduce.PhoenixInputFormat;
+import org.apache.phoenix.mapreduce.PhoenixRecordWritable;
+import org.apache.phoenix.schema.TableAlreadyExistsException;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.NotThreadSafe;
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.Locale;
+import java.util.Properties;
+
+/**
+ * Test class for Hive PhoenixInputFormat
+ */
+@NotThreadSafe
+@Category(ParallelStatsDisabledTest.class)
+public class HivePhoenixInputFormatTest extends ParallelStatsDisabledIT {
+ private static final Logger LOG = LoggerFactory.getLogger(
+ HivePhoenixInputFormatTest.class);
+ private static final String TABLE_NAME = "HivePhoenixInputFormatTest"
+ .toUpperCase(Locale.ROOT);
+ private static final String DDL = "CREATE TABLE "
+ + TABLE_NAME
+ + " (V1 varchar NOT NULL PRIMARY KEY, V2 integer)";
+ private static final int SPLITS = 128;
+
+ /*
+ *
+ * This test will create phoenix table with 128 splits and compare
+ * performance of split generation in serial/parallel
+ *
+ * */
+ @Test
+ public void testGetSplitsSerialOrParallel() throws IOException,SQLException {
+ PhoenixInputFormat<PhoenixRecordWritable> inputFormat =
+ new PhoenixInputFormat<PhoenixRecordWritable>();
+ long start,end;
+
+ // create table with N splits
+ System.out.println(
+ String.format("generate testing table with %s splits",
+ String.valueOf(SPLITS)));
+ setupTestTable();
+ // setup configuration required for PhoenixInputFormat
+ Configuration conf = getUtility().getConfiguration();
+ JobConf jobConf = new JobConf(conf);
+ configureTestInput(jobConf);
+
+
+ // test get splits in serial
+ start = System.currentTimeMillis();
+ jobConf.set("hive.phoenix.split.parallel.threshold","0");
+ InputSplit[] inputSplitsSerial = inputFormat.getSplits(jobConf,SPLITS);
+ end = System.currentTimeMillis();
+ long durationInSerial=end - start;
+ System.out.println(String.format(
Review Comment:
We need to use assertions and no use in printing those in the logs.
##########
phoenix-hive-base/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixInputFormat.java:
##########
@@ -121,73 +126,192 @@ public InputSplit[] getSplits(JobConf jobConf, int numSplits) throws IOException
private List<InputSplit> generateSplits(final JobConf jobConf, final QueryPlan qplan,
final List<KeyRange> splits, String query) throws
IOException {
- if (qplan == null){
+ if (qplan == null) {
throw new NullPointerException();
- }if (splits == null){
+ }
+ if (splits == null) {
throw new NullPointerException();
}
final List<InputSplit> psplits = new ArrayList<>(splits.size());
- Path[] tablePaths = FileInputFormat.getInputPaths(ShimLoader.getHadoopShims()
- .newJobContext(new Job(jobConf)));
- boolean splitByStats = jobConf.getBoolean(PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
+ final Path[] tablePaths = FileInputFormat.getInputPaths(
+ ShimLoader.getHadoopShims().newJobContext(new Job(jobConf)));
+ final boolean splitByStats = jobConf.getBoolean(
+ PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
false);
-
+ final int parallelThreshould = jobConf.getInt(
+ "hive.phoenix.split.parallel.threshold",
+ 32);
setScanCacheSize(jobConf);
+ if (
+ (parallelThreshould <= 0)
+ ||
+ (qplan.getScans().size() < parallelThreshould)
+ ) {
+ LOG.info("generate splits in serial");
+ for (final List<Scan> scans : qplan.getScans()) {
+ psplits.addAll(
+ generateSplitsInternal(
+ jobConf,
+ qplan,
+ splits,
+ query,
+ scans,
+ splitByStats,
+ tablePaths)
+ );
+ }
+ } else {
+ final int parallism = jobConf.getInt(
+ "hive.phoenix.split.parallel.level",
+ Runtime.getRuntime().availableProcessors() * 2);
+ ExecutorService executorService = Executors.newFixedThreadPool(
+ parallism);
+ LOG.info("generate splits in parallel with {} threads", parallism);
- // Adding Localization
- try (org.apache.hadoop.hbase.client.Connection connection = ConnectionFactory.createConnection(PhoenixConnectionUtil.getConfiguration(jobConf))) {
- RegionLocator regionLocator = connection.getRegionLocator(TableName.valueOf(qplan
- .getTableRef().getTable().getPhysicalName().toString()));
+ List<Future<List<InputSplit>>> tasks = new ArrayList<>();
- for (List<Scan> scans : qplan.getScans()) {
+ try {
+ for (final List<Scan> scans : qplan.getScans()) {
+ Future<List<InputSplit>> task = executorService.submit(
+ new Callable<List<InputSplit>>() {
+ @Override
+ public List<InputSplit> call() throws Exception {
+ return generateSplitsInternal(jobConf,
+ qplan,
+ splits,
+ query,
+ scans,
+ splitByStats,
+ tablePaths);
+ }
+ });
+ tasks.add(task);
+ }
+ for (Future<List<InputSplit>> task : tasks) {
+ psplits.addAll(task.get());
+ }
+ } catch (ExecutionException | InterruptedException exception) {
+ throw new IOException("failed to get splits,reason:",
+ exception);
+ } finally {
+ executorService.shutdown();
+ }
+ }
+ return psplits;
+ }
+ /**
+ * This method is used to generate splits for each scan list.
+ * @param jobConf MapReduce Job Configuration
+ * @param qplan phoenix query plan
+ * @param splits phoenix table splits
+ * @param query phoenix query statement
+ * @param scans scan list slice of query plan
+ * @param splitByStats split by stat enabled
+ * @param tablePaths table paths
+ * @return List of Input Splits
+ * @throws IOException if function fails
+ */
+ private List<InputSplit> generateSplitsInternal(final JobConf jobConf,
+ final QueryPlan qplan,
+ final List<KeyRange> splits,
+ final String query,
+ final List<Scan> scans,
+ final boolean splitByStats,
+ final Path[] tablePaths) throws IOException {
+
+ final List<InputSplit> psplits = new ArrayList<>(scans.size());
+ try (org.apache.hadoop.hbase.client.Connection connection =
Review Comment:
The connection creation can be shared and reuse when generating the inputsplit.
##########
phoenix-hive-base/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixInputFormat.java:
##########
@@ -121,73 +126,192 @@ public InputSplit[] getSplits(JobConf jobConf, int numSplits) throws IOException
private List<InputSplit> generateSplits(final JobConf jobConf, final QueryPlan qplan,
final List<KeyRange> splits, String query) throws
IOException {
- if (qplan == null){
+ if (qplan == null) {
throw new NullPointerException();
- }if (splits == null){
+ }
+ if (splits == null) {
throw new NullPointerException();
}
final List<InputSplit> psplits = new ArrayList<>(splits.size());
- Path[] tablePaths = FileInputFormat.getInputPaths(ShimLoader.getHadoopShims()
- .newJobContext(new Job(jobConf)));
- boolean splitByStats = jobConf.getBoolean(PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
+ final Path[] tablePaths = FileInputFormat.getInputPaths(
+ ShimLoader.getHadoopShims().newJobContext(new Job(jobConf)));
+ final boolean splitByStats = jobConf.getBoolean(
+ PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
false);
-
+ final int parallelThreshould = jobConf.getInt(
+ "hive.phoenix.split.parallel.threshold",
+ 32);
setScanCacheSize(jobConf);
+ if (
+ (parallelThreshould <= 0)
+ ||
+ (qplan.getScans().size() < parallelThreshould)
+ ) {
+ LOG.info("generate splits in serial");
+ for (final List<Scan> scans : qplan.getScans()) {
+ psplits.addAll(
+ generateSplitsInternal(
+ jobConf,
+ qplan,
+ splits,
+ query,
+ scans,
+ splitByStats,
+ tablePaths)
+ );
+ }
+ } else {
+ final int parallism = jobConf.getInt(
+ "hive.phoenix.split.parallel.level",
+ Runtime.getRuntime().availableProcessors() * 2);
+ ExecutorService executorService = Executors.newFixedThreadPool(
+ parallism);
+ LOG.info("generate splits in parallel with {} threads", parallism);
- // Adding Localization
- try (org.apache.hadoop.hbase.client.Connection connection = ConnectionFactory.createConnection(PhoenixConnectionUtil.getConfiguration(jobConf))) {
- RegionLocator regionLocator = connection.getRegionLocator(TableName.valueOf(qplan
- .getTableRef().getTable().getPhysicalName().toString()));
+ List<Future<List<InputSplit>>> tasks = new ArrayList<>();
- for (List<Scan> scans : qplan.getScans()) {
+ try {
+ for (final List<Scan> scans : qplan.getScans()) {
+ Future<List<InputSplit>> task = executorService.submit(
+ new Callable<List<InputSplit>>() {
+ @Override
+ public List<InputSplit> call() throws Exception {
+ return generateSplitsInternal(jobConf,
+ qplan,
+ splits,
+ query,
+ scans,
+ splitByStats,
+ tablePaths);
+ }
+ });
+ tasks.add(task);
+ }
+ for (Future<List<InputSplit>> task : tasks) {
+ psplits.addAll(task.get());
+ }
+ } catch (ExecutionException | InterruptedException exception) {
+ throw new IOException("failed to get splits,reason:",
+ exception);
+ } finally {
+ executorService.shutdown();
+ }
+ }
+ return psplits;
+ }
+ /**
+ * This method is used to generate splits for each scan list.
+ * @param jobConf MapReduce Job Configuration
+ * @param qplan phoenix query plan
+ * @param splits phoenix table splits
+ * @param query phoenix query statement
+ * @param scans scan list slice of query plan
+ * @param splitByStats split by stat enabled
+ * @param tablePaths table paths
+ * @return List of Input Splits
+ * @throws IOException if function fails
+ */
+ private List<InputSplit> generateSplitsInternal(final JobConf jobConf,
+ final QueryPlan qplan,
+ final List<KeyRange> splits,
+ final String query,
+ final List<Scan> scans,
+ final boolean splitByStats,
+ final Path[] tablePaths) throws IOException {
+
+ final List<InputSplit> psplits = new ArrayList<>(scans.size());
+ try (org.apache.hadoop.hbase.client.Connection connection =
+ ConnectionFactory.createConnection(
+ PhoenixConnectionUtil.getConfiguration(jobConf))) {
+ RegionLocator regionLocator =
+ connection.getRegionLocator(TableName.valueOf(
Review Comment:
Region locator also can be shared for each call.
##########
phoenix-hive-base/src/test/java/org/apache/phoenix/hive/HivePhoenixInputFormatTest.java:
##########
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.hive;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.mapred.InputSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.phoenix.end2end.ParallelStatsDisabledIT;
+import org.apache.phoenix.end2end.ParallelStatsDisabledTest;
+import org.apache.phoenix.hive.constants.PhoenixStorageHandlerConstants;
+import org.apache.phoenix.hive.mapreduce.PhoenixInputFormat;
+import org.apache.phoenix.mapreduce.PhoenixRecordWritable;
+import org.apache.phoenix.schema.TableAlreadyExistsException;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.NotThreadSafe;
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.Locale;
+import java.util.Properties;
+
+/**
+ * Test class for Hive PhoenixInputFormat
+ */
+@NotThreadSafe
+@Category(ParallelStatsDisabledTest.class)
+public class HivePhoenixInputFormatTest extends ParallelStatsDisabledIT {
+ private static final Logger LOG = LoggerFactory.getLogger(
+ HivePhoenixInputFormatTest.class);
+ private static final String TABLE_NAME = "HivePhoenixInputFormatTest"
+ .toUpperCase(Locale.ROOT);
+ private static final String DDL = "CREATE TABLE "
+ + TABLE_NAME
+ + " (V1 varchar NOT NULL PRIMARY KEY, V2 integer)";
+ private static final int SPLITS = 128;
+
+ /*
+ *
+ * This test will create phoenix table with 128 splits and compare
+ * performance of split generation in serial/parallel
+ *
+ * */
+ @Test
+ public void testGetSplitsSerialOrParallel() throws IOException,SQLException {
+ PhoenixInputFormat<PhoenixRecordWritable> inputFormat =
+ new PhoenixInputFormat<PhoenixRecordWritable>();
+ long start,end;
+
+ // create table with N splits
+ System.out.println(
+ String.format("generate testing table with %s splits",
+ String.valueOf(SPLITS)));
+ setupTestTable();
+ // setup configuration required for PhoenixInputFormat
+ Configuration conf = getUtility().getConfiguration();
+ JobConf jobConf = new JobConf(conf);
+ configureTestInput(jobConf);
+
+
+ // test get splits in serial
+ start = System.currentTimeMillis();
+ jobConf.set("hive.phoenix.split.parallel.threshold","0");
+ InputSplit[] inputSplitsSerial = inputFormat.getSplits(jobConf,SPLITS);
+ end = System.currentTimeMillis();
+ long durationInSerial=end - start;
+ System.out.println(String.format(
+ "get split in serial requires:%s ms",
+ String.valueOf(durationInSerial)));
+
+ // test get splits in parallel
+ start = System.currentTimeMillis();
+ jobConf.set("hive.phoenix.split.parallel.threshold", "1");
+ InputSplit[] inputSplitsParallel = inputFormat.getSplits(
+ jobConf,
+ SPLITS);
+ end = System.currentTimeMillis();
+ long durationInParallel=end - start;
+
+ System.out.println(String.format(
+ "get split in parallel requires:%s ms",
+ String.valueOf(durationInParallel)));
+
+ // Test if performance of parallel method is better than serial method
+ Assert.assertTrue(durationInParallel < durationInSerial);
+ // Test if the input split returned by serial method and parallel method are the same
+ Assert.assertTrue(inputSplitsParallel.length==SPLITS);
+ Assert.assertTrue(
+ inputSplitsParallel.length == inputSplitsSerial.length
+ );
+ for (final InputSplit inputSplitParallel:inputSplitsParallel){
+ boolean match=false;
+ for (final InputSplit inputSplitSerial:inputSplitsSerial){
+ if (inputSplitParallel.equals(inputSplitSerial)){
+ match=true;
+ break;
+ }
+ }
+ Assert.assertTrue(match);
+ }
+ }
+
+ private static void setupTestTable() throws SQLException {
+ final byte [] start=new byte[0];
+ final byte [] end = Bytes.createMaxByteArray(4);
+ final byte[][] splits = Bytes.split(start, end, SPLITS-2);
+ createTestTableWithBinarySplit(getUrl(),DDL, splits ,null);
+ }
+
+ private static void buildPreparedSqlWithBinarySplits(
+ StringBuffer sb,
+ int splits)
+ {
Review Comment:
code formatting required.
##########
phoenix-hive-base/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixInputFormat.java:
##########
@@ -121,73 +126,192 @@ public InputSplit[] getSplits(JobConf jobConf, int numSplits) throws IOException
private List<InputSplit> generateSplits(final JobConf jobConf, final QueryPlan qplan,
final List<KeyRange> splits, String query) throws
IOException {
- if (qplan == null){
+ if (qplan == null) {
throw new NullPointerException();
- }if (splits == null){
+ }
+ if (splits == null) {
throw new NullPointerException();
}
final List<InputSplit> psplits = new ArrayList<>(splits.size());
- Path[] tablePaths = FileInputFormat.getInputPaths(ShimLoader.getHadoopShims()
- .newJobContext(new Job(jobConf)));
- boolean splitByStats = jobConf.getBoolean(PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
+ final Path[] tablePaths = FileInputFormat.getInputPaths(
+ ShimLoader.getHadoopShims().newJobContext(new Job(jobConf)));
+ final boolean splitByStats = jobConf.getBoolean(
+ PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
false);
-
+ final int parallelThreshould = jobConf.getInt(
+ "hive.phoenix.split.parallel.threshold",
+ 32);
setScanCacheSize(jobConf);
+ if (
+ (parallelThreshould <= 0)
+ ||
+ (qplan.getScans().size() < parallelThreshould)
+ ) {
+ LOG.info("generate splits in serial");
+ for (final List<Scan> scans : qplan.getScans()) {
+ psplits.addAll(
+ generateSplitsInternal(
+ jobConf,
+ qplan,
+ splits,
+ query,
+ scans,
+ splitByStats,
+ tablePaths)
+ );
+ }
+ } else {
+ final int parallism = jobConf.getInt(
Review Comment:
Whats the difference between this parallelism level config and parallel threshold.
##########
phoenix-hive-base/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixInputFormat.java:
##########
@@ -121,73 +126,192 @@ public InputSplit[] getSplits(JobConf jobConf, int numSplits) throws IOException
private List<InputSplit> generateSplits(final JobConf jobConf, final QueryPlan qplan,
final List<KeyRange> splits, String query) throws
IOException {
- if (qplan == null){
+ if (qplan == null) {
throw new NullPointerException();
- }if (splits == null){
+ }
+ if (splits == null) {
throw new NullPointerException();
}
final List<InputSplit> psplits = new ArrayList<>(splits.size());
- Path[] tablePaths = FileInputFormat.getInputPaths(ShimLoader.getHadoopShims()
- .newJobContext(new Job(jobConf)));
- boolean splitByStats = jobConf.getBoolean(PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
+ final Path[] tablePaths = FileInputFormat.getInputPaths(
+ ShimLoader.getHadoopShims().newJobContext(new Job(jobConf)));
+ final boolean splitByStats = jobConf.getBoolean(
+ PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
false);
-
+ final int parallelThreshould = jobConf.getInt(
+ "hive.phoenix.split.parallel.threshold",
+ 32);
setScanCacheSize(jobConf);
+ if (
+ (parallelThreshould <= 0)
+ ||
+ (qplan.getScans().size() < parallelThreshould)
+ ) {
+ LOG.info("generate splits in serial");
+ for (final List<Scan> scans : qplan.getScans()) {
+ psplits.addAll(
+ generateSplitsInternal(
+ jobConf,
+ qplan,
+ splits,
+ query,
+ scans,
+ splitByStats,
+ tablePaths)
+ );
+ }
+ } else {
+ final int parallism = jobConf.getInt(
+ "hive.phoenix.split.parallel.level",
+ Runtime.getRuntime().availableProcessors() * 2);
+ ExecutorService executorService = Executors.newFixedThreadPool(
+ parallism);
+ LOG.info("generate splits in parallel with {} threads", parallism);
- // Adding Localization
- try (org.apache.hadoop.hbase.client.Connection connection = ConnectionFactory.createConnection(PhoenixConnectionUtil.getConfiguration(jobConf))) {
- RegionLocator regionLocator = connection.getRegionLocator(TableName.valueOf(qplan
- .getTableRef().getTable().getPhysicalName().toString()));
+ List<Future<List<InputSplit>>> tasks = new ArrayList<>();
- for (List<Scan> scans : qplan.getScans()) {
+ try {
+ for (final List<Scan> scans : qplan.getScans()) {
+ Future<List<InputSplit>> task = executorService.submit(
+ new Callable<List<InputSplit>>() {
+ @Override
+ public List<InputSplit> call() throws Exception {
+ return generateSplitsInternal(jobConf,
+ qplan,
+ splits,
+ query,
+ scans,
+ splitByStats,
+ tablePaths);
+ }
+ });
+ tasks.add(task);
+ }
+ for (Future<List<InputSplit>> task : tasks) {
+ psplits.addAll(task.get());
+ }
+ } catch (ExecutionException | InterruptedException exception) {
+ throw new IOException("failed to get splits,reason:",
+ exception);
+ } finally {
+ executorService.shutdown();
+ }
+ }
+ return psplits;
+ }
+ /**
+ * This method is used to generate splits for each scan list.
+ * @param jobConf MapReduce Job Configuration
+ * @param qplan phoenix query plan
+ * @param splits phoenix table splits
+ * @param query phoenix query statement
+ * @param scans scan list slice of query plan
+ * @param splitByStats split by stat enabled
+ * @param tablePaths table paths
+ * @return List of Input Splits
+ * @throws IOException if function fails
+ */
+ private List<InputSplit> generateSplitsInternal(final JobConf jobConf,
+ final QueryPlan qplan,
+ final List<KeyRange> splits,
+ final String query,
+ final List<Scan> scans,
+ final boolean splitByStats,
+ final Path[] tablePaths) throws IOException {
+
+ final List<InputSplit> psplits = new ArrayList<>(scans.size());
+ try (org.apache.hadoop.hbase.client.Connection connection =
+ ConnectionFactory.createConnection(
+ PhoenixConnectionUtil.getConfiguration(jobConf))) {
+ RegionLocator regionLocator =
+ connection.getRegionLocator(TableName.valueOf(
+ qplan.getTableRef().getTable()
+ .getPhysicalName().toString()));
PhoenixInputSplit inputSplit;
- HRegionLocation location = regionLocator.getRegionLocation(scans.get(0).getStartRow()
- , false);
- long regionSize = CompatUtil.getSize(regionLocator, connection.getAdmin(), location);
- String regionLocation = PhoenixStorageHandlerUtil.getRegionLocation(location, LOG);
+ HRegionLocation location = regionLocator.getRegionLocation(
+ scans.get(0).getStartRow(),
+ false);
+ long regionSize = CompatUtil.getSize(regionLocator,
+ connection.getAdmin(),
+ location);
+ String regionLocation =
+ PhoenixStorageHandlerUtil.getRegionLocation(location,
+ LOG);
if (splitByStats) {
for (Scan aScan : scans) {
if (LOG.isDebugEnabled()) {
- LOG.debug("Split for scan : " + aScan + "with scanAttribute : " + aScan
- .getAttributesMap() + " [scanCache, cacheBlock, scanBatch] : [" +
- aScan.getCaching() + ", " + aScan.getCacheBlocks() + ", " + aScan
- .getBatch() + "] and regionLocation : " + regionLocation);
+ LOG.debug("Split for scan : "
+ + aScan
+ + "with scanAttribute : "
+ + aScan.getAttributesMap()
+ + " [scanCache, cacheBlock, scanBatch] : ["
+ + aScan.getCaching()
+ + ", "
+ + aScan.getCacheBlocks()
+ + ", "
+ + aScan.getBatch()
+ + "] and regionLocation : "
+ + regionLocation);
}
- inputSplit = new PhoenixInputSplit(new ArrayList<>(Arrays.asList(aScan)), tablePaths[0],
- regionLocation, regionSize);
+ inputSplit =
+ new PhoenixInputSplit(
+ new ArrayList<>(Arrays.asList(aScan)),
+ tablePaths[0],
+ regionLocation,
+ regionSize);
inputSplit.setQuery(query);
psplits.add(inputSplit);
}
} else {
if (LOG.isDebugEnabled()) {
- LOG.debug("Scan count[" + scans.size() + "] : " + Bytes.toStringBinary(scans
- .get(0).getStartRow()) + " ~ " + Bytes.toStringBinary(scans.get(scans
- .size() - 1).getStopRow()));
- LOG.debug("First scan : " + scans.get(0) + "with scanAttribute : " + scans
- .get(0).getAttributesMap() + " [scanCache, cacheBlock, scanBatch] : " +
- "[" + scans.get(0).getCaching() + ", " + scans.get(0).getCacheBlocks()
- + ", " + scans.get(0).getBatch() + "] and regionLocation : " +
- regionLocation);
+ LOG.debug("Scan count["
Review Comment:
format properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] stoty commented on pull request #79: PHOENIX-6698 hive-connector will take long time to generate splits fo…
Posted by GitBox <gi...@apache.org>.
stoty commented on PR #79:
URL: https://github.com/apache/phoenix-connectors/pull/79#issuecomment-1136908804
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 6m 46s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 18s | master passed |
| +1 :green_heart: | compile | 1m 12s | master passed |
| +1 :green_heart: | checkstyle | 0m 19s | master passed |
| +1 :green_heart: | javadoc | 0m 25s | master passed |
| +0 :ok: | spotbugs | 2m 11s | phoenix-hive-base in master has 46 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 54s | the patch passed |
| +1 :green_heart: | compile | 1m 11s | the patch passed |
| +1 :green_heart: | javac | 1m 11s | the patch passed |
| -1 :x: | checkstyle | 0m 19s | phoenix-hive-base: The patch generated 45 new + 130 unchanged - 32 fixed = 175 total (was 162) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 25s | the patch passed |
| +1 :green_heart: | spotbugs | 1m 59s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 49m 55s | phoenix-hive-base in the patch failed. |
| -1 :x: | asflicense | 0m 13s | The patch generated 2 ASF License warnings. |
| | | 127m 49s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/9/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix-connectors/pull/79 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile |
| uname | Linux ace0e52f9cec 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-connectors-personality.sh |
| git revision | master / 9ed127f |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/9/artifact/yetus-general-check/output/diff-checkstyle-phoenix-hive-base.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/9/artifact/yetus-general-check/output/patch-unit-phoenix-hive-base.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/9/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/9/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 1699 (vs. ulimit of 30000) |
| modules | C: phoenix-hive-base U: phoenix-hive-base |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/9/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] stoty commented on pull request #79: PHOENIX-6698 hive-connector will take long time to generate splits fo…
Posted by GitBox <gi...@apache.org>.
stoty commented on PR #79:
URL: https://github.com/apache/phoenix-connectors/pull/79#issuecomment-1123296531
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 42s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 26s | master passed |
| +1 :green_heart: | compile | 1m 6s | master passed |
| +1 :green_heart: | checkstyle | 0m 18s | master passed |
| +1 :green_heart: | javadoc | 0m 27s | master passed |
| +0 :ok: | spotbugs | 2m 0s | phoenix-hive-base in master has 46 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 18m 48s | the patch passed |
| +1 :green_heart: | compile | 1m 7s | the patch passed |
| +1 :green_heart: | javac | 1m 7s | the patch passed |
| -1 :x: | checkstyle | 0m 18s | phoenix-hive-base: The patch generated 4 new + 52 unchanged - 32 fixed = 56 total (was 84) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 26s | the patch passed |
| +1 :green_heart: | spotbugs | 1m 56s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 53m 22s | phoenix-hive-base in the patch failed. |
| -1 :x: | asflicense | 0m 13s | The patch generated 2 ASF License warnings. |
| | | 118m 52s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix-connectors/pull/79 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile |
| uname | Linux 73d06dc4ab23 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-connectors-personality.sh |
| git revision | master / 9ed127f |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/2/artifact/yetus-general-check/output/diff-checkstyle-phoenix-hive-base.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/2/artifact/yetus-general-check/output/patch-unit-phoenix-hive-base.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/2/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/2/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 1803 (vs. ulimit of 30000) |
| modules | C: phoenix-hive-base U: phoenix-hive-base |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/2/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] jichen20210919 commented on a diff in pull request #79: PHOENIX-6698 hive-connector will take long time to generate splits fo…
Posted by GitBox <gi...@apache.org>.
jichen20210919 commented on code in PR #79:
URL: https://github.com/apache/phoenix-connectors/pull/79#discussion_r873477213
##########
phoenix-hive-base/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixInputFormat.java:
##########
@@ -121,73 +126,192 @@ public InputSplit[] getSplits(JobConf jobConf, int numSplits) throws IOException
private List<InputSplit> generateSplits(final JobConf jobConf, final QueryPlan qplan,
final List<KeyRange> splits, String query) throws
IOException {
- if (qplan == null){
+ if (qplan == null) {
throw new NullPointerException();
- }if (splits == null){
+ }
+ if (splits == null) {
throw new NullPointerException();
}
final List<InputSplit> psplits = new ArrayList<>(splits.size());
- Path[] tablePaths = FileInputFormat.getInputPaths(ShimLoader.getHadoopShims()
- .newJobContext(new Job(jobConf)));
- boolean splitByStats = jobConf.getBoolean(PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
+ final Path[] tablePaths = FileInputFormat.getInputPaths(
+ ShimLoader.getHadoopShims().newJobContext(new Job(jobConf)));
+ final boolean splitByStats = jobConf.getBoolean(
+ PhoenixStorageHandlerConstants.SPLIT_BY_STATS,
false);
-
+ final int parallelThreshould = jobConf.getInt(
+ "hive.phoenix.split.parallel.threshold",
+ 32);
setScanCacheSize(jobConf);
+ if (
+ (parallelThreshould <= 0)
+ ||
+ (qplan.getScans().size() < parallelThreshould)
+ ) {
+ LOG.info("generate splits in serial");
+ for (final List<Scan> scans : qplan.getScans()) {
+ psplits.addAll(
+ generateSplitsInternal(
+ jobConf,
+ qplan,
+ splits,
+ query,
+ scans,
+ splitByStats,
+ tablePaths)
+ );
+ }
+ } else {
+ final int parallism = jobConf.getInt(
Review Comment:
parallelism level config is used to control the worker threads count for parallel split method, parallel threshold is used to control which split-generation method is used, serial or parallel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] stoty commented on pull request #79: PHOENIX-6698 hive-connector will take long time to generate splits fo…
Posted by GitBox <gi...@apache.org>.
stoty commented on PR #79:
URL: https://github.com/apache/phoenix-connectors/pull/79#issuecomment-1129704965
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 59s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 43s | master passed |
| +1 :green_heart: | compile | 1m 10s | master passed |
| +1 :green_heart: | checkstyle | 0m 17s | master passed |
| +1 :green_heart: | javadoc | 0m 25s | master passed |
| +0 :ok: | spotbugs | 2m 10s | phoenix-hive-base in master has 46 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 12s | the patch passed |
| +1 :green_heart: | compile | 1m 9s | the patch passed |
| +1 :green_heart: | javac | 1m 9s | the patch passed |
| -1 :x: | checkstyle | 0m 18s | phoenix-hive-base: The patch generated 39 new + 52 unchanged - 32 fixed = 91 total (was 84) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 26s | the patch passed |
| +1 :green_heart: | spotbugs | 2m 0s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 59m 41s | phoenix-hive-base in the patch failed. |
| -1 :x: | asflicense | 0m 12s | The patch generated 2 ASF License warnings. |
| | | 131m 21s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/5/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix-connectors/pull/79 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile |
| uname | Linux 47e776d62a5c 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-connectors-personality.sh |
| git revision | master / 9ed127f |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/5/artifact/yetus-general-check/output/diff-checkstyle-phoenix-hive-base.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/5/artifact/yetus-general-check/output/patch-unit-phoenix-hive-base.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/5/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/5/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 1611 (vs. ulimit of 30000) |
| modules | C: phoenix-hive-base U: phoenix-hive-base |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/5/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] stoty commented on pull request #79: PHOENIX-6698 hive-connector will take long time to generate splits fo…
Posted by GitBox <gi...@apache.org>.
stoty commented on PR #79:
URL: https://github.com/apache/phoenix-connectors/pull/79#issuecomment-1138113288
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 6m 15s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 49s | master passed |
| +1 :green_heart: | compile | 1m 20s | master passed |
| +1 :green_heart: | checkstyle | 0m 22s | master passed |
| +1 :green_heart: | javadoc | 0m 30s | master passed |
| +0 :ok: | spotbugs | 2m 21s | phoenix-hive-base in master has 46 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 12s | the patch passed |
| +1 :green_heart: | compile | 1m 13s | the patch passed |
| +1 :green_heart: | javac | 1m 13s | the patch passed |
| -1 :x: | checkstyle | 0m 22s | phoenix-hive-base: The patch generated 42 new + 130 unchanged - 32 fixed = 172 total (was 162) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 28s | the patch passed |
| +1 :green_heart: | spotbugs | 2m 0s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 76m 54s | phoenix-hive-base in the patch failed. |
| -1 :x: | asflicense | 0m 13s | The patch generated 2 ASF License warnings. |
| | | 155m 47s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/11/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix-connectors/pull/79 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile |
| uname | Linux f98789364a67 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-connectors-personality.sh |
| git revision | master / 9ed127f |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/11/artifact/yetus-general-check/output/diff-checkstyle-phoenix-hive-base.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/11/artifact/yetus-general-check/output/patch-unit-phoenix-hive-base.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/11/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/11/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 1919 (vs. ulimit of 30000) |
| modules | C: phoenix-hive-base U: phoenix-hive-base |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/11/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] stoty commented on pull request #79: PHOENIX-6698 hive-connector will take long time to generate splits fo…
Posted by GitBox <gi...@apache.org>.
stoty commented on PR #79:
URL: https://github.com/apache/phoenix-connectors/pull/79#issuecomment-1136687697
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 58s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 41s | master passed |
| +1 :green_heart: | compile | 1m 11s | master passed |
| +1 :green_heart: | checkstyle | 0m 19s | master passed |
| +1 :green_heart: | javadoc | 0m 26s | master passed |
| +0 :ok: | spotbugs | 2m 13s | phoenix-hive-base in master has 46 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 9s | the patch passed |
| +1 :green_heart: | compile | 1m 10s | the patch passed |
| +1 :green_heart: | javac | 1m 10s | the patch passed |
| -1 :x: | checkstyle | 0m 19s | phoenix-hive-base: The patch generated 45 new + 130 unchanged - 32 fixed = 175 total (was 162) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 25s | the patch passed |
| +1 :green_heart: | spotbugs | 2m 3s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 76m 36s | phoenix-hive-base in the patch failed. |
| -1 :x: | asflicense | 0m 13s | The patch generated 2 ASF License warnings. |
| | | 149m 22s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/7/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix-connectors/pull/79 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile |
| uname | Linux 3b51daef3fed 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-connectors-personality.sh |
| git revision | master / 9ed127f |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/7/artifact/yetus-general-check/output/diff-checkstyle-phoenix-hive-base.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/7/artifact/yetus-general-check/output/patch-unit-phoenix-hive-base.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/7/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/7/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 1668 (vs. ulimit of 30000) |
| modules | C: phoenix-hive-base U: phoenix-hive-base |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-79/7/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org