You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@phoenix.apache.org by GitBox <gi...@apache.org> on 2022/05/15 23:25:50 UTC
[GitHub] [phoenix-connectors] chrajeshbabu opened a new pull request, #80: PHOENIX-6694 Avoid unnecessary calls of fetching table meta data to r…
chrajeshbabu opened a new pull request, #80:
URL: https://github.com/apache/phoenix-connectors/pull/80
…egion servers holding the system tables in batch oriented jobs in spark or hive otherwise those RS become hotspot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] chrajeshbabu commented on pull request #80: PHOENIX-6694 Avoid unnecessary calls of fetching table meta data to r…
Posted by GitBox <gi...@apache.org>.
chrajeshbabu commented on PR #80:
URL: https://github.com/apache/phoenix-connectors/pull/80#issuecomment-1144233048
> +1 to Istvan's suggestion on a unit test, but I can't think of an easy way to verify that this is effective 🤔
@stoty @joshelser I have tried to add a test case to cover the ptable bytes parsing and checking in PhoenixTestingDataSource but somehow this not picking in the tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] stoty commented on pull request #80: PHOENIX-6694 Avoid unnecessary calls of fetching table meta data to r…
Posted by GitBox <gi...@apache.org>.
stoty commented on PR #80:
URL: https://github.com/apache/phoenix-connectors/pull/80#issuecomment-1144558046
+1 LGTM.
If you make a PR for the test, we'll try to help figuring why it is not being run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] stoty commented on pull request #80: PHOENIX-6694 Avoid unnecessary calls of fetching table meta data to r…
Posted by GitBox <gi...@apache.org>.
stoty commented on PR #80:
URL: https://github.com/apache/phoenix-connectors/pull/80#issuecomment-1127516065
Also, the current patch doesn't have a test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] stoty commented on a diff in pull request #80: PHOENIX-6694 Avoid unnecessary calls of fetching table meta data to r…
Posted by GitBox <gi...@apache.org>.
stoty commented on code in PR #80:
URL: https://github.com/apache/phoenix-connectors/pull/80#discussion_r874344921
##########
phoenix-spark-base/src/main/java/org/apache/phoenix/spark/datasource/v2/reader/PhoenixInputPartitionReader.java:
##########
@@ -94,6 +99,10 @@ private QueryPlan getQueryPlan() throws SQLException {
}
try (Connection conn = DriverManager.getConnection(
JDBC_PROTOCOL + JDBC_PROTOCOL_SEPARATOR + zkUrl, overridingProps)) {
+ PTable pTable = PTable.parseFrom(options.getTableBytes());
+ org.apache.phoenix.schema.PTable table = PTableImpl.createFromProto(pTable);
+ PhoenixConnection phoenixConnection = conn.unwrap(PhoenixConnection.class);
+ phoenixConnection.addTable(table, System.currentTimeMillis());
Review Comment:
Interesting point about the timestamp.
The point of this patch is to avoid hammering the system tables with a huge number of parallel requests.
I think that if we have executor starvation, then the jobs will not be started immediately, and the syscat load is not really a problem.
Can you think of a case of when the jobs are delayed enough for this to matter, but enough of them start up synchrounously for that to be a problem (I don't know enough about Spark to tell) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] chrajeshbabu commented on a diff in pull request #80: PHOENIX-6694 Avoid unnecessary calls of fetching table meta data to r…
Posted by GitBox <gi...@apache.org>.
chrajeshbabu commented on code in PR #80:
URL: https://github.com/apache/phoenix-connectors/pull/80#discussion_r887369394
##########
phoenix-spark-base/src/main/java/org/apache/phoenix/spark/datasource/v2/reader/PhoenixInputPartitionReader.java:
##########
@@ -81,7 +86,7 @@ Properties getOverriddenPropsFromOptions() {
return options.getOverriddenProps();
}
- private QueryPlan getQueryPlan() throws SQLException {
+ private QueryPlan getQueryPlan() throws SQLException, InvalidProtocolBufferException {
Review Comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] stoty commented on pull request #80: PHOENIX-6694 Avoid unnecessary calls of fetching table meta data to r…
Posted by GitBox <gi...@apache.org>.
stoty commented on PR #80:
URL: https://github.com/apache/phoenix-connectors/pull/80#issuecomment-1144321121
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 6m 39s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 14m 32s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 22s | master passed |
| +1 :green_heart: | compile | 2m 29s | master passed |
| +1 :green_heart: | checkstyle | 0m 28s | master passed |
| +1 :green_heart: | javadoc | 0m 37s | master passed |
| +0 :ok: | spotbugs | 3m 5s | phoenix-spark-base in master has 40 extant spotbugs warnings. |
| +0 :ok: | spotbugs | 1m 2s | phoenix5-spark3 in master has 22 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 23m 16s | the patch passed |
| +1 :green_heart: | compile | 2m 29s | the patch passed |
| +1 :green_heart: | javac | 2m 29s | the patch passed |
| -1 :x: | checkstyle | 0m 16s | phoenix-spark-base: The patch generated 21 new + 128 unchanged - 9 fixed = 149 total (was 137) |
| -1 :x: | checkstyle | 0m 13s | phoenix5-spark3: The patch generated 15 new + 122 unchanged - 5 fixed = 137 total (was 127) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 0m 36s | the patch passed |
| +1 :green_heart: | spotbugs | 3m 48s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 50m 13s | phoenix-spark-base in the patch failed. |
| +1 :green_heart: | unit | 0m 56s | phoenix5-spark3 in the patch passed. |
| -1 :x: | asflicense | 0m 21s | The patch generated 2 ASF License warnings. |
| | | 137m 52s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-80/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix-connectors/pull/80 |
| Optional Tests | dupname asflicense javac javadoc unit xml compile spotbugs hbaseanti checkstyle |
| uname | Linux 61a201f09b53 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-connectors-personality.sh |
| git revision | master / 9ed127f |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-80/2/artifact/yetus-general-check/output/diff-checkstyle-phoenix-spark-base.txt |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-80/2/artifact/yetus-general-check/output/diff-checkstyle-phoenix5-spark3.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-80/2/artifact/yetus-general-check/output/patch-unit-phoenix-spark-base.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-80/2/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-80/2/artifact/yetus-general-check/output/patch-asflicense-problems.txt |
| Max. process+thread count | 1834 (vs. ulimit of 30000) |
| modules | C: phoenix-spark-base phoenix5-spark3 U: . |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-80/2/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] chrajeshbabu commented on a diff in pull request #80: PHOENIX-6694 Avoid unnecessary calls of fetching table meta data to r…
Posted by GitBox <gi...@apache.org>.
chrajeshbabu commented on code in PR #80:
URL: https://github.com/apache/phoenix-connectors/pull/80#discussion_r887369726
##########
phoenix-spark-base/src/main/java/org/apache/phoenix/spark/datasource/v2/reader/PhoenixDataSourceReader.java:
##########
@@ -181,10 +184,12 @@ public List<InputPartition<InternalRow>> planInputPartitions() {
// Get the region size
long regionSize = CompatUtil.getSize(regionLocator, connection.getAdmin(), location);
-
+ byte[] tableBytes = PTableImpl.toProto(queryPlan.getTableRef().getTable()).
Review Comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] joshelser commented on a diff in pull request #80: PHOENIX-6694 Avoid unnecessary calls of fetching table meta data to r…
Posted by GitBox <gi...@apache.org>.
joshelser commented on code in PR #80:
URL: https://github.com/apache/phoenix-connectors/pull/80#discussion_r880977239
##########
phoenix-spark-base/src/main/java/org/apache/phoenix/spark/datasource/v2/reader/PhoenixInputPartitionReader.java:
##########
@@ -94,6 +99,10 @@ private QueryPlan getQueryPlan() throws SQLException {
}
try (Connection conn = DriverManager.getConnection(
JDBC_PROTOCOL + JDBC_PROTOCOL_SEPARATOR + zkUrl, overridingProps)) {
+ PTable pTable = PTable.parseFrom(options.getTableBytes());
+ org.apache.phoenix.schema.PTable table = PTableImpl.createFromProto(pTable);
+ PhoenixConnection phoenixConnection = conn.unwrap(PhoenixConnection.class);
+ phoenixConnection.addTable(table, System.currentTimeMillis());
Review Comment:
> Can you think of a case of when the jobs are delayed enough for this to matter, but enough of them start up synchrounously for the generated load to be a problem (I don't know enough about Spark to tell) ?
This used to be somethign that would happen in MapReduce with lots of mappers (where the state of the world might change from job submissions until the last mappers actually got scheduled and ran). Honestly, I don't think it's something we can really optimize anyways (and it's not the "common" path that people would be running `alter table` commands multiple times a day).
> the jobs will not be started immediately, and the syscat load is not really a problem.
That's a good point too. I didn't think about that side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] stoty commented on pull request #80: PHOENIX-6694 Avoid unnecessary calls of fetching table meta data to r…
Posted by GitBox <gi...@apache.org>.
stoty commented on PR #80:
URL: https://github.com/apache/phoenix-connectors/pull/80#issuecomment-1127121682
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 6m 28s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 6s | master passed |
| +1 :green_heart: | compile | 1m 59s | master passed |
| +1 :green_heart: | checkstyle | 0m 18s | master passed |
| +1 :green_heart: | javadoc | 0m 24s | master passed |
| +0 :ok: | spotbugs | 2m 48s | phoenix-spark-base in master has 40 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 15m 38s | root in the patch failed. |
| +1 :green_heart: | compile | 1m 48s | the patch passed |
| +1 :green_heart: | javac | 1m 48s | the patch passed |
| -1 :x: | checkstyle | 0m 16s | phoenix-spark-base: The patch generated 22 new + 127 unchanged - 10 fixed = 149 total (was 137) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed |
| +1 :green_heart: | spotbugs | 2m 38s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 49m 1s | phoenix-spark-base in the patch failed. |
| +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. |
| | | 122m 1s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-80/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix-connectors/pull/80 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile |
| uname | Linux 9974d96df439 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-connectors-personality.sh |
| git revision | master / 9ed127f |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| mvninstall | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-80/1/artifact/yetus-general-check/output/patch-mvninstall-root.txt |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-80/1/artifact/yetus-general-check/output/diff-checkstyle-phoenix-spark-base.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-80/1/artifact/yetus-general-check/output/patch-unit-phoenix-spark-base.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-80/1/testReport/ |
| Max. process+thread count | 1603 (vs. ulimit of 30000) |
| modules | C: phoenix-spark-base U: phoenix-spark-base |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-80/1/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] joshelser commented on a diff in pull request #80: PHOENIX-6694 Avoid unnecessary calls of fetching table meta data to r…
Posted by GitBox <gi...@apache.org>.
joshelser commented on code in PR #80:
URL: https://github.com/apache/phoenix-connectors/pull/80#discussion_r874129143
##########
phoenix-spark-base/src/main/java/org/apache/phoenix/spark/datasource/v2/reader/PhoenixInputPartitionReader.java:
##########
@@ -94,6 +99,10 @@ private QueryPlan getQueryPlan() throws SQLException {
}
try (Connection conn = DriverManager.getConnection(
JDBC_PROTOCOL + JDBC_PROTOCOL_SEPARATOR + zkUrl, overridingProps)) {
+ PTable pTable = PTable.parseFrom(options.getTableBytes());
+ org.apache.phoenix.schema.PTable table = PTableImpl.createFromProto(pTable);
+ PhoenixConnection phoenixConnection = conn.unwrap(PhoenixConnection.class);
+ phoenixConnection.addTable(table, System.currentTimeMillis());
Review Comment:
Please add a log message that we are adding a cached version of the PTable here (rather than resolving the real table).
On the off chance that there is some problem (from when we read the table until when we ran this code), it would be good to remind anyone looking at the log that we're doing this trick.
##########
phoenix-spark-base/src/main/java/org/apache/phoenix/spark/datasource/v2/reader/PhoenixInputPartitionReader.java:
##########
@@ -81,7 +86,7 @@ Properties getOverriddenPropsFromOptions() {
return options.getOverriddenProps();
}
- private QueryPlan getQueryPlan() throws SQLException {
+ private QueryPlan getQueryPlan() throws SQLException, InvalidProtocolBufferException {
Review Comment:
I think you could catch the `InvalidProtocolBufferException` and throw it as a RuntimeException in `getQueryPlan()`, rather than doing it in `initialize()`
##########
phoenix-spark-base/src/main/java/org/apache/phoenix/spark/datasource/v2/reader/PhoenixInputPartitionReader.java:
##########
@@ -94,6 +99,10 @@ private QueryPlan getQueryPlan() throws SQLException {
}
try (Connection conn = DriverManager.getConnection(
JDBC_PROTOCOL + JDBC_PROTOCOL_SEPARATOR + zkUrl, overridingProps)) {
+ PTable pTable = PTable.parseFrom(options.getTableBytes());
+ org.apache.phoenix.schema.PTable table = PTableImpl.createFromProto(pTable);
+ PhoenixConnection phoenixConnection = conn.unwrap(PhoenixConnection.class);
+ phoenixConnection.addTable(table, System.currentTimeMillis());
Review Comment:
Also, do we want to record the time in which the PTable was cached and use that rather than `System.currentTimeMillis()`?
My thinking is probably "no". If we use the original time we read the PTable and a long time elapsed (really big job, starvation on the Executor side), it could be hour(s) until this code is actually executed. If the `UPDATE_CACHE_FREQUENCY` isn't increased to a large value, it might cause the Phoenix client to re-fetch the PTable again? (just guessing).
Have you thought about this case already?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] stoty commented on a diff in pull request #80: PHOENIX-6694 Avoid unnecessary calls of fetching table meta data to r…
Posted by GitBox <gi...@apache.org>.
stoty commented on code in PR #80:
URL: https://github.com/apache/phoenix-connectors/pull/80#discussion_r873581804
##########
phoenix-spark-base/src/main/java/org/apache/phoenix/spark/datasource/v2/reader/PhoenixDataSourceReader.java:
##########
@@ -181,10 +184,12 @@ public List<InputPartition<InternalRow>> planInputPartitions() {
// Get the region size
long regionSize = CompatUtil.getSize(regionLocator, connection.getAdmin(), location);
-
+ byte[] tableBytes = PTableImpl.toProto(queryPlan.getTableRef().getTable()).
Review Comment:
Can we have a bit more descripte name ?
Like pTableCacheBytes ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] stoty commented on a diff in pull request #80: PHOENIX-6694 Avoid unnecessary calls of fetching table meta data to r…
Posted by GitBox <gi...@apache.org>.
stoty commented on code in PR #80:
URL: https://github.com/apache/phoenix-connectors/pull/80#discussion_r874344921
##########
phoenix-spark-base/src/main/java/org/apache/phoenix/spark/datasource/v2/reader/PhoenixInputPartitionReader.java:
##########
@@ -94,6 +99,10 @@ private QueryPlan getQueryPlan() throws SQLException {
}
try (Connection conn = DriverManager.getConnection(
JDBC_PROTOCOL + JDBC_PROTOCOL_SEPARATOR + zkUrl, overridingProps)) {
+ PTable pTable = PTable.parseFrom(options.getTableBytes());
+ org.apache.phoenix.schema.PTable table = PTableImpl.createFromProto(pTable);
+ PhoenixConnection phoenixConnection = conn.unwrap(PhoenixConnection.class);
+ phoenixConnection.addTable(table, System.currentTimeMillis());
Review Comment:
Interesting point about the timestamp.
The point of this patch is to avoid hammering the system tables with a huge number of parallel requests.
I think that if we have executor starvation, then the jobs will not be started immediately, and the syscat load is not really a problem.
Can you think of a case of when the jobs are delayed enough for this to matter, but enough of them start up synchrounously for the generated load to be a problem (I don't know enough about Spark to tell) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix-connectors] chrajeshbabu merged pull request #80: PHOENIX-6694 Avoid unnecessary calls of fetching table meta data to r…
Posted by GitBox <gi...@apache.org>.
chrajeshbabu merged PR #80:
URL: https://github.com/apache/phoenix-connectors/pull/80
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org