You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by ankitsinghal <gi...@git.apache.org> on 2017/07/21 08:48:46 UTC
[GitHub] phoenix pull request #268: PHOENIX-4010 Hash Join cache may not be send to a...
GitHub user ankitsinghal opened a pull request:
https://github.com/apache/phoenix/pull/268
PHOENIX-4010 Hash Join cache may not be send to all regionservers whe…
…n we have stale HBase meta cache
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ankitsinghal/phoenix PHOENIX-4010
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/phoenix/pull/268.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #268
----
commit ebf6dbf6a9a9d8bb408cec70f99f3fb5abc8ddcc
Author: Ankit Singhal <an...@gmail.com>
Date: 2017-07-21T08:47:31Z
PHOENIX-4010 Hash Join cache may not be send to all regionservers when we have stale HBase meta cache
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request #268: PHOENIX-4010 Hash Join cache may not be send to a...
Posted by JamesRTaylor <gi...@git.apache.org>.
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/268#discussion_r129152257
--- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/HashJoinIT.java ---
@@ -3453,4 +3496,27 @@ public void testJoinWithSetMaxRows() throws Exception {
conn.close();
}
}
+
+ public static class InvalidateHashCacheRandomly extends SimpleRegionObserver {
--- End diff --
These join tests run on the same cluster as all other tests derived from ParallelStatsDisabledIT. Rather than having a random test failure, let's move this to a dedicated test derived from BaseUniqueNamesOwnClusterIT and have it consistently fail.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix issue #268: PHOENIX-4010 Hash Join cache may not be send to all regi...
Posted by ankitsinghal <gi...@git.apache.org>.
Github user ankitsinghal commented on the issue:
https://github.com/apache/phoenix/pull/268
@JamesRTaylor , can you please review, I have removed the dependency on PHOENIX-3163.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request #268: PHOENIX-4010 Hash Join cache may not be send to a...
Posted by ankitsinghal <gi...@git.apache.org>.
Github user ankitsinghal commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/268#discussion_r129589111
--- Diff: phoenix-core/src/main/java/org/apache/phoenix/compile/UpsertCompiler.java ---
@@ -804,6 +804,7 @@ public Long getEstimatedBytesToScan() throws SQLException {
////////////////////////////////////////////////////////////////////
// UPSERT SELECT run client-side
/////////////////////////////////////////////////////////////////////
+ ScanUtil.setClientSideUpsertSelect(queryPlan.getContext().getScan());
--- End diff --
this change is actually a part of PHOENIX-3163, let me see if we can improve.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request #268: PHOENIX-4010 Hash Join cache may not be send to a...
Posted by JamesRTaylor <gi...@git.apache.org>.
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/268#discussion_r129187485
--- Diff: phoenix-core/src/main/java/org/apache/phoenix/compile/UpsertCompiler.java ---
@@ -804,6 +804,7 @@ public Long getEstimatedBytesToScan() throws SQLException {
////////////////////////////////////////////////////////////////////
// UPSERT SELECT run client-side
/////////////////////////////////////////////////////////////////////
+ ScanUtil.setClientSideUpsertSelect(queryPlan.getContext().getScan());
--- End diff --
This would be better to encapsulate in the StatementPlan if we even need it. Why do we need it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request #268: PHOENIX-4010 Hash Join cache may not be send to a...
Posted by JamesRTaylor <gi...@git.apache.org>.
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/268#discussion_r129187099
--- Diff: phoenix-core/src/main/java/org/apache/phoenix/cache/ServerCacheClient.java ---
@@ -303,73 +318,84 @@ public TaskExecutionMetricsHolder getTaskExecutionMetric() {
* @throws SQLException
* @throws IllegalStateException if hashed table cannot be removed on any region server on which it was added
*/
- private void removeServerCache(final byte[] cacheId, Set<HRegionLocation> servers) throws SQLException {
- ConnectionQueryServices services = connection.getQueryServices();
- Throwable lastThrowable = null;
- TableRef cacheUsingTableRef = cacheUsingTableRefMap.get(Bytes.mapKey(cacheId));
- final PTable cacheUsingTable = cacheUsingTableRef.getTable();
- byte[] tableName = cacheUsingTableRef.getTable().getPhysicalName().getBytes();
- HTableInterface iterateOverTable = services.getTable(tableName);
- try {
- List<HRegionLocation> locations = services.getAllTableRegions(tableName);
- Set<HRegionLocation> remainingOnServers = new HashSet<HRegionLocation>(servers);
- /**
- * Allow for the possibility that the region we based where to send our cache has split and been
- * relocated to another region server *after* we sent it, but before we removed it. To accommodate
- * this, we iterate through the current metadata boundaries and remove the cache once for each
- * server that we originally sent to.
- */
- if (LOG.isDebugEnabled()) {LOG.debug(addCustomAnnotations("Removing Cache " + cacheId + " from servers.", connection));}
- for (HRegionLocation entry : locations) {
- if (remainingOnServers.contains(entry)) { // Call once per server
- try {
+ private void removeServerCache(final ServerCache cache, Set<HRegionLocation> servers) throws SQLException {
+ HTableInterface iterateOverTable = null;
+ final byte[] cacheId = cache.getId();
+ try {
+ List<byte[]> keys = cache.getKeys();
+ ConnectionQueryServices services = connection.getQueryServices();
+ Throwable lastThrowable = null;
+ TableRef cacheUsingTableRef = cacheUsingTableRefMap.get(Bytes.mapKey(cacheId));
+ final PTable cacheUsingTable = cacheUsingTableRef.getTable();
+ byte[] tableName = cacheUsingTableRef.getTable().getPhysicalName().getBytes();
+ iterateOverTable = services.getTable(tableName);
+
+ List<HRegionLocation> locations = services.getAllTableRegions(tableName);
+ Set<HRegionLocation> remainingOnServers = new HashSet<HRegionLocation>(servers);
+ /**
+ * Allow for the possibility that the region we based where to send our cache has split and been relocated
+ * to another region server *after* we sent it, but before we removed it. To accommodate this, we iterate
+ * through the current metadata boundaries and remove the cache once for each server that we originally sent
+ * to.
+ */
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(addCustomAnnotations("Removing Cache " + cacheId + " from servers.", connection));
+ }
+ for (HRegionLocation entry : locations) {
+ // Call once per server
+ if (remainingOnServers.contains(entry)
+ || (keys != null && ByteUtil.contains(keys,entry.getRegionInfo().getStartKey()))) {
--- End diff --
Why is this extra check required? Is this when the Hash Cache is sent over to the RS that doesn't have it? If so, can we document that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request #268: PHOENIX-4010 Hash Join cache may not be send to a...
Posted by JamesRTaylor <gi...@git.apache.org>.
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/268#discussion_r129601553
--- Diff: phoenix-core/src/main/java/org/apache/phoenix/compile/UpsertCompiler.java ---
@@ -804,6 +804,7 @@ public Long getEstimatedBytesToScan() throws SQLException {
////////////////////////////////////////////////////////////////////
// UPSERT SELECT run client-side
/////////////////////////////////////////////////////////////////////
+ ScanUtil.setClientSideUpsertSelect(queryPlan.getContext().getScan());
--- End diff --
Let's do those changes in a separate patch, please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix issue #268: PHOENIX-4010 Hash Join cache may not be send to all regi...
Posted by ankitsinghal <gi...@git.apache.org>.
Github user ankitsinghal commented on the issue:
https://github.com/apache/phoenix/pull/268
Thanks @JamesRTaylor , have taken care the review comments in latest commit. Let me know if it is fine now
---
[GitHub] phoenix pull request #268: PHOENIX-4010 Hash Join cache may not be send to a...
Posted by JamesRTaylor <gi...@git.apache.org>.
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/268#discussion_r137629055
--- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/BaseJoinIT.java ---
@@ -30,14 +30,19 @@
import java.util.Map;
import java.util.regex.Pattern;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.phoenix.cache.ServerCacheClient;
+import org.apache.phoenix.end2end.HashJoinCacheIT.InvalidateHashCacheRandomly;
+import org.apache.phoenix.util.ReadOnlyProps;
import org.apache.phoenix.util.SchemaUtil;
import org.apache.phoenix.util.StringUtil;
import org.junit.Before;
+import org.junit.BeforeClass;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.Maps;
-public abstract class BaseJoinIT extends ParallelStatsDisabledIT {
+public abstract class BaseJoinIT extends BaseUniqueNamesOwnClusterIT {
--- End diff --
Can we keep BaseJoinIT derived from ParallelStatsDisabledIT? Otherwise, there's a lot more mini clusters created which increases our end-to-end test time run more and more.
If the reason is because you're adding InvalidateHashCacheRandomly, just add it instead dynamically to the table created in your test. To prevent flapping, you need to poll until you the descriptor matches what you've changed. See PartialIndexRebuilderIT.addWriteFailingCoprocessor() for an example you can generalize and/or copy/paste.
---
[GitHub] phoenix pull request #268: PHOENIX-4010 Hash Join cache may not be send to a...
Posted by JamesRTaylor <gi...@git.apache.org>.
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/268#discussion_r129188027
--- Diff: phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java ---
@@ -108,6 +126,10 @@ public TableResultIterator(MutationState mutationState, Scan scan, ScanMetricsHo
this.scanIterator = UNINITIALIZED_SCANNER;
this.renewLeaseThreshold = renewLeaseThreshold;
this.scanGrouper = scanGrouper;
+ this.hashCacheClient = new HashCacheClient(plan.getContext().getConnection());
+ this.caches = caches;
+ this.retry=plan.getContext().getConnection().getQueryServices().getConfiguration()
--- End diff --
Can we use getProps() here instead of going through the Configuration object? We've found the synchronization done by Configuration to be a bottleneck in the past (and not necessary since these properties don't change).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request #268: PHOENIX-4010 Hash Join cache may not be send to a...
Posted by JamesRTaylor <gi...@git.apache.org>.
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/268#discussion_r137628509
--- Diff: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/BaseScannerRegionObserver.java ---
@@ -109,12 +110,16 @@
/** Exposed for testing */
public static final String SCANNER_OPENED_TRACE_INFO = "Scanner opened on server";
+ public static final String CLIENT_SIDE_UPSERT_SELECT = "_ClientSideUpsertSelect";
+ protected Configuration rawConf;
--- End diff --
What's the reason for this new rawConf member variable? Where is it used?
---
[GitHub] phoenix issue #268: PHOENIX-4010 Hash Join cache may not be send to all regi...
Posted by ankitsinghal <gi...@git.apache.org>.
Github user ankitsinghal commented on the issue:
https://github.com/apache/phoenix/pull/268
Ping @JamesRTaylor
---
[GitHub] phoenix pull request #268: PHOENIX-4010 Hash Join cache may not be send to a...
Posted by JamesRTaylor <gi...@git.apache.org>.
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/268#discussion_r129186906
--- Diff: phoenix-core/src/main/java/org/apache/phoenix/cache/ServerCacheClient.java ---
@@ -303,73 +318,84 @@ public TaskExecutionMetricsHolder getTaskExecutionMetric() {
* @throws SQLException
* @throws IllegalStateException if hashed table cannot be removed on any region server on which it was added
*/
- private void removeServerCache(final byte[] cacheId, Set<HRegionLocation> servers) throws SQLException {
- ConnectionQueryServices services = connection.getQueryServices();
- Throwable lastThrowable = null;
- TableRef cacheUsingTableRef = cacheUsingTableRefMap.get(Bytes.mapKey(cacheId));
- final PTable cacheUsingTable = cacheUsingTableRef.getTable();
- byte[] tableName = cacheUsingTableRef.getTable().getPhysicalName().getBytes();
- HTableInterface iterateOverTable = services.getTable(tableName);
- try {
- List<HRegionLocation> locations = services.getAllTableRegions(tableName);
- Set<HRegionLocation> remainingOnServers = new HashSet<HRegionLocation>(servers);
- /**
- * Allow for the possibility that the region we based where to send our cache has split and been
- * relocated to another region server *after* we sent it, but before we removed it. To accommodate
- * this, we iterate through the current metadata boundaries and remove the cache once for each
- * server that we originally sent to.
- */
- if (LOG.isDebugEnabled()) {LOG.debug(addCustomAnnotations("Removing Cache " + cacheId + " from servers.", connection));}
- for (HRegionLocation entry : locations) {
- if (remainingOnServers.contains(entry)) { // Call once per server
- try {
+ private void removeServerCache(final ServerCache cache, Set<HRegionLocation> servers) throws SQLException {
+ HTableInterface iterateOverTable = null;
+ final byte[] cacheId = cache.getId();
+ try {
+ List<byte[]> keys = cache.getKeys();
+ ConnectionQueryServices services = connection.getQueryServices();
+ Throwable lastThrowable = null;
+ TableRef cacheUsingTableRef = cacheUsingTableRefMap.get(Bytes.mapKey(cacheId));
+ final PTable cacheUsingTable = cacheUsingTableRef.getTable();
+ byte[] tableName = cacheUsingTableRef.getTable().getPhysicalName().getBytes();
+ iterateOverTable = services.getTable(tableName);
+
+ List<HRegionLocation> locations = services.getAllTableRegions(tableName);
+ Set<HRegionLocation> remainingOnServers = new HashSet<HRegionLocation>(servers);
--- End diff --
Instead of Set<HRegionLocation> can we change this to Set<ServerName>? It'd make it more clear IMHO
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request #268: PHOENIX-4010 Hash Join cache may not be send to a...
Posted by JamesRTaylor <gi...@git.apache.org>.
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/268#discussion_r129185736
--- Diff: phoenix-core/src/main/java/org/apache/phoenix/iterate/TableResultIterator.java ---
@@ -163,8 +185,38 @@ public Tuple next() throws SQLException {
}
}
plan.getContext().getConnection().getQueryServices().clearTableRegionCache(htable.getTableName());
- this.scanIterator =
- plan.iterator(scanGrouper, newScan);
+ if (e1 instanceof HashJoinCacheNotFoundException) {
+ logger.debug(
+ "Retrying when Hash Join cache is not found on the server ,by sending the cache again");
+ if (retry <= 0) { throw e1; }
+ retry--;
+ try {
+ Long cacheId = ((HashJoinCacheNotFoundException)e1).getCacheId();
+ if (!hashCacheClient.addHashCacheToServer(newScan.getStartRow(),
+ ServerCacheClient.getCacheForId(caches, cacheId), plan.getTableRef().getTable())) { throw e1; }
+
+ try {
+ if(ScanUtil.isClientSideUpsertSelect(newScan)) {
--- End diff --
Why is this necessary? Can we generalize this is a better way?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix issue #268: PHOENIX-4010 Hash Join cache may not be send to all regi...
Posted by JamesRTaylor <gi...@git.apache.org>.
Github user JamesRTaylor commented on the issue:
https://github.com/apache/phoenix/pull/268
+1
---
[GitHub] phoenix pull request #268: PHOENIX-4010 Hash Join cache may not be send to a...
Posted by JamesRTaylor <gi...@git.apache.org>.
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/268#discussion_r129185421
--- Diff: phoenix-core/src/main/java/org/apache/phoenix/cache/ServerCacheClient.java ---
@@ -394,4 +420,82 @@ public static String idToString(byte[] uuid) {
}
return regionStartKey;
}
+
+ public boolean addServerCache(byte[] startkeyOfRegion, ServerCache cache, HashCacheFactory cacheFactory,
+ byte[] txState, PTable pTable) throws Exception {
+ HTableInterface table = null;
+ byte[] cacheId = cache.getId();
+ try {
+ ConnectionQueryServices services = connection.getQueryServices();
+ byte[] tableName = pTable.getPhysicalName().getBytes();
+ table = services.getTable(tableName);
+ boolean success = addServerCache(table, startkeyOfRegion, pTable, cacheId, cache.getCachePtr(), cacheFactory, txState);
+ //track keys so that we can remove the hash table cache from the new regionservers where the cache was re-sent
+ cache.addKey(startkeyOfRegion);
+ return success;
+ } finally {
+ Closeables.closeQuietly(table);
+ }
+ }
+
+ public boolean addServerCache(HTableInterface htable, byte[] key, final PTable cacheUsingTable, final byte[] cacheId,
+ final ImmutableBytesWritable cachePtr, final ServerCacheFactory cacheFactory, final byte[] txState)
+ throws Exception {
+ byte[] keyInRegion = getKeyInRegion(key);
+ final Map<byte[], AddServerCacheResponse> results;
+ try {
+ results = htable.coprocessorService(ServerCachingService.class, keyInRegion, keyInRegion,
+ new Batch.Call<ServerCachingService, AddServerCacheResponse>() {
+ @Override
+ public AddServerCacheResponse call(ServerCachingService instance) throws IOException {
+ ServerRpcController controller = new ServerRpcController();
+ BlockingRpcCallback<AddServerCacheResponse> rpcCallback = new BlockingRpcCallback<AddServerCacheResponse>();
+ AddServerCacheRequest.Builder builder = AddServerCacheRequest.newBuilder();
+ final byte[] tenantIdBytes;
+ if (cacheUsingTable.isMultiTenant()) {
+ try {
+ tenantIdBytes = connection.getTenantId() == null ? null
+ : ScanUtil.getTenantIdBytes(cacheUsingTable.getRowKeySchema(),
+ cacheUsingTable.getBucketNum() != null, connection.getTenantId(),
+ cacheUsingTable.getViewIndexId() != null);
+ } catch (SQLException e) {
+ throw new IOException(e);
+ }
+ } else {
+ tenantIdBytes = connection.getTenantId() == null ? null
+ : connection.getTenantId().getBytes();
+ }
+ if (tenantIdBytes != null) {
+ builder.setTenantId(ByteStringer.wrap(tenantIdBytes));
+ }
+ builder.setCacheId(ByteStringer.wrap(cacheId));
+ builder.setCachePtr(org.apache.phoenix.protobuf.ProtobufUtil.toProto(cachePtr));
+ builder.setHasProtoBufIndexMaintainer(true);
+ ServerCacheFactoryProtos.ServerCacheFactory.Builder svrCacheFactoryBuider = ServerCacheFactoryProtos.ServerCacheFactory
+ .newBuilder();
+ svrCacheFactoryBuider.setClassName(cacheFactory.getClass().getName());
+ builder.setCacheFactory(svrCacheFactoryBuider.build());
+ builder.setTxState(ByteStringer.wrap(txState));
+ instance.addServerCache(controller, builder.build(), rpcCallback);
+ if (controller.getFailedOn() != null) { throw controller.getFailedOn(); }
+ return rpcCallback.get();
+ }
+ });
+ } catch (Throwable t) {
+ throw new Exception(t);
+ }
+ if (results != null && results.size() == 1) { return results.values().iterator().next().getReturn(); }
+ return false;
+
+ }
+
+ public static ServerCache getCacheForId(List<ServerCache> caches, Long cacheId) {
--- End diff --
How about using a Map<ImmutableBytesPtr,ServerCache> instead of a List and switching id to ImmutableBytesPtr? Then you can get rid of this static method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---