You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@phoenix.apache.org by "virajjasani (via GitHub)" <gi...@apache.org> on 2024/08/16 01:53:12 UTC
[PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
virajjasani opened a new pull request, #1955:
URL: https://github.com/apache/phoenix/pull/1955
Jira: PHOENIX-7386
If table has UPDATE_CACHE_FREQUENCY set to non-default i.e. anything other than ALWAYS, depending on the value, PTable objects are cached until that duration at the client side metadata cache. One of the common cases for creating index is to create index in CREATE_DISABLE state.
When any index is in DISABLE, CREATE_DISABLE or PENDING_ACTIVE state, Phoenix client does not include PTable of the corresponding index as part of the IndexMaintainer objects to server. Since indexes are expected to be either dropped or in building state from the above disabled states, it is crucial for Phoenix client to override the non-default value of UPDATE_CACHE_FREQUENCY on the base table. This would help avoid any data integrity issue because until the index state becomes BUILDING or ACTIVE, the client will continue to override UPDATE_CACHE_FREQUENCY to default value to let the UPSERT and/or SELECT queries to initiate getTable() RPC and override the client side metadata cache value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1720258899
##########
phoenix-core-client/src/main/java/org/apache/phoenix/schema/MetaDataClient.java:
##########
@@ -785,19 +785,33 @@ private boolean avoidRpcToGetTable(boolean alwaysHitServer, Long resolvedTimesta
final long effectiveUpdateCacheFreq;
final String ucfInfoForLogging; // Only used for logging purposes
+ boolean overrideUcfToDefault =
Review Comment:
Sounds good, let me extend this support as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1719195541
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheFreqWithDDLValidationIT.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.end2end;
+
+import org.apache.phoenix.coprocessorclient.BaseScannerRegionObserverConstants;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Maps;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.junit.BeforeClass;
+
+import java.util.Map;
+
+//@Category(NeedsOwnMiniClusterTest.class)
+// Uncomment as part of PHOENIX-7381
Review Comment:
FYI @palashc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1723935098
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheFreqIT.java:
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.end2end;
+
+import com.google.protobuf.RpcCallback;
+import com.google.protobuf.RpcController;
+import org.apache.hadoop.hbase.DoNotRetryIOException;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.compile.ExplainPlan;
+import org.apache.phoenix.compile.ExplainPlanAttributes;
+import org.apache.phoenix.coprocessor.MetaDataEndpointImpl;
+import org.apache.phoenix.coprocessor.generated.MetaDataProtos;
+import org.apache.phoenix.coprocessorclient.BaseScannerRegionObserverConstants;
+import org.apache.phoenix.exception.PhoenixIOException;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.jdbc.PhoenixPreparedStatement;
+import org.apache.phoenix.protobuf.ProtobufUtil;
+import org.apache.phoenix.query.BaseTest;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Maps;
+import org.apache.phoenix.util.ClientUtil;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.Map;
+
+import static org.apache.phoenix.query.QueryServices.DISABLE_VIEW_SUBTREE_VALIDATION;
+
+@Category(NeedsOwnMiniClusterTest.class)
+public class UpdateCacheFreqIT extends BaseTest {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(UpdateCacheFreqIT.class);
+
+ private static void initCluster() throws Exception {
+ Map<String, String> props = Maps.newConcurrentMap();
+ props.put(BaseScannerRegionObserverConstants.PHOENIX_MAX_LOOKBACK_AGE_CONF_KEY,
+ Integer.toString(60 * 60 * 1000));
+ props.put(QueryServices.DEFAULT_UPDATE_CACHE_FREQUENCY_ATRRIB, "ALWAYS");
+ props.put(QueryServices.LAST_DDL_TIMESTAMP_VALIDATION_ENABLED, Boolean.toString(false));
+ props.put(QueryServices.PHOENIX_METADATA_INVALIDATE_CACHE_ENABLED, Boolean.toString(false));
+ props.put(QueryServices.TASK_HANDLING_INITIAL_DELAY_MS_ATTRIB,
+ Long.toString(Long.MAX_VALUE));
+ props.put(DISABLE_VIEW_SUBTREE_VALIDATION, "true");
+ setUpTestDriver(new ReadOnlyProps(props.entrySet().iterator()));
+ }
+
+ @BeforeClass
+ public static synchronized void doSetup() throws Exception {
+ initCluster();
+ }
+
+ public static class TestMetaDataEndpointImpl extends MetaDataEndpointImpl {
+
+ @Override
+ public void getTable(RpcController controller, MetaDataProtos.GetTableRequest request,
+ RpcCallback<MetaDataProtos.MetaDataResponse> done) {
+ LOGGER.error("Not expected to get getTable() for {}",
+ Bytes.toString(request.getTableName().toByteArray()));
+ ProtobufUtil.setControllerException(controller,
+ ClientUtil.createIOException(
+ SchemaUtil.getPhysicalTableName(
+ PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES,
+ false).toString(),
+ new DoNotRetryIOException("Not allowed")));
+
+ }
+
+ }
+
+ @Test
+ public void testUcfWithNoGetTableCalls() throws Throwable {
+ final String tableName = generateUniqueName();
+ final String indexName = "IDX_" + tableName;
+ final String view01 = "v01_" + tableName;
+ final String view02 = "v02_" + tableName;
+ final String index_view01 = "idx_v01_" + tableName;
+ final String index_view02 = "idx_v02_" + tableName;
+ final String index_view03 = "idx_v03_" + tableName;
+ final String index_view04 = "idx_v04_" + tableName;
+
+ try (Connection conn = DriverManager.getConnection(getUrl())) {
+ final Statement stmt = conn.createStatement();
+
+ stmt.execute("CREATE TABLE " + tableName
+ + " (COL1 CHAR(10) NOT NULL, COL2 CHAR(5) NOT NULL, COL3 VARCHAR,"
+ + " COL4 VARCHAR CONSTRAINT pk PRIMARY KEY(COL1, COL2))"
+ + " UPDATE_CACHE_FREQUENCY=20000");
+ stmt.execute("CREATE INDEX " + indexName
+ + " ON " + tableName + " (COL3) INCLUDE (COL4)");
+ stmt.execute("CREATE VIEW " + view01
+ + " (VCOL1 CHAR(8), COL5 VARCHAR) AS SELECT * FROM " + tableName
+ + " WHERE COL1 = 'col1'");
+ stmt.execute("CREATE VIEW " + view02 + " (VCOL2 CHAR(10), COL6 VARCHAR)"
+ + " AS SELECT * FROM " + view01 + " WHERE VCOL1 = 'vcol1'");
+
+ stmt.execute("CREATE INDEX " + index_view01 + " ON " + view01 + " (COL5) INCLUDE "
+ + "(COL1, COL2, COL3)");
+ stmt.execute("CREATE INDEX " + index_view02 + " ON " + view02 + " (COL6) INCLUDE "
+ + "(COL1, COL2, COL3)");
+ stmt.execute("CREATE INDEX " + index_view03 + " ON " + view01 + " (COL5) INCLUDE "
+ + "(COL2, COL1)");
+ stmt.execute("CREATE INDEX " + index_view04 + " ON " + view02 + " (COL6) INCLUDE "
+ + "(COL2, COL1)");
+
+ stmt.execute("UPSERT INTO " + view02
+ + " (col2, vcol2, col5, col6) values ('0001', 'vcol2_01', 'col5_01', " +
+ "'col6_01')");
+ conn.commit();
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0002', 'vcol2_02', 'col5_02', 'col6_02')");
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0003', 'vcol2_03', 'col5_03', 'col6_03')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0004', 'vcol2', 'col3_04', 'col4_04', 'col5_04')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0005', 'vcol-2', 'col3_05', 'col4_05', 'col5_05')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0006', 'vcol-1', 'col3_06', 'col4_06', 'col5_06')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0007', 'vcol1', 'col3_07', 'col4_07', 'col5_07')");
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0008', 'vcol2_08', 'col5_08', 'col6_02')");
+ conn.commit();
+
+ final Statement statement = conn.createStatement();
+ ResultSet rs =
+ statement.executeQuery(
+ "SELECT COL2, VCOL1, VCOL2, COL5, COL6 FROM " + view02);
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertFalse(rs.next());
+
+ String query = "SELECT COL1, COL2, COL3, COL4 FROM " + tableName + " WHERE COL3 = " +
+ "'col3_04'";
+ rs = statement.executeQuery(query);
+ Assert.assertTrue(rs.next());
+ Assert.assertFalse(rs.next());
+
+ ExplainPlan explainPlan = conn.prepareStatement(query)
+ .unwrap(PhoenixPreparedStatement.class).optimizeQuery()
+ .getExplainPlan();
+ ExplainPlanAttributes explainPlanAttributes =
+ explainPlan.getPlanStepsAsAttributes();
+ Assert.assertEquals(indexName, explainPlanAttributes.getTableName());
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ }
+ }
+
+ @Test
+ public void testUcfWithDisabledIndex() throws Throwable {
+ final String tableName = generateUniqueName();
+ final String indexName = "IDX_" + tableName;
+
+ Connection conn = DriverManager.getConnection(getUrl());
+ try {
+ final Statement stmt = conn.createStatement();
+
+ stmt.execute("CREATE TABLE " + tableName
+ + " (COL1 CHAR(10) NOT NULL, COL2 CHAR(5) NOT NULL, COL3 VARCHAR,"
+ + " COL4 VARCHAR CONSTRAINT pk PRIMARY KEY(COL1, COL2))"
+ + " UPDATE_CACHE_FREQUENCY=10000");
+ stmt.execute("CREATE INDEX " + indexName
+ + " ON " + tableName + " (COL3) INCLUDE (COL4)");
+
+ stmt.execute("ALTER INDEX " + indexName + " ON " + tableName + " DISABLE");
+
+ Thread.sleep(11000);
+ stmt.execute("UPSERT INTO " + tableName
+ + " (col1, col2, col3, col4) values ('c001', "
+ + "'c002', 'c003', 'c004')");
+ conn.commit();
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+
+ stmt.execute("UPSERT INTO " + tableName
+ + " (col1, col2, col3, col4) values ('c011', "
+ + "'c012', 'c013', 'c014')");
+ conn.commit();
+ throw new RuntimeException("Should not reach here");
+ } catch (PhoenixIOException e) {
Review Comment:
The reason why i am using ALTER TABLE to set index state is because otherwise using async index means running jobs to get the index state updated right? We want to avoid that and directly check index table state to begin with. The testing is same but avoiding async index status update is better as it tends to make tests flaky. We don't know by when the index state will be updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1720601097
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheFreqWithDDLValidationIT.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.end2end;
+
+import org.apache.phoenix.coprocessorclient.BaseScannerRegionObserverConstants;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Maps;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.junit.BeforeClass;
+
+import java.util.Map;
+
+//@Category(NeedsOwnMiniClusterTest.class)
+// Uncomment as part of PHOENIX-7381
Review Comment:
This is not the case for ALWAYS, this is some non default positive value e.g. 20s or 10s. The idea is, even when we have metadata caching based DDL timestamp validation enabled and we have not yet updated UCF to NEVER for those tables with some positive UCF value, we need to ensure that even DDL timestamp validation API does not get called with every query if UCF has some positive value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1724102616
##########
phoenix-core-client/src/main/java/org/apache/phoenix/schema/MetaDataClient.java:
##########
@@ -785,19 +785,37 @@ private boolean avoidRpcToGetTable(boolean alwaysHitServer, Long resolvedTimesta
final long effectiveUpdateCacheFreq;
final String ucfInfoForLogging; // Only used for logging purposes
+ boolean overrideUcfToDefault = false;
+ if (table.getType() == INDEX) {
+ overrideUcfToDefault =
+ PIndexState.PENDING_DISABLE.equals(table.getIndexState()) ||
+ !IndexMaintainer.sendIndexMaintainer(table);
+ }
+ if (!overrideUcfToDefault && !table.getIndexes().isEmpty()) {
+ List<PTable> indexes = table.getIndexes();
+ List<PTable> maintainedIndexes =
+ Lists.newArrayList(IndexMaintainer.maintainedIndexes(indexes.iterator()));
+ // The maintainedIndexes contain only the indexes that are used by clients
+ // while generating the mutations. If all the indexes are usable by clients,
+ // we don't need to override UPDATE_CACHE_FREQUENCY. However, if any index is
+ // not in usable state by the client mutations, we should override
+ // UPDATE_CACHE_FREQUENCY to default value so that we make getTable() RPC calls
+ // until all index states change to ACTIVE, BUILDING or other usable states.
+ overrideUcfToDefault = indexes.size() != maintainedIndexes.size();
+ }
+
// What if the table is created with UPDATE_CACHE_FREQUENCY explicitly set to ALWAYS?
// i.e. explicitly set to 0. We should ideally be checking for something like
// hasUpdateCacheFrequency().
//always fetch an Index in PENDING_DISABLE state to retrieve server timestamp
//QueryOptimizer needs that to decide whether the index can be used
- if (PIndexState.PENDING_DISABLE.equals(table.getIndexState())) {
+ if (overrideUcfToDefault) {
Review Comment:
Exactly, we will need some level of re-iteration of PHOENIX-6883 with UCF. Ultimately, when client sets UCF to some value other than ALWAYS and NEVER, we need to ensure no getTable() or validateDdlTimestamp() RPC calls are initiated by clients for the duration of UCF. Its good that now we know this, so we will re-iterate on PHOENIX-6883 :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1720904297
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheFreqWithDDLValidationIT.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.end2end;
+
+import org.apache.phoenix.coprocessorclient.BaseScannerRegionObserverConstants;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Maps;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.junit.BeforeClass;
+
+import java.util.Map;
+
+//@Category(NeedsOwnMiniClusterTest.class)
+// Uncomment as part of PHOENIX-7381
Review Comment:
My point is, when we uncomment this test as part of PHOENIX-7381, the tests will likely pass. Until PHOENIX-7381, these tests will not pass because even for 10/20s UCF, they still perform validate ddl timestamp and that makes getTable() call internally, hence the test fails as of now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1723763990
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheFreqIT.java:
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.end2end;
+
+import com.google.protobuf.RpcCallback;
+import com.google.protobuf.RpcController;
+import org.apache.hadoop.hbase.DoNotRetryIOException;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.compile.ExplainPlan;
+import org.apache.phoenix.compile.ExplainPlanAttributes;
+import org.apache.phoenix.coprocessor.MetaDataEndpointImpl;
+import org.apache.phoenix.coprocessor.generated.MetaDataProtos;
+import org.apache.phoenix.coprocessorclient.BaseScannerRegionObserverConstants;
+import org.apache.phoenix.exception.PhoenixIOException;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.jdbc.PhoenixPreparedStatement;
+import org.apache.phoenix.protobuf.ProtobufUtil;
+import org.apache.phoenix.query.BaseTest;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Maps;
+import org.apache.phoenix.util.ClientUtil;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.Map;
+
+import static org.apache.phoenix.query.QueryServices.DISABLE_VIEW_SUBTREE_VALIDATION;
+
+@Category(NeedsOwnMiniClusterTest.class)
+public class UpdateCacheFreqIT extends BaseTest {
Review Comment:
Done
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheFreqIT.java:
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.end2end;
+
+import com.google.protobuf.RpcCallback;
+import com.google.protobuf.RpcController;
+import org.apache.hadoop.hbase.DoNotRetryIOException;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.compile.ExplainPlan;
+import org.apache.phoenix.compile.ExplainPlanAttributes;
+import org.apache.phoenix.coprocessor.MetaDataEndpointImpl;
+import org.apache.phoenix.coprocessor.generated.MetaDataProtos;
+import org.apache.phoenix.coprocessorclient.BaseScannerRegionObserverConstants;
+import org.apache.phoenix.exception.PhoenixIOException;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.jdbc.PhoenixPreparedStatement;
+import org.apache.phoenix.protobuf.ProtobufUtil;
+import org.apache.phoenix.query.BaseTest;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Maps;
+import org.apache.phoenix.util.ClientUtil;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.Map;
+
+import static org.apache.phoenix.query.QueryServices.DISABLE_VIEW_SUBTREE_VALIDATION;
+
+@Category(NeedsOwnMiniClusterTest.class)
+public class UpdateCacheFreqIT extends BaseTest {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(UpdateCacheFreqIT.class);
+
+ private static void initCluster() throws Exception {
+ Map<String, String> props = Maps.newConcurrentMap();
+ props.put(BaseScannerRegionObserverConstants.PHOENIX_MAX_LOOKBACK_AGE_CONF_KEY,
+ Integer.toString(60 * 60 * 1000));
+ props.put(QueryServices.DEFAULT_UPDATE_CACHE_FREQUENCY_ATRRIB, "ALWAYS");
+ props.put(QueryServices.LAST_DDL_TIMESTAMP_VALIDATION_ENABLED, Boolean.toString(false));
+ props.put(QueryServices.PHOENIX_METADATA_INVALIDATE_CACHE_ENABLED, Boolean.toString(false));
+ props.put(QueryServices.TASK_HANDLING_INITIAL_DELAY_MS_ATTRIB,
+ Long.toString(Long.MAX_VALUE));
+ props.put(DISABLE_VIEW_SUBTREE_VALIDATION, "true");
+ setUpTestDriver(new ReadOnlyProps(props.entrySet().iterator()));
+ }
+
+ @BeforeClass
+ public static synchronized void doSetup() throws Exception {
+ initCluster();
+ }
+
+ public static class TestMetaDataEndpointImpl extends MetaDataEndpointImpl {
+
+ @Override
+ public void getTable(RpcController controller, MetaDataProtos.GetTableRequest request,
+ RpcCallback<MetaDataProtos.MetaDataResponse> done) {
+ LOGGER.error("Not expected to get getTable() for {}",
+ Bytes.toString(request.getTableName().toByteArray()));
+ ProtobufUtil.setControllerException(controller,
+ ClientUtil.createIOException(
+ SchemaUtil.getPhysicalTableName(
+ PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES,
+ false).toString(),
+ new DoNotRetryIOException("Not allowed")));
+
+ }
+
+ }
+
+ @Test
+ public void testUcfWithNoGetTableCalls() throws Throwable {
+ final String tableName = generateUniqueName();
+ final String indexName = "IDX_" + tableName;
+ final String view01 = "v01_" + tableName;
+ final String view02 = "v02_" + tableName;
+ final String index_view01 = "idx_v01_" + tableName;
+ final String index_view02 = "idx_v02_" + tableName;
+ final String index_view03 = "idx_v03_" + tableName;
+ final String index_view04 = "idx_v04_" + tableName;
+
+ try (Connection conn = DriverManager.getConnection(getUrl())) {
+ final Statement stmt = conn.createStatement();
+
+ stmt.execute("CREATE TABLE " + tableName
+ + " (COL1 CHAR(10) NOT NULL, COL2 CHAR(5) NOT NULL, COL3 VARCHAR,"
+ + " COL4 VARCHAR CONSTRAINT pk PRIMARY KEY(COL1, COL2))"
+ + " UPDATE_CACHE_FREQUENCY=20000");
+ stmt.execute("CREATE INDEX " + indexName
+ + " ON " + tableName + " (COL3) INCLUDE (COL4)");
+ stmt.execute("CREATE VIEW " + view01
+ + " (VCOL1 CHAR(8), COL5 VARCHAR) AS SELECT * FROM " + tableName
+ + " WHERE COL1 = 'col1'");
+ stmt.execute("CREATE VIEW " + view02 + " (VCOL2 CHAR(10), COL6 VARCHAR)"
+ + " AS SELECT * FROM " + view01 + " WHERE VCOL1 = 'vcol1'");
+
+ stmt.execute("CREATE INDEX " + index_view01 + " ON " + view01 + " (COL5) INCLUDE "
+ + "(COL1, COL2, COL3)");
+ stmt.execute("CREATE INDEX " + index_view02 + " ON " + view02 + " (COL6) INCLUDE "
+ + "(COL1, COL2, COL3)");
+ stmt.execute("CREATE INDEX " + index_view03 + " ON " + view01 + " (COL5) INCLUDE "
+ + "(COL2, COL1)");
+ stmt.execute("CREATE INDEX " + index_view04 + " ON " + view02 + " (COL6) INCLUDE "
+ + "(COL2, COL1)");
+
+ stmt.execute("UPSERT INTO " + view02
+ + " (col2, vcol2, col5, col6) values ('0001', 'vcol2_01', 'col5_01', " +
+ "'col6_01')");
+ conn.commit();
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0002', 'vcol2_02', 'col5_02', 'col6_02')");
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0003', 'vcol2_03', 'col5_03', 'col6_03')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0004', 'vcol2', 'col3_04', 'col4_04', 'col5_04')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0005', 'vcol-2', 'col3_05', 'col4_05', 'col5_05')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0006', 'vcol-1', 'col3_06', 'col4_06', 'col5_06')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0007', 'vcol1', 'col3_07', 'col4_07', 'col5_07')");
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0008', 'vcol2_08', 'col5_08', 'col6_02')");
+ conn.commit();
+
+ final Statement statement = conn.createStatement();
+ ResultSet rs =
+ statement.executeQuery(
+ "SELECT COL2, VCOL1, VCOL2, COL5, COL6 FROM " + view02);
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertFalse(rs.next());
+
+ String query = "SELECT COL1, COL2, COL3, COL4 FROM " + tableName + " WHERE COL3 = " +
+ "'col3_04'";
+ rs = statement.executeQuery(query);
+ Assert.assertTrue(rs.next());
+ Assert.assertFalse(rs.next());
+
+ ExplainPlan explainPlan = conn.prepareStatement(query)
+ .unwrap(PhoenixPreparedStatement.class).optimizeQuery()
+ .getExplainPlan();
+ ExplainPlanAttributes explainPlanAttributes =
+ explainPlan.getPlanStepsAsAttributes();
+ Assert.assertEquals(indexName, explainPlanAttributes.getTableName());
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ }
+ }
+
+ @Test
+ public void testUcfWithDisabledIndex() throws Throwable {
+ final String tableName = generateUniqueName();
+ final String indexName = "IDX_" + tableName;
+
+ Connection conn = DriverManager.getConnection(getUrl());
+ try {
+ final Statement stmt = conn.createStatement();
+
+ stmt.execute("CREATE TABLE " + tableName
+ + " (COL1 CHAR(10) NOT NULL, COL2 CHAR(5) NOT NULL, COL3 VARCHAR,"
+ + " COL4 VARCHAR CONSTRAINT pk PRIMARY KEY(COL1, COL2))"
+ + " UPDATE_CACHE_FREQUENCY=10000");
+ stmt.execute("CREATE INDEX " + indexName
+ + " ON " + tableName + " (COL3) INCLUDE (COL4)");
+
+ stmt.execute("ALTER INDEX " + indexName + " ON " + tableName + " DISABLE");
+
+ Thread.sleep(11000);
+ stmt.execute("UPSERT INTO " + tableName
+ + " (col1, col2, col3, col4) values ('c001', "
+ + "'c002', 'c003', 'c004')");
+ conn.commit();
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+
+ stmt.execute("UPSERT INTO " + tableName
+ + " (col1, col2, col3, col4) values ('c011', "
+ + "'c012', 'c013', 'c014')");
+ conn.commit();
+ throw new RuntimeException("Should not reach here");
+ } catch (PhoenixIOException e) {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1722392715
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheFreqWithDDLValidationIT.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.end2end;
+
+import org.apache.phoenix.coprocessorclient.BaseScannerRegionObserverConstants;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Maps;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.junit.BeforeClass;
+
+import java.util.Map;
+
+//@Category(NeedsOwnMiniClusterTest.class)
+// Uncomment as part of PHOENIX-7381
Review Comment:
Okay got it, I will look at these these tests as part of PHOENIX-7381.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1723931298
##########
phoenix-core-client/src/main/java/org/apache/phoenix/schema/MetaDataClient.java:
##########
@@ -785,19 +785,37 @@ private boolean avoidRpcToGetTable(boolean alwaysHitServer, Long resolvedTimesta
final long effectiveUpdateCacheFreq;
final String ucfInfoForLogging; // Only used for logging purposes
+ boolean overrideUcfToDefault = false;
+ if (table.getType() == INDEX) {
+ overrideUcfToDefault =
+ PIndexState.PENDING_DISABLE.equals(table.getIndexState()) ||
+ !IndexMaintainer.sendIndexMaintainer(table);
+ }
+ if (!overrideUcfToDefault && !table.getIndexes().isEmpty()) {
+ List<PTable> indexes = table.getIndexes();
+ List<PTable> maintainedIndexes =
+ Lists.newArrayList(IndexMaintainer.maintainedIndexes(indexes.iterator()));
+ // The maintainedIndexes contain only the indexes that are used by clients
+ // while generating the mutations. If all the indexes are usable by clients,
+ // we don't need to override UPDATE_CACHE_FREQUENCY. However, if any index is
+ // not in usable state by the client mutations, we should override
+ // UPDATE_CACHE_FREQUENCY to default value so that we make getTable() RPC calls
+ // until all index states change to ACTIVE, BUILDING or other usable states.
+ overrideUcfToDefault = indexes.size() != maintainedIndexes.size();
+ }
+
// What if the table is created with UPDATE_CACHE_FREQUENCY explicitly set to ALWAYS?
// i.e. explicitly set to 0. We should ideally be checking for something like
// hasUpdateCacheFrequency().
//always fetch an Index in PENDING_DISABLE state to retrieve server timestamp
//QueryOptimizer needs that to decide whether the index can be used
- if (PIndexState.PENDING_DISABLE.equals(table.getIndexState())) {
+ if (overrideUcfToDefault) {
Review Comment:
It seems we are not directly using ALWAYS anywhere but get connection level prop: `phoenix.default.update.cache.frequency`. While if this is not set, obviously it gets default value of `ALWAYS`, but if `phoenix.default.update.cache.frequency` is set at Connection level, it means we need to use it. It basically means that the client really does not care for `ALWAYS` at all, it is risky for client to even use this prop at Connection level but whoever uses it, must know what consequences they will deal with.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1722389988
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheFreqIT.java:
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.end2end;
+
+import com.google.protobuf.RpcCallback;
+import com.google.protobuf.RpcController;
+import org.apache.hadoop.hbase.DoNotRetryIOException;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.compile.ExplainPlan;
+import org.apache.phoenix.compile.ExplainPlanAttributes;
+import org.apache.phoenix.coprocessor.MetaDataEndpointImpl;
+import org.apache.phoenix.coprocessor.generated.MetaDataProtos;
+import org.apache.phoenix.coprocessorclient.BaseScannerRegionObserverConstants;
+import org.apache.phoenix.exception.PhoenixIOException;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.jdbc.PhoenixPreparedStatement;
+import org.apache.phoenix.protobuf.ProtobufUtil;
+import org.apache.phoenix.query.BaseTest;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Maps;
+import org.apache.phoenix.util.ClientUtil;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.Map;
+
+import static org.apache.phoenix.query.QueryServices.DISABLE_VIEW_SUBTREE_VALIDATION;
+
+@Category(NeedsOwnMiniClusterTest.class)
+public class UpdateCacheFreqIT extends BaseTest {
Review Comment:
The only reason why i could not use existing UpdateCacheIT is because we need to set these configs before the cluster comes up and we want to validate getTable() RPC calls by switching the coproc implementations between some queries. Achieving this with existing test did not seem feasible, hence the new class. And also, created `UpdateCacheFreqWithDDLValidationIT` to run these exact tests just by using metadata caching redesign configs.
But if you have any other test name in mind, i can use it. Maybe `UpdateCacheWithGetTableIT`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1723843918
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheFreqIT.java:
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.end2end;
+
+import com.google.protobuf.RpcCallback;
+import com.google.protobuf.RpcController;
+import org.apache.hadoop.hbase.DoNotRetryIOException;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.compile.ExplainPlan;
+import org.apache.phoenix.compile.ExplainPlanAttributes;
+import org.apache.phoenix.coprocessor.MetaDataEndpointImpl;
+import org.apache.phoenix.coprocessor.generated.MetaDataProtos;
+import org.apache.phoenix.coprocessorclient.BaseScannerRegionObserverConstants;
+import org.apache.phoenix.exception.PhoenixIOException;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.jdbc.PhoenixPreparedStatement;
+import org.apache.phoenix.protobuf.ProtobufUtil;
+import org.apache.phoenix.query.BaseTest;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Maps;
+import org.apache.phoenix.util.ClientUtil;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.Map;
+
+import static org.apache.phoenix.query.QueryServices.DISABLE_VIEW_SUBTREE_VALIDATION;
+
+@Category(NeedsOwnMiniClusterTest.class)
+public class UpdateCacheFreqIT extends BaseTest {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(UpdateCacheFreqIT.class);
+
+ private static void initCluster() throws Exception {
+ Map<String, String> props = Maps.newConcurrentMap();
+ props.put(BaseScannerRegionObserverConstants.PHOENIX_MAX_LOOKBACK_AGE_CONF_KEY,
+ Integer.toString(60 * 60 * 1000));
+ props.put(QueryServices.DEFAULT_UPDATE_CACHE_FREQUENCY_ATRRIB, "ALWAYS");
+ props.put(QueryServices.LAST_DDL_TIMESTAMP_VALIDATION_ENABLED, Boolean.toString(false));
+ props.put(QueryServices.PHOENIX_METADATA_INVALIDATE_CACHE_ENABLED, Boolean.toString(false));
+ props.put(QueryServices.TASK_HANDLING_INITIAL_DELAY_MS_ATTRIB,
+ Long.toString(Long.MAX_VALUE));
+ props.put(DISABLE_VIEW_SUBTREE_VALIDATION, "true");
+ setUpTestDriver(new ReadOnlyProps(props.entrySet().iterator()));
+ }
+
+ @BeforeClass
+ public static synchronized void doSetup() throws Exception {
+ initCluster();
+ }
+
+ public static class TestMetaDataEndpointImpl extends MetaDataEndpointImpl {
+
+ @Override
+ public void getTable(RpcController controller, MetaDataProtos.GetTableRequest request,
+ RpcCallback<MetaDataProtos.MetaDataResponse> done) {
+ LOGGER.error("Not expected to get getTable() for {}",
+ Bytes.toString(request.getTableName().toByteArray()));
+ ProtobufUtil.setControllerException(controller,
+ ClientUtil.createIOException(
+ SchemaUtil.getPhysicalTableName(
+ PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES,
+ false).toString(),
+ new DoNotRetryIOException("Not allowed")));
+
+ }
+
+ }
+
+ @Test
+ public void testUcfWithNoGetTableCalls() throws Throwable {
+ final String tableName = generateUniqueName();
+ final String indexName = "IDX_" + tableName;
+ final String view01 = "v01_" + tableName;
+ final String view02 = "v02_" + tableName;
+ final String index_view01 = "idx_v01_" + tableName;
+ final String index_view02 = "idx_v02_" + tableName;
+ final String index_view03 = "idx_v03_" + tableName;
+ final String index_view04 = "idx_v04_" + tableName;
+
+ try (Connection conn = DriverManager.getConnection(getUrl())) {
+ final Statement stmt = conn.createStatement();
+
+ stmt.execute("CREATE TABLE " + tableName
+ + " (COL1 CHAR(10) NOT NULL, COL2 CHAR(5) NOT NULL, COL3 VARCHAR,"
+ + " COL4 VARCHAR CONSTRAINT pk PRIMARY KEY(COL1, COL2))"
+ + " UPDATE_CACHE_FREQUENCY=20000");
+ stmt.execute("CREATE INDEX " + indexName
+ + " ON " + tableName + " (COL3) INCLUDE (COL4)");
+ stmt.execute("CREATE VIEW " + view01
+ + " (VCOL1 CHAR(8), COL5 VARCHAR) AS SELECT * FROM " + tableName
+ + " WHERE COL1 = 'col1'");
+ stmt.execute("CREATE VIEW " + view02 + " (VCOL2 CHAR(10), COL6 VARCHAR)"
+ + " AS SELECT * FROM " + view01 + " WHERE VCOL1 = 'vcol1'");
+
+ stmt.execute("CREATE INDEX " + index_view01 + " ON " + view01 + " (COL5) INCLUDE "
+ + "(COL1, COL2, COL3)");
+ stmt.execute("CREATE INDEX " + index_view02 + " ON " + view02 + " (COL6) INCLUDE "
+ + "(COL1, COL2, COL3)");
+ stmt.execute("CREATE INDEX " + index_view03 + " ON " + view01 + " (COL5) INCLUDE "
+ + "(COL2, COL1)");
+ stmt.execute("CREATE INDEX " + index_view04 + " ON " + view02 + " (COL6) INCLUDE "
+ + "(COL2, COL1)");
+
+ stmt.execute("UPSERT INTO " + view02
+ + " (col2, vcol2, col5, col6) values ('0001', 'vcol2_01', 'col5_01', " +
+ "'col6_01')");
+ conn.commit();
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0002', 'vcol2_02', 'col5_02', 'col6_02')");
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0003', 'vcol2_03', 'col5_03', 'col6_03')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0004', 'vcol2', 'col3_04', 'col4_04', 'col5_04')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0005', 'vcol-2', 'col3_05', 'col4_05', 'col5_05')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0006', 'vcol-1', 'col3_06', 'col4_06', 'col5_06')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0007', 'vcol1', 'col3_07', 'col4_07', 'col5_07')");
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0008', 'vcol2_08', 'col5_08', 'col6_02')");
+ conn.commit();
+
+ final Statement statement = conn.createStatement();
+ ResultSet rs =
+ statement.executeQuery(
+ "SELECT COL2, VCOL1, VCOL2, COL5, COL6 FROM " + view02);
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertFalse(rs.next());
+
+ String query = "SELECT COL1, COL2, COL3, COL4 FROM " + tableName + " WHERE COL3 = " +
+ "'col3_04'";
+ rs = statement.executeQuery(query);
+ Assert.assertTrue(rs.next());
+ Assert.assertFalse(rs.next());
+
+ ExplainPlan explainPlan = conn.prepareStatement(query)
+ .unwrap(PhoenixPreparedStatement.class).optimizeQuery()
+ .getExplainPlan();
+ ExplainPlanAttributes explainPlanAttributes =
+ explainPlan.getPlanStepsAsAttributes();
+ Assert.assertEquals(indexName, explainPlanAttributes.getTableName());
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ }
+ }
+
+ @Test
+ public void testUcfWithDisabledIndex() throws Throwable {
+ final String tableName = generateUniqueName();
+ final String indexName = "IDX_" + tableName;
+
+ Connection conn = DriverManager.getConnection(getUrl());
+ try {
+ final Statement stmt = conn.createStatement();
+
+ stmt.execute("CREATE TABLE " + tableName
+ + " (COL1 CHAR(10) NOT NULL, COL2 CHAR(5) NOT NULL, COL3 VARCHAR,"
+ + " COL4 VARCHAR CONSTRAINT pk PRIMARY KEY(COL1, COL2))"
+ + " UPDATE_CACHE_FREQUENCY=10000");
+ stmt.execute("CREATE INDEX " + indexName
+ + " ON " + tableName + " (COL3) INCLUDE (COL4)");
+
+ stmt.execute("ALTER INDEX " + indexName + " ON " + tableName + " DISABLE");
+
+ Thread.sleep(11000);
+ stmt.execute("UPSERT INTO " + tableName
+ + " (col1, col2, col3, col4) values ('c001', "
+ + "'c002', 'c003', 'c004')");
+ conn.commit();
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+
+ stmt.execute("UPSERT INTO " + tableName
+ + " (col1, col2, col3, col4) values ('c011', "
+ + "'c012', 'c013', 'c014')");
+ conn.commit();
+ throw new RuntimeException("Should not reach here");
+ } catch (PhoenixIOException e) {
Review Comment:
@virajjasani Maybe I am missing something but the current tests still only check whether there are no getTable calls when all indexes are active and that there are getTable calls when there is some index in a "disabled" state.
Should there be a test where index is created in CREATE_DISABLE state to begin with, client should be overriding UCF and doing getTable calls, then we change the index state to BUILDING OR ACTIVE - after one more getTable call, client should stop overriding UCF and not do any getTable calls. Since this is exactly what would happen for atleast our prod use-case, I think it would be good to have that tested. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani merged PR #1955:
URL: https://github.com/apache/phoenix/pull/1955
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1720144218
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheFreqWithDDLValidationIT.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.end2end;
+
+import org.apache.phoenix.coprocessorclient.BaseScannerRegionObserverConstants;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Maps;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.junit.BeforeClass;
+
+import java.util.Map;
+
+//@Category(NeedsOwnMiniClusterTest.class)
+// Uncomment as part of PHOENIX-7381
Review Comment:
When enabling last_ddl_timestamp validation, we would want to set default UCF to NEVER so that clients do not perform a server RPC unless required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1723932006
##########
phoenix-core-client/src/main/java/org/apache/phoenix/schema/MetaDataClient.java:
##########
@@ -785,19 +785,37 @@ private boolean avoidRpcToGetTable(boolean alwaysHitServer, Long resolvedTimesta
final long effectiveUpdateCacheFreq;
final String ucfInfoForLogging; // Only used for logging purposes
+ boolean overrideUcfToDefault = false;
+ if (table.getType() == INDEX) {
+ overrideUcfToDefault =
+ PIndexState.PENDING_DISABLE.equals(table.getIndexState()) ||
+ !IndexMaintainer.sendIndexMaintainer(table);
+ }
+ if (!overrideUcfToDefault && !table.getIndexes().isEmpty()) {
+ List<PTable> indexes = table.getIndexes();
+ List<PTable> maintainedIndexes =
+ Lists.newArrayList(IndexMaintainer.maintainedIndexes(indexes.iterator()));
+ // The maintainedIndexes contain only the indexes that are used by clients
+ // while generating the mutations. If all the indexes are usable by clients,
+ // we don't need to override UPDATE_CACHE_FREQUENCY. However, if any index is
+ // not in usable state by the client mutations, we should override
+ // UPDATE_CACHE_FREQUENCY to default value so that we make getTable() RPC calls
+ // until all index states change to ACTIVE, BUILDING or other usable states.
+ overrideUcfToDefault = indexes.size() != maintainedIndexes.size();
+ }
+
// What if the table is created with UPDATE_CACHE_FREQUENCY explicitly set to ALWAYS?
// i.e. explicitly set to 0. We should ideally be checking for something like
// hasUpdateCacheFrequency().
//always fetch an Index in PENDING_DISABLE state to retrieve server timestamp
//QueryOptimizer needs that to decide whether the index can be used
- if (PIndexState.PENDING_DISABLE.equals(table.getIndexState())) {
+ if (overrideUcfToDefault) {
Review Comment:
> This is why I had added a separate config for `DEFAULT_UPDATE_CACHE_FREQUENCY_FOR_PENDING_DISABLED_INDEX` and set that to ALWAYS.
Yes, i can remove this also with this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1723983534
##########
phoenix-core-client/src/main/java/org/apache/phoenix/schema/MetaDataClient.java:
##########
@@ -785,19 +785,37 @@ private boolean avoidRpcToGetTable(boolean alwaysHitServer, Long resolvedTimesta
final long effectiveUpdateCacheFreq;
final String ucfInfoForLogging; // Only used for logging purposes
+ boolean overrideUcfToDefault = false;
+ if (table.getType() == INDEX) {
+ overrideUcfToDefault =
+ PIndexState.PENDING_DISABLE.equals(table.getIndexState()) ||
+ !IndexMaintainer.sendIndexMaintainer(table);
+ }
+ if (!overrideUcfToDefault && !table.getIndexes().isEmpty()) {
+ List<PTable> indexes = table.getIndexes();
+ List<PTable> maintainedIndexes =
+ Lists.newArrayList(IndexMaintainer.maintainedIndexes(indexes.iterator()));
+ // The maintainedIndexes contain only the indexes that are used by clients
+ // while generating the mutations. If all the indexes are usable by clients,
+ // we don't need to override UPDATE_CACHE_FREQUENCY. However, if any index is
+ // not in usable state by the client mutations, we should override
+ // UPDATE_CACHE_FREQUENCY to default value so that we make getTable() RPC calls
+ // until all index states change to ACTIVE, BUILDING or other usable states.
+ overrideUcfToDefault = indexes.size() != maintainedIndexes.size();
+ }
+
// What if the table is created with UPDATE_CACHE_FREQUENCY explicitly set to ALWAYS?
// i.e. explicitly set to 0. We should ideally be checking for something like
// hasUpdateCacheFrequency().
//always fetch an Index in PENDING_DISABLE state to retrieve server timestamp
//QueryOptimizer needs that to decide whether the index can be used
- if (PIndexState.PENDING_DISABLE.equals(table.getIndexState())) {
+ if (overrideUcfToDefault) {
Review Comment:
For PHOENIX-6883, we wanted to set default UCF to NEVER and that is why this config was introduced. At this point, I am a little confused about how this UCF change and PHOENIX-6883 would work together in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1723815604
##########
phoenix-core-client/src/main/java/org/apache/phoenix/schema/MetaDataClient.java:
##########
@@ -785,19 +785,37 @@ private boolean avoidRpcToGetTable(boolean alwaysHitServer, Long resolvedTimesta
final long effectiveUpdateCacheFreq;
final String ucfInfoForLogging; // Only used for logging purposes
+ boolean overrideUcfToDefault = false;
+ if (table.getType() == INDEX) {
+ overrideUcfToDefault =
+ PIndexState.PENDING_DISABLE.equals(table.getIndexState()) ||
+ !IndexMaintainer.sendIndexMaintainer(table);
+ }
+ if (!overrideUcfToDefault && !table.getIndexes().isEmpty()) {
+ List<PTable> indexes = table.getIndexes();
+ List<PTable> maintainedIndexes =
+ Lists.newArrayList(IndexMaintainer.maintainedIndexes(indexes.iterator()));
+ // The maintainedIndexes contain only the indexes that are used by clients
+ // while generating the mutations. If all the indexes are usable by clients,
+ // we don't need to override UPDATE_CACHE_FREQUENCY. However, if any index is
+ // not in usable state by the client mutations, we should override
+ // UPDATE_CACHE_FREQUENCY to default value so that we make getTable() RPC calls
+ // until all index states change to ACTIVE, BUILDING or other usable states.
+ overrideUcfToDefault = indexes.size() != maintainedIndexes.size();
+ }
+
// What if the table is created with UPDATE_CACHE_FREQUENCY explicitly set to ALWAYS?
// i.e. explicitly set to 0. We should ideally be checking for something like
// hasUpdateCacheFrequency().
//always fetch an Index in PENDING_DISABLE state to retrieve server timestamp
//QueryOptimizer needs that to decide whether the index can be used
- if (PIndexState.PENDING_DISABLE.equals(table.getIndexState())) {
+ if (overrideUcfToDefault) {
Review Comment:
@virajjasani Default today is ALWAYS so this would work but what if default is something else? Should we explicitly override to ALWAYS?
This is why I had added a separate config for `DEFAULT_UPDATE_CACHE_FREQUENCY_FOR_PENDING_DISABLED_INDEX` and set that to ALWAYS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1722390676
##########
phoenix-core-client/src/main/java/org/apache/phoenix/schema/MetaDataClient.java:
##########
@@ -785,19 +785,33 @@ private boolean avoidRpcToGetTable(boolean alwaysHitServer, Long resolvedTimesta
final long effectiveUpdateCacheFreq;
final String ucfInfoForLogging; // Only used for logging purposes
+ boolean overrideUcfToDefault =
Review Comment:
@virajjasani thanks, this looks good now!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1722392016
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheFreqIT.java:
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.end2end;
+
+import com.google.protobuf.RpcCallback;
+import com.google.protobuf.RpcController;
+import org.apache.hadoop.hbase.DoNotRetryIOException;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.compile.ExplainPlan;
+import org.apache.phoenix.compile.ExplainPlanAttributes;
+import org.apache.phoenix.coprocessor.MetaDataEndpointImpl;
+import org.apache.phoenix.coprocessor.generated.MetaDataProtos;
+import org.apache.phoenix.coprocessorclient.BaseScannerRegionObserverConstants;
+import org.apache.phoenix.exception.PhoenixIOException;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.jdbc.PhoenixPreparedStatement;
+import org.apache.phoenix.protobuf.ProtobufUtil;
+import org.apache.phoenix.query.BaseTest;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Maps;
+import org.apache.phoenix.util.ClientUtil;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.Map;
+
+import static org.apache.phoenix.query.QueryServices.DISABLE_VIEW_SUBTREE_VALIDATION;
+
+@Category(NeedsOwnMiniClusterTest.class)
+public class UpdateCacheFreqIT extends BaseTest {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(UpdateCacheFreqIT.class);
+
+ private static void initCluster() throws Exception {
+ Map<String, String> props = Maps.newConcurrentMap();
+ props.put(BaseScannerRegionObserverConstants.PHOENIX_MAX_LOOKBACK_AGE_CONF_KEY,
+ Integer.toString(60 * 60 * 1000));
+ props.put(QueryServices.DEFAULT_UPDATE_CACHE_FREQUENCY_ATRRIB, "ALWAYS");
+ props.put(QueryServices.LAST_DDL_TIMESTAMP_VALIDATION_ENABLED, Boolean.toString(false));
+ props.put(QueryServices.PHOENIX_METADATA_INVALIDATE_CACHE_ENABLED, Boolean.toString(false));
+ props.put(QueryServices.TASK_HANDLING_INITIAL_DELAY_MS_ATTRIB,
+ Long.toString(Long.MAX_VALUE));
+ props.put(DISABLE_VIEW_SUBTREE_VALIDATION, "true");
+ setUpTestDriver(new ReadOnlyProps(props.entrySet().iterator()));
+ }
+
+ @BeforeClass
+ public static synchronized void doSetup() throws Exception {
+ initCluster();
+ }
+
+ public static class TestMetaDataEndpointImpl extends MetaDataEndpointImpl {
+
+ @Override
+ public void getTable(RpcController controller, MetaDataProtos.GetTableRequest request,
+ RpcCallback<MetaDataProtos.MetaDataResponse> done) {
+ LOGGER.error("Not expected to get getTable() for {}",
+ Bytes.toString(request.getTableName().toByteArray()));
+ ProtobufUtil.setControllerException(controller,
+ ClientUtil.createIOException(
+ SchemaUtil.getPhysicalTableName(
+ PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES,
+ false).toString(),
+ new DoNotRetryIOException("Not allowed")));
+
+ }
+
+ }
+
+ @Test
+ public void testUcfWithNoGetTableCalls() throws Throwable {
+ final String tableName = generateUniqueName();
+ final String indexName = "IDX_" + tableName;
+ final String view01 = "v01_" + tableName;
+ final String view02 = "v02_" + tableName;
+ final String index_view01 = "idx_v01_" + tableName;
+ final String index_view02 = "idx_v02_" + tableName;
+ final String index_view03 = "idx_v03_" + tableName;
+ final String index_view04 = "idx_v04_" + tableName;
+
+ try (Connection conn = DriverManager.getConnection(getUrl())) {
+ final Statement stmt = conn.createStatement();
+
+ stmt.execute("CREATE TABLE " + tableName
+ + " (COL1 CHAR(10) NOT NULL, COL2 CHAR(5) NOT NULL, COL3 VARCHAR,"
+ + " COL4 VARCHAR CONSTRAINT pk PRIMARY KEY(COL1, COL2))"
+ + " UPDATE_CACHE_FREQUENCY=20000");
+ stmt.execute("CREATE INDEX " + indexName
+ + " ON " + tableName + " (COL3) INCLUDE (COL4)");
+ stmt.execute("CREATE VIEW " + view01
+ + " (VCOL1 CHAR(8), COL5 VARCHAR) AS SELECT * FROM " + tableName
+ + " WHERE COL1 = 'col1'");
+ stmt.execute("CREATE VIEW " + view02 + " (VCOL2 CHAR(10), COL6 VARCHAR)"
+ + " AS SELECT * FROM " + view01 + " WHERE VCOL1 = 'vcol1'");
+
+ stmt.execute("CREATE INDEX " + index_view01 + " ON " + view01 + " (COL5) INCLUDE "
+ + "(COL1, COL2, COL3)");
+ stmt.execute("CREATE INDEX " + index_view02 + " ON " + view02 + " (COL6) INCLUDE "
+ + "(COL1, COL2, COL3)");
+ stmt.execute("CREATE INDEX " + index_view03 + " ON " + view01 + " (COL5) INCLUDE "
+ + "(COL2, COL1)");
+ stmt.execute("CREATE INDEX " + index_view04 + " ON " + view02 + " (COL6) INCLUDE "
+ + "(COL2, COL1)");
+
+ stmt.execute("UPSERT INTO " + view02
+ + " (col2, vcol2, col5, col6) values ('0001', 'vcol2_01', 'col5_01', " +
+ "'col6_01')");
+ conn.commit();
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0002', 'vcol2_02', 'col5_02', 'col6_02')");
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0003', 'vcol2_03', 'col5_03', 'col6_03')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0004', 'vcol2', 'col3_04', 'col4_04', 'col5_04')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0005', 'vcol-2', 'col3_05', 'col4_05', 'col5_05')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0006', 'vcol-1', 'col3_06', 'col4_06', 'col5_06')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0007', 'vcol1', 'col3_07', 'col4_07', 'col5_07')");
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0008', 'vcol2_08', 'col5_08', 'col6_02')");
+ conn.commit();
+
+ final Statement statement = conn.createStatement();
+ ResultSet rs =
+ statement.executeQuery(
+ "SELECT COL2, VCOL1, VCOL2, COL5, COL6 FROM " + view02);
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertFalse(rs.next());
+
+ String query = "SELECT COL1, COL2, COL3, COL4 FROM " + tableName + " WHERE COL3 = " +
+ "'col3_04'";
+ rs = statement.executeQuery(query);
+ Assert.assertTrue(rs.next());
+ Assert.assertFalse(rs.next());
+
+ ExplainPlan explainPlan = conn.prepareStatement(query)
+ .unwrap(PhoenixPreparedStatement.class).optimizeQuery()
+ .getExplainPlan();
+ ExplainPlanAttributes explainPlanAttributes =
+ explainPlan.getPlanStepsAsAttributes();
+ Assert.assertEquals(indexName, explainPlanAttributes.getTableName());
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ }
+ }
+
+ @Test
+ public void testUcfWithDisabledIndex() throws Throwable {
+ final String tableName = generateUniqueName();
+ final String indexName = "IDX_" + tableName;
+
+ Connection conn = DriverManager.getConnection(getUrl());
+ try {
+ final Statement stmt = conn.createStatement();
+
+ stmt.execute("CREATE TABLE " + tableName
+ + " (COL1 CHAR(10) NOT NULL, COL2 CHAR(5) NOT NULL, COL3 VARCHAR,"
+ + " COL4 VARCHAR CONSTRAINT pk PRIMARY KEY(COL1, COL2))"
+ + " UPDATE_CACHE_FREQUENCY=10000");
+ stmt.execute("CREATE INDEX " + indexName
+ + " ON " + tableName + " (COL3) INCLUDE (COL4)");
+
+ stmt.execute("ALTER INDEX " + indexName + " ON " + tableName + " DISABLE");
+
+ Thread.sleep(11000);
+ stmt.execute("UPSERT INTO " + tableName
+ + " (col1, col2, col3, col4) values ('c001', "
+ + "'c002', 'c003', 'c004')");
+ conn.commit();
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+
+ stmt.execute("UPSERT INTO " + tableName
+ + " (col1, col2, col3, col4) values ('c011', "
+ + "'c012', 'c013', 'c014')");
+ conn.commit();
+ throw new RuntimeException("Should not reach here");
+ } catch (PhoenixIOException e) {
Review Comment:
Except for Index jobs, do we any way to directly work on Index PTable object, bypassing data table PTable? If we rely on index jobs, it would be difficult to simulate when to switch over to new test coproc in the middle of job execution right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1722397847
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheFreqIT.java:
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.end2end;
+
+import com.google.protobuf.RpcCallback;
+import com.google.protobuf.RpcController;
+import org.apache.hadoop.hbase.DoNotRetryIOException;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.compile.ExplainPlan;
+import org.apache.phoenix.compile.ExplainPlanAttributes;
+import org.apache.phoenix.coprocessor.MetaDataEndpointImpl;
+import org.apache.phoenix.coprocessor.generated.MetaDataProtos;
+import org.apache.phoenix.coprocessorclient.BaseScannerRegionObserverConstants;
+import org.apache.phoenix.exception.PhoenixIOException;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.jdbc.PhoenixPreparedStatement;
+import org.apache.phoenix.protobuf.ProtobufUtil;
+import org.apache.phoenix.query.BaseTest;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Maps;
+import org.apache.phoenix.util.ClientUtil;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.Map;
+
+import static org.apache.phoenix.query.QueryServices.DISABLE_VIEW_SUBTREE_VALIDATION;
+
+@Category(NeedsOwnMiniClusterTest.class)
+public class UpdateCacheFreqIT extends BaseTest {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(UpdateCacheFreqIT.class);
+
+ private static void initCluster() throws Exception {
+ Map<String, String> props = Maps.newConcurrentMap();
+ props.put(BaseScannerRegionObserverConstants.PHOENIX_MAX_LOOKBACK_AGE_CONF_KEY,
+ Integer.toString(60 * 60 * 1000));
+ props.put(QueryServices.DEFAULT_UPDATE_CACHE_FREQUENCY_ATRRIB, "ALWAYS");
+ props.put(QueryServices.LAST_DDL_TIMESTAMP_VALIDATION_ENABLED, Boolean.toString(false));
+ props.put(QueryServices.PHOENIX_METADATA_INVALIDATE_CACHE_ENABLED, Boolean.toString(false));
+ props.put(QueryServices.TASK_HANDLING_INITIAL_DELAY_MS_ATTRIB,
+ Long.toString(Long.MAX_VALUE));
+ props.put(DISABLE_VIEW_SUBTREE_VALIDATION, "true");
+ setUpTestDriver(new ReadOnlyProps(props.entrySet().iterator()));
+ }
+
+ @BeforeClass
+ public static synchronized void doSetup() throws Exception {
+ initCluster();
+ }
+
+ public static class TestMetaDataEndpointImpl extends MetaDataEndpointImpl {
+
+ @Override
+ public void getTable(RpcController controller, MetaDataProtos.GetTableRequest request,
+ RpcCallback<MetaDataProtos.MetaDataResponse> done) {
+ LOGGER.error("Not expected to get getTable() for {}",
+ Bytes.toString(request.getTableName().toByteArray()));
+ ProtobufUtil.setControllerException(controller,
+ ClientUtil.createIOException(
+ SchemaUtil.getPhysicalTableName(
+ PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES,
+ false).toString(),
+ new DoNotRetryIOException("Not allowed")));
+
+ }
+
+ }
+
+ @Test
+ public void testUcfWithNoGetTableCalls() throws Throwable {
+ final String tableName = generateUniqueName();
+ final String indexName = "IDX_" + tableName;
+ final String view01 = "v01_" + tableName;
+ final String view02 = "v02_" + tableName;
+ final String index_view01 = "idx_v01_" + tableName;
+ final String index_view02 = "idx_v02_" + tableName;
+ final String index_view03 = "idx_v03_" + tableName;
+ final String index_view04 = "idx_v04_" + tableName;
+
+ try (Connection conn = DriverManager.getConnection(getUrl())) {
+ final Statement stmt = conn.createStatement();
+
+ stmt.execute("CREATE TABLE " + tableName
+ + " (COL1 CHAR(10) NOT NULL, COL2 CHAR(5) NOT NULL, COL3 VARCHAR,"
+ + " COL4 VARCHAR CONSTRAINT pk PRIMARY KEY(COL1, COL2))"
+ + " UPDATE_CACHE_FREQUENCY=20000");
+ stmt.execute("CREATE INDEX " + indexName
+ + " ON " + tableName + " (COL3) INCLUDE (COL4)");
+ stmt.execute("CREATE VIEW " + view01
+ + " (VCOL1 CHAR(8), COL5 VARCHAR) AS SELECT * FROM " + tableName
+ + " WHERE COL1 = 'col1'");
+ stmt.execute("CREATE VIEW " + view02 + " (VCOL2 CHAR(10), COL6 VARCHAR)"
+ + " AS SELECT * FROM " + view01 + " WHERE VCOL1 = 'vcol1'");
+
+ stmt.execute("CREATE INDEX " + index_view01 + " ON " + view01 + " (COL5) INCLUDE "
+ + "(COL1, COL2, COL3)");
+ stmt.execute("CREATE INDEX " + index_view02 + " ON " + view02 + " (COL6) INCLUDE "
+ + "(COL1, COL2, COL3)");
+ stmt.execute("CREATE INDEX " + index_view03 + " ON " + view01 + " (COL5) INCLUDE "
+ + "(COL2, COL1)");
+ stmt.execute("CREATE INDEX " + index_view04 + " ON " + view02 + " (COL6) INCLUDE "
+ + "(COL2, COL1)");
+
+ stmt.execute("UPSERT INTO " + view02
+ + " (col2, vcol2, col5, col6) values ('0001', 'vcol2_01', 'col5_01', " +
+ "'col6_01')");
+ conn.commit();
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0002', 'vcol2_02', 'col5_02', 'col6_02')");
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0003', 'vcol2_03', 'col5_03', 'col6_03')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0004', 'vcol2', 'col3_04', 'col4_04', 'col5_04')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0005', 'vcol-2', 'col3_05', 'col4_05', 'col5_05')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0006', 'vcol-1', 'col3_06', 'col4_06', 'col5_06')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0007', 'vcol1', 'col3_07', 'col4_07', 'col5_07')");
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0008', 'vcol2_08', 'col5_08', 'col6_02')");
+ conn.commit();
+
+ final Statement statement = conn.createStatement();
+ ResultSet rs =
+ statement.executeQuery(
+ "SELECT COL2, VCOL1, VCOL2, COL5, COL6 FROM " + view02);
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertFalse(rs.next());
+
+ String query = "SELECT COL1, COL2, COL3, COL4 FROM " + tableName + " WHERE COL3 = " +
+ "'col3_04'";
+ rs = statement.executeQuery(query);
+ Assert.assertTrue(rs.next());
+ Assert.assertFalse(rs.next());
+
+ ExplainPlan explainPlan = conn.prepareStatement(query)
+ .unwrap(PhoenixPreparedStatement.class).optimizeQuery()
+ .getExplainPlan();
+ ExplainPlanAttributes explainPlanAttributes =
+ explainPlan.getPlanStepsAsAttributes();
+ Assert.assertEquals(indexName, explainPlanAttributes.getTableName());
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ }
+ }
+
+ @Test
+ public void testUcfWithDisabledIndex() throws Throwable {
+ final String tableName = generateUniqueName();
+ final String indexName = "IDX_" + tableName;
+
+ Connection conn = DriverManager.getConnection(getUrl());
+ try {
+ final Statement stmt = conn.createStatement();
+
+ stmt.execute("CREATE TABLE " + tableName
+ + " (COL1 CHAR(10) NOT NULL, COL2 CHAR(5) NOT NULL, COL3 VARCHAR,"
+ + " COL4 VARCHAR CONSTRAINT pk PRIMARY KEY(COL1, COL2))"
+ + " UPDATE_CACHE_FREQUENCY=10000");
+ stmt.execute("CREATE INDEX " + indexName
+ + " ON " + tableName + " (COL3) INCLUDE (COL4)");
+
+ stmt.execute("ALTER INDEX " + indexName + " ON " + tableName + " DISABLE");
+
+ Thread.sleep(11000);
+ stmt.execute("UPSERT INTO " + tableName
+ + " (col1, col2, col3, col4) values ('c001', "
+ + "'c002', 'c003', 'c004')");
+ conn.commit();
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+
+ stmt.execute("UPSERT INTO " + tableName
+ + " (col1, col2, col3, col4) values ('c011', "
+ + "'c012', 'c013', 'c014')");
+ conn.commit();
+ throw new RuntimeException("Should not reach here");
+ } catch (PhoenixIOException e) {
Review Comment:
What if in the end of this test:
1. we change index state using ALTER command (can use a different client for this)
- maybe this will also need the original coproc?
2. switch to normal co-proc and do a query on the table so that the getTable goes through
3. then we switch to new coproc and try another query to see if there was a getTable or not
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1722397847
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheFreqIT.java:
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.end2end;
+
+import com.google.protobuf.RpcCallback;
+import com.google.protobuf.RpcController;
+import org.apache.hadoop.hbase.DoNotRetryIOException;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.compile.ExplainPlan;
+import org.apache.phoenix.compile.ExplainPlanAttributes;
+import org.apache.phoenix.coprocessor.MetaDataEndpointImpl;
+import org.apache.phoenix.coprocessor.generated.MetaDataProtos;
+import org.apache.phoenix.coprocessorclient.BaseScannerRegionObserverConstants;
+import org.apache.phoenix.exception.PhoenixIOException;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.jdbc.PhoenixPreparedStatement;
+import org.apache.phoenix.protobuf.ProtobufUtil;
+import org.apache.phoenix.query.BaseTest;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Maps;
+import org.apache.phoenix.util.ClientUtil;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.Map;
+
+import static org.apache.phoenix.query.QueryServices.DISABLE_VIEW_SUBTREE_VALIDATION;
+
+@Category(NeedsOwnMiniClusterTest.class)
+public class UpdateCacheFreqIT extends BaseTest {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(UpdateCacheFreqIT.class);
+
+ private static void initCluster() throws Exception {
+ Map<String, String> props = Maps.newConcurrentMap();
+ props.put(BaseScannerRegionObserverConstants.PHOENIX_MAX_LOOKBACK_AGE_CONF_KEY,
+ Integer.toString(60 * 60 * 1000));
+ props.put(QueryServices.DEFAULT_UPDATE_CACHE_FREQUENCY_ATRRIB, "ALWAYS");
+ props.put(QueryServices.LAST_DDL_TIMESTAMP_VALIDATION_ENABLED, Boolean.toString(false));
+ props.put(QueryServices.PHOENIX_METADATA_INVALIDATE_CACHE_ENABLED, Boolean.toString(false));
+ props.put(QueryServices.TASK_HANDLING_INITIAL_DELAY_MS_ATTRIB,
+ Long.toString(Long.MAX_VALUE));
+ props.put(DISABLE_VIEW_SUBTREE_VALIDATION, "true");
+ setUpTestDriver(new ReadOnlyProps(props.entrySet().iterator()));
+ }
+
+ @BeforeClass
+ public static synchronized void doSetup() throws Exception {
+ initCluster();
+ }
+
+ public static class TestMetaDataEndpointImpl extends MetaDataEndpointImpl {
+
+ @Override
+ public void getTable(RpcController controller, MetaDataProtos.GetTableRequest request,
+ RpcCallback<MetaDataProtos.MetaDataResponse> done) {
+ LOGGER.error("Not expected to get getTable() for {}",
+ Bytes.toString(request.getTableName().toByteArray()));
+ ProtobufUtil.setControllerException(controller,
+ ClientUtil.createIOException(
+ SchemaUtil.getPhysicalTableName(
+ PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES,
+ false).toString(),
+ new DoNotRetryIOException("Not allowed")));
+
+ }
+
+ }
+
+ @Test
+ public void testUcfWithNoGetTableCalls() throws Throwable {
+ final String tableName = generateUniqueName();
+ final String indexName = "IDX_" + tableName;
+ final String view01 = "v01_" + tableName;
+ final String view02 = "v02_" + tableName;
+ final String index_view01 = "idx_v01_" + tableName;
+ final String index_view02 = "idx_v02_" + tableName;
+ final String index_view03 = "idx_v03_" + tableName;
+ final String index_view04 = "idx_v04_" + tableName;
+
+ try (Connection conn = DriverManager.getConnection(getUrl())) {
+ final Statement stmt = conn.createStatement();
+
+ stmt.execute("CREATE TABLE " + tableName
+ + " (COL1 CHAR(10) NOT NULL, COL2 CHAR(5) NOT NULL, COL3 VARCHAR,"
+ + " COL4 VARCHAR CONSTRAINT pk PRIMARY KEY(COL1, COL2))"
+ + " UPDATE_CACHE_FREQUENCY=20000");
+ stmt.execute("CREATE INDEX " + indexName
+ + " ON " + tableName + " (COL3) INCLUDE (COL4)");
+ stmt.execute("CREATE VIEW " + view01
+ + " (VCOL1 CHAR(8), COL5 VARCHAR) AS SELECT * FROM " + tableName
+ + " WHERE COL1 = 'col1'");
+ stmt.execute("CREATE VIEW " + view02 + " (VCOL2 CHAR(10), COL6 VARCHAR)"
+ + " AS SELECT * FROM " + view01 + " WHERE VCOL1 = 'vcol1'");
+
+ stmt.execute("CREATE INDEX " + index_view01 + " ON " + view01 + " (COL5) INCLUDE "
+ + "(COL1, COL2, COL3)");
+ stmt.execute("CREATE INDEX " + index_view02 + " ON " + view02 + " (COL6) INCLUDE "
+ + "(COL1, COL2, COL3)");
+ stmt.execute("CREATE INDEX " + index_view03 + " ON " + view01 + " (COL5) INCLUDE "
+ + "(COL2, COL1)");
+ stmt.execute("CREATE INDEX " + index_view04 + " ON " + view02 + " (COL6) INCLUDE "
+ + "(COL2, COL1)");
+
+ stmt.execute("UPSERT INTO " + view02
+ + " (col2, vcol2, col5, col6) values ('0001', 'vcol2_01', 'col5_01', " +
+ "'col6_01')");
+ conn.commit();
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0002', 'vcol2_02', 'col5_02', 'col6_02')");
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0003', 'vcol2_03', 'col5_03', 'col6_03')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0004', 'vcol2', 'col3_04', 'col4_04', 'col5_04')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0005', 'vcol-2', 'col3_05', 'col4_05', 'col5_05')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0006', 'vcol-1', 'col3_06', 'col4_06', 'col5_06')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0007', 'vcol1', 'col3_07', 'col4_07', 'col5_07')");
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0008', 'vcol2_08', 'col5_08', 'col6_02')");
+ conn.commit();
+
+ final Statement statement = conn.createStatement();
+ ResultSet rs =
+ statement.executeQuery(
+ "SELECT COL2, VCOL1, VCOL2, COL5, COL6 FROM " + view02);
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertFalse(rs.next());
+
+ String query = "SELECT COL1, COL2, COL3, COL4 FROM " + tableName + " WHERE COL3 = " +
+ "'col3_04'";
+ rs = statement.executeQuery(query);
+ Assert.assertTrue(rs.next());
+ Assert.assertFalse(rs.next());
+
+ ExplainPlan explainPlan = conn.prepareStatement(query)
+ .unwrap(PhoenixPreparedStatement.class).optimizeQuery()
+ .getExplainPlan();
+ ExplainPlanAttributes explainPlanAttributes =
+ explainPlan.getPlanStepsAsAttributes();
+ Assert.assertEquals(indexName, explainPlanAttributes.getTableName());
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ }
+ }
+
+ @Test
+ public void testUcfWithDisabledIndex() throws Throwable {
+ final String tableName = generateUniqueName();
+ final String indexName = "IDX_" + tableName;
+
+ Connection conn = DriverManager.getConnection(getUrl());
+ try {
+ final Statement stmt = conn.createStatement();
+
+ stmt.execute("CREATE TABLE " + tableName
+ + " (COL1 CHAR(10) NOT NULL, COL2 CHAR(5) NOT NULL, COL3 VARCHAR,"
+ + " COL4 VARCHAR CONSTRAINT pk PRIMARY KEY(COL1, COL2))"
+ + " UPDATE_CACHE_FREQUENCY=10000");
+ stmt.execute("CREATE INDEX " + indexName
+ + " ON " + tableName + " (COL3) INCLUDE (COL4)");
+
+ stmt.execute("ALTER INDEX " + indexName + " ON " + tableName + " DISABLE");
+
+ Thread.sleep(11000);
+ stmt.execute("UPSERT INTO " + tableName
+ + " (col1, col2, col3, col4) values ('c001', "
+ + "'c002', 'c003', 'c004')");
+ conn.commit();
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+
+ stmt.execute("UPSERT INTO " + tableName
+ + " (col1, col2, col3, col4) values ('c011', "
+ + "'c012', 'c013', 'c014')");
+ conn.commit();
+ throw new RuntimeException("Should not reach here");
+ } catch (PhoenixIOException e) {
Review Comment:
What if in the middle of this test:
1. we change index state using ALTER command
2. switch to normal co-proc and do a query on the table so that the getTable goes through
3. then we switch to new coproc and try another query to see if there was a getTable or not
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheFreqIT.java:
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.end2end;
+
+import com.google.protobuf.RpcCallback;
+import com.google.protobuf.RpcController;
+import org.apache.hadoop.hbase.DoNotRetryIOException;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.compile.ExplainPlan;
+import org.apache.phoenix.compile.ExplainPlanAttributes;
+import org.apache.phoenix.coprocessor.MetaDataEndpointImpl;
+import org.apache.phoenix.coprocessor.generated.MetaDataProtos;
+import org.apache.phoenix.coprocessorclient.BaseScannerRegionObserverConstants;
+import org.apache.phoenix.exception.PhoenixIOException;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.jdbc.PhoenixPreparedStatement;
+import org.apache.phoenix.protobuf.ProtobufUtil;
+import org.apache.phoenix.query.BaseTest;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Maps;
+import org.apache.phoenix.util.ClientUtil;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.Map;
+
+import static org.apache.phoenix.query.QueryServices.DISABLE_VIEW_SUBTREE_VALIDATION;
+
+@Category(NeedsOwnMiniClusterTest.class)
+public class UpdateCacheFreqIT extends BaseTest {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(UpdateCacheFreqIT.class);
+
+ private static void initCluster() throws Exception {
+ Map<String, String> props = Maps.newConcurrentMap();
+ props.put(BaseScannerRegionObserverConstants.PHOENIX_MAX_LOOKBACK_AGE_CONF_KEY,
+ Integer.toString(60 * 60 * 1000));
+ props.put(QueryServices.DEFAULT_UPDATE_CACHE_FREQUENCY_ATRRIB, "ALWAYS");
+ props.put(QueryServices.LAST_DDL_TIMESTAMP_VALIDATION_ENABLED, Boolean.toString(false));
+ props.put(QueryServices.PHOENIX_METADATA_INVALIDATE_CACHE_ENABLED, Boolean.toString(false));
+ props.put(QueryServices.TASK_HANDLING_INITIAL_DELAY_MS_ATTRIB,
+ Long.toString(Long.MAX_VALUE));
+ props.put(DISABLE_VIEW_SUBTREE_VALIDATION, "true");
+ setUpTestDriver(new ReadOnlyProps(props.entrySet().iterator()));
+ }
+
+ @BeforeClass
+ public static synchronized void doSetup() throws Exception {
+ initCluster();
+ }
+
+ public static class TestMetaDataEndpointImpl extends MetaDataEndpointImpl {
+
+ @Override
+ public void getTable(RpcController controller, MetaDataProtos.GetTableRequest request,
+ RpcCallback<MetaDataProtos.MetaDataResponse> done) {
+ LOGGER.error("Not expected to get getTable() for {}",
+ Bytes.toString(request.getTableName().toByteArray()));
+ ProtobufUtil.setControllerException(controller,
+ ClientUtil.createIOException(
+ SchemaUtil.getPhysicalTableName(
+ PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES,
+ false).toString(),
+ new DoNotRetryIOException("Not allowed")));
+
+ }
+
+ }
+
+ @Test
+ public void testUcfWithNoGetTableCalls() throws Throwable {
+ final String tableName = generateUniqueName();
+ final String indexName = "IDX_" + tableName;
+ final String view01 = "v01_" + tableName;
+ final String view02 = "v02_" + tableName;
+ final String index_view01 = "idx_v01_" + tableName;
+ final String index_view02 = "idx_v02_" + tableName;
+ final String index_view03 = "idx_v03_" + tableName;
+ final String index_view04 = "idx_v04_" + tableName;
+
+ try (Connection conn = DriverManager.getConnection(getUrl())) {
+ final Statement stmt = conn.createStatement();
+
+ stmt.execute("CREATE TABLE " + tableName
+ + " (COL1 CHAR(10) NOT NULL, COL2 CHAR(5) NOT NULL, COL3 VARCHAR,"
+ + " COL4 VARCHAR CONSTRAINT pk PRIMARY KEY(COL1, COL2))"
+ + " UPDATE_CACHE_FREQUENCY=20000");
+ stmt.execute("CREATE INDEX " + indexName
+ + " ON " + tableName + " (COL3) INCLUDE (COL4)");
+ stmt.execute("CREATE VIEW " + view01
+ + " (VCOL1 CHAR(8), COL5 VARCHAR) AS SELECT * FROM " + tableName
+ + " WHERE COL1 = 'col1'");
+ stmt.execute("CREATE VIEW " + view02 + " (VCOL2 CHAR(10), COL6 VARCHAR)"
+ + " AS SELECT * FROM " + view01 + " WHERE VCOL1 = 'vcol1'");
+
+ stmt.execute("CREATE INDEX " + index_view01 + " ON " + view01 + " (COL5) INCLUDE "
+ + "(COL1, COL2, COL3)");
+ stmt.execute("CREATE INDEX " + index_view02 + " ON " + view02 + " (COL6) INCLUDE "
+ + "(COL1, COL2, COL3)");
+ stmt.execute("CREATE INDEX " + index_view03 + " ON " + view01 + " (COL5) INCLUDE "
+ + "(COL2, COL1)");
+ stmt.execute("CREATE INDEX " + index_view04 + " ON " + view02 + " (COL6) INCLUDE "
+ + "(COL2, COL1)");
+
+ stmt.execute("UPSERT INTO " + view02
+ + " (col2, vcol2, col5, col6) values ('0001', 'vcol2_01', 'col5_01', " +
+ "'col6_01')");
+ conn.commit();
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0002', 'vcol2_02', 'col5_02', 'col6_02')");
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0003', 'vcol2_03', 'col5_03', 'col6_03')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0004', 'vcol2', 'col3_04', 'col4_04', 'col5_04')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0005', 'vcol-2', 'col3_05', 'col4_05', 'col5_05')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0006', 'vcol-1', 'col3_06', 'col4_06', 'col5_06')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0007', 'vcol1', 'col3_07', 'col4_07', 'col5_07')");
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0008', 'vcol2_08', 'col5_08', 'col6_02')");
+ conn.commit();
+
+ final Statement statement = conn.createStatement();
+ ResultSet rs =
+ statement.executeQuery(
+ "SELECT COL2, VCOL1, VCOL2, COL5, COL6 FROM " + view02);
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertFalse(rs.next());
+
+ String query = "SELECT COL1, COL2, COL3, COL4 FROM " + tableName + " WHERE COL3 = " +
+ "'col3_04'";
+ rs = statement.executeQuery(query);
+ Assert.assertTrue(rs.next());
+ Assert.assertFalse(rs.next());
+
+ ExplainPlan explainPlan = conn.prepareStatement(query)
+ .unwrap(PhoenixPreparedStatement.class).optimizeQuery()
+ .getExplainPlan();
+ ExplainPlanAttributes explainPlanAttributes =
+ explainPlan.getPlanStepsAsAttributes();
+ Assert.assertEquals(indexName, explainPlanAttributes.getTableName());
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ }
+ }
+
+ @Test
+ public void testUcfWithDisabledIndex() throws Throwable {
+ final String tableName = generateUniqueName();
+ final String indexName = "IDX_" + tableName;
+
+ Connection conn = DriverManager.getConnection(getUrl());
+ try {
+ final Statement stmt = conn.createStatement();
+
+ stmt.execute("CREATE TABLE " + tableName
+ + " (COL1 CHAR(10) NOT NULL, COL2 CHAR(5) NOT NULL, COL3 VARCHAR,"
+ + " COL4 VARCHAR CONSTRAINT pk PRIMARY KEY(COL1, COL2))"
+ + " UPDATE_CACHE_FREQUENCY=10000");
+ stmt.execute("CREATE INDEX " + indexName
+ + " ON " + tableName + " (COL3) INCLUDE (COL4)");
+
+ stmt.execute("ALTER INDEX " + indexName + " ON " + tableName + " DISABLE");
+
+ Thread.sleep(11000);
+ stmt.execute("UPSERT INTO " + tableName
+ + " (col1, col2, col3, col4) values ('c001', "
+ + "'c002', 'c003', 'c004')");
+ conn.commit();
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+
+ stmt.execute("UPSERT INTO " + tableName
+ + " (col1, col2, col3, col4) values ('c011', "
+ + "'c012', 'c013', 'c014')");
+ conn.commit();
+ throw new RuntimeException("Should not reach here");
+ } catch (PhoenixIOException e) {
Review Comment:
What if in the end of this test:
1. we change index state using ALTER command
2. switch to normal co-proc and do a query on the table so that the getTable goes through
3. then we switch to new coproc and try another query to see if there was a getTable or not
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1722397847
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheFreqIT.java:
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.end2end;
+
+import com.google.protobuf.RpcCallback;
+import com.google.protobuf.RpcController;
+import org.apache.hadoop.hbase.DoNotRetryIOException;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.compile.ExplainPlan;
+import org.apache.phoenix.compile.ExplainPlanAttributes;
+import org.apache.phoenix.coprocessor.MetaDataEndpointImpl;
+import org.apache.phoenix.coprocessor.generated.MetaDataProtos;
+import org.apache.phoenix.coprocessorclient.BaseScannerRegionObserverConstants;
+import org.apache.phoenix.exception.PhoenixIOException;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.jdbc.PhoenixPreparedStatement;
+import org.apache.phoenix.protobuf.ProtobufUtil;
+import org.apache.phoenix.query.BaseTest;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Maps;
+import org.apache.phoenix.util.ClientUtil;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.Map;
+
+import static org.apache.phoenix.query.QueryServices.DISABLE_VIEW_SUBTREE_VALIDATION;
+
+@Category(NeedsOwnMiniClusterTest.class)
+public class UpdateCacheFreqIT extends BaseTest {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(UpdateCacheFreqIT.class);
+
+ private static void initCluster() throws Exception {
+ Map<String, String> props = Maps.newConcurrentMap();
+ props.put(BaseScannerRegionObserverConstants.PHOENIX_MAX_LOOKBACK_AGE_CONF_KEY,
+ Integer.toString(60 * 60 * 1000));
+ props.put(QueryServices.DEFAULT_UPDATE_CACHE_FREQUENCY_ATRRIB, "ALWAYS");
+ props.put(QueryServices.LAST_DDL_TIMESTAMP_VALIDATION_ENABLED, Boolean.toString(false));
+ props.put(QueryServices.PHOENIX_METADATA_INVALIDATE_CACHE_ENABLED, Boolean.toString(false));
+ props.put(QueryServices.TASK_HANDLING_INITIAL_DELAY_MS_ATTRIB,
+ Long.toString(Long.MAX_VALUE));
+ props.put(DISABLE_VIEW_SUBTREE_VALIDATION, "true");
+ setUpTestDriver(new ReadOnlyProps(props.entrySet().iterator()));
+ }
+
+ @BeforeClass
+ public static synchronized void doSetup() throws Exception {
+ initCluster();
+ }
+
+ public static class TestMetaDataEndpointImpl extends MetaDataEndpointImpl {
+
+ @Override
+ public void getTable(RpcController controller, MetaDataProtos.GetTableRequest request,
+ RpcCallback<MetaDataProtos.MetaDataResponse> done) {
+ LOGGER.error("Not expected to get getTable() for {}",
+ Bytes.toString(request.getTableName().toByteArray()));
+ ProtobufUtil.setControllerException(controller,
+ ClientUtil.createIOException(
+ SchemaUtil.getPhysicalTableName(
+ PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES,
+ false).toString(),
+ new DoNotRetryIOException("Not allowed")));
+
+ }
+
+ }
+
+ @Test
+ public void testUcfWithNoGetTableCalls() throws Throwable {
+ final String tableName = generateUniqueName();
+ final String indexName = "IDX_" + tableName;
+ final String view01 = "v01_" + tableName;
+ final String view02 = "v02_" + tableName;
+ final String index_view01 = "idx_v01_" + tableName;
+ final String index_view02 = "idx_v02_" + tableName;
+ final String index_view03 = "idx_v03_" + tableName;
+ final String index_view04 = "idx_v04_" + tableName;
+
+ try (Connection conn = DriverManager.getConnection(getUrl())) {
+ final Statement stmt = conn.createStatement();
+
+ stmt.execute("CREATE TABLE " + tableName
+ + " (COL1 CHAR(10) NOT NULL, COL2 CHAR(5) NOT NULL, COL3 VARCHAR,"
+ + " COL4 VARCHAR CONSTRAINT pk PRIMARY KEY(COL1, COL2))"
+ + " UPDATE_CACHE_FREQUENCY=20000");
+ stmt.execute("CREATE INDEX " + indexName
+ + " ON " + tableName + " (COL3) INCLUDE (COL4)");
+ stmt.execute("CREATE VIEW " + view01
+ + " (VCOL1 CHAR(8), COL5 VARCHAR) AS SELECT * FROM " + tableName
+ + " WHERE COL1 = 'col1'");
+ stmt.execute("CREATE VIEW " + view02 + " (VCOL2 CHAR(10), COL6 VARCHAR)"
+ + " AS SELECT * FROM " + view01 + " WHERE VCOL1 = 'vcol1'");
+
+ stmt.execute("CREATE INDEX " + index_view01 + " ON " + view01 + " (COL5) INCLUDE "
+ + "(COL1, COL2, COL3)");
+ stmt.execute("CREATE INDEX " + index_view02 + " ON " + view02 + " (COL6) INCLUDE "
+ + "(COL1, COL2, COL3)");
+ stmt.execute("CREATE INDEX " + index_view03 + " ON " + view01 + " (COL5) INCLUDE "
+ + "(COL2, COL1)");
+ stmt.execute("CREATE INDEX " + index_view04 + " ON " + view02 + " (COL6) INCLUDE "
+ + "(COL2, COL1)");
+
+ stmt.execute("UPSERT INTO " + view02
+ + " (col2, vcol2, col5, col6) values ('0001', 'vcol2_01', 'col5_01', " +
+ "'col6_01')");
+ conn.commit();
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0002', 'vcol2_02', 'col5_02', 'col6_02')");
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0003', 'vcol2_03', 'col5_03', 'col6_03')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0004', 'vcol2', 'col3_04', 'col4_04', 'col5_04')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0005', 'vcol-2', 'col3_05', 'col4_05', 'col5_05')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0006', 'vcol-1', 'col3_06', 'col4_06', 'col5_06')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0007', 'vcol1', 'col3_07', 'col4_07', 'col5_07')");
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0008', 'vcol2_08', 'col5_08', 'col6_02')");
+ conn.commit();
+
+ final Statement statement = conn.createStatement();
+ ResultSet rs =
+ statement.executeQuery(
+ "SELECT COL2, VCOL1, VCOL2, COL5, COL6 FROM " + view02);
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertFalse(rs.next());
+
+ String query = "SELECT COL1, COL2, COL3, COL4 FROM " + tableName + " WHERE COL3 = " +
+ "'col3_04'";
+ rs = statement.executeQuery(query);
+ Assert.assertTrue(rs.next());
+ Assert.assertFalse(rs.next());
+
+ ExplainPlan explainPlan = conn.prepareStatement(query)
+ .unwrap(PhoenixPreparedStatement.class).optimizeQuery()
+ .getExplainPlan();
+ ExplainPlanAttributes explainPlanAttributes =
+ explainPlan.getPlanStepsAsAttributes();
+ Assert.assertEquals(indexName, explainPlanAttributes.getTableName());
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ }
+ }
+
+ @Test
+ public void testUcfWithDisabledIndex() throws Throwable {
+ final String tableName = generateUniqueName();
+ final String indexName = "IDX_" + tableName;
+
+ Connection conn = DriverManager.getConnection(getUrl());
+ try {
+ final Statement stmt = conn.createStatement();
+
+ stmt.execute("CREATE TABLE " + tableName
+ + " (COL1 CHAR(10) NOT NULL, COL2 CHAR(5) NOT NULL, COL3 VARCHAR,"
+ + " COL4 VARCHAR CONSTRAINT pk PRIMARY KEY(COL1, COL2))"
+ + " UPDATE_CACHE_FREQUENCY=10000");
+ stmt.execute("CREATE INDEX " + indexName
+ + " ON " + tableName + " (COL3) INCLUDE (COL4)");
+
+ stmt.execute("ALTER INDEX " + indexName + " ON " + tableName + " DISABLE");
+
+ Thread.sleep(11000);
+ stmt.execute("UPSERT INTO " + tableName
+ + " (col1, col2, col3, col4) values ('c001', "
+ + "'c002', 'c003', 'c004')");
+ conn.commit();
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+
+ stmt.execute("UPSERT INTO " + tableName
+ + " (col1, col2, col3, col4) values ('c011', "
+ + "'c012', 'c013', 'c014')");
+ conn.commit();
+ throw new RuntimeException("Should not reach here");
+ } catch (PhoenixIOException e) {
Review Comment:
What if in the end of this test:
1. we change index state using ALTER command (can use a different client for this)
2. switch to normal co-proc and do a query on the table so that the getTable goes through
3. then we switch to new coproc and try another query to see if there was a getTable or not
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1722386983
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheFreqIT.java:
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.end2end;
+
+import com.google.protobuf.RpcCallback;
+import com.google.protobuf.RpcController;
+import org.apache.hadoop.hbase.DoNotRetryIOException;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.compile.ExplainPlan;
+import org.apache.phoenix.compile.ExplainPlanAttributes;
+import org.apache.phoenix.coprocessor.MetaDataEndpointImpl;
+import org.apache.phoenix.coprocessor.generated.MetaDataProtos;
+import org.apache.phoenix.coprocessorclient.BaseScannerRegionObserverConstants;
+import org.apache.phoenix.exception.PhoenixIOException;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.jdbc.PhoenixPreparedStatement;
+import org.apache.phoenix.protobuf.ProtobufUtil;
+import org.apache.phoenix.query.BaseTest;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Maps;
+import org.apache.phoenix.util.ClientUtil;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.Map;
+
+import static org.apache.phoenix.query.QueryServices.DISABLE_VIEW_SUBTREE_VALIDATION;
+
+@Category(NeedsOwnMiniClusterTest.class)
+public class UpdateCacheFreqIT extends BaseTest {
Review Comment:
@virajjasani We already have a test class `UpdateCacheIT`. Should we re-use that for these tests? If not, maybe we can have some different name for this test class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1720144218
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheFreqWithDDLValidationIT.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.end2end;
+
+import org.apache.phoenix.coprocessorclient.BaseScannerRegionObserverConstants;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Maps;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.junit.BeforeClass;
+
+import java.util.Map;
+
+//@Category(NeedsOwnMiniClusterTest.class)
+// Uncomment as part of PHOENIX-7381
Review Comment:
When enabling last_ddl_timestamp validation, we would want to set default UCF to NEVER so that clients do not perform a server RPC unless required. Not sure if UCF=always makes sense with enabling 6883 re-design.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1720601097
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheFreqWithDDLValidationIT.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.end2end;
+
+import org.apache.phoenix.coprocessorclient.BaseScannerRegionObserverConstants;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Maps;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.junit.BeforeClass;
+
+import java.util.Map;
+
+//@Category(NeedsOwnMiniClusterTest.class)
+// Uncomment as part of PHOENIX-7381
Review Comment:
This is not ALWAYS, this is some non default positive value e.g. 20s or 10s.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1722484469
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheFreqIT.java:
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.end2end;
+
+import com.google.protobuf.RpcCallback;
+import com.google.protobuf.RpcController;
+import org.apache.hadoop.hbase.DoNotRetryIOException;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.compile.ExplainPlan;
+import org.apache.phoenix.compile.ExplainPlanAttributes;
+import org.apache.phoenix.coprocessor.MetaDataEndpointImpl;
+import org.apache.phoenix.coprocessor.generated.MetaDataProtos;
+import org.apache.phoenix.coprocessorclient.BaseScannerRegionObserverConstants;
+import org.apache.phoenix.exception.PhoenixIOException;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.jdbc.PhoenixPreparedStatement;
+import org.apache.phoenix.protobuf.ProtobufUtil;
+import org.apache.phoenix.query.BaseTest;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Maps;
+import org.apache.phoenix.util.ClientUtil;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.Map;
+
+import static org.apache.phoenix.query.QueryServices.DISABLE_VIEW_SUBTREE_VALIDATION;
+
+@Category(NeedsOwnMiniClusterTest.class)
+public class UpdateCacheFreqIT extends BaseTest {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(UpdateCacheFreqIT.class);
+
+ private static void initCluster() throws Exception {
+ Map<String, String> props = Maps.newConcurrentMap();
+ props.put(BaseScannerRegionObserverConstants.PHOENIX_MAX_LOOKBACK_AGE_CONF_KEY,
+ Integer.toString(60 * 60 * 1000));
+ props.put(QueryServices.DEFAULT_UPDATE_CACHE_FREQUENCY_ATRRIB, "ALWAYS");
+ props.put(QueryServices.LAST_DDL_TIMESTAMP_VALIDATION_ENABLED, Boolean.toString(false));
+ props.put(QueryServices.PHOENIX_METADATA_INVALIDATE_CACHE_ENABLED, Boolean.toString(false));
+ props.put(QueryServices.TASK_HANDLING_INITIAL_DELAY_MS_ATTRIB,
+ Long.toString(Long.MAX_VALUE));
+ props.put(DISABLE_VIEW_SUBTREE_VALIDATION, "true");
+ setUpTestDriver(new ReadOnlyProps(props.entrySet().iterator()));
+ }
+
+ @BeforeClass
+ public static synchronized void doSetup() throws Exception {
+ initCluster();
+ }
+
+ public static class TestMetaDataEndpointImpl extends MetaDataEndpointImpl {
+
+ @Override
+ public void getTable(RpcController controller, MetaDataProtos.GetTableRequest request,
+ RpcCallback<MetaDataProtos.MetaDataResponse> done) {
+ LOGGER.error("Not expected to get getTable() for {}",
+ Bytes.toString(request.getTableName().toByteArray()));
+ ProtobufUtil.setControllerException(controller,
+ ClientUtil.createIOException(
+ SchemaUtil.getPhysicalTableName(
+ PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES,
+ false).toString(),
+ new DoNotRetryIOException("Not allowed")));
+
+ }
+
+ }
+
+ @Test
+ public void testUcfWithNoGetTableCalls() throws Throwable {
+ final String tableName = generateUniqueName();
+ final String indexName = "IDX_" + tableName;
+ final String view01 = "v01_" + tableName;
+ final String view02 = "v02_" + tableName;
+ final String index_view01 = "idx_v01_" + tableName;
+ final String index_view02 = "idx_v02_" + tableName;
+ final String index_view03 = "idx_v03_" + tableName;
+ final String index_view04 = "idx_v04_" + tableName;
+
+ try (Connection conn = DriverManager.getConnection(getUrl())) {
+ final Statement stmt = conn.createStatement();
+
+ stmt.execute("CREATE TABLE " + tableName
+ + " (COL1 CHAR(10) NOT NULL, COL2 CHAR(5) NOT NULL, COL3 VARCHAR,"
+ + " COL4 VARCHAR CONSTRAINT pk PRIMARY KEY(COL1, COL2))"
+ + " UPDATE_CACHE_FREQUENCY=20000");
+ stmt.execute("CREATE INDEX " + indexName
+ + " ON " + tableName + " (COL3) INCLUDE (COL4)");
+ stmt.execute("CREATE VIEW " + view01
+ + " (VCOL1 CHAR(8), COL5 VARCHAR) AS SELECT * FROM " + tableName
+ + " WHERE COL1 = 'col1'");
+ stmt.execute("CREATE VIEW " + view02 + " (VCOL2 CHAR(10), COL6 VARCHAR)"
+ + " AS SELECT * FROM " + view01 + " WHERE VCOL1 = 'vcol1'");
+
+ stmt.execute("CREATE INDEX " + index_view01 + " ON " + view01 + " (COL5) INCLUDE "
+ + "(COL1, COL2, COL3)");
+ stmt.execute("CREATE INDEX " + index_view02 + " ON " + view02 + " (COL6) INCLUDE "
+ + "(COL1, COL2, COL3)");
+ stmt.execute("CREATE INDEX " + index_view03 + " ON " + view01 + " (COL5) INCLUDE "
+ + "(COL2, COL1)");
+ stmt.execute("CREATE INDEX " + index_view04 + " ON " + view02 + " (COL6) INCLUDE "
+ + "(COL2, COL1)");
+
+ stmt.execute("UPSERT INTO " + view02
+ + " (col2, vcol2, col5, col6) values ('0001', 'vcol2_01', 'col5_01', " +
+ "'col6_01')");
+ conn.commit();
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0002', 'vcol2_02', 'col5_02', 'col6_02')");
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0003', 'vcol2_03', 'col5_03', 'col6_03')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0004', 'vcol2', 'col3_04', 'col4_04', 'col5_04')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0005', 'vcol-2', 'col3_05', 'col4_05', 'col5_05')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0006', 'vcol-1', 'col3_06', 'col4_06', 'col5_06')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0007', 'vcol1', 'col3_07', 'col4_07', 'col5_07')");
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0008', 'vcol2_08', 'col5_08', 'col6_02')");
+ conn.commit();
+
+ final Statement statement = conn.createStatement();
+ ResultSet rs =
+ statement.executeQuery(
+ "SELECT COL2, VCOL1, VCOL2, COL5, COL6 FROM " + view02);
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertFalse(rs.next());
+
+ String query = "SELECT COL1, COL2, COL3, COL4 FROM " + tableName + " WHERE COL3 = " +
+ "'col3_04'";
+ rs = statement.executeQuery(query);
+ Assert.assertTrue(rs.next());
+ Assert.assertFalse(rs.next());
+
+ ExplainPlan explainPlan = conn.prepareStatement(query)
+ .unwrap(PhoenixPreparedStatement.class).optimizeQuery()
+ .getExplainPlan();
+ ExplainPlanAttributes explainPlanAttributes =
+ explainPlan.getPlanStepsAsAttributes();
+ Assert.assertEquals(indexName, explainPlanAttributes.getTableName());
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ }
+ }
+
+ @Test
+ public void testUcfWithDisabledIndex() throws Throwable {
+ final String tableName = generateUniqueName();
+ final String indexName = "IDX_" + tableName;
+
+ Connection conn = DriverManager.getConnection(getUrl());
+ try {
+ final Statement stmt = conn.createStatement();
+
+ stmt.execute("CREATE TABLE " + tableName
+ + " (COL1 CHAR(10) NOT NULL, COL2 CHAR(5) NOT NULL, COL3 VARCHAR,"
+ + " COL4 VARCHAR CONSTRAINT pk PRIMARY KEY(COL1, COL2))"
+ + " UPDATE_CACHE_FREQUENCY=10000");
+ stmt.execute("CREATE INDEX " + indexName
+ + " ON " + tableName + " (COL3) INCLUDE (COL4)");
+
+ stmt.execute("ALTER INDEX " + indexName + " ON " + tableName + " DISABLE");
+
+ Thread.sleep(11000);
+ stmt.execute("UPSERT INTO " + tableName
+ + " (col1, col2, col3, col4) values ('c001', "
+ + "'c002', 'c003', 'c004')");
+ conn.commit();
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+
+ stmt.execute("UPSERT INTO " + tableName
+ + " (col1, col2, col3, col4) values ('c011', "
+ + "'c012', 'c013', 'c014')");
+ conn.commit();
+ throw new RuntimeException("Should not reach here");
+ } catch (PhoenixIOException e) {
Review Comment:
Query on the table will go through because the whole Data table PTable is in the client cache. We need query to Index table specifically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1720144218
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheFreqWithDDLValidationIT.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.end2end;
+
+import org.apache.phoenix.coprocessorclient.BaseScannerRegionObserverConstants;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Maps;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.junit.BeforeClass;
+
+import java.util.Map;
+
+//@Category(NeedsOwnMiniClusterTest.class)
+// Uncomment as part of PHOENIX-7381
Review Comment:
When enabling last_ddl_timestamp validation, we would want to set default UCF to NEVER.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1720142790
##########
phoenix-core-client/src/main/java/org/apache/phoenix/schema/MetaDataClient.java:
##########
@@ -785,19 +785,33 @@ private boolean avoidRpcToGetTable(boolean alwaysHitServer, Long resolvedTimesta
final long effectiveUpdateCacheFreq;
final String ucfInfoForLogging; // Only used for logging purposes
+ boolean overrideUcfToDefault =
Review Comment:
@virajjasani Should we override UCF if the ptable passed to this method is an index in any "unusable" state and not just PENDING_DISABLE?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1722390423
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheFreqIT.java:
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.end2end;
+
+import com.google.protobuf.RpcCallback;
+import com.google.protobuf.RpcController;
+import org.apache.hadoop.hbase.DoNotRetryIOException;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.compile.ExplainPlan;
+import org.apache.phoenix.compile.ExplainPlanAttributes;
+import org.apache.phoenix.coprocessor.MetaDataEndpointImpl;
+import org.apache.phoenix.coprocessor.generated.MetaDataProtos;
+import org.apache.phoenix.coprocessorclient.BaseScannerRegionObserverConstants;
+import org.apache.phoenix.exception.PhoenixIOException;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.jdbc.PhoenixPreparedStatement;
+import org.apache.phoenix.protobuf.ProtobufUtil;
+import org.apache.phoenix.query.BaseTest;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Maps;
+import org.apache.phoenix.util.ClientUtil;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.Map;
+
+import static org.apache.phoenix.query.QueryServices.DISABLE_VIEW_SUBTREE_VALIDATION;
+
+@Category(NeedsOwnMiniClusterTest.class)
+public class UpdateCacheFreqIT extends BaseTest {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(UpdateCacheFreqIT.class);
+
+ private static void initCluster() throws Exception {
+ Map<String, String> props = Maps.newConcurrentMap();
+ props.put(BaseScannerRegionObserverConstants.PHOENIX_MAX_LOOKBACK_AGE_CONF_KEY,
+ Integer.toString(60 * 60 * 1000));
+ props.put(QueryServices.DEFAULT_UPDATE_CACHE_FREQUENCY_ATRRIB, "ALWAYS");
+ props.put(QueryServices.LAST_DDL_TIMESTAMP_VALIDATION_ENABLED, Boolean.toString(false));
+ props.put(QueryServices.PHOENIX_METADATA_INVALIDATE_CACHE_ENABLED, Boolean.toString(false));
+ props.put(QueryServices.TASK_HANDLING_INITIAL_DELAY_MS_ATTRIB,
+ Long.toString(Long.MAX_VALUE));
+ props.put(DISABLE_VIEW_SUBTREE_VALIDATION, "true");
+ setUpTestDriver(new ReadOnlyProps(props.entrySet().iterator()));
+ }
+
+ @BeforeClass
+ public static synchronized void doSetup() throws Exception {
+ initCluster();
+ }
+
+ public static class TestMetaDataEndpointImpl extends MetaDataEndpointImpl {
+
+ @Override
+ public void getTable(RpcController controller, MetaDataProtos.GetTableRequest request,
+ RpcCallback<MetaDataProtos.MetaDataResponse> done) {
+ LOGGER.error("Not expected to get getTable() for {}",
+ Bytes.toString(request.getTableName().toByteArray()));
+ ProtobufUtil.setControllerException(controller,
+ ClientUtil.createIOException(
+ SchemaUtil.getPhysicalTableName(
+ PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES,
+ false).toString(),
+ new DoNotRetryIOException("Not allowed")));
+
+ }
+
+ }
+
+ @Test
+ public void testUcfWithNoGetTableCalls() throws Throwable {
+ final String tableName = generateUniqueName();
+ final String indexName = "IDX_" + tableName;
+ final String view01 = "v01_" + tableName;
+ final String view02 = "v02_" + tableName;
+ final String index_view01 = "idx_v01_" + tableName;
+ final String index_view02 = "idx_v02_" + tableName;
+ final String index_view03 = "idx_v03_" + tableName;
+ final String index_view04 = "idx_v04_" + tableName;
+
+ try (Connection conn = DriverManager.getConnection(getUrl())) {
+ final Statement stmt = conn.createStatement();
+
+ stmt.execute("CREATE TABLE " + tableName
+ + " (COL1 CHAR(10) NOT NULL, COL2 CHAR(5) NOT NULL, COL3 VARCHAR,"
+ + " COL4 VARCHAR CONSTRAINT pk PRIMARY KEY(COL1, COL2))"
+ + " UPDATE_CACHE_FREQUENCY=20000");
+ stmt.execute("CREATE INDEX " + indexName
+ + " ON " + tableName + " (COL3) INCLUDE (COL4)");
+ stmt.execute("CREATE VIEW " + view01
+ + " (VCOL1 CHAR(8), COL5 VARCHAR) AS SELECT * FROM " + tableName
+ + " WHERE COL1 = 'col1'");
+ stmt.execute("CREATE VIEW " + view02 + " (VCOL2 CHAR(10), COL6 VARCHAR)"
+ + " AS SELECT * FROM " + view01 + " WHERE VCOL1 = 'vcol1'");
+
+ stmt.execute("CREATE INDEX " + index_view01 + " ON " + view01 + " (COL5) INCLUDE "
+ + "(COL1, COL2, COL3)");
+ stmt.execute("CREATE INDEX " + index_view02 + " ON " + view02 + " (COL6) INCLUDE "
+ + "(COL1, COL2, COL3)");
+ stmt.execute("CREATE INDEX " + index_view03 + " ON " + view01 + " (COL5) INCLUDE "
+ + "(COL2, COL1)");
+ stmt.execute("CREATE INDEX " + index_view04 + " ON " + view02 + " (COL6) INCLUDE "
+ + "(COL2, COL1)");
+
+ stmt.execute("UPSERT INTO " + view02
+ + " (col2, vcol2, col5, col6) values ('0001', 'vcol2_01', 'col5_01', " +
+ "'col6_01')");
+ conn.commit();
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0002', 'vcol2_02', 'col5_02', 'col6_02')");
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0003', 'vcol2_03', 'col5_03', 'col6_03')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0004', 'vcol2', 'col3_04', 'col4_04', 'col5_04')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0005', 'vcol-2', 'col3_05', 'col4_05', 'col5_05')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0006', 'vcol-1', 'col3_06', 'col4_06', 'col5_06')");
+ stmt.execute("UPSERT INTO " + view01 + " (col2, vcol1, col3, col4, col5) values "
+ + "('0007', 'vcol1', 'col3_07', 'col4_07', 'col5_07')");
+ stmt.execute("UPSERT INTO " + view02
+ +
+ " (col2, vcol2, col5, col6) values ('0008', 'vcol2_08', 'col5_08', 'col6_02')");
+ conn.commit();
+
+ final Statement statement = conn.createStatement();
+ ResultSet rs =
+ statement.executeQuery(
+ "SELECT COL2, VCOL1, VCOL2, COL5, COL6 FROM " + view02);
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertTrue(rs.next());
+ Assert.assertFalse(rs.next());
+
+ String query = "SELECT COL1, COL2, COL3, COL4 FROM " + tableName + " WHERE COL3 = " +
+ "'col3_04'";
+ rs = statement.executeQuery(query);
+ Assert.assertTrue(rs.next());
+ Assert.assertFalse(rs.next());
+
+ ExplainPlan explainPlan = conn.prepareStatement(query)
+ .unwrap(PhoenixPreparedStatement.class).optimizeQuery()
+ .getExplainPlan();
+ ExplainPlanAttributes explainPlanAttributes =
+ explainPlan.getPlanStepsAsAttributes();
+ Assert.assertEquals(indexName, explainPlanAttributes.getTableName());
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ }
+ }
+
+ @Test
+ public void testUcfWithDisabledIndex() throws Throwable {
+ final String tableName = generateUniqueName();
+ final String indexName = "IDX_" + tableName;
+
+ Connection conn = DriverManager.getConnection(getUrl());
+ try {
+ final Statement stmt = conn.createStatement();
+
+ stmt.execute("CREATE TABLE " + tableName
+ + " (COL1 CHAR(10) NOT NULL, COL2 CHAR(5) NOT NULL, COL3 VARCHAR,"
+ + " COL4 VARCHAR CONSTRAINT pk PRIMARY KEY(COL1, COL2))"
+ + " UPDATE_CACHE_FREQUENCY=10000");
+ stmt.execute("CREATE INDEX " + indexName
+ + " ON " + tableName + " (COL3) INCLUDE (COL4)");
+
+ stmt.execute("ALTER INDEX " + indexName + " ON " + tableName + " DISABLE");
+
+ Thread.sleep(11000);
+ stmt.execute("UPSERT INTO " + tableName
+ + " (col1, col2, col3, col4) values ('c001', "
+ + "'c002', 'c003', 'c004')");
+ conn.commit();
+
+ TestUtil.removeCoprocessor(conn, "SYSTEM.CATALOG", MetaDataEndpointImpl.class);
+ TestUtil.addCoprocessor(conn, "SYSTEM.CATALOG", TestMetaDataEndpointImpl.class);
+
+ stmt.execute("UPSERT INTO " + tableName
+ + " (col1, col2, col3, col4) values ('c011', "
+ + "'c012', 'c013', 'c014')");
+ conn.commit();
+ throw new RuntimeException("Should not reach here");
+ } catch (PhoenixIOException e) {
Review Comment:
@virajjasani Do you think we should also test that once a "disabled" index is in BUILDING or ACTIVE state on the server, client learns about it and then stops overriding the UCF value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7386 Override UPDATE_CACHE_FREQUENCY if table has disabled indexes [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1955:
URL: https://github.com/apache/phoenix/pull/1955#discussion_r1722391750
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheFreqIT.java:
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.phoenix.end2end;
+
+import com.google.protobuf.RpcCallback;
+import com.google.protobuf.RpcController;
+import org.apache.hadoop.hbase.DoNotRetryIOException;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.phoenix.compile.ExplainPlan;
+import org.apache.phoenix.compile.ExplainPlanAttributes;
+import org.apache.phoenix.coprocessor.MetaDataEndpointImpl;
+import org.apache.phoenix.coprocessor.generated.MetaDataProtos;
+import org.apache.phoenix.coprocessorclient.BaseScannerRegionObserverConstants;
+import org.apache.phoenix.exception.PhoenixIOException;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.jdbc.PhoenixPreparedStatement;
+import org.apache.phoenix.protobuf.ProtobufUtil;
+import org.apache.phoenix.query.BaseTest;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.thirdparty.com.google.common.collect.Maps;
+import org.apache.phoenix.util.ClientUtil;
+import org.apache.phoenix.util.ReadOnlyProps;
+import org.apache.phoenix.util.SchemaUtil;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.Map;
+
+import static org.apache.phoenix.query.QueryServices.DISABLE_VIEW_SUBTREE_VALIDATION;
+
+@Category(NeedsOwnMiniClusterTest.class)
+public class UpdateCacheFreqIT extends BaseTest {
Review Comment:
Since this is mostly about indexes and positive UCF values, something on those lines would be great, whatever you decide should be good!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org