You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@phoenix.apache.org by "palashc (via GitHub)" <gi...@apache.org> on 2024/03/19 20:59:23 UTC
[PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
palashc opened a new pull request, #1859:
URL: https://github.com/apache/phoenix/pull/1859
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1561779159
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheAcrossDifferentClientsIT.java:
##########
@@ -104,17 +105,19 @@ public void testUpdateCacheFrequencyWithAddAndDropTable() throws Exception {
} catch (TableNotFoundException e) {
//Expected
}
- rs = conn2.createStatement().executeQuery("select * from "+tableName);
try {
+ rs = conn2.createStatement().executeQuery("select * from "+tableName);
rs.next();
fail("Should throw org.apache.hadoop.hbase.TableNotFoundException since the latest metadata " +
"wasn't fetched");
- } catch (PhoenixIOException ex) {
Review Comment:
client-1 dropped the table, client-2 will get TableNotFoundException i.e. SQLException when validating last ddl timestamp.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1563255982
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/AlterTableWithViewsIT.java:
##########
@@ -1149,10 +1152,8 @@ public void testDroppingIndexedColDropsViewIndex() throws Exception {
byte[] viewIndexPhysicalTable = viewIndex.getPhysicalName().getBytes();
assertNotNull("Can't find view index", viewIndex);
assertEquals("Unexpected number of indexes ", 2, view.getIndexes().size());
- assertEquals("Unexpected index ", fullNameViewIndex1 , view.getIndexes().get(0).getName()
Review Comment:
Need to look into this test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1563419971
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewMetadataIT.java:
##########
@@ -1173,10 +1174,10 @@ private void helpTestQueryForViewOnTableThatHasIndex(Statement s1,
s1.execute("create index " + indexName + " ON " + tableName
+ " (col2)");
- try (ResultSet rs = s2.executeQuery("explain select /*+ INDEX("
+ try (ResultSet rs = s2.executeQuery("select /*+ INDEX("
Review Comment:
Fixed the problem in https://issues.apache.org/jira/browse/PHOENIX-7307
Will revert this change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1563317917
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/AlterTableWithViewsIT.java:
##########
@@ -1149,10 +1152,8 @@ public void testDroppingIndexedColDropsViewIndex() throws Exception {
byte[] viewIndexPhysicalTable = viewIndex.getPhysicalName().getBytes();
assertNotNull("Can't find view index", viewIndex);
assertEquals("Unexpected number of indexes ", 2, view.getIndexes().size());
- assertEquals("Unexpected index ", fullNameViewIndex1 , view.getIndexes().get(0).getName()
Review Comment:
We do not invalidate cache when we drop indexes on the server side as part of a drop column statement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "shahrs87 (via GitHub)" <gi...@apache.org>.
shahrs87 commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1569445197
##########
phoenix-core/src/it/java/org/apache/phoenix/monitoring/PhoenixTableLevelMetricsIT.java:
##########
@@ -1276,14 +1281,24 @@ public void testMetricsWithIndexUsage() throws Exception {
assertTrue(metricExists);
metricExists = false;
//assert BaseTable is not being queried
- for (PhoenixTableMetric metric : getPhoenixTableClientMetrics().get(dataTable)) {
- if (metric.getMetricType().equals(SELECT_SQL_COUNTER)) {
- metricExists = true;
- assertMetricValue(metric, SELECT_SQL_COUNTER, 0, CompareOp.EQ);
- break;
+ //if client is validating last_ddl_timestamps with ucf=never,
+ //there will be no metrics for base table (like getTable RPC times/counts).
+ boolean lastDDLTimestampValidationEnabled =
+ conn.unwrap(PhoenixConnection.class).getQueryServices().getProps().getBoolean(
Review Comment:
Do you want to use [ValidateLastDDLTimestampUtil#getValidateLastDdlTimestampEnabled](https://github.com/apache/phoenix/blob/PHOENIX-6883-feature/phoenix-core-client/src/main/java/org/apache/phoenix/util/ValidateLastDDLTimestampUtil.java#L73C27-L73C61) method instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1562968430
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/IndexAsyncThresholdIT.java:
##########
@@ -162,9 +162,7 @@ public void testAsyncIndexCreation() throws Exception {
exception = (SQLException) e;
}
connection.commit();
- PTableKey key = new PTableKey(null, this.tableName);
- PMetaData metaCache = connection.unwrap(PhoenixConnection.class).getMetaDataCache();
- List<PTable> indexes = metaCache.getTableRef(key).getTable().getIndexes();
+ List<PTable> indexes = connection.unwrap(PhoenixConnection.class).getTable(this.tableName).getIndexes();
Review Comment:
This is to make sure we fetch the latest ptable. I think such changes are okay to keep even if UCF=always.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1563208179
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/LoadSystemTableSnapshotIT.java:
##########
@@ -167,6 +167,8 @@ public void testPhoenixUpgrade() throws Exception {
serverProps.put(QueryServices.IS_NAMESPACE_MAPPING_ENABLED, "true");
Map<String, String> clientProps = Maps.newHashMapWithExpectedSize(2);
clientProps.put(QueryServices.IS_NAMESPACE_MAPPING_ENABLED, "true");
+ clientProps.put(QueryServices.DEFAULT_UPDATE_CACHE_FREQUENCY_ATRRIB, Long.toString(0L));
Review Comment:
Remove this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1563256224
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewMetadataIT.java:
##########
@@ -1173,10 +1174,10 @@ private void helpTestQueryForViewOnTableThatHasIndex(Statement s1,
s1.execute("create index " + indexName + " ON " + tableName
+ " (col2)");
- try (ResultSet rs = s2.executeQuery("explain select /*+ INDEX("
+ try (ResultSet rs = s2.executeQuery("select /*+ INDEX("
Review Comment:
will look into this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1561780735
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/MutableIndexFailureIT.java:
##########
@@ -176,6 +178,7 @@ protected static Map<String,String> getServerProps(){
* because we want to control it's execution ourselves
*/
serverProps.put(QueryServices.INDEX_REBUILD_TASK_INITIAL_DELAY, Long.toString(Long.MAX_VALUE));
+ serverProps.put(REGIONSERVER_COPROCESSOR_CONF_KEY, PhoenixRegionServerEndpoint.class.getName());
Review Comment:
changed this and another test where it was missing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1562960469
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/LogicalTableNameExtendedIT.java:
##########
@@ -209,21 +209,26 @@ public void testHint() throws Exception {
createIndexOnTable(conn, tableName, indexName2);
populateTable(conn, tableName, 1, 2);
- // Test hint
String tableSelect = "SELECT V1,V2,V3 FROM " + tableName;
ResultSet rs = conn.createStatement().executeQuery("EXPLAIN " + tableSelect);
- assertEquals(true, QueryUtil.getExplainPlan(rs).contains(indexName));
+ String plan = QueryUtil.getExplainPlan(rs);
Review Comment:
There is assumption on the ordering of index PTables inside the parent table PTable. Optimizer can pick any of them but the test assumes only the first one in the list will get picked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1562953056
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/AlterTableWithViewsIT.java:
##########
@@ -1149,10 +1152,8 @@ public void testDroppingIndexedColDropsViewIndex() throws Exception {
byte[] viewIndexPhysicalTable = viewIndex.getPhysicalName().getBytes();
assertNotNull("Can't find view index", viewIndex);
assertEquals("Unexpected number of indexes ", 2, view.getIndexes().size());
- assertEquals("Unexpected index ", fullNameViewIndex1 , view.getIndexes().get(0).getName()
Review Comment:
The asserts here assume a certain ordering of indexes in the view's PTable but that ordering does not hold if the view PTable was refreshed in the client's cache. So I changed the asserts to look for the index name in a list of expected index names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1558009123
##########
phoenix-core-client/src/main/java/org/apache/phoenix/execute/MutationState.java:
##########
@@ -1240,6 +1241,15 @@ private void send(Iterator<TableRef> tableRefIterator) throws SQLException {
String tableName = tableRef.getTable().getTableName().toString();
mc.updateCache(tenantId, schemaName, tableName, true);
}
+ } catch (MetaDataEntityNotFoundException e) {
Review Comment:
Removed that change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "shahrs87 (via GitHub)" <gi...@apache.org>.
shahrs87 commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1556370118
##########
phoenix-core-client/src/main/java/org/apache/phoenix/execute/MutationState.java:
##########
@@ -1240,6 +1241,15 @@ private void send(Iterator<TableRef> tableRefIterator) throws SQLException {
String tableName = tableRef.getTable().getTableName().toString();
mc.updateCache(tenantId, schemaName, tableName, true);
}
+ } catch (MetaDataEntityNotFoundException e) {
Review Comment:
Why would `ValidateLastDDLTimestampUtil#validateLastDDLTimestamp` throw `MetaDataEntityNotFoundException` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "shahrs87 (via GitHub)" <gi...@apache.org>.
shahrs87 commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1571120003
##########
phoenix-core/src/it/java/org/apache/phoenix/query/MetaDataCachingIT.java:
##########
@@ -51,6 +51,10 @@ public class MetaDataCachingIT extends BaseTest {
private static final Logger LOGGER = LoggerFactory.getLogger(MetaDataCachingIT.class);
private final Random RAND = new Random(11);
+ private boolean isLastDDLTimestampValidationEnabled = config.getBoolean(
Review Comment:
Instead of duplicating this line in 3 different tests, do you mind creating a util method in `ValidateLastDDLTimestampUtil` something like `getValidateLastDdlTimestampEnabled(Configuration config)` similar to `getValidateLastDdlTimestampEnabled(PhoenixConnection connection)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1562956356
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/LogicalTableNameBaseIT.java:
##########
@@ -530,13 +531,16 @@ protected void validateIndex(Connection connection, String tableName, boolean is
}
public static void renameAndDropPhysicalTable(Connection conn, String tenantId, String schema, String tableName, String physicalName, boolean isNamespaceEnabled) throws Exception {
+ // if client is validating last_ddl_timestamp, this change in physical table name should be visible to the client
Review Comment:
We will have to make the change in our light fork then. Keeping this change does not affect the test even with UCF=always.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#issuecomment-2041306628
@shahrs87 Let me know if we can get this one in now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1562971229
##########
phoenix-core/src/it/java/org/apache/phoenix/monitoring/PhoenixTableLevelMetricsIT.java:
##########
@@ -1276,14 +1281,24 @@ public void testMetricsWithIndexUsage() throws Exception {
assertTrue(metricExists);
metricExists = false;
//assert BaseTable is not being queried
- for (PhoenixTableMetric metric : getPhoenixTableClientMetrics().get(dataTable)) {
- if (metric.getMetricType().equals(SELECT_SQL_COUNTER)) {
- metricExists = true;
- assertMetricValue(metric, SELECT_SQL_COUNTER, 0, CompareOp.EQ);
- break;
+ //if client is validating last_ddl_timestamps with ucf=never,
+ //there will be no metrics for base table (like getTable RPC times/counts).
Review Comment:
Since the query plan picks the index, client never interacts with the base table directly if UCF=never and hence metrics related objects are not initialized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1563230909
##########
phoenix-core/src/it/java/org/apache/phoenix/jdbc/HighAvailabilityTestingUtility.java:
##########
@@ -536,6 +539,12 @@ private static void setUpDefaultHBaseConfig(Configuration conf) {
// Hadoop cluster settings to avoid failing tests
conf.setInt(DFS_REPLICATION_KEY, 1); // we only need one replica for testing
+
+ // Phoenix Region Server Endpoint needed for metadata caching
+ if (QueryServicesOptions.DEFAULT_LAST_DDL_TIMESTAMP_VALIDATION_ENABLED) {
Review Comment:
remove the if check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "shahrs87 (via GitHub)" <gi...@apache.org>.
shahrs87 commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1569412298
##########
phoenix-core-client/src/main/java/org/apache/phoenix/schema/MetaDataClient.java:
##########
@@ -1776,7 +1778,18 @@ public MutationState createIndex(CreateIndexStatement statement, byte[][] splits
return buildIndexAtTimeStamp(table, statement.getTable());
}
- return buildIndex(table, tableRef);
+ MutationState state = buildIndex(table, tableRef);
+ // If client is validating LAST_DDL_TIMESTAMPS, parent's last_ddl_timestamp changed
+ // so remove it from client's cache. It will be refreshed when table is accessed next time.
+ boolean lastDDLTimestampValidationEnabled =
Review Comment:
Do you want to use [ValidateLastDDLTimestampUtil#getValidateLastDdlTimestampEnabled](https://github.com/apache/phoenix/blob/PHOENIX-6883-feature/phoenix-core-client/src/main/java/org/apache/phoenix/util/ValidateLastDDLTimestampUtil.java#L73C27-L73C61) method instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1563020743
##########
phoenix-core/src/it/java/org/apache/phoenix/query/MetaDataCachingIT.java:
##########
@@ -142,18 +146,25 @@ public void testSystemTablesAreInCache() throws Exception {
*/
@Test
public void testGlobalClientCacheMetrics() throws Exception {
- int numThreads = 5;
+ int numThreads = 1;
Review Comment:
not sure I remember this one, I will revert this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1571084136
##########
phoenix-core/src/it/java/org/apache/phoenix/query/MetaDataCachingIT.java:
##########
@@ -152,12 +156,19 @@ public void testGlobalClientCacheMetrics() throws Exception {
simulateWorkload("testGlobalClientCacheMetrics", numTables, numThreads, numMaxDML);
// only 1 miss when the table is created
+ int numExpectedMisses = 1;
+ if (QueryServicesOptions.DEFAULT_LAST_DDL_TIMESTAMP_VALIDATION_ENABLED) {
Review Comment:
I will make the change in tests to get the value from the config object in BaseTest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#issuecomment-2065439873
@shahrs87 Latest test report - https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1859/22/testReport/
`UpsertSelectIT.testUpsertSelectForAgg` passes locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1571134045
##########
phoenix-core/src/it/java/org/apache/phoenix/rpc/UpdateCacheIT.java:
##########
@@ -69,6 +70,10 @@
@Category(ParallelStatsDisabledTest.class)
public class UpdateCacheIT extends ParallelStatsDisabledIT {
+ private boolean isLastDDLTimestampValidationEnabled
+ = QueryServicesOptions.DEFAULT_LAST_DDL_TIMESTAMP_VALIDATION_ENABLED
Review Comment:
will use config
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1563309487
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheAcrossDifferentClientsIT.java:
##########
@@ -104,17 +105,19 @@ public void testUpdateCacheFrequencyWithAddAndDropTable() throws Exception {
} catch (TableNotFoundException e) {
//Expected
}
- rs = conn2.createStatement().executeQuery("select * from "+tableName);
try {
+ rs = conn2.createStatement().executeQuery("select * from "+tableName);
rs.next();
fail("Should throw org.apache.hadoop.hbase.TableNotFoundException since the latest metadata " +
"wasn't fetched");
- } catch (PhoenixIOException ex) {
Review Comment:
@shahrs87 The difference comes from where the exception is thrown.
- The test has given UCF=1000000000 when creating the table. So when client-2 from old design tried to query the dropped table, it does not refresh its cache. `executeQuery` does not error out. `rs.next()` throws `PhoenixIOException`
```
org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: N000001
at org.apache.phoenix.util.ClientUtil.parseServerException(ClientUtil.java:70)
at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1464)
at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1398)
at org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:188)
at org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:94)
at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:883)
```
- With new design, we fail in `executeQuery` itself and throw `TableNotFoundException` when validating timestamps.
```
org.apache.phoenix.schema.TableNotFoundException: ERROR 1012 (42M03): Table undefined.
at org.apache.phoenix.exception.SQLExceptionCode$9.newException(SQLExceptionCode.java:239) ~[classes/:?]
at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:229) ~[classes/:?]
at org.apache.phoenix.util.ClientUtil.parseRemoteException(ClientUtil.java:115) ~[classes/:?]
at org.apache.phoenix.util.ClientUtil.parseServerExceptionOrNull(ClientUtil.java:80) ~[classes/:?]
at org.apache.phoenix.util.ClientUtil.parseServerException(ClientUtil.java:66) ~[classes/:?]
at org.apache.phoenix.util.ValidateLastDDLTimestampUtil.validateLastDDLTimestamp(ValidateLastDDLTimestampUtil.java:112) ~[classes/:?]
at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:390) ~[classes/:?]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "shahrs87 (via GitHub)" <gi...@apache.org>.
shahrs87 commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1556441800
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheAcrossDifferentClientsIT.java:
##########
@@ -104,17 +105,19 @@ public void testUpdateCacheFrequencyWithAddAndDropTable() throws Exception {
} catch (TableNotFoundException e) {
//Expected
}
- rs = conn2.createStatement().executeQuery("select * from "+tableName);
try {
+ rs = conn2.createStatement().executeQuery("select * from "+tableName);
rs.next();
fail("Should throw org.apache.hadoop.hbase.TableNotFoundException since the latest metadata " +
"wasn't fetched");
- } catch (PhoenixIOException ex) {
Review Comment:
Why is this change required?
##########
phoenix-core-client/src/main/java/org/apache/phoenix/util/ValidateLastDDLTimestampUtil.java:
##########
@@ -67,6 +67,7 @@ public static String getInfoString(PName tenantId, List<TableRef> tableRefs) {
/**
* Get whether last ddl timestamp validation is enabled on the connection
+ *
Review Comment:
nit: remove extra line.
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpgradeIT.java:
##########
@@ -731,13 +730,14 @@ public void testLastDDLTimestampBootstrap() throws Exception {
}
private void nullDDLTimestamps(Connection conn) throws SQLException {
+ //ignore system tables since that can interfere with other tests.
String pkCols = TENANT_ID + ", " + TABLE_SCHEM +
", " + TABLE_NAME + ", " + COLUMN_NAME + ", " + COLUMN_FAMILY;
String upsertSql =
"UPSERT INTO " + SYSTEM_CATALOG_NAME + " (" + pkCols + ", " +
LAST_DDL_TIMESTAMP + ")" + " " +
"SELECT " + pkCols + ", NULL FROM " + SYSTEM_CATALOG_NAME + " " +
- "WHERE " + TABLE_TYPE + " IS NOT NULL";
+ "WHERE " + TABLE_TYPE + " " + " != '" + PTableType.SYSTEM.getSerializedValue() + "'";
Review Comment:
why is this change required?
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewIT.java:
##########
@@ -660,6 +661,14 @@ public void testCreateChildViewWithBaseTableIndex(boolean localIndex)
fail();
} catch (Exception ignore) {
}
+
+ PTable view = PhoenixRuntime.getTable(conn, fullViewName);
Review Comment:
why is this change required?
##########
phoenix-core-client/src/main/java/org/apache/phoenix/util/ValidateLastDDLTimestampUtil.java:
##########
@@ -172,6 +173,8 @@ public static void validateLastDDLTimestamp(PhoenixConnection conn,
requestBuilder.addLastDDLTimestampRequests(innerBuilder);
}
}
+
Review Comment:
nit: remove extra couple of lines.
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/MutableIndexFailureIT.java:
##########
@@ -176,6 +178,7 @@ protected static Map<String,String> getServerProps(){
* because we want to control it's execution ourselves
*/
serverProps.put(QueryServices.INDEX_REBUILD_TASK_INITIAL_DELAY, Long.toString(Long.MAX_VALUE));
+ serverProps.put(REGIONSERVER_COPROCESSOR_CONF_KEY, PhoenixRegionServerEndpoint.class.getName());
Review Comment:
Why are we setting `REGIONSERVER_COPROCESSOR_CONF_KEY` to `PhoenixRegionServerEndpoint` and not `PhoenixRegionServerEndpointTestImpl` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "shahrs87 (via GitHub)" <gi...@apache.org>.
shahrs87 commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1562878900
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/AlterTableWithViewsIT.java:
##########
@@ -1149,10 +1152,8 @@ public void testDroppingIndexedColDropsViewIndex() throws Exception {
byte[] viewIndexPhysicalTable = viewIndex.getPhysicalName().getBytes();
assertNotNull("Can't find view index", viewIndex);
assertEquals("Unexpected number of indexes ", 2, view.getIndexes().size());
- assertEquals("Unexpected index ", fullNameViewIndex1 , view.getIndexes().get(0).getName()
Review Comment:
Why is this change needed?
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/AlterTableWithViewsIT.java:
##########
@@ -1174,21 +1175,16 @@ public void testDroppingIndexedColDropsViewIndex() throws Exception {
}
// verify index metadata was dropped
- try {
- viewConn.createStatement().execute("SELECT * FROM " + fullNameViewIndex1 );
- fail("Index metadata should have been dropped");
- } catch (TableNotFoundException e) {
- }
-
pconn = viewConn.unwrap(PhoenixConnection.class);
- view = pconn.getTable(new PTableKey(tenantId, viewOfTable ));
+ view = pconn.getTableNoCache(viewOfTable);
+ assertEquals("Unexpected number of indexes ", 1, view.getIndexes().size());
+ assertEquals("Unexpected index ", fullNameViewIndex2 , view.getIndexes().get(0).getName().getString());
+ assertNotEquals("Dropped index should not be in view metadata ", fullNameViewIndex1 , view.getIndexes().get(0).getName().getString());
try {
- viewIndex = pconn.getTable(new PTableKey(tenantId, fullNameViewIndex1 ));
+ viewIndex = pconn.getTableNoCache(fullNameViewIndex1);
Review Comment:
why is this change required?
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/LogicalTableNameBaseIT.java:
##########
@@ -530,13 +531,16 @@ protected void validateIndex(Connection connection, String tableName, boolean is
}
public static void renameAndDropPhysicalTable(Connection conn, String tenantId, String schema, String tableName, String physicalName, boolean isNamespaceEnabled) throws Exception {
+ // if client is validating last_ddl_timestamp, this change in physical table name should be visible to the client
Review Comment:
Do we need this change given that default UCF is set to ALWAYS?
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/AlterTableWithViewsIT.java:
##########
@@ -1174,21 +1175,16 @@ public void testDroppingIndexedColDropsViewIndex() throws Exception {
}
// verify index metadata was dropped
- try {
Review Comment:
why is this change required?
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewMetadataIT.java:
##########
@@ -1173,10 +1174,10 @@ private void helpTestQueryForViewOnTableThatHasIndex(Statement s1,
s1.execute("create index " + indexName + " ON " + tableName
+ " (col2)");
- try (ResultSet rs = s2.executeQuery("explain select /*+ INDEX("
+ try (ResultSet rs = s2.executeQuery("select /*+ INDEX("
Review Comment:
why this change?
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/IndexAsyncThresholdIT.java:
##########
@@ -162,9 +162,7 @@ public void testAsyncIndexCreation() throws Exception {
exception = (SQLException) e;
}
connection.commit();
- PTableKey key = new PTableKey(null, this.tableName);
- PMetaData metaCache = connection.unwrap(PhoenixConnection.class).getMetaDataCache();
- List<PTable> indexes = metaCache.getTableRef(key).getTable().getIndexes();
+ List<PTable> indexes = connection.unwrap(PhoenixConnection.class).getTable(this.tableName).getIndexes();
Review Comment:
why this change?
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/LogicalTableNameExtendedIT.java:
##########
@@ -209,21 +209,26 @@ public void testHint() throws Exception {
createIndexOnTable(conn, tableName, indexName2);
populateTable(conn, tableName, 1, 2);
- // Test hint
String tableSelect = "SELECT V1,V2,V3 FROM " + tableName;
ResultSet rs = conn.createStatement().executeQuery("EXPLAIN " + tableSelect);
- assertEquals(true, QueryUtil.getExplainPlan(rs).contains(indexName));
+ String plan = QueryUtil.getExplainPlan(rs);
Review Comment:
why is this change required?
##########
phoenix-core/src/test/java/org/apache/phoenix/jdbc/ParallelPhoenixConnectionFailureTest.java:
##########
@@ -51,9 +52,12 @@ public class ParallelPhoenixConnectionFailureTest extends BaseTest {
@Test
public void testExecuteQueryChainFailure() throws SQLException {
HBaseTestingUtility hbaseTestingUtility = new HBaseTestingUtility();
-
- PhoenixConnection conn1 = (PhoenixConnection) DriverManager.getConnection(url);
- PhoenixConnection conn2 = (PhoenixConnection) DriverManager.getConnection(url);
+ Properties props = new Properties();
+ // disable last_ddl_timestamp validation for tests which use ConnectionLessQueryService
+ // since there is no hbase connection to get list of live region servers
+ props.put(QueryServices.LAST_DDL_TIMESTAMP_VALIDATION_ENABLED, Boolean.toString(false));
Review Comment:
The default is false. Do we need to set it to false again?
##########
phoenix-core/src/it/java/org/apache/phoenix/query/MetaDataCachingIT.java:
##########
@@ -142,18 +146,25 @@ public void testSystemTablesAreInCache() throws Exception {
*/
@Test
public void testGlobalClientCacheMetrics() throws Exception {
- int numThreads = 5;
+ int numThreads = 1;
Review Comment:
why this change?
##########
phoenix-core/src/test/java/org/apache/phoenix/query/BaseConnectionlessQueryTest.java:
##########
@@ -86,7 +88,11 @@ private static void startServer(String url) throws Exception {
// only load the test driver if we are testing locally - for integration tests, we want to
// test on a wider scale
if (PhoenixEmbeddedDriver.isTestUrl(url)) {
- driver = initDriver(ReadOnlyProps.EMPTY_PROPS);
+ Map<String, String> props = Maps.newHashMapWithExpectedSize(1);
+ // disable last_ddl_timestamp validation for tests which use ConnectionLessQueryService
+ // since there is no hbase connection to get list of live region servers
+ props.put(QueryServices.LAST_DDL_TIMESTAMP_VALIDATION_ENABLED, Boolean.toString(false));
Review Comment:
The default is false. Do we need to set it to false again?
##########
phoenix-core/src/it/java/org/apache/phoenix/monitoring/PhoenixTableLevelMetricsIT.java:
##########
@@ -1276,14 +1281,24 @@ public void testMetricsWithIndexUsage() throws Exception {
assertTrue(metricExists);
metricExists = false;
//assert BaseTable is not being queried
- for (PhoenixTableMetric metric : getPhoenixTableClientMetrics().get(dataTable)) {
- if (metric.getMetricType().equals(SELECT_SQL_COUNTER)) {
- metricExists = true;
- assertMetricValue(metric, SELECT_SQL_COUNTER, 0, CompareOp.EQ);
- break;
+ //if client is validating last_ddl_timestamps with ucf=never,
+ //there will be no metrics for base table (like getTable RPC times/counts).
Review Comment:
Can you explain more on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1562954852
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/AlterTableWithViewsIT.java:
##########
@@ -1174,21 +1175,16 @@ public void testDroppingIndexedColDropsViewIndex() throws Exception {
}
// verify index metadata was dropped
- try {
- viewConn.createStatement().execute("SELECT * FROM " + fullNameViewIndex1 );
- fail("Index metadata should have been dropped");
- } catch (TableNotFoundException e) {
- }
-
pconn = viewConn.unwrap(PhoenixConnection.class);
- view = pconn.getTable(new PTableKey(tenantId, viewOfTable ));
+ view = pconn.getTableNoCache(viewOfTable);
+ assertEquals("Unexpected number of indexes ", 1, view.getIndexes().size());
+ assertEquals("Unexpected index ", fullNameViewIndex2 , view.getIndexes().get(0).getName().getString());
+ assertNotEquals("Dropped index should not be in view metadata ", fullNameViewIndex1 , view.getIndexes().get(0).getName().getString());
try {
- viewIndex = pconn.getTable(new PTableKey(tenantId, fullNameViewIndex1 ));
+ viewIndex = pconn.getTableNoCache(fullNameViewIndex1);
Review Comment:
This is to make sure that the view index is fetched from the server and not from the client's cache. The test is verifying whether the view index was dropped on the server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1569809009
##########
phoenix-core/src/it/java/org/apache/phoenix/query/MetaDataCachingIT.java:
##########
@@ -152,12 +156,19 @@ public void testGlobalClientCacheMetrics() throws Exception {
simulateWorkload("testGlobalClientCacheMetrics", numTables, numThreads, numMaxDML);
// only 1 miss when the table is created
+ int numExpectedMisses = 1;
+ if (QueryServicesOptions.DEFAULT_LAST_DDL_TIMESTAMP_VALIDATION_ENABLED) {
Review Comment:
@shahrs87 I have made the change in MetaDataClient.java as that is production code. For the test code, maybe it is not required? Also in some tests, we do not have a connection object which can be used directly for `getValidateLastDdlTimestampEnabled`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "shahrs87 (via GitHub)" <gi...@apache.org>.
shahrs87 commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1569452771
##########
phoenix-core/src/it/java/org/apache/phoenix/rpc/UpdateCacheConnectionLevelPropIT.java:
##########
@@ -61,6 +62,9 @@ public class UpdateCacheConnectionLevelPropIT extends ParallelStatsDisabledIT {
private static Connection conn1;
private static Connection conn2;
private static ConnectionQueryServices spyForConn2;
+ private boolean isLastDDLTimestampValidationEnabled
Review Comment:
Why not to get the last ddl timestamp validation enabled flag from the connection properties?
##########
phoenix-core/src/it/java/org/apache/phoenix/rpc/UpdateCacheIT.java:
##########
@@ -69,6 +70,10 @@
@Category(ParallelStatsDisabledTest.class)
public class UpdateCacheIT extends ParallelStatsDisabledIT {
+ private boolean isLastDDLTimestampValidationEnabled
+ = QueryServicesOptions.DEFAULT_LAST_DDL_TIMESTAMP_VALIDATION_ENABLED
Review Comment:
Why not to get the last ddl timestamp validation enabled flag from the connection properties?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1569663494
##########
phoenix-core/src/it/java/org/apache/phoenix/query/MetaDataCachingIT.java:
##########
@@ -152,12 +156,19 @@ public void testGlobalClientCacheMetrics() throws Exception {
simulateWorkload("testGlobalClientCacheMetrics", numTables, numThreads, numMaxDML);
// only 1 miss when the table is created
+ int numExpectedMisses = 1;
+ if (QueryServicesOptions.DEFAULT_LAST_DDL_TIMESTAMP_VALIDATION_ENABLED) {
Review Comment:
Not sure if there is a connection here in the test to use?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1569324361
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/AlterTableWithViewsIT.java:
##########
@@ -1149,10 +1152,8 @@ public void testDroppingIndexedColDropsViewIndex() throws Exception {
byte[] viewIndexPhysicalTable = viewIndex.getPhysicalName().getBytes();
assertNotNull("Can't find view index", viewIndex);
assertEquals("Unexpected number of indexes ", 2, view.getIndexes().size());
- assertEquals("Unexpected index ", fullNameViewIndex1 , view.getIndexes().get(0).getName()
Review Comment:
Reverted
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewMetadataIT.java:
##########
@@ -1173,10 +1174,10 @@ private void helpTestQueryForViewOnTableThatHasIndex(Statement s1,
s1.execute("create index " + indexName + " ON " + tableName
+ " (col2)");
- try (ResultSet rs = s2.executeQuery("explain select /*+ INDEX("
+ try (ResultSet rs = s2.executeQuery("select /*+ INDEX("
Review Comment:
reverted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "shahrs87 (via GitHub)" <gi...@apache.org>.
shahrs87 commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1571130674
##########
phoenix-core-client/src/main/java/org/apache/phoenix/compile/FromCompiler.java:
##########
@@ -297,7 +297,9 @@ public static ColumnResolver getIndexResolver(SingleTableStatement statement,
public static ColumnResolver getResolver(SingleTableStatement statement, PhoenixConnection connection, Map<String, UDFParseNode> udfParseNodes)
throws SQLException {
- SingleTableColumnResolver visitor = new SingleTableColumnResolver(connection, statement.getTable(), true, 0, udfParseNodes);
+ SingleTableColumnResolver visitor
Review Comment:
I see that we are calling this resolver from `MetaDataClient#createIndex `and `CreateIndexCompiler#compile` method. So we want to bypass cache for createIndex and resolve table always. So lets rename this method to `getResolverForCreateIndex` and add a comment on why we are setting `alwaysHitServer` to true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1569663907
##########
phoenix-core/src/it/java/org/apache/phoenix/rpc/UpdateCacheConnectionLevelPropIT.java:
##########
@@ -61,6 +62,9 @@ public class UpdateCacheConnectionLevelPropIT extends ParallelStatsDisabledIT {
private static Connection conn1;
private static Connection conn2;
private static ConnectionQueryServices spyForConn2;
+ private boolean isLastDDLTimestampValidationEnabled
Review Comment:
Not sure if there is a connection here in the test to use?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "shahrs87 (via GitHub)" <gi...@apache.org>.
shahrs87 commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1569452071
##########
phoenix-core/src/it/java/org/apache/phoenix/query/MetaDataCachingIT.java:
##########
@@ -152,12 +156,19 @@ public void testGlobalClientCacheMetrics() throws Exception {
simulateWorkload("testGlobalClientCacheMetrics", numTables, numThreads, numMaxDML);
// only 1 miss when the table is created
+ int numExpectedMisses = 1;
+ if (QueryServicesOptions.DEFAULT_LAST_DDL_TIMESTAMP_VALIDATION_ENABLED) {
Review Comment:
Why not to get the last ddl timestamp validation enabled flag from the connection properties?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1561756250
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpgradeIT.java:
##########
@@ -731,13 +730,14 @@ public void testLastDDLTimestampBootstrap() throws Exception {
}
private void nullDDLTimestamps(Connection conn) throws SQLException {
+ //ignore system tables since that can interfere with other tests.
String pkCols = TENANT_ID + ", " + TABLE_SCHEM +
", " + TABLE_NAME + ", " + COLUMN_NAME + ", " + COLUMN_FAMILY;
String upsertSql =
"UPSERT INTO " + SYSTEM_CATALOG_NAME + " (" + pkCols + ", " +
LAST_DDL_TIMESTAMP + ")" + " " +
"SELECT " + pkCols + ", NULL FROM " + SYSTEM_CATALOG_NAME + " " +
- "WHERE " + TABLE_TYPE + " IS NOT NULL";
+ "WHERE " + TABLE_TYPE + " " + " != '" + PTableType.SYSTEM.getSerializedValue() + "'";
Review Comment:
Tests in this class passed when run individually but fail when the whole test class is run. One test `testLastDDLTimestampBootstrap()` in the class was interfering with the other tests. That test nulls out all LAST_DDL_TSs for test purpose and did not ignore system tables.
The change is in the helper method `nullDDLTimestamps` which nulls out all tables to ignore SYSTEM tables.
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpgradeIT.java:
##########
@@ -731,13 +730,14 @@ public void testLastDDLTimestampBootstrap() throws Exception {
}
private void nullDDLTimestamps(Connection conn) throws SQLException {
+ //ignore system tables since that can interfere with other tests.
String pkCols = TENANT_ID + ", " + TABLE_SCHEM +
", " + TABLE_NAME + ", " + COLUMN_NAME + ", " + COLUMN_FAMILY;
String upsertSql =
"UPSERT INTO " + SYSTEM_CATALOG_NAME + " (" + pkCols + ", " +
LAST_DDL_TIMESTAMP + ")" + " " +
"SELECT " + pkCols + ", NULL FROM " + SYSTEM_CATALOG_NAME + " " +
- "WHERE " + TABLE_TYPE + " IS NOT NULL";
+ "WHERE " + TABLE_TYPE + " " + " != '" + PTableType.SYSTEM.getSerializedValue() + "'";
Review Comment:
Tests in this class passed when run individually but failed when the whole test class is run. One test `testLastDDLTimestampBootstrap()` in the class was interfering with the other tests. That test nulls out all LAST_DDL_TSs for test purpose and did not ignore system tables.
The change is in the helper method `nullDDLTimestamps` which nulls out all tables to ignore SYSTEM tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1561768305
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewIT.java:
##########
@@ -660,6 +661,14 @@ public void testCreateChildViewWithBaseTableIndex(boolean localIndex)
fail();
} catch (Exception ignore) {
}
+
+ PTable view = PhoenixRuntime.getTable(conn, fullViewName);
Review Comment:
I can not seem to remember why I added these lines, those PTable variables are not even accessed later. I will remove this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1563254443
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/UpdateCacheAcrossDifferentClientsIT.java:
##########
@@ -104,17 +105,19 @@ public void testUpdateCacheFrequencyWithAddAndDropTable() throws Exception {
} catch (TableNotFoundException e) {
//Expected
}
- rs = conn2.createStatement().executeQuery("select * from "+tableName);
try {
+ rs = conn2.createStatement().executeQuery("select * from "+tableName);
rs.next();
fail("Should throw org.apache.hadoop.hbase.TableNotFoundException since the latest metadata " +
"wasn't fetched");
- } catch (PhoenixIOException ex) {
Review Comment:
Need to look into this one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "palashc (via GitHub)" <gi...@apache.org>.
palashc commented on code in PR #1859:
URL: https://github.com/apache/phoenix/pull/1859#discussion_r1563344594
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/AlterTableWithViewsIT.java:
##########
@@ -1149,10 +1152,8 @@ public void testDroppingIndexedColDropsViewIndex() throws Exception {
byte[] viewIndexPhysicalTable = viewIndex.getPhysicalName().getBytes();
assertNotNull("Can't find view index", viewIndex);
assertEquals("Unexpected number of indexes ", 2, view.getIndexes().size());
- assertEquals("Unexpected index ", fullNameViewIndex1 , view.getIndexes().get(0).getName()
Review Comment:
Will make that design change as part of https://issues.apache.org/jira/browse/PHOENIX-7305 and revert this part of the test change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7166 : Enable feature flags introduced by metadata caching redesign [phoenix]
Posted by "shahrs87 (via GitHub)" <gi...@apache.org>.
shahrs87 merged PR #1859:
URL: https://github.com/apache/phoenix/pull/1859
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org