You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@phoenix.apache.org by "haridsv (via GitHub)" <gi...@apache.org> on 2023/12/14 13:56:17 UTC
[PR] PHOENIX-7014: Query compiler/optimizer changes along with some PHOENIX-7015 changes [phoenix]
haridsv opened a new pull request, #1766:
URL: https://github.com/apache/phoenix/pull/1766
This PR replaces #1738 to include some experimental scanner changes for PHOENIX-7015.
This has the changes needed for PHOENIX-7014. It short-circuits the existing optimizer logic to produce a plan that is suitable for CDC queries. It also reimplements the query hints so that it is simpler and easier to maintain using regex. It includes a PoC query test and some debug test cases for short term reference (can be removed before merging to master). Currently `select *` works but if you specify column names, you get an `IndexOutOfBoundsException` that I will look into later.
This PR also goes one step further to generating a mock JSON string (PoC in #1738) to be able to generate results from the actual data (though, not per the expected format). I temporary added Gson as a dependency to generate the result, but this will be replaced with BSON once the JSON support is available. Here is a quick demo of how this works:
```
0: jdbc:phoenix:localhost> create table t (k INTEGER PRIMARY KEY, v1 INTEGER);
No rows affected (0.651 seconds)
0: jdbc:phoenix:localhost> upsert into t (k, v1) VALUES (1, 100);
1 row affected (0.067 seconds)
0: jdbc:phoenix:localhost> upsert into t (k, v1) VALUES (2, 200);
1 row affected (0.067 seconds)
0: jdbc:phoenix:localhost> upsert into t (k, v1) VALUES (1, 101);
1 row affected (0.067 seconds)
0: jdbc:phoenix:localhost> create cdc tchanges on t(PHOENIX_ROW_TIMESTAMP());
2 rows affected (5.686 seconds)
0: jdbc:phoenix:localhost> select * from tchanges;
+--------------------------+---+------------+
| PHOENIX_ROW_TIMESTAMP() | K | CDC JSON |
+--------------------------+---+------------+
| 2023-12-06 | 1 | {"V1":100} |
| 2023-12-06 | 2 | {"V1":200} |
| 2023-12-06 | 1 | {"V1":101} |
+--------------------------+---+------------+
3 rows selected (0.022 seconds
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7014: Query compiler/optimizer changes along with some PHOENIX-7015 changes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #1766:
URL: https://github.com/apache/phoenix/pull/1766#discussion_r1434375182
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/CDCMiscIT.java:
##########
@@ -269,4 +273,109 @@ public void testDropCDCIndex () throws SQLException {
}
}
+ private void assertResultSet(ResultSet rs) throws Exception{
+ Gson gson = new Gson();
+ assertEquals(true, rs.next());
+ assertEquals(1, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 100d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(true, rs.next());
+ assertEquals(2, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 200d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(true, rs.next());
+ assertEquals(1, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 101d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(false, rs.next());
+ rs.close();
+ }
+
+ @Test
+ public void testSelectCDC() throws Exception {
+ Properties props = new Properties();
+ props.put(QueryServices.TASK_HANDLING_INTERVAL_MS_ATTRIB, Long.toString(Long.MAX_VALUE));
+ props.put("hbase.client.scanner.timeout.period", "6000000");
+ props.put("phoenix.query.timeoutMs", "6000000");
+ props.put("zookeeper.session.timeout", "6000000");
+ props.put("hbase.rpc.timeout", "6000000");
+ Connection conn = DriverManager.getConnection(getUrl(), props);
+ String tableName = generateUniqueName();
+ conn.createStatement().execute(
+ "CREATE TABLE " + tableName + " ( k INTEGER PRIMARY KEY," + " v1 INTEGER)");
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (1, 100)");
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (2, 200)");
+ conn.commit();
+ Thread.sleep(1000);
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (1, 101)");
+ conn.commit();
+ String cdcName = generateUniqueName();
+ String cdc_sql = "CREATE CDC " + cdcName
Review Comment:
> If we do this, won't we also need to project the index status via CDC object
I think that can be later improvement but supporting creation of async index is necessary because creating sync index for large table will anyways be too expensive or not possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7014: Query compiler/optimizer changes along with some PHOENIX-7015 changes [phoenix]
Posted by "haridsv (via GitHub)" <gi...@apache.org>.
haridsv commented on code in PR #1766:
URL: https://github.com/apache/phoenix/pull/1766#discussion_r1433516465
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/CDCMiscIT.java:
##########
@@ -269,4 +273,109 @@ public void testDropCDCIndex () throws SQLException {
}
}
+ private void assertResultSet(ResultSet rs) throws Exception{
+ Gson gson = new Gson();
+ assertEquals(true, rs.next());
+ assertEquals(1, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 100d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(true, rs.next());
+ assertEquals(2, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 200d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(true, rs.next());
+ assertEquals(1, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 101d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(false, rs.next());
+ rs.close();
+ }
+
+ @Test
+ public void testSelectCDC() throws Exception {
+ Properties props = new Properties();
+ props.put(QueryServices.TASK_HANDLING_INTERVAL_MS_ATTRIB, Long.toString(Long.MAX_VALUE));
+ props.put("hbase.client.scanner.timeout.period", "6000000");
+ props.put("phoenix.query.timeoutMs", "6000000");
+ props.put("zookeeper.session.timeout", "6000000");
+ props.put("hbase.rpc.timeout", "6000000");
+ Connection conn = DriverManager.getConnection(getUrl(), props);
+ String tableName = generateUniqueName();
+ conn.createStatement().execute(
+ "CREATE TABLE " + tableName + " ( k INTEGER PRIMARY KEY," + " v1 INTEGER)");
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (1, 100)");
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (2, 200)");
+ conn.commit();
+ Thread.sleep(1000);
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (1, 101)");
+ conn.commit();
+ String cdcName = generateUniqueName();
+ String cdc_sql = "CREATE CDC " + cdcName
Review Comment:
> Everytime we run CREATE CDC do we need to pass PHOENIX_ROW_TIMESTAMP ? If that is something mandatory we shouldn't make it part of the interface and the system should do it implicitly under the hood.
The feature allows any timestamp like column to be used instead of the PHOENIX_ROW_TIMESTMAP that is why this flexibility is being allowed, however I am not 100% sure about the use cases, I will have an offline discussion on this, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7014: Query compiler/optimizer changes along with some PHOENIX-7015 changes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #1766:
URL: https://github.com/apache/phoenix/pull/1766#issuecomment-1868326096
With follow-ups to come in separate PRs, merging this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7014: Query compiler/optimizer changes along with some PHOENIX-7015 changes [phoenix]
Posted by "tkhurana (via GitHub)" <gi...@apache.org>.
tkhurana commented on code in PR #1766:
URL: https://github.com/apache/phoenix/pull/1766#discussion_r1434502531
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/CDCMiscIT.java:
##########
@@ -269,4 +273,109 @@ public void testDropCDCIndex () throws SQLException {
}
}
+ private void assertResultSet(ResultSet rs) throws Exception{
+ Gson gson = new Gson();
+ assertEquals(true, rs.next());
+ assertEquals(1, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 100d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(true, rs.next());
+ assertEquals(2, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 200d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(true, rs.next());
+ assertEquals(1, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 101d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(false, rs.next());
+ rs.close();
+ }
+
+ @Test
+ public void testSelectCDC() throws Exception {
+ Properties props = new Properties();
+ props.put(QueryServices.TASK_HANDLING_INTERVAL_MS_ATTRIB, Long.toString(Long.MAX_VALUE));
+ props.put("hbase.client.scanner.timeout.period", "6000000");
+ props.put("phoenix.query.timeoutMs", "6000000");
+ props.put("zookeeper.session.timeout", "6000000");
+ props.put("hbase.rpc.timeout", "6000000");
+ Connection conn = DriverManager.getConnection(getUrl(), props);
+ String tableName = generateUniqueName();
+ conn.createStatement().execute(
+ "CREATE TABLE " + tableName + " ( k INTEGER PRIMARY KEY," + " v1 INTEGER)");
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (1, 100)");
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (2, 200)");
+ conn.commit();
+ Thread.sleep(1000);
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (1, 101)");
+ conn.commit();
+ String cdcName = generateUniqueName();
+ String cdc_sql = "CREATE CDC " + cdcName
Review Comment:
> If we do this, won't we also need to project the index status via CDC object
@haridsv I didn't quite understand what you mean by index status here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7014: Query compiler/optimizer changes along with some PHOENIX-7015 changes [phoenix]
Posted by "tkhurana (via GitHub)" <gi...@apache.org>.
tkhurana commented on code in PR #1766:
URL: https://github.com/apache/phoenix/pull/1766#discussion_r1431784177
##########
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/CDCGlobalIndexRegionScanner.java:
##########
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import com.google.gson.Gson;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellBuilder;
+import org.apache.hadoop.hbase.CellBuilderFactory;
+import org.apache.hadoop.hbase.CellBuilderType;
+import org.apache.hadoop.hbase.CellComparator;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.regionserver.RegionScanner;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.execute.TupleProjector;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.tuple.ResultTuple;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EncodedColumnsUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.ScanUtil;
+import org.apache.phoenix.util.ServerUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+import static org.apache.phoenix.coprocessor.BaseScannerRegionObserver.CDC_JSON_COL_QUALIFIER;
+import static org.apache.phoenix.coprocessor.BaseScannerRegionObserver.DATA_COL_QUALIFIER_TO_NAME_MAP;
+import static org.apache.phoenix.coprocessor.BaseScannerRegionObserver.DATA_COL_QUALIFIER_TO_TYPE_MAP;
+
+public class CDCGlobalIndexRegionScanner extends UncoveredGlobalIndexRegionScanner {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(CDCGlobalIndexRegionScanner.class);
+
+ private Map<ImmutableBytesPtr, String> dataColQualNameMap;
+ private Map<ImmutableBytesPtr, PDataType> dataColQualTypeMap;
+ // Map<dataRowKey: Map<TS: Map<qualifier: Cell>>>
+ private Map<ImmutableBytesPtr, Map<Long, Map<ImmutableBytesPtr, Cell>>> dataRowChanges =
Review Comment:
This is pretty complicated. Maybe you could introduce some abstraction which represents a change as a object. Then you could have a collection of changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7014: Query compiler/optimizer changes along with some PHOENIX-7015 changes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani merged PR #1766:
URL: https://github.com/apache/phoenix/pull/1766
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7014: Query compiler/optimizer changes along with some PHOENIX-7015 changes [phoenix]
Posted by "haridsv (via GitHub)" <gi...@apache.org>.
haridsv commented on code in PR #1766:
URL: https://github.com/apache/phoenix/pull/1766#discussion_r1432258142
##########
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/CDCGlobalIndexRegionScanner.java:
##########
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.coprocessor;
+
+import com.google.gson.Gson;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellBuilder;
+import org.apache.hadoop.hbase.CellBuilderFactory;
+import org.apache.hadoop.hbase.CellBuilderType;
+import org.apache.hadoop.hbase.CellComparator;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.regionserver.Region;
+import org.apache.hadoop.hbase.regionserver.RegionScanner;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.phoenix.execute.TupleProjector;
+import org.apache.phoenix.hbase.index.util.ImmutableBytesPtr;
+import org.apache.phoenix.index.IndexMaintainer;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.tuple.ResultTuple;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.CDCUtil;
+import org.apache.phoenix.util.EncodedColumnsUtil;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.ScanUtil;
+import org.apache.phoenix.util.ServerUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+import static org.apache.phoenix.coprocessor.BaseScannerRegionObserver.CDC_JSON_COL_QUALIFIER;
+import static org.apache.phoenix.coprocessor.BaseScannerRegionObserver.DATA_COL_QUALIFIER_TO_NAME_MAP;
+import static org.apache.phoenix.coprocessor.BaseScannerRegionObserver.DATA_COL_QUALIFIER_TO_TYPE_MAP;
+
+public class CDCGlobalIndexRegionScanner extends UncoveredGlobalIndexRegionScanner {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(CDCGlobalIndexRegionScanner.class);
+
+ private Map<ImmutableBytesPtr, String> dataColQualNameMap;
+ private Map<ImmutableBytesPtr, PDataType> dataColQualTypeMap;
+ // Map<dataRowKey: Map<TS: Map<qualifier: Cell>>>
+ private Map<ImmutableBytesPtr, Map<Long, Map<ImmutableBytesPtr, Cell>>> dataRowChanges =
Review Comment:
This was meant to be a quick PoC sort of code and should go away as @TheNamesRai is already reworking the logic to build the row without having to use this intermediate data structure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7014: Query compiler/optimizer changes along with some PHOENIX-7015 changes [phoenix]
Posted by "tkhurana (via GitHub)" <gi...@apache.org>.
tkhurana commented on code in PR #1766:
URL: https://github.com/apache/phoenix/pull/1766#discussion_r1433131522
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/CDCMiscIT.java:
##########
@@ -269,4 +273,109 @@ public void testDropCDCIndex () throws SQLException {
}
}
+ private void assertResultSet(ResultSet rs) throws Exception{
+ Gson gson = new Gson();
+ assertEquals(true, rs.next());
+ assertEquals(1, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 100d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(true, rs.next());
+ assertEquals(2, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 200d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(true, rs.next());
+ assertEquals(1, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 101d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(false, rs.next());
+ rs.close();
+ }
+
+ @Test
+ public void testSelectCDC() throws Exception {
+ Properties props = new Properties();
+ props.put(QueryServices.TASK_HANDLING_INTERVAL_MS_ATTRIB, Long.toString(Long.MAX_VALUE));
+ props.put("hbase.client.scanner.timeout.period", "6000000");
+ props.put("phoenix.query.timeoutMs", "6000000");
+ props.put("zookeeper.session.timeout", "6000000");
+ props.put("hbase.rpc.timeout", "6000000");
+ Connection conn = DriverManager.getConnection(getUrl(), props);
+ String tableName = generateUniqueName();
+ conn.createStatement().execute(
+ "CREATE TABLE " + tableName + " ( k INTEGER PRIMARY KEY," + " v1 INTEGER)");
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (1, 100)");
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (2, 200)");
+ conn.commit();
+ Thread.sleep(1000);
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (1, 101)");
+ conn.commit();
+ String cdcName = generateUniqueName();
+ String cdc_sql = "CREATE CDC " + cdcName
Review Comment:
Everytime we run `CREATE CDC` do we need to pass `PHOENIX_ROW_TIMESTAMP` ? If that is something mandatory we shouldn't make it part of the interface and the system should do it implicitly under the hood.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7014: Query compiler/optimizer changes along with some PHOENIX-7015 changes [phoenix]
Posted by "haridsv (via GitHub)" <gi...@apache.org>.
haridsv commented on code in PR #1766:
URL: https://github.com/apache/phoenix/pull/1766#discussion_r1433518168
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/CDCMiscIT.java:
##########
@@ -269,4 +273,109 @@ public void testDropCDCIndex () throws SQLException {
}
}
+ private void assertResultSet(ResultSet rs) throws Exception{
+ Gson gson = new Gson();
+ assertEquals(true, rs.next());
+ assertEquals(1, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 100d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(true, rs.next());
+ assertEquals(2, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 200d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(true, rs.next());
+ assertEquals(1, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 101d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(false, rs.next());
+ rs.close();
+ }
+
+ @Test
+ public void testSelectCDC() throws Exception {
+ Properties props = new Properties();
+ props.put(QueryServices.TASK_HANDLING_INTERVAL_MS_ATTRIB, Long.toString(Long.MAX_VALUE));
+ props.put("hbase.client.scanner.timeout.period", "6000000");
+ props.put("phoenix.query.timeoutMs", "6000000");
+ props.put("zookeeper.session.timeout", "6000000");
+ props.put("hbase.rpc.timeout", "6000000");
+ Connection conn = DriverManager.getConnection(getUrl(), props);
+ String tableName = generateUniqueName();
+ conn.createStatement().execute(
+ "CREATE TABLE " + tableName + " ( k INTEGER PRIMARY KEY," + " v1 INTEGER)");
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (1, 100)");
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (2, 200)");
+ conn.commit();
+ Thread.sleep(1000);
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (1, 101)");
+ conn.commit();
+ String cdcName = generateUniqueName();
+ String cdc_sql = "CREATE CDC " + cdcName
Review Comment:
> For bigger tables the index needs to be created async and then built explicitly using IndexTool.
If we do this, won't we also need to project the index status via CDC object, since we intend to keep the index hidden as an implementation detail? Do you think we should have an offline mode for CREATE CDC itself?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7014: Query compiler/optimizer changes along with some PHOENIX-7015 changes [phoenix]
Posted by "haridsv (via GitHub)" <gi...@apache.org>.
haridsv commented on PR #1766:
URL: https://github.com/apache/phoenix/pull/1766#issuecomment-1897824167
PR #1794 is the follow up that implements the query functionality to the spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7014: Query compiler/optimizer changes along with some PHOENIX-7015 changes [phoenix]
Posted by "haridsv (via GitHub)" <gi...@apache.org>.
haridsv commented on code in PR #1766:
URL: https://github.com/apache/phoenix/pull/1766#discussion_r1435014971
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/CDCMiscIT.java:
##########
@@ -269,4 +273,109 @@ public void testDropCDCIndex () throws SQLException {
}
}
+ private void assertResultSet(ResultSet rs) throws Exception{
+ Gson gson = new Gson();
+ assertEquals(true, rs.next());
+ assertEquals(1, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 100d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(true, rs.next());
+ assertEquals(2, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 200d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(true, rs.next());
+ assertEquals(1, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 101d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(false, rs.next());
+ rs.close();
+ }
+
+ @Test
+ public void testSelectCDC() throws Exception {
+ Properties props = new Properties();
+ props.put(QueryServices.TASK_HANDLING_INTERVAL_MS_ATTRIB, Long.toString(Long.MAX_VALUE));
+ props.put("hbase.client.scanner.timeout.period", "6000000");
+ props.put("phoenix.query.timeoutMs", "6000000");
+ props.put("zookeeper.session.timeout", "6000000");
+ props.put("hbase.rpc.timeout", "6000000");
+ Connection conn = DriverManager.getConnection(getUrl(), props);
+ String tableName = generateUniqueName();
+ conn.createStatement().execute(
+ "CREATE TABLE " + tableName + " ( k INTEGER PRIMARY KEY," + " v1 INTEGER)");
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (1, 100)");
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (2, 200)");
+ conn.commit();
+ Thread.sleep(1000);
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (1, 101)");
+ conn.commit();
+ String cdcName = generateUniqueName();
+ String cdc_sql = "CREATE CDC " + cdcName
Review Comment:
> I didn't quite understand what you mean by index status here ?
I was referring to the async index creation lifecycle, which should be attributed to the CDC so that user can query and know when it is done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7014: Query compiler/optimizer changes along with some PHOENIX-7015 changes [phoenix]
Posted by "tkhurana (via GitHub)" <gi...@apache.org>.
tkhurana commented on code in PR #1766:
URL: https://github.com/apache/phoenix/pull/1766#discussion_r1435279079
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/CDCMiscIT.java:
##########
@@ -269,4 +273,109 @@ public void testDropCDCIndex () throws SQLException {
}
}
+ private void assertResultSet(ResultSet rs) throws Exception{
+ Gson gson = new Gson();
+ assertEquals(true, rs.next());
+ assertEquals(1, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 100d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(true, rs.next());
+ assertEquals(2, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 200d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(true, rs.next());
+ assertEquals(1, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 101d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(false, rs.next());
+ rs.close();
+ }
+
+ @Test
+ public void testSelectCDC() throws Exception {
+ Properties props = new Properties();
+ props.put(QueryServices.TASK_HANDLING_INTERVAL_MS_ATTRIB, Long.toString(Long.MAX_VALUE));
+ props.put("hbase.client.scanner.timeout.period", "6000000");
+ props.put("phoenix.query.timeoutMs", "6000000");
+ props.put("zookeeper.session.timeout", "6000000");
+ props.put("hbase.rpc.timeout", "6000000");
+ Connection conn = DriverManager.getConnection(getUrl(), props);
+ String tableName = generateUniqueName();
+ conn.createStatement().execute(
+ "CREATE TABLE " + tableName + " ( k INTEGER PRIMARY KEY," + " v1 INTEGER)");
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (1, 100)");
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (2, 200)");
+ conn.commit();
+ Thread.sleep(1000);
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (1, 101)");
+ conn.commit();
+ String cdcName = generateUniqueName();
+ String cdc_sql = "CREATE CDC " + cdcName
Review Comment:
OK, we can tackle that in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7014: Query compiler/optimizer changes along with some PHOENIX-7015 changes [phoenix]
Posted by "haridsv (via GitHub)" <gi...@apache.org>.
haridsv commented on code in PR #1766:
URL: https://github.com/apache/phoenix/pull/1766#discussion_r1465296893
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/CDCMiscIT.java:
##########
@@ -269,4 +273,109 @@ public void testDropCDCIndex () throws SQLException {
}
}
+ private void assertResultSet(ResultSet rs) throws Exception{
+ Gson gson = new Gson();
+ assertEquals(true, rs.next());
+ assertEquals(1, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 100d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(true, rs.next());
+ assertEquals(2, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 200d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(true, rs.next());
+ assertEquals(1, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 101d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(false, rs.next());
+ rs.close();
+ }
+
+ @Test
+ public void testSelectCDC() throws Exception {
+ Properties props = new Properties();
+ props.put(QueryServices.TASK_HANDLING_INTERVAL_MS_ATTRIB, Long.toString(Long.MAX_VALUE));
+ props.put("hbase.client.scanner.timeout.period", "6000000");
+ props.put("phoenix.query.timeoutMs", "6000000");
+ props.put("zookeeper.session.timeout", "6000000");
+ props.put("hbase.rpc.timeout", "6000000");
+ Connection conn = DriverManager.getConnection(getUrl(), props);
+ String tableName = generateUniqueName();
+ conn.createStatement().execute(
+ "CREATE TABLE " + tableName + " ( k INTEGER PRIMARY KEY," + " v1 INTEGER)");
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (1, 100)");
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (2, 200)");
+ conn.commit();
+ Thread.sleep(1000);
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (1, 101)");
+ conn.commit();
+ String cdcName = generateUniqueName();
+ String cdc_sql = "CREATE CDC " + cdcName
Review Comment:
@tkhurana This is addressed as part of #1802.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7014: Query compiler/optimizer changes along with some PHOENIX-7015 changes [phoenix]
Posted by "haridsv (via GitHub)" <gi...@apache.org>.
haridsv commented on code in PR #1766:
URL: https://github.com/apache/phoenix/pull/1766#discussion_r1465296893
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/CDCMiscIT.java:
##########
@@ -269,4 +273,109 @@ public void testDropCDCIndex () throws SQLException {
}
}
+ private void assertResultSet(ResultSet rs) throws Exception{
+ Gson gson = new Gson();
+ assertEquals(true, rs.next());
+ assertEquals(1, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 100d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(true, rs.next());
+ assertEquals(2, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 200d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(true, rs.next());
+ assertEquals(1, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 101d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(false, rs.next());
+ rs.close();
+ }
+
+ @Test
+ public void testSelectCDC() throws Exception {
+ Properties props = new Properties();
+ props.put(QueryServices.TASK_HANDLING_INTERVAL_MS_ATTRIB, Long.toString(Long.MAX_VALUE));
+ props.put("hbase.client.scanner.timeout.period", "6000000");
+ props.put("phoenix.query.timeoutMs", "6000000");
+ props.put("zookeeper.session.timeout", "6000000");
+ props.put("hbase.rpc.timeout", "6000000");
+ Connection conn = DriverManager.getConnection(getUrl(), props);
+ String tableName = generateUniqueName();
+ conn.createStatement().execute(
+ "CREATE TABLE " + tableName + " ( k INTEGER PRIMARY KEY," + " v1 INTEGER)");
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (1, 100)");
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (2, 200)");
+ conn.commit();
+ Thread.sleep(1000);
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (1, 101)");
+ conn.commit();
+ String cdcName = generateUniqueName();
+ String cdc_sql = "CREATE CDC " + cdcName
Review Comment:
@tkhurana Async index creation is addressed as part of #1802.
Also removed the need to explicitly specify `PHOENIX_ROW_TIMESTAMP()` as we dropped the support for user specified timestamp column.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7014: Query compiler/optimizer changes along with some PHOENIX-7015 changes [phoenix]
Posted by "haridsv (via GitHub)" <gi...@apache.org>.
haridsv commented on code in PR #1766:
URL: https://github.com/apache/phoenix/pull/1766#discussion_r1435015447
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/CDCMiscIT.java:
##########
@@ -269,4 +273,109 @@ public void testDropCDCIndex () throws SQLException {
}
}
+ private void assertResultSet(ResultSet rs) throws Exception{
+ Gson gson = new Gson();
+ assertEquals(true, rs.next());
+ assertEquals(1, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 100d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(true, rs.next());
+ assertEquals(2, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 200d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(true, rs.next());
+ assertEquals(1, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 101d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(false, rs.next());
+ rs.close();
+ }
+
+ @Test
+ public void testSelectCDC() throws Exception {
+ Properties props = new Properties();
+ props.put(QueryServices.TASK_HANDLING_INTERVAL_MS_ATTRIB, Long.toString(Long.MAX_VALUE));
+ props.put("hbase.client.scanner.timeout.period", "6000000");
+ props.put("phoenix.query.timeoutMs", "6000000");
+ props.put("zookeeper.session.timeout", "6000000");
+ props.put("hbase.rpc.timeout", "6000000");
+ Connection conn = DriverManager.getConnection(getUrl(), props);
+ String tableName = generateUniqueName();
+ conn.createStatement().execute(
+ "CREATE TABLE " + tableName + " ( k INTEGER PRIMARY KEY," + " v1 INTEGER)");
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (1, 100)");
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (2, 200)");
+ conn.commit();
+ Thread.sleep(1000);
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (1, 101)");
+ conn.commit();
+ String cdcName = generateUniqueName();
+ String cdc_sql = "CREATE CDC " + cdcName
Review Comment:
@tkhurana In any case, I am not planning to address these as part of this PR, so are you OK to merge this as is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7014: Query compiler/optimizer changes along with some PHOENIX-7015 changes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #1766:
URL: https://github.com/apache/phoenix/pull/1766#discussion_r1434377696
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/CDCMiscIT.java:
##########
@@ -269,4 +273,109 @@ public void testDropCDCIndex () throws SQLException {
}
}
+ private void assertResultSet(ResultSet rs) throws Exception{
+ Gson gson = new Gson();
+ assertEquals(true, rs.next());
+ assertEquals(1, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 100d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(true, rs.next());
+ assertEquals(2, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 200d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(true, rs.next());
+ assertEquals(1, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 101d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(false, rs.next());
+ rs.close();
+ }
+
+ @Test
+ public void testSelectCDC() throws Exception {
+ Properties props = new Properties();
+ props.put(QueryServices.TASK_HANDLING_INTERVAL_MS_ATTRIB, Long.toString(Long.MAX_VALUE));
+ props.put("hbase.client.scanner.timeout.period", "6000000");
+ props.put("phoenix.query.timeoutMs", "6000000");
+ props.put("zookeeper.session.timeout", "6000000");
+ props.put("hbase.rpc.timeout", "6000000");
+ Connection conn = DriverManager.getConnection(getUrl(), props);
+ String tableName = generateUniqueName();
+ conn.createStatement().execute(
+ "CREATE TABLE " + tableName + " ( k INTEGER PRIMARY KEY," + " v1 INTEGER)");
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (1, 100)");
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (2, 200)");
+ conn.commit();
+ Thread.sleep(1000);
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (1, 101)");
+ conn.commit();
+ String cdcName = generateUniqueName();
+ String cdc_sql = "CREATE CDC " + cdcName
Review Comment:
Let's see if Tanuj has any other recommendations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7014: Query compiler/optimizer changes along with some PHOENIX-7015 changes [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #1766:
URL: https://github.com/apache/phoenix/pull/1766#issuecomment-1866735458
@tkhurana if you are good with the current state with any additional comments to be taken care of in subsequent PRs against feature branch (or final patch against master branch), we can merge this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-7014: Query compiler/optimizer changes along with some PHOENIX-7015 changes [phoenix]
Posted by "tkhurana (via GitHub)" <gi...@apache.org>.
tkhurana commented on code in PR #1766:
URL: https://github.com/apache/phoenix/pull/1766#discussion_r1433133100
##########
phoenix-core/src/it/java/org/apache/phoenix/end2end/CDCMiscIT.java:
##########
@@ -269,4 +273,109 @@ public void testDropCDCIndex () throws SQLException {
}
}
+ private void assertResultSet(ResultSet rs) throws Exception{
+ Gson gson = new Gson();
+ assertEquals(true, rs.next());
+ assertEquals(1, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 100d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(true, rs.next());
+ assertEquals(2, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 200d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(true, rs.next());
+ assertEquals(1, rs.getInt(2));
+ assertEquals(new HashMap(){{put("V1", 101d);}}, gson.fromJson(rs.getString(3),
+ HashMap.class));
+ assertEquals(false, rs.next());
+ rs.close();
+ }
+
+ @Test
+ public void testSelectCDC() throws Exception {
+ Properties props = new Properties();
+ props.put(QueryServices.TASK_HANDLING_INTERVAL_MS_ATTRIB, Long.toString(Long.MAX_VALUE));
+ props.put("hbase.client.scanner.timeout.period", "6000000");
+ props.put("phoenix.query.timeoutMs", "6000000");
+ props.put("zookeeper.session.timeout", "6000000");
+ props.put("hbase.rpc.timeout", "6000000");
+ Connection conn = DriverManager.getConnection(getUrl(), props);
+ String tableName = generateUniqueName();
+ conn.createStatement().execute(
+ "CREATE TABLE " + tableName + " ( k INTEGER PRIMARY KEY," + " v1 INTEGER)");
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (1, 100)");
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (2, 200)");
+ conn.commit();
+ Thread.sleep(1000);
+ conn.createStatement().execute("UPSERT INTO " + tableName + " (k, v1) VALUES (1, 101)");
+ conn.commit();
+ String cdcName = generateUniqueName();
+ String cdc_sql = "CREATE CDC " + cdcName
Review Comment:
Also it seems the uncovered index that is being created as part of CREATE CDC is being created synchronously. That will only work for small tables. For bigger tables the index needs to be created async and then built explicitly using IndexTool.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org