You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@phoenix.apache.org by twdsilva <gi...@git.apache.org> on 2018/10/30 20:59:32 UTC
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
GitHub user twdsilva opened a pull request:
https://github.com/apache/phoenix/pull/402
PHOENIX-4981 Add tests for ORDER BY, GROUP BY and salted tables using…
… phoenix-spark]
@karanmehta93 thanks for the review nice catch, I modified the SparkContext variable to be volatile.
@ChinmaySKulkarni can you please review? I refactored the AggregateIT, OrderByIT and SaltedIT so that it can be used to run queries using phoenix-spark. I created Base*IT based on these tests with two subclasses (one for phoenix , one for phoenix-spark). I added a QueryBuilder to generate a SQL query that is used to setup the spark sql query. I also added SparkResultSet that implements the JDBC interface so that the existing tests can be reused without much changes.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/twdsilva/phoenix PHOENIX-4981
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/phoenix/pull/402.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #402
----
commit c241e5320760384a1710839b28f2e1a2f16d38fd
Author: Thomas D'Silva <td...@...>
Date: 2018-10-19T05:00:01Z
PHOENIX-4981 Add tests for ORDER BY, GROUP BY and salted tables using phoenix-spark
----
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by ChinmaySKulkarni <gi...@git.apache.org>.
Github user ChinmaySKulkarni commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r229561770
--- Diff: phoenix-spark/src/it/java/org/apache/phoenix/spark/AggregateIT.java ---
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.spark;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+import org.apache.phoenix.end2end.BaseAggregateIT;
+import org.apache.phoenix.util.QueryBuilder;
+
+public class AggregateIT extends BaseAggregateIT {
+
+ @Override
+ protected ResultSet executeQueryThrowsException(Connection conn, QueryBuilder queryBuilder,
+ String expectedPhoenixExceptionMsg, String expectedSparkExceptionMsg) {
+ ResultSet rs = null;
+ try {
+ rs = executeQuery(conn, queryBuilder);
+ }
+ catch(Exception e) {
+ assertTrue(e.getMessage().contains(expectedSparkExceptionMsg));
+ }
+ return rs;
+ }
+
+ @Override
+ protected ResultSet executeQuery(Connection conn, QueryBuilder queryBuilder) throws SQLException {
+ return SparkUtil.executeQuery(conn, queryBuilder, getUrl(), config);
+ }
+
+ @Override
+ protected void testCountNullInNonEmptyKeyValueCF(int columnEncodedBytes) throws Exception {
+ try (Connection conn = DriverManager.getConnection(getUrl())) {
+ //Type is INT
+ String intTableName=generateUniqueName();
+ String sql="create table " + intTableName + " (mykey integer not null primary key, A.COLA integer, B.COLB integer) "
+ + "IMMUTABLE_ROWS=true, IMMUTABLE_STORAGE_SCHEME = ONE_CELL_PER_COLUMN, COLUMN_ENCODED_BYTES = " + columnEncodedBytes + ", DISABLE_WAL=true";
+
+ conn.createStatement().execute(sql);
+ conn.createStatement().execute("UPSERT INTO "+intTableName+" VALUES (1,1)");
+ conn.createStatement().execute("UPSERT INTO "+intTableName+" VALUES (2,1)");
+ conn.createStatement().execute("UPSERT INTO "+intTableName+" VALUES (3,1,2)");
+ conn.createStatement().execute("UPSERT INTO "+intTableName+" VALUES (4,1)");
+ conn.createStatement().execute("UPSERT INTO "+intTableName+" VALUES (5,1)");
+ conn.commit();
+
+ sql="select count(*) from "+intTableName;
+ QueryBuilder queryBuilder = new QueryBuilder();
+ queryBuilder.setSelectExpression("COUNT(*)");
+ queryBuilder.setFullTableName(intTableName);
--- End diff --
You can instead do method chaining here since you have a fluent interface.
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by ChinmaySKulkarni <gi...@git.apache.org>.
Github user ChinmaySKulkarni commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r230161996
--- Diff: phoenix-spark/src/it/java/org/apache/phoenix/spark/SparkUtil.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.spark;
+
+import com.google.common.base.Joiner;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.util.QueryBuilder;
+import org.apache.spark.SparkConf;
+import org.apache.spark.SparkContext;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.SQLContext;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.execution.SparkPlan;
+import org.mortbay.log.Log;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.List;
+
+public class SparkUtil {
+
+ public static SparkContext getSparkContext() {
+ return SparkSession.builder().appName("Java Spark Tests").master("local[2]")
+ .config("spark.ui.showConsoleProgress", false).getOrCreate().sparkContext();
+ }
+
+ public static SQLContext getSqlContext() {
+ return SparkSession.builder().appName("Java Spark Tests").master("local[2]")
--- End diff --
nit: Extract all these strings as static final member variables
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by twdsilva <gi...@git.apache.org>.
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r230130599
--- Diff: phoenix-spark/src/it/java/org/apache/phoenix/spark/OrderByIT.java ---
@@ -0,0 +1,444 @@
+package org.apache.phoenix.spark;
+
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.List;
+import java.util.Properties;
+
+import org.apache.phoenix.end2end.BaseOrderByIT;
+import org.apache.phoenix.util.PropertiesUtil;
+import org.apache.phoenix.util.QueryBuilder;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.SQLContext;
+import org.junit.Ignore;
+import org.junit.Test;
+
+import com.google.common.collect.Lists;
+
+import scala.Option;
+import scala.collection.JavaConverters;
+
+public class OrderByIT extends BaseOrderByIT {
+
+ @Override
+ protected ResultSet executeQuery(Connection conn, QueryBuilder queryBuilder) throws SQLException {
+ return SparkUtil.executeQuery(conn, queryBuilder, getUrl(), config);
+ }
+
+ @Test
+ public void testOrderByWithJoin() throws Exception {
+ Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
+ try (Connection conn = DriverManager.getConnection(getUrl(), props)) {
+ conn.setAutoCommit(false);
+ String tableName1 = generateUniqueName();
+ String ddl = "CREATE TABLE " + tableName1 +
+ " (a_string varchar not null, cf1.a integer, cf1.b varchar, col1 integer, cf2.c varchar, cf2.d integer " +
+ " CONSTRAINT pk PRIMARY KEY (a_string))\n";
+ createTestTable(getUrl(), ddl);
+ String dml = "UPSERT INTO " + tableName1 + " VALUES(?,?,?,?,?,?)";
+ PreparedStatement stmt = conn.prepareStatement(dml);
+ stmt.setString(1, "a");
+ stmt.setInt(2, 40);
+ stmt.setString(3, "aa");
+ stmt.setInt(4, 10);
+ stmt.setString(5, "bb");
+ stmt.setInt(6, 20);
+ stmt.execute();
+ stmt.setString(1, "c");
+ stmt.setInt(2, 30);
+ stmt.setString(3, "cc");
+ stmt.setInt(4, 50);
+ stmt.setString(5, "dd");
+ stmt.setInt(6, 60);
+ stmt.execute();
+ stmt.setString(1, "b");
+ stmt.setInt(2, 40);
+ stmt.setString(3, "bb");
+ stmt.setInt(4, 5);
+ stmt.setString(5, "aa");
+ stmt.setInt(6, 80);
+ stmt.execute();
+ conn.commit();
+
+ String tableName2 = generateUniqueName();
+ ddl = "CREATE TABLE " + tableName2 +
+ " (a_string varchar not null, col1 integer" +
+ " CONSTRAINT pk PRIMARY KEY (a_string))\n";
+ createTestTable(getUrl(), ddl);
+
+ dml = "UPSERT INTO " + tableName2 + " VALUES(?, ?)";
+ stmt = conn.prepareStatement(dml);
+ stmt.setString(1, "a");
+ stmt.setInt(2, 40);
+ stmt.execute();
+ stmt.setString(1, "b");
+ stmt.setInt(2, 20);
+ stmt.execute();
+ stmt.setString(1, "c");
+ stmt.setInt(2, 30);
+ stmt.execute();
+ conn.commit();
+
+ List<String> table1Columns = Lists.newArrayList("A_STRING", "CF1.A", "CF1.B", "COL1", "CF2.C", "CF2.D");
+ SQLContext sqlContext = new SQLContext(SparkUtil.getSparkContext());
+ Dataset phoenixDataSet =
+ new PhoenixRDD(SparkUtil.getSparkContext(), tableName1,
+ JavaConverters.collectionAsScalaIterableConverter(table1Columns)
+ .asScala().toSeq(),
--- End diff --
done
---
[GitHub] phoenix issue #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and salted...
Posted by twdsilva <gi...@git.apache.org>.
Github user twdsilva commented on the issue:
https://github.com/apache/phoenix/pull/402
@ChinmaySKulkarni Thanks for the review, I have updated the PR please take a look.
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by ChinmaySKulkarni <gi...@git.apache.org>.
Github user ChinmaySKulkarni commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r229927243
--- Diff: phoenix-core/src/main/java/org/apache/phoenix/util/QueryBuilder.java ---
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.util;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.phoenix.parse.HintNode;
+
+import java.util.Collections;
+import java.util.List;
+
+import static org.apache.phoenix.util.SchemaUtil.getEscapedFullColumnName;
+
+public class QueryBuilder {
+
+ private String fullTableName;
+ // regular columns that are in the select clause
+ private List<String> selectColumns = Collections.emptyList();
+
+ // columns that are required for expressions in the select clause
+ private List<String> selectExpressionColumns = Collections.emptyList();
+ // expression string in the select clause (for eg COL1 || COL2)
+ private String selectExpression;
+ private String whereClause;
+ private String orderByClause;
+ private String groupByClause;
+ private String havingClause;
+ private HintNode.Hint hint;
+ private boolean escapeCols;
+ private boolean distinct;
+ private int limit;
--- End diff --
Ok makes sense
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by ChinmaySKulkarni <gi...@git.apache.org>.
Github user ChinmaySKulkarni commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r229565710
--- Diff: phoenix-spark/src/it/java/org/apache/phoenix/spark/SparkUtil.java ---
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.spark;
+
+import com.google.common.base.Joiner;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.util.QueryBuilder;
+import org.apache.spark.SparkConf;
+import org.apache.spark.SparkContext;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.SQLContext;
+import org.apache.spark.sql.execution.SparkPlan;
+import org.mortbay.log.Log;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.List;
+
+public class SparkUtil {
+
+ private static volatile SparkContext INSTANCE = null;
+
+ public static SparkContext getSparkContext() {
+ if (INSTANCE == null) {
+ synchronized (SparkUtil.class) {
+ if (INSTANCE == null) {
+ SparkConf conf = new SparkConf()
+ .setAppName("Java Spark Tests")
+ .setMaster("local[2]") // 2 threads, some parallelism
+ .set("spark.ui.showConsoleProgress", "false");// Disable printing stage progress
+ INSTANCE = new SparkContext(conf);
+ }
+ }
+ }
+ return INSTANCE;
+ }
+
+ public static ResultSet executeQuery(Connection conn, QueryBuilder queryBuilder, String url, Configuration config)
+ throws SQLException {
+ SQLContext sqlContext = new SQLContext(SparkUtil.getSparkContext());
--- End diff --
It looks as though `SQLContext` is [deprecated](https://spark.apache.org/docs/2.3.0/api/java/org/apache/spark/sql/SQLContext.html#SQLContext-org.apache.spark.SparkContext-). Quoting: _Deprecated. Use SparkSession.builder instead. Since 2.0.0._
---
[GitHub] phoenix issue #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and salted...
Posted by twdsilva <gi...@git.apache.org>.
Github user twdsilva commented on the issue:
https://github.com/apache/phoenix/pull/402
Thank @ChinmaySKulkarni , fixed the nits, will get this committed.
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by twdsilva <gi...@git.apache.org>.
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r229889139
--- Diff: phoenix-spark/pom.xml ---
@@ -487,6 +487,16 @@
<testSourceDirectory>src/it/scala</testSourceDirectory>
<testResources><testResource><directory>src/it/resources</directory></testResource></testResources>
<plugins>
+<!--
--- End diff --
I had commented this by mistake, fixed now.
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by ChinmaySKulkarni <gi...@git.apache.org>.
Github user ChinmaySKulkarni commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r229563160
--- Diff: phoenix-spark/src/it/java/org/apache/phoenix/spark/AggregateIT.java ---
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.spark;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+import org.apache.phoenix.end2end.BaseAggregateIT;
+import org.apache.phoenix.util.QueryBuilder;
+
+public class AggregateIT extends BaseAggregateIT {
+
+ @Override
+ protected ResultSet executeQueryThrowsException(Connection conn, QueryBuilder queryBuilder,
+ String expectedPhoenixExceptionMsg, String expectedSparkExceptionMsg) {
+ ResultSet rs = null;
+ try {
+ rs = executeQuery(conn, queryBuilder);
+ }
+ catch(Exception e) {
+ assertTrue(e.getMessage().contains(expectedSparkExceptionMsg));
+ }
+ return rs;
+ }
+
+ @Override
+ protected ResultSet executeQuery(Connection conn, QueryBuilder queryBuilder) throws SQLException {
+ return SparkUtil.executeQuery(conn, queryBuilder, getUrl(), config);
+ }
+
+ @Override
+ protected void testCountNullInNonEmptyKeyValueCF(int columnEncodedBytes) throws Exception {
+ try (Connection conn = DriverManager.getConnection(getUrl())) {
+ //Type is INT
+ String intTableName=generateUniqueName();
+ String sql="create table " + intTableName + " (mykey integer not null primary key, A.COLA integer, B.COLB integer) "
+ + "IMMUTABLE_ROWS=true, IMMUTABLE_STORAGE_SCHEME = ONE_CELL_PER_COLUMN, COLUMN_ENCODED_BYTES = " + columnEncodedBytes + ", DISABLE_WAL=true";
+
+ conn.createStatement().execute(sql);
+ conn.createStatement().execute("UPSERT INTO "+intTableName+" VALUES (1,1)");
+ conn.createStatement().execute("UPSERT INTO "+intTableName+" VALUES (2,1)");
+ conn.createStatement().execute("UPSERT INTO "+intTableName+" VALUES (3,1,2)");
+ conn.createStatement().execute("UPSERT INTO "+intTableName+" VALUES (4,1)");
+ conn.createStatement().execute("UPSERT INTO "+intTableName+" VALUES (5,1)");
+ conn.commit();
+
+ sql="select count(*) from "+intTableName;
+ QueryBuilder queryBuilder = new QueryBuilder();
+ queryBuilder.setSelectExpression("COUNT(*)");
+ queryBuilder.setFullTableName(intTableName);
+ ResultSet rs = executeQuery(conn, queryBuilder);
+ assertTrue(rs.next());
+ assertEquals(5, rs.getLong(1));
+
+ sql="select count(*) from "+intTableName + " where b.colb is not null";
+ queryBuilder.setWhereClause("`B.COLB` IS NOT NULL");
+ rs = executeQuery(conn, queryBuilder);
+ assertTrue(rs.next());
+ assertEquals(1, rs.getLong(1));
+
+ sql="select count(*) from "+intTableName + " where b.colb is null";
+ queryBuilder.setWhereClause("`B.COLB` IS NULL");
--- End diff --
It seems like the only difference between this test and the one in phoenix-core is the backticks provided to querybuilder setter methods. I'm guessing this is a result of how `SparkUtil` executes queries. Please correct me if I'm wrong, but if not, can we further reuse the code from the phoenix-core tests instead of having more duplication?
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by ChinmaySKulkarni <gi...@git.apache.org>.
Github user ChinmaySKulkarni commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r229564032
--- Diff: phoenix-spark/src/it/java/org/apache/phoenix/spark/OrderByIT.java ---
@@ -0,0 +1,444 @@
+package org.apache.phoenix.spark;
+
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.List;
+import java.util.Properties;
+
+import org.apache.phoenix.end2end.BaseOrderByIT;
+import org.apache.phoenix.util.PropertiesUtil;
+import org.apache.phoenix.util.QueryBuilder;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.SQLContext;
+import org.junit.Ignore;
+import org.junit.Test;
+
+import com.google.common.collect.Lists;
+
+import scala.Option;
+import scala.collection.JavaConverters;
+
+public class OrderByIT extends BaseOrderByIT {
+
+ @Override
+ protected ResultSet executeQuery(Connection conn, QueryBuilder queryBuilder) throws SQLException {
+ return SparkUtil.executeQuery(conn, queryBuilder, getUrl(), config);
+ }
+
+ @Test
+ public void testOrderByWithJoin() throws Exception {
+ Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
+ try (Connection conn = DriverManager.getConnection(getUrl(), props)) {
+ conn.setAutoCommit(false);
+ String tableName1 = generateUniqueName();
+ String ddl = "CREATE TABLE " + tableName1 +
+ " (a_string varchar not null, cf1.a integer, cf1.b varchar, col1 integer, cf2.c varchar, cf2.d integer " +
+ " CONSTRAINT pk PRIMARY KEY (a_string))\n";
+ createTestTable(getUrl(), ddl);
+ String dml = "UPSERT INTO " + tableName1 + " VALUES(?,?,?,?,?,?)";
+ PreparedStatement stmt = conn.prepareStatement(dml);
+ stmt.setString(1, "a");
+ stmt.setInt(2, 40);
+ stmt.setString(3, "aa");
+ stmt.setInt(4, 10);
+ stmt.setString(5, "bb");
+ stmt.setInt(6, 20);
+ stmt.execute();
+ stmt.setString(1, "c");
+ stmt.setInt(2, 30);
+ stmt.setString(3, "cc");
+ stmt.setInt(4, 50);
+ stmt.setString(5, "dd");
+ stmt.setInt(6, 60);
+ stmt.execute();
+ stmt.setString(1, "b");
+ stmt.setInt(2, 40);
+ stmt.setString(3, "bb");
+ stmt.setInt(4, 5);
+ stmt.setString(5, "aa");
+ stmt.setInt(6, 80);
+ stmt.execute();
+ conn.commit();
+
+ String tableName2 = generateUniqueName();
+ ddl = "CREATE TABLE " + tableName2 +
+ " (a_string varchar not null, col1 integer" +
+ " CONSTRAINT pk PRIMARY KEY (a_string))\n";
+ createTestTable(getUrl(), ddl);
+
+ dml = "UPSERT INTO " + tableName2 + " VALUES(?, ?)";
+ stmt = conn.prepareStatement(dml);
+ stmt.setString(1, "a");
+ stmt.setInt(2, 40);
+ stmt.execute();
+ stmt.setString(1, "b");
+ stmt.setInt(2, 20);
+ stmt.execute();
+ stmt.setString(1, "c");
+ stmt.setInt(2, 30);
+ stmt.execute();
+ conn.commit();
+
+ List<String> table1Columns = Lists.newArrayList("A_STRING", "CF1.A", "CF1.B", "COL1", "CF2.C", "CF2.D");
+ SQLContext sqlContext = new SQLContext(SparkUtil.getSparkContext());
+ Dataset phoenixDataSet =
+ new PhoenixRDD(SparkUtil.getSparkContext(), tableName1,
+ JavaConverters.collectionAsScalaIterableConverter(table1Columns)
+ .asScala().toSeq(),
--- End diff --
Can you add some comments here
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by twdsilva <gi...@git.apache.org>.
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r229897850
--- Diff: phoenix-spark/src/it/java/org/apache/phoenix/spark/AggregateIT.java ---
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.spark;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+import org.apache.phoenix.end2end.BaseAggregateIT;
+import org.apache.phoenix.util.QueryBuilder;
+
+public class AggregateIT extends BaseAggregateIT {
+
+ @Override
+ protected ResultSet executeQueryThrowsException(Connection conn, QueryBuilder queryBuilder,
+ String expectedPhoenixExceptionMsg, String expectedSparkExceptionMsg) {
+ ResultSet rs = null;
+ try {
+ rs = executeQuery(conn, queryBuilder);
+ }
+ catch(Exception e) {
+ assertTrue(e.getMessage().contains(expectedSparkExceptionMsg));
+ }
+ return rs;
+ }
+
+ @Override
+ protected ResultSet executeQuery(Connection conn, QueryBuilder queryBuilder) throws SQLException {
+ return SparkUtil.executeQuery(conn, queryBuilder, getUrl(), config);
+ }
+
+ @Override
+ protected void testCountNullInNonEmptyKeyValueCF(int columnEncodedBytes) throws Exception {
+ try (Connection conn = DriverManager.getConnection(getUrl())) {
+ //Type is INT
+ String intTableName=generateUniqueName();
+ String sql="create table " + intTableName + " (mykey integer not null primary key, A.COLA integer, B.COLB integer) "
+ + "IMMUTABLE_ROWS=true, IMMUTABLE_STORAGE_SCHEME = ONE_CELL_PER_COLUMN, COLUMN_ENCODED_BYTES = " + columnEncodedBytes + ", DISABLE_WAL=true";
+
+ conn.createStatement().execute(sql);
+ conn.createStatement().execute("UPSERT INTO "+intTableName+" VALUES (1,1)");
+ conn.createStatement().execute("UPSERT INTO "+intTableName+" VALUES (2,1)");
+ conn.createStatement().execute("UPSERT INTO "+intTableName+" VALUES (3,1,2)");
+ conn.createStatement().execute("UPSERT INTO "+intTableName+" VALUES (4,1)");
+ conn.createStatement().execute("UPSERT INTO "+intTableName+" VALUES (5,1)");
+ conn.commit();
+
+ sql="select count(*) from "+intTableName;
+ QueryBuilder queryBuilder = new QueryBuilder();
+ queryBuilder.setSelectExpression("COUNT(*)");
+ queryBuilder.setFullTableName(intTableName);
--- End diff --
Done.
---
[GitHub] phoenix issue #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and salted...
Posted by ChinmaySKulkarni <gi...@git.apache.org>.
Github user ChinmaySKulkarni commented on the issue:
https://github.com/apache/phoenix/pull/402
@twdsilva A couple of minor nits, otherwise looks good to go. Thanks!
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by ChinmaySKulkarni <gi...@git.apache.org>.
Github user ChinmaySKulkarni commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r229563250
--- Diff: phoenix-spark/src/it/java/org/apache/phoenix/spark/AggregateIT.java ---
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.spark;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+import org.apache.phoenix.end2end.BaseAggregateIT;
+import org.apache.phoenix.util.QueryBuilder;
+
+public class AggregateIT extends BaseAggregateIT {
+
+ @Override
+ protected ResultSet executeQueryThrowsException(Connection conn, QueryBuilder queryBuilder,
+ String expectedPhoenixExceptionMsg, String expectedSparkExceptionMsg) {
+ ResultSet rs = null;
+ try {
+ rs = executeQuery(conn, queryBuilder);
--- End diff --
Similar question/comment about `Assert.fail` here as well.
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by twdsilva <gi...@git.apache.org>.
Github user twdsilva closed the pull request at:
https://github.com/apache/phoenix/pull/402
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by twdsilva <gi...@git.apache.org>.
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r229900126
--- Diff: phoenix-spark/src/it/java/org/apache/phoenix/spark/AggregateIT.java ---
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.spark;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+import org.apache.phoenix.end2end.BaseAggregateIT;
+import org.apache.phoenix.util.QueryBuilder;
+
+public class AggregateIT extends BaseAggregateIT {
+
+ @Override
+ protected ResultSet executeQueryThrowsException(Connection conn, QueryBuilder queryBuilder,
+ String expectedPhoenixExceptionMsg, String expectedSparkExceptionMsg) {
+ ResultSet rs = null;
+ try {
+ rs = executeQuery(conn, queryBuilder);
--- End diff --
fixed.
---
[GitHub] phoenix issue #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and salted...
Posted by twdsilva <gi...@git.apache.org>.
Github user twdsilva commented on the issue:
https://github.com/apache/phoenix/pull/402
@ChinmaySKulkarni I removed the use of the deprecated SqlContext constructor, please take a look.
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by twdsilva <gi...@git.apache.org>.
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r229889444
--- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/AggregateIT.java ---
@@ -17,507 +17,29 @@
*/
package org.apache.phoenix.end2end;
-import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertTrue;
-import static org.junit.Assert.fail;
-import static org.apache.phoenix.util.TestUtil.assertResultSet;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.schema.AmbiguousColumnException;
+import org.apache.phoenix.util.PropertiesUtil;
+import org.apache.phoenix.util.QueryBuilder;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Test;
-import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
-import java.sql.Statement;
-import java.util.List;
import java.util.Properties;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.phoenix.compile.QueryPlan;
-import org.apache.phoenix.jdbc.PhoenixConnection;
-import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
-import org.apache.phoenix.jdbc.PhoenixStatement;
-import org.apache.phoenix.query.KeyRange;
-import org.apache.phoenix.schema.AmbiguousColumnException;
-import org.apache.phoenix.schema.types.PChar;
-import org.apache.phoenix.schema.types.PInteger;
-import org.apache.phoenix.util.ByteUtil;
-import org.apache.phoenix.util.PropertiesUtil;
-import org.apache.phoenix.util.QueryUtil;
-import org.apache.phoenix.util.TestUtil;
-import org.junit.Test;
-
-
-public class AggregateIT extends ParallelStatsDisabledIT {
- private static void initData(Connection conn, String tableName) throws SQLException {
- conn.createStatement().execute("create table " + tableName +
- " (id varchar not null primary key,\n" +
- " uri varchar, appcpu integer)");
- insertRow(conn, tableName, "Report1", 10, 1);
- insertRow(conn, tableName, "Report2", 10, 2);
- insertRow(conn, tableName, "Report3", 30, 3);
- insertRow(conn, tableName, "Report4", 30, 4);
- insertRow(conn, tableName, "SOQL1", 10, 5);
- insertRow(conn, tableName, "SOQL2", 10, 6);
- insertRow(conn, tableName, "SOQL3", 30, 7);
- insertRow(conn, tableName, "SOQL4", 30, 8);
- conn.commit();
- }
-
- private static void insertRow(Connection conn, String tableName, String uri, int appcpu, int id) throws SQLException {
- PreparedStatement statement = conn.prepareStatement("UPSERT INTO " + tableName + "(id, uri, appcpu) values (?,?,?)");
- statement.setString(1, "id" + id);
- statement.setString(2, uri);
- statement.setInt(3, appcpu);
- statement.executeUpdate();
- }
-
- @Test
- public void testDuplicateTrailingAggExpr() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- conn.createStatement().execute("create table " + tableName +
- " (nam VARCHAR(20), address VARCHAR(20), id BIGINT "
- + "constraint my_pk primary key (id))");
- PreparedStatement statement = conn.prepareStatement("UPSERT INTO " + tableName + "(nam, address, id) values (?,?,?)");
- statement.setString(1, "pulkit");
- statement.setString(2, "badaun");
- statement.setInt(3, 1);
- statement.executeUpdate();
- conn.commit();
- Statement stmt = conn.createStatement();
- ResultSet rs = stmt.executeQuery("select distinct 'harshit' as \"test_column\", trim(nam), trim(nam) from " + tableName);
- assertTrue(rs.next());
- assertEquals("harshit", rs.getString(1));
- assertEquals("pulkit", rs.getString(2));
- assertEquals("pulkit", rs.getString(3));
- conn.close();
- }
-
- @Test
- public void testExpressionInGroupBy() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- String ddl = " create table " + tableName + "(tgb_id integer NOT NULL,utc_date_epoch integer NOT NULL,tgb_name varchar(40),ack_success_count integer" +
- ",ack_success_one_ack_count integer, CONSTRAINT pk_tgb_counter PRIMARY KEY(tgb_id, utc_date_epoch))";
- String query = "SELECT tgb_id, tgb_name, (utc_date_epoch/10)*10 AS utc_epoch_hour,SUM(ack_success_count + ack_success_one_ack_count) AS ack_tx_sum" +
- " FROM " + tableName + " GROUP BY tgb_id, tgb_name, utc_epoch_hour";
-
- createTestTable(getUrl(), ddl);
- String dml = "UPSERT INTO " + tableName + " VALUES(?,?,?,?,?)";
- PreparedStatement stmt = conn.prepareStatement(dml);
- stmt.setInt(1, 1);
- stmt.setInt(2, 1000);
- stmt.setString(3, "aaa");
- stmt.setInt(4, 1);
- stmt.setInt(5, 1);
- stmt.execute();
- stmt.setInt(1, 2);
- stmt.setInt(2, 2000);
- stmt.setString(3, "bbb");
- stmt.setInt(4, 2);
- stmt.setInt(5, 2);
- stmt.execute();
- conn.commit();
-
- ResultSet rs = conn.createStatement().executeQuery(query);
- assertTrue(rs.next());
- assertEquals(1,rs.getInt(1));
- assertEquals("aaa",rs.getString(2));
- assertEquals(1000,rs.getInt(3));
- assertEquals(2,rs.getInt(4));
- assertTrue(rs.next());
- assertEquals(2,rs.getInt(1));
- assertEquals("bbb",rs.getString(2));
- assertEquals(2000,rs.getInt(3));
- assertEquals(4,rs.getInt(4));
- assertFalse(rs.next());
- rs.close();
- conn.close();
- }
-
- @Test
- public void testBooleanInGroupBy() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- String ddl = " create table " + tableName + "(id varchar primary key,v1 boolean, v2 integer, v3 integer)";
-
- createTestTable(getUrl(), ddl);
- PreparedStatement stmt = conn.prepareStatement("UPSERT INTO " + tableName + "(id,v2,v3) VALUES(?,?,?)");
- stmt.setString(1, "a");
- stmt.setInt(2, 1);
- stmt.setInt(3, 1);
- stmt.execute();
- stmt.close();
- stmt = conn.prepareStatement("UPSERT INTO " + tableName + " VALUES(?,?,?,?)");
- stmt.setString(1, "b");
- stmt.setBoolean(2, false);
- stmt.setInt(3, 2);
- stmt.setInt(4, 2);
- stmt.execute();
- stmt.setString(1, "c");
- stmt.setBoolean(2, true);
- stmt.setInt(3, 3);
- stmt.setInt(4, 3);
- stmt.execute();
- conn.commit();
-
- String[] gbs = {"v1,v2,v3","v1,v3,v2","v2,v1,v3"};
- for (String gb : gbs) {
- ResultSet rs = conn.createStatement().executeQuery("SELECT v1, v2, v3 from " + tableName + " group by " + gb);
- assertTrue(rs.next());
- assertEquals(false,rs.getBoolean("v1"));
- assertTrue(rs.wasNull());
- assertEquals(1,rs.getInt("v2"));
- assertEquals(1,rs.getInt("v3"));
- assertTrue(rs.next());
- assertEquals(false,rs.getBoolean("v1"));
- assertFalse(rs.wasNull());
- assertEquals(2,rs.getInt("v2"));
- assertEquals(2,rs.getInt("v3"));
- assertTrue(rs.next());
- assertEquals(true,rs.getBoolean("v1"));
- assertEquals(3,rs.getInt("v2"));
- assertEquals(3,rs.getInt("v3"));
- assertFalse(rs.next());
- rs.close();
- }
- conn.close();
- }
-
- @Test
- public void testScanUri() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- initData(conn, tableName);
- Statement stmt = conn.createStatement();
- ResultSet rs = stmt.executeQuery("select uri from " + tableName);
- assertTrue(rs.next());
- assertEquals("Report1", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("Report2", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("Report3", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("Report4", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("SOQL1", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("SOQL2", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("SOQL3", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("SOQL4", rs.getString(1));
- assertFalse(rs.next());
- conn.close();
- }
-
- @Test
- public void testCount() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- initData(conn, tableName);
- Statement stmt = conn.createStatement();
- ResultSet rs = stmt.executeQuery("select count(1) from " + tableName);
- assertTrue(rs.next());
- assertEquals(8, rs.getInt(1));
- assertFalse(rs.next());
- conn.close();
- }
-
- @Test
- public void testGroupByCase() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- String groupBy1 = "select " +
- "case when uri LIKE 'Report%' then 'Reports' else 'Other' END category" +
- ", avg(appcpu) from " + tableName +
- " group by category";
-
- String groupBy2 = "select " +
- "case uri when 'Report%' then 'Reports' else 'Other' END category" +
- ", avg(appcpu) from " + tableName +
- " group by appcpu, category";
-
- String groupBy3 = "select " +
- "case uri when 'Report%' then 'Reports' else 'Other' END category" +
- ", avg(appcpu) from " + tableName +
- " group by avg(appcpu), category";
- initData(conn, tableName);
- conn.createStatement().executeQuery(groupBy1);
- conn.createStatement().executeQuery(groupBy2);
- // TODO: validate query results
- try {
- conn.createStatement().executeQuery(groupBy3);
- fail();
- } catch (SQLException e) {
- assertTrue(e.getMessage().contains("Aggregate expressions may not be used in GROUP BY"));
- }
- conn.close();
- }
-
-
- @Test
- public void testGroupByArray() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
-
- String tableName = generateUniqueName();
- conn.createStatement().execute("CREATE TABLE " + tableName + "(\n" +
- " a VARCHAR NOT NULL,\n" +
- " b VARCHAR,\n" +
- " c INTEGER,\n" +
- " d VARCHAR,\n" +
- " e VARCHAR ARRAY,\n" +
- " f BIGINT,\n" +
- " g BIGINT,\n" +
- " CONSTRAINT pk PRIMARY KEY(a)\n" +
- ")");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('1', 'val', 100, 'a', ARRAY ['b'], 1, 2)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('2', 'val', 100, 'a', ARRAY ['b'], 3, 4)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('3', 'val', 100, 'a', ARRAY ['b','c'], 5, 6)");
- conn.commit();
-
- ResultSet rs = conn.createStatement().executeQuery("SELECT c, SUM(f + g) AS sumone, d, e\n" +
- "FROM " + tableName + "\n" +
- "WHERE b = 'val'\n" +
- " AND a IN ('1','2','3')\n" +
- "GROUP BY c, d, e\n" +
- "ORDER BY sumone DESC");
- assertTrue(rs.next());
- assertEquals(100, rs.getInt(1));
- assertEquals(11, rs.getLong(2));
- assertTrue(rs.next());
- assertEquals(100, rs.getInt(1));
- assertEquals(10, rs.getLong(2));
- assertFalse(rs.next());
- conn.close();
- }
-
- @Test
- public void testGroupByOrderPreserving() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
-
- conn.createStatement().execute("CREATE TABLE " + tableName + "(ORGANIZATION_ID char(15) not null, \n" +
- "JOURNEY_ID char(15) not null, \n" +
- "DATASOURCE SMALLINT not null, \n" +
- "MATCH_STATUS TINYINT not null, \n" +
- "EXTERNAL_DATASOURCE_KEY varchar(30), \n" +
- "ENTITY_ID char(15) not null, \n" +
- "CONSTRAINT PK PRIMARY KEY (\n" +
- " ORGANIZATION_ID, \n" +
- " JOURNEY_ID, \n" +
- " DATASOURCE, \n" +
- " MATCH_STATUS,\n" +
- " EXTERNAL_DATASOURCE_KEY,\n" +
- " ENTITY_ID))");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('000001111122222', '333334444455555', 0, 0, 'abc', '666667777788888')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('000001111122222', '333334444455555', 0, 0, 'abcd', '666667777788889')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('000001111122222', '333334444455555', 0, 0, 'abc', '666667777788899')");
- conn.commit();
- String query =
- "SELECT COUNT(1), EXTERNAL_DATASOURCE_KEY As DUP_COUNT\n" +
- " FROM " + tableName + " \n" +
- " WHERE JOURNEY_ID='333334444455555' AND \n" +
- " DATASOURCE=0 AND MATCH_STATUS <= 1 and \n" +
- " ORGANIZATION_ID='000001111122222' \n" +
- " GROUP BY MATCH_STATUS, EXTERNAL_DATASOURCE_KEY \n" +
- " HAVING COUNT(1) > 1";
- ResultSet rs = conn.createStatement().executeQuery(query);
- assertTrue(rs.next());
- assertEquals(2,rs.getInt(1));
- assertEquals("abc", rs.getString(2));
- assertFalse(rs.next());
-
- rs = conn.createStatement().executeQuery("EXPLAIN " + query);
- assertEquals(
- "CLIENT PARALLEL 1-WAY RANGE SCAN OVER " + tableName + " ['000001111122222','333334444455555',0,*] - ['000001111122222','333334444455555',0,1]\n" +
- " SERVER FILTER BY FIRST KEY ONLY\n" +
- " SERVER AGGREGATE INTO ORDERED DISTINCT ROWS BY [MATCH_STATUS, EXTERNAL_DATASOURCE_KEY]\n" +
- "CLIENT FILTER BY COUNT(1) > 1",QueryUtil.getExplainPlan(rs));
- }
-
- @Test
- public void testGroupByOrderPreservingDescSort() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- conn.createStatement().execute("CREATE TABLE " + tableName + " (k1 char(1) not null, k2 char(1) not null, constraint pk primary key (k1,k2)) split on ('ac','jc','nc')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 'a')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 'b')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 'c')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 'd')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 'a')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 'b')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 'c')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 'd')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 'a')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 'b')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 'c')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 'd')");
- conn.commit();
- String query = "SELECT k1,count(*) FROM " + tableName + " GROUP BY k1 ORDER BY k1 DESC";
- ResultSet rs = conn.createStatement().executeQuery(query);
- assertTrue(rs.next());
- assertEquals("n", rs.getString(1));
- assertEquals(4, rs.getInt(2));
- assertTrue(rs.next());
- assertEquals("j", rs.getString(1));
- assertEquals(4, rs.getInt(2));
- assertTrue(rs.next());
- assertEquals("a", rs.getString(1));
- assertEquals(4, rs.getInt(2));
- assertFalse(rs.next());
- rs = conn.createStatement().executeQuery("EXPLAIN " + query);
- assertEquals(
- "CLIENT PARALLEL 1-WAY REVERSE FULL SCAN OVER " + tableName + "\n" +
- " SERVER FILTER BY FIRST KEY ONLY\n" +
- " SERVER AGGREGATE INTO ORDERED DISTINCT ROWS BY [K1]", QueryUtil.getExplainPlan(rs));
- }
-
- @Test
- public void testSumGroupByOrderPreservingDesc() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
+import static org.apache.phoenix.util.TestUtil.assertResultSet;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
--- End diff --
done
---
[GitHub] phoenix issue #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and salted...
Posted by ChinmaySKulkarni <gi...@git.apache.org>.
Github user ChinmaySKulkarni commented on the issue:
https://github.com/apache/phoenix/pull/402
@twdsilva some comments and questions. Overall looks really good!
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by ChinmaySKulkarni <gi...@git.apache.org>.
Github user ChinmaySKulkarni commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r229561126
--- Diff: phoenix-spark/pom.xml ---
@@ -487,6 +487,16 @@
<testSourceDirectory>src/it/scala</testSourceDirectory>
<testResources><testResource><directory>src/it/resources</directory></testResource></testResources>
<plugins>
+<!--
--- End diff --
Remove this commented section
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by ChinmaySKulkarni <gi...@git.apache.org>.
Github user ChinmaySKulkarni commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r229564917
--- Diff: phoenix-spark/src/main/java/org/apache/phoenix/spark/SparkResultSet.java ---
@@ -0,0 +1,1056 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.spark;
+
+import org.apache.phoenix.exception.SQLExceptionCode;
+import org.apache.phoenix.exception.SQLExceptionInfo;
+import org.apache.phoenix.util.SQLCloseable;
+import org.apache.spark.sql.Row;
+
+import java.io.InputStream;
+import java.io.Reader;
+import java.math.BigDecimal;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.sql.Array;
+import java.sql.Blob;
+import java.sql.Clob;
+import java.sql.Date;
+import java.sql.NClob;
+import java.sql.Ref;
+import java.sql.ResultSet;
+import java.sql.ResultSetMetaData;
+import java.sql.RowId;
+import java.sql.SQLException;
+import java.sql.SQLFeatureNotSupportedException;
+import java.sql.SQLWarning;
+import java.sql.SQLXML;
+import java.sql.Statement;
+import java.sql.Time;
+import java.sql.Timestamp;
+import java.util.Arrays;
+import java.util.Calendar;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Helper class to convert a List of Rows returns from a dataset to a sql ResultSet
--- End diff --
nit: 'returned'
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by twdsilva <gi...@git.apache.org>.
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r229888377
--- Diff: phoenix-core/src/main/java/org/apache/phoenix/util/QueryBuilder.java ---
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.util;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.phoenix.parse.HintNode;
+
+import java.util.Collections;
+import java.util.List;
+
+import static org.apache.phoenix.util.SchemaUtil.getEscapedFullColumnName;
+
+public class QueryBuilder {
+
+ private String fullTableName;
+ // regular columns that are in the select clause
+ private List<String> selectColumns = Collections.emptyList();
+
+ // columns that are required for expressions in the select clause
+ private List<String> selectExpressionColumns = Collections.emptyList();
+ // expression string in the select clause (for eg COL1 || COL2)
+ private String selectExpression;
+ private String whereClause;
+ private String orderByClause;
+ private String groupByClause;
+ private String havingClause;
+ private HintNode.Hint hint;
+ private boolean escapeCols;
+ private boolean distinct;
+ private int limit;
--- End diff --
IMHO I don't think that necessary as we don't have an object that represents a Query, the build() just returns a a string.
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by twdsilva <gi...@git.apache.org>.
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r229901798
--- Diff: phoenix-spark/src/main/java/org/apache/phoenix/spark/SparkResultSet.java ---
@@ -0,0 +1,1056 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.spark;
+
+import org.apache.phoenix.exception.SQLExceptionCode;
+import org.apache.phoenix.exception.SQLExceptionInfo;
+import org.apache.phoenix.util.SQLCloseable;
+import org.apache.spark.sql.Row;
+
+import java.io.InputStream;
+import java.io.Reader;
+import java.math.BigDecimal;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.sql.Array;
+import java.sql.Blob;
+import java.sql.Clob;
+import java.sql.Date;
+import java.sql.NClob;
+import java.sql.Ref;
+import java.sql.ResultSet;
+import java.sql.ResultSetMetaData;
+import java.sql.RowId;
+import java.sql.SQLException;
+import java.sql.SQLFeatureNotSupportedException;
+import java.sql.SQLWarning;
+import java.sql.SQLXML;
+import java.sql.Statement;
+import java.sql.Time;
+import java.sql.Timestamp;
+import java.util.Arrays;
+import java.util.Calendar;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Helper class to convert a List of Rows returns from a dataset to a sql ResultSet
--- End diff --
fixed.
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by ChinmaySKulkarni <gi...@git.apache.org>.
Github user ChinmaySKulkarni commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r230161691
--- Diff: phoenix-spark/src/it/java/org/apache/phoenix/spark/OrderByIT.java ---
@@ -0,0 +1,460 @@
+package org.apache.phoenix.spark;
+
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.List;
+import java.util.Properties;
+
+import org.apache.phoenix.end2end.BaseOrderByIT;
+import org.apache.phoenix.util.PropertiesUtil;
+import org.apache.phoenix.util.QueryBuilder;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.SQLContext;
+import org.junit.Ignore;
+import org.junit.Test;
+
+import com.google.common.collect.Lists;
+
+import scala.Option;
+import scala.collection.JavaConverters;
+
+public class OrderByIT extends BaseOrderByIT {
+
+ @Override
+ protected ResultSet executeQueryThrowsException(Connection conn, QueryBuilder queryBuilder,
+ String expectedPhoenixExceptionMsg, String expectedSparkExceptionMsg) {
+ ResultSet rs = null;
+ try {
+ rs = executeQuery(conn, queryBuilder);
+ fail();
+ }
+ catch(Exception e) {
+ assertTrue(e.getMessage().contains(expectedSparkExceptionMsg));
+ }
+ return rs;
+ }
+
+ @Override
+ protected ResultSet executeQuery(Connection conn, QueryBuilder queryBuilder) throws SQLException {
+ return SparkUtil.executeQuery(conn, queryBuilder, getUrl(), config);
+ }
+
+ @Test
+ public void testOrderByWithJoin() throws Exception {
+ Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
+ try (Connection conn = DriverManager.getConnection(getUrl(), props)) {
+ conn.setAutoCommit(false);
+ String tableName1 = generateUniqueName();
+ String ddl = "CREATE TABLE " + tableName1 +
+ " (a_string varchar not null, cf1.a integer, cf1.b varchar, col1 integer, cf2.c varchar, cf2.d integer " +
+ " CONSTRAINT pk PRIMARY KEY (a_string))\n";
+ createTestTable(getUrl(), ddl);
+ String dml = "UPSERT INTO " + tableName1 + " VALUES(?,?,?,?,?,?)";
+ PreparedStatement stmt = conn.prepareStatement(dml);
+ stmt.setString(1, "a");
+ stmt.setInt(2, 40);
+ stmt.setString(3, "aa");
+ stmt.setInt(4, 10);
+ stmt.setString(5, "bb");
+ stmt.setInt(6, 20);
+ stmt.execute();
+ stmt.setString(1, "c");
+ stmt.setInt(2, 30);
+ stmt.setString(3, "cc");
+ stmt.setInt(4, 50);
+ stmt.setString(5, "dd");
+ stmt.setInt(6, 60);
+ stmt.execute();
+ stmt.setString(1, "b");
+ stmt.setInt(2, 40);
+ stmt.setString(3, "bb");
+ stmt.setInt(4, 5);
+ stmt.setString(5, "aa");
+ stmt.setInt(6, 80);
+ stmt.execute();
+ conn.commit();
+
+ String tableName2 = generateUniqueName();
+ ddl = "CREATE TABLE " + tableName2 +
+ " (a_string varchar not null, col1 integer" +
+ " CONSTRAINT pk PRIMARY KEY (a_string))\n";
+ createTestTable(getUrl(), ddl);
+
+ dml = "UPSERT INTO " + tableName2 + " VALUES(?, ?)";
+ stmt = conn.prepareStatement(dml);
+ stmt.setString(1, "a");
+ stmt.setInt(2, 40);
+ stmt.execute();
+ stmt.setString(1, "b");
+ stmt.setInt(2, 20);
+ stmt.execute();
+ stmt.setString(1, "c");
+ stmt.setInt(2, 30);
+ stmt.execute();
+ conn.commit();
+
+ // create two PhoenixRDDs using the table namea and columns that are required for the JOIN query
--- End diff --
nit: typo
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by ChinmaySKulkarni <gi...@git.apache.org>.
Github user ChinmaySKulkarni commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r229561363
--- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/ParallelStatsDisabledIT.java ---
@@ -41,4 +53,29 @@ public static final void doSetup() throws Exception {
public static void tearDownMiniCluster() throws Exception {
BaseTest.tearDownMiniClusterIfBeyondThreshold();
}
+
+ protected ResultSet executeQuery(Connection conn, QueryBuilder queryBuilder) throws SQLException {
+ PreparedStatement statement = conn.prepareStatement(queryBuilder.build());
+ ResultSet rs = statement.executeQuery();
+ return rs;
+ }
+
+ protected ResultSet executeQueryThrowsException(Connection conn, QueryBuilder queryBuilder,
+ String expectedPhoenixExceptionMsg, String expectedSparkExceptionMsg) {
+ ResultSet rs = null;
+ try {
+ rs = executeQuery(conn, queryBuilder);
+ }
+ catch(Exception e) {
+ assertTrue(e.getMessage().contains(expectedPhoenixExceptionMsg));
+ }
+ return rs;
--- End diff --
Do we ever want code to reach here? Or do we want to `Assert.fail` if the exception doesn't occur?
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by ChinmaySKulkarni <gi...@git.apache.org>.
Github user ChinmaySKulkarni commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r230161876
--- Diff: phoenix-spark/src/it/java/org/apache/phoenix/spark/AggregateIT.java ---
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.spark;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+import org.apache.phoenix.end2end.BaseAggregateIT;
+import org.apache.phoenix.util.QueryBuilder;
+
+public class AggregateIT extends BaseAggregateIT {
+
+ @Override
+ protected ResultSet executeQueryThrowsException(Connection conn, QueryBuilder queryBuilder,
+ String expectedPhoenixExceptionMsg, String expectedSparkExceptionMsg) {
+ ResultSet rs = null;
+ try {
+ rs = executeQuery(conn, queryBuilder);
+ }
+ catch(Exception e) {
+ assertTrue(e.getMessage().contains(expectedSparkExceptionMsg));
+ }
+ return rs;
+ }
+
+ @Override
+ protected ResultSet executeQuery(Connection conn, QueryBuilder queryBuilder) throws SQLException {
+ return SparkUtil.executeQuery(conn, queryBuilder, getUrl(), config);
+ }
+
+ @Override
+ protected void testCountNullInNonEmptyKeyValueCF(int columnEncodedBytes) throws Exception {
+ try (Connection conn = DriverManager.getConnection(getUrl())) {
+ //Type is INT
+ String intTableName=generateUniqueName();
+ String sql="create table " + intTableName + " (mykey integer not null primary key, A.COLA integer, B.COLB integer) "
+ + "IMMUTABLE_ROWS=true, IMMUTABLE_STORAGE_SCHEME = ONE_CELL_PER_COLUMN, COLUMN_ENCODED_BYTES = " + columnEncodedBytes + ", DISABLE_WAL=true";
+
+ conn.createStatement().execute(sql);
+ conn.createStatement().execute("UPSERT INTO "+intTableName+" VALUES (1,1)");
+ conn.createStatement().execute("UPSERT INTO "+intTableName+" VALUES (2,1)");
+ conn.createStatement().execute("UPSERT INTO "+intTableName+" VALUES (3,1,2)");
+ conn.createStatement().execute("UPSERT INTO "+intTableName+" VALUES (4,1)");
+ conn.createStatement().execute("UPSERT INTO "+intTableName+" VALUES (5,1)");
+ conn.commit();
+
+ sql="select count(*) from "+intTableName;
+ QueryBuilder queryBuilder = new QueryBuilder();
+ queryBuilder.setSelectExpression("COUNT(*)");
+ queryBuilder.setFullTableName(intTableName);
+ ResultSet rs = executeQuery(conn, queryBuilder);
+ assertTrue(rs.next());
+ assertEquals(5, rs.getLong(1));
+
+ sql="select count(*) from "+intTableName + " where b.colb is not null";
+ queryBuilder.setWhereClause("`B.COLB` IS NOT NULL");
+ rs = executeQuery(conn, queryBuilder);
+ assertTrue(rs.next());
+ assertEquals(1, rs.getLong(1));
+
+ sql="select count(*) from "+intTableName + " where b.colb is null";
+ queryBuilder.setWhereClause("`B.COLB` IS NULL");
--- End diff --
Ok let's leave it the way it is for now.
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by ChinmaySKulkarni <gi...@git.apache.org>.
Github user ChinmaySKulkarni commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r229554215
--- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/AggregateIT.java ---
@@ -17,507 +17,29 @@
*/
package org.apache.phoenix.end2end;
-import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertTrue;
-import static org.junit.Assert.fail;
-import static org.apache.phoenix.util.TestUtil.assertResultSet;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.schema.AmbiguousColumnException;
+import org.apache.phoenix.util.PropertiesUtil;
+import org.apache.phoenix.util.QueryBuilder;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Test;
-import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
-import java.sql.Statement;
-import java.util.List;
import java.util.Properties;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.phoenix.compile.QueryPlan;
-import org.apache.phoenix.jdbc.PhoenixConnection;
-import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
-import org.apache.phoenix.jdbc.PhoenixStatement;
-import org.apache.phoenix.query.KeyRange;
-import org.apache.phoenix.schema.AmbiguousColumnException;
-import org.apache.phoenix.schema.types.PChar;
-import org.apache.phoenix.schema.types.PInteger;
-import org.apache.phoenix.util.ByteUtil;
-import org.apache.phoenix.util.PropertiesUtil;
-import org.apache.phoenix.util.QueryUtil;
-import org.apache.phoenix.util.TestUtil;
-import org.junit.Test;
-
-
-public class AggregateIT extends ParallelStatsDisabledIT {
- private static void initData(Connection conn, String tableName) throws SQLException {
- conn.createStatement().execute("create table " + tableName +
- " (id varchar not null primary key,\n" +
- " uri varchar, appcpu integer)");
- insertRow(conn, tableName, "Report1", 10, 1);
- insertRow(conn, tableName, "Report2", 10, 2);
- insertRow(conn, tableName, "Report3", 30, 3);
- insertRow(conn, tableName, "Report4", 30, 4);
- insertRow(conn, tableName, "SOQL1", 10, 5);
- insertRow(conn, tableName, "SOQL2", 10, 6);
- insertRow(conn, tableName, "SOQL3", 30, 7);
- insertRow(conn, tableName, "SOQL4", 30, 8);
- conn.commit();
- }
-
- private static void insertRow(Connection conn, String tableName, String uri, int appcpu, int id) throws SQLException {
- PreparedStatement statement = conn.prepareStatement("UPSERT INTO " + tableName + "(id, uri, appcpu) values (?,?,?)");
- statement.setString(1, "id" + id);
- statement.setString(2, uri);
- statement.setInt(3, appcpu);
- statement.executeUpdate();
- }
-
- @Test
- public void testDuplicateTrailingAggExpr() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- conn.createStatement().execute("create table " + tableName +
- " (nam VARCHAR(20), address VARCHAR(20), id BIGINT "
- + "constraint my_pk primary key (id))");
- PreparedStatement statement = conn.prepareStatement("UPSERT INTO " + tableName + "(nam, address, id) values (?,?,?)");
- statement.setString(1, "pulkit");
- statement.setString(2, "badaun");
- statement.setInt(3, 1);
- statement.executeUpdate();
- conn.commit();
- Statement stmt = conn.createStatement();
- ResultSet rs = stmt.executeQuery("select distinct 'harshit' as \"test_column\", trim(nam), trim(nam) from " + tableName);
- assertTrue(rs.next());
- assertEquals("harshit", rs.getString(1));
- assertEquals("pulkit", rs.getString(2));
- assertEquals("pulkit", rs.getString(3));
- conn.close();
- }
-
- @Test
- public void testExpressionInGroupBy() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- String ddl = " create table " + tableName + "(tgb_id integer NOT NULL,utc_date_epoch integer NOT NULL,tgb_name varchar(40),ack_success_count integer" +
- ",ack_success_one_ack_count integer, CONSTRAINT pk_tgb_counter PRIMARY KEY(tgb_id, utc_date_epoch))";
- String query = "SELECT tgb_id, tgb_name, (utc_date_epoch/10)*10 AS utc_epoch_hour,SUM(ack_success_count + ack_success_one_ack_count) AS ack_tx_sum" +
- " FROM " + tableName + " GROUP BY tgb_id, tgb_name, utc_epoch_hour";
-
- createTestTable(getUrl(), ddl);
- String dml = "UPSERT INTO " + tableName + " VALUES(?,?,?,?,?)";
- PreparedStatement stmt = conn.prepareStatement(dml);
- stmt.setInt(1, 1);
- stmt.setInt(2, 1000);
- stmt.setString(3, "aaa");
- stmt.setInt(4, 1);
- stmt.setInt(5, 1);
- stmt.execute();
- stmt.setInt(1, 2);
- stmt.setInt(2, 2000);
- stmt.setString(3, "bbb");
- stmt.setInt(4, 2);
- stmt.setInt(5, 2);
- stmt.execute();
- conn.commit();
-
- ResultSet rs = conn.createStatement().executeQuery(query);
- assertTrue(rs.next());
- assertEquals(1,rs.getInt(1));
- assertEquals("aaa",rs.getString(2));
- assertEquals(1000,rs.getInt(3));
- assertEquals(2,rs.getInt(4));
- assertTrue(rs.next());
- assertEquals(2,rs.getInt(1));
- assertEquals("bbb",rs.getString(2));
- assertEquals(2000,rs.getInt(3));
- assertEquals(4,rs.getInt(4));
- assertFalse(rs.next());
- rs.close();
- conn.close();
- }
-
- @Test
- public void testBooleanInGroupBy() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- String ddl = " create table " + tableName + "(id varchar primary key,v1 boolean, v2 integer, v3 integer)";
-
- createTestTable(getUrl(), ddl);
- PreparedStatement stmt = conn.prepareStatement("UPSERT INTO " + tableName + "(id,v2,v3) VALUES(?,?,?)");
- stmt.setString(1, "a");
- stmt.setInt(2, 1);
- stmt.setInt(3, 1);
- stmt.execute();
- stmt.close();
- stmt = conn.prepareStatement("UPSERT INTO " + tableName + " VALUES(?,?,?,?)");
- stmt.setString(1, "b");
- stmt.setBoolean(2, false);
- stmt.setInt(3, 2);
- stmt.setInt(4, 2);
- stmt.execute();
- stmt.setString(1, "c");
- stmt.setBoolean(2, true);
- stmt.setInt(3, 3);
- stmt.setInt(4, 3);
- stmt.execute();
- conn.commit();
-
- String[] gbs = {"v1,v2,v3","v1,v3,v2","v2,v1,v3"};
- for (String gb : gbs) {
- ResultSet rs = conn.createStatement().executeQuery("SELECT v1, v2, v3 from " + tableName + " group by " + gb);
- assertTrue(rs.next());
- assertEquals(false,rs.getBoolean("v1"));
- assertTrue(rs.wasNull());
- assertEquals(1,rs.getInt("v2"));
- assertEquals(1,rs.getInt("v3"));
- assertTrue(rs.next());
- assertEquals(false,rs.getBoolean("v1"));
- assertFalse(rs.wasNull());
- assertEquals(2,rs.getInt("v2"));
- assertEquals(2,rs.getInt("v3"));
- assertTrue(rs.next());
- assertEquals(true,rs.getBoolean("v1"));
- assertEquals(3,rs.getInt("v2"));
- assertEquals(3,rs.getInt("v3"));
- assertFalse(rs.next());
- rs.close();
- }
- conn.close();
- }
-
- @Test
- public void testScanUri() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- initData(conn, tableName);
- Statement stmt = conn.createStatement();
- ResultSet rs = stmt.executeQuery("select uri from " + tableName);
- assertTrue(rs.next());
- assertEquals("Report1", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("Report2", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("Report3", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("Report4", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("SOQL1", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("SOQL2", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("SOQL3", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("SOQL4", rs.getString(1));
- assertFalse(rs.next());
- conn.close();
- }
-
- @Test
- public void testCount() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- initData(conn, tableName);
- Statement stmt = conn.createStatement();
- ResultSet rs = stmt.executeQuery("select count(1) from " + tableName);
- assertTrue(rs.next());
- assertEquals(8, rs.getInt(1));
- assertFalse(rs.next());
- conn.close();
- }
-
- @Test
- public void testGroupByCase() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- String groupBy1 = "select " +
- "case when uri LIKE 'Report%' then 'Reports' else 'Other' END category" +
- ", avg(appcpu) from " + tableName +
- " group by category";
-
- String groupBy2 = "select " +
- "case uri when 'Report%' then 'Reports' else 'Other' END category" +
- ", avg(appcpu) from " + tableName +
- " group by appcpu, category";
-
- String groupBy3 = "select " +
- "case uri when 'Report%' then 'Reports' else 'Other' END category" +
- ", avg(appcpu) from " + tableName +
- " group by avg(appcpu), category";
- initData(conn, tableName);
- conn.createStatement().executeQuery(groupBy1);
- conn.createStatement().executeQuery(groupBy2);
- // TODO: validate query results
- try {
- conn.createStatement().executeQuery(groupBy3);
- fail();
- } catch (SQLException e) {
- assertTrue(e.getMessage().contains("Aggregate expressions may not be used in GROUP BY"));
- }
- conn.close();
- }
-
-
- @Test
- public void testGroupByArray() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
-
- String tableName = generateUniqueName();
- conn.createStatement().execute("CREATE TABLE " + tableName + "(\n" +
- " a VARCHAR NOT NULL,\n" +
- " b VARCHAR,\n" +
- " c INTEGER,\n" +
- " d VARCHAR,\n" +
- " e VARCHAR ARRAY,\n" +
- " f BIGINT,\n" +
- " g BIGINT,\n" +
- " CONSTRAINT pk PRIMARY KEY(a)\n" +
- ")");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('1', 'val', 100, 'a', ARRAY ['b'], 1, 2)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('2', 'val', 100, 'a', ARRAY ['b'], 3, 4)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('3', 'val', 100, 'a', ARRAY ['b','c'], 5, 6)");
- conn.commit();
-
- ResultSet rs = conn.createStatement().executeQuery("SELECT c, SUM(f + g) AS sumone, d, e\n" +
- "FROM " + tableName + "\n" +
- "WHERE b = 'val'\n" +
- " AND a IN ('1','2','3')\n" +
- "GROUP BY c, d, e\n" +
- "ORDER BY sumone DESC");
- assertTrue(rs.next());
- assertEquals(100, rs.getInt(1));
- assertEquals(11, rs.getLong(2));
- assertTrue(rs.next());
- assertEquals(100, rs.getInt(1));
- assertEquals(10, rs.getLong(2));
- assertFalse(rs.next());
- conn.close();
- }
-
- @Test
- public void testGroupByOrderPreserving() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
-
- conn.createStatement().execute("CREATE TABLE " + tableName + "(ORGANIZATION_ID char(15) not null, \n" +
- "JOURNEY_ID char(15) not null, \n" +
- "DATASOURCE SMALLINT not null, \n" +
- "MATCH_STATUS TINYINT not null, \n" +
- "EXTERNAL_DATASOURCE_KEY varchar(30), \n" +
- "ENTITY_ID char(15) not null, \n" +
- "CONSTRAINT PK PRIMARY KEY (\n" +
- " ORGANIZATION_ID, \n" +
- " JOURNEY_ID, \n" +
- " DATASOURCE, \n" +
- " MATCH_STATUS,\n" +
- " EXTERNAL_DATASOURCE_KEY,\n" +
- " ENTITY_ID))");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('000001111122222', '333334444455555', 0, 0, 'abc', '666667777788888')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('000001111122222', '333334444455555', 0, 0, 'abcd', '666667777788889')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('000001111122222', '333334444455555', 0, 0, 'abc', '666667777788899')");
- conn.commit();
- String query =
- "SELECT COUNT(1), EXTERNAL_DATASOURCE_KEY As DUP_COUNT\n" +
- " FROM " + tableName + " \n" +
- " WHERE JOURNEY_ID='333334444455555' AND \n" +
- " DATASOURCE=0 AND MATCH_STATUS <= 1 and \n" +
- " ORGANIZATION_ID='000001111122222' \n" +
- " GROUP BY MATCH_STATUS, EXTERNAL_DATASOURCE_KEY \n" +
- " HAVING COUNT(1) > 1";
- ResultSet rs = conn.createStatement().executeQuery(query);
- assertTrue(rs.next());
- assertEquals(2,rs.getInt(1));
- assertEquals("abc", rs.getString(2));
- assertFalse(rs.next());
-
- rs = conn.createStatement().executeQuery("EXPLAIN " + query);
- assertEquals(
- "CLIENT PARALLEL 1-WAY RANGE SCAN OVER " + tableName + " ['000001111122222','333334444455555',0,*] - ['000001111122222','333334444455555',0,1]\n" +
- " SERVER FILTER BY FIRST KEY ONLY\n" +
- " SERVER AGGREGATE INTO ORDERED DISTINCT ROWS BY [MATCH_STATUS, EXTERNAL_DATASOURCE_KEY]\n" +
- "CLIENT FILTER BY COUNT(1) > 1",QueryUtil.getExplainPlan(rs));
- }
-
- @Test
- public void testGroupByOrderPreservingDescSort() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- conn.createStatement().execute("CREATE TABLE " + tableName + " (k1 char(1) not null, k2 char(1) not null, constraint pk primary key (k1,k2)) split on ('ac','jc','nc')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 'a')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 'b')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 'c')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 'd')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 'a')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 'b')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 'c')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 'd')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 'a')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 'b')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 'c')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 'd')");
- conn.commit();
- String query = "SELECT k1,count(*) FROM " + tableName + " GROUP BY k1 ORDER BY k1 DESC";
- ResultSet rs = conn.createStatement().executeQuery(query);
- assertTrue(rs.next());
- assertEquals("n", rs.getString(1));
- assertEquals(4, rs.getInt(2));
- assertTrue(rs.next());
- assertEquals("j", rs.getString(1));
- assertEquals(4, rs.getInt(2));
- assertTrue(rs.next());
- assertEquals("a", rs.getString(1));
- assertEquals(4, rs.getInt(2));
- assertFalse(rs.next());
- rs = conn.createStatement().executeQuery("EXPLAIN " + query);
- assertEquals(
- "CLIENT PARALLEL 1-WAY REVERSE FULL SCAN OVER " + tableName + "\n" +
- " SERVER FILTER BY FIRST KEY ONLY\n" +
- " SERVER AGGREGATE INTO ORDERED DISTINCT ROWS BY [K1]", QueryUtil.getExplainPlan(rs));
- }
-
- @Test
- public void testSumGroupByOrderPreservingDesc() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
+import static org.apache.phoenix.util.TestUtil.assertResultSet;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
--- End diff --
nit: Looks like this diff is just due to reordering imports. Please refactor
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by ChinmaySKulkarni <gi...@git.apache.org>.
Github user ChinmaySKulkarni commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r229559932
--- Diff: phoenix-core/src/main/java/org/apache/phoenix/util/QueryBuilder.java ---
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.util;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.phoenix.parse.HintNode;
+
+import java.util.Collections;
+import java.util.List;
+
+import static org.apache.phoenix.util.SchemaUtil.getEscapedFullColumnName;
+
+public class QueryBuilder {
+
+ private String fullTableName;
+ // regular columns that are in the select clause
+ private List<String> selectColumns = Collections.emptyList();
+
+ // columns that are required for expressions in the select clause
+ private List<String> selectExpressionColumns = Collections.emptyList();
+ // expression string in the select clause (for eg COL1 || COL2)
+ private String selectExpression;
+ private String whereClause;
+ private String orderByClause;
+ private String groupByClause;
+ private String havingClause;
+ private HintNode.Hint hint;
+ private boolean escapeCols;
+ private boolean distinct;
+ private int limit;
--- End diff --
Can we make all member variables final and have the builder's build method set all values in a private constructor so that we follow the builder pattern more closely?
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by ChinmaySKulkarni <gi...@git.apache.org>.
Github user ChinmaySKulkarni commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r229563545
--- Diff: phoenix-spark/src/it/java/org/apache/phoenix/spark/OrderByIT.java ---
@@ -0,0 +1,444 @@
+package org.apache.phoenix.spark;
+
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.List;
+import java.util.Properties;
+
+import org.apache.phoenix.end2end.BaseOrderByIT;
+import org.apache.phoenix.util.PropertiesUtil;
+import org.apache.phoenix.util.QueryBuilder;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.SQLContext;
+import org.junit.Ignore;
+import org.junit.Test;
+
+import com.google.common.collect.Lists;
+
+import scala.Option;
+import scala.collection.JavaConverters;
+
+public class OrderByIT extends BaseOrderByIT {
+
+ @Override
+ protected ResultSet executeQuery(Connection conn, QueryBuilder queryBuilder) throws SQLException {
+ return SparkUtil.executeQuery(conn, queryBuilder, getUrl(), config);
--- End diff --
With more tests being added for phoenix-spark, we will be overriding this method each time. Can we have a base class with overridden methods (that use `SparkUtil` APIs) for all phoenix-spark tests instead?
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by twdsilva <gi...@git.apache.org>.
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r229898379
--- Diff: phoenix-spark/src/it/java/org/apache/phoenix/spark/AggregateIT.java ---
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.spark;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+import org.apache.phoenix.end2end.BaseAggregateIT;
+import org.apache.phoenix.util.QueryBuilder;
+
+public class AggregateIT extends BaseAggregateIT {
+
+ @Override
+ protected ResultSet executeQueryThrowsException(Connection conn, QueryBuilder queryBuilder,
+ String expectedPhoenixExceptionMsg, String expectedSparkExceptionMsg) {
+ ResultSet rs = null;
+ try {
+ rs = executeQuery(conn, queryBuilder);
+ }
+ catch(Exception e) {
+ assertTrue(e.getMessage().contains(expectedSparkExceptionMsg));
+ }
+ return rs;
+ }
+
+ @Override
+ protected ResultSet executeQuery(Connection conn, QueryBuilder queryBuilder) throws SQLException {
+ return SparkUtil.executeQuery(conn, queryBuilder, getUrl(), config);
+ }
+
+ @Override
+ protected void testCountNullInNonEmptyKeyValueCF(int columnEncodedBytes) throws Exception {
+ try (Connection conn = DriverManager.getConnection(getUrl())) {
+ //Type is INT
+ String intTableName=generateUniqueName();
+ String sql="create table " + intTableName + " (mykey integer not null primary key, A.COLA integer, B.COLB integer) "
+ + "IMMUTABLE_ROWS=true, IMMUTABLE_STORAGE_SCHEME = ONE_CELL_PER_COLUMN, COLUMN_ENCODED_BYTES = " + columnEncodedBytes + ", DISABLE_WAL=true";
+
+ conn.createStatement().execute(sql);
+ conn.createStatement().execute("UPSERT INTO "+intTableName+" VALUES (1,1)");
+ conn.createStatement().execute("UPSERT INTO "+intTableName+" VALUES (2,1)");
+ conn.createStatement().execute("UPSERT INTO "+intTableName+" VALUES (3,1,2)");
+ conn.createStatement().execute("UPSERT INTO "+intTableName+" VALUES (4,1)");
+ conn.createStatement().execute("UPSERT INTO "+intTableName+" VALUES (5,1)");
+ conn.commit();
+
+ sql="select count(*) from "+intTableName;
+ QueryBuilder queryBuilder = new QueryBuilder();
+ queryBuilder.setSelectExpression("COUNT(*)");
+ queryBuilder.setFullTableName(intTableName);
+ ResultSet rs = executeQuery(conn, queryBuilder);
+ assertTrue(rs.next());
+ assertEquals(5, rs.getLong(1));
+
+ sql="select count(*) from "+intTableName + " where b.colb is not null";
+ queryBuilder.setWhereClause("`B.COLB` IS NOT NULL");
+ rs = executeQuery(conn, queryBuilder);
+ assertTrue(rs.next());
+ assertEquals(1, rs.getLong(1));
+
+ sql="select count(*) from "+intTableName + " where b.colb is null";
+ queryBuilder.setWhereClause("`B.COLB` IS NULL");
--- End diff --
If a column name contains a dot that spark sql requires the backticks. Automatically generating the sql for this is difficult especially when columns as part of expressions etc.
---
[GitHub] phoenix issue #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and salted...
Posted by twdsilva <gi...@git.apache.org>.
Github user twdsilva commented on the issue:
https://github.com/apache/phoenix/pull/402
I also had to bump up the spark version to 2.3.2 as this version has more sql support, in order to get the tests to pass.
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by twdsilva <gi...@git.apache.org>.
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r229885734
--- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/AggregateIT.java ---
@@ -17,507 +17,29 @@
*/
package org.apache.phoenix.end2end;
-import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertTrue;
-import static org.junit.Assert.fail;
-import static org.apache.phoenix.util.TestUtil.assertResultSet;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.schema.AmbiguousColumnException;
+import org.apache.phoenix.util.PropertiesUtil;
+import org.apache.phoenix.util.QueryBuilder;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Test;
-import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
-import java.sql.Statement;
-import java.util.List;
import java.util.Properties;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.phoenix.compile.QueryPlan;
-import org.apache.phoenix.jdbc.PhoenixConnection;
-import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
-import org.apache.phoenix.jdbc.PhoenixStatement;
-import org.apache.phoenix.query.KeyRange;
-import org.apache.phoenix.schema.AmbiguousColumnException;
-import org.apache.phoenix.schema.types.PChar;
-import org.apache.phoenix.schema.types.PInteger;
-import org.apache.phoenix.util.ByteUtil;
-import org.apache.phoenix.util.PropertiesUtil;
-import org.apache.phoenix.util.QueryUtil;
-import org.apache.phoenix.util.TestUtil;
-import org.junit.Test;
-
-
-public class AggregateIT extends ParallelStatsDisabledIT {
- private static void initData(Connection conn, String tableName) throws SQLException {
- conn.createStatement().execute("create table " + tableName +
- " (id varchar not null primary key,\n" +
- " uri varchar, appcpu integer)");
- insertRow(conn, tableName, "Report1", 10, 1);
- insertRow(conn, tableName, "Report2", 10, 2);
- insertRow(conn, tableName, "Report3", 30, 3);
- insertRow(conn, tableName, "Report4", 30, 4);
- insertRow(conn, tableName, "SOQL1", 10, 5);
- insertRow(conn, tableName, "SOQL2", 10, 6);
- insertRow(conn, tableName, "SOQL3", 30, 7);
- insertRow(conn, tableName, "SOQL4", 30, 8);
- conn.commit();
- }
-
- private static void insertRow(Connection conn, String tableName, String uri, int appcpu, int id) throws SQLException {
- PreparedStatement statement = conn.prepareStatement("UPSERT INTO " + tableName + "(id, uri, appcpu) values (?,?,?)");
- statement.setString(1, "id" + id);
- statement.setString(2, uri);
- statement.setInt(3, appcpu);
- statement.executeUpdate();
- }
-
- @Test
- public void testDuplicateTrailingAggExpr() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- conn.createStatement().execute("create table " + tableName +
- " (nam VARCHAR(20), address VARCHAR(20), id BIGINT "
- + "constraint my_pk primary key (id))");
- PreparedStatement statement = conn.prepareStatement("UPSERT INTO " + tableName + "(nam, address, id) values (?,?,?)");
- statement.setString(1, "pulkit");
- statement.setString(2, "badaun");
- statement.setInt(3, 1);
- statement.executeUpdate();
- conn.commit();
- Statement stmt = conn.createStatement();
- ResultSet rs = stmt.executeQuery("select distinct 'harshit' as \"test_column\", trim(nam), trim(nam) from " + tableName);
- assertTrue(rs.next());
- assertEquals("harshit", rs.getString(1));
- assertEquals("pulkit", rs.getString(2));
- assertEquals("pulkit", rs.getString(3));
- conn.close();
- }
-
- @Test
- public void testExpressionInGroupBy() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- String ddl = " create table " + tableName + "(tgb_id integer NOT NULL,utc_date_epoch integer NOT NULL,tgb_name varchar(40),ack_success_count integer" +
- ",ack_success_one_ack_count integer, CONSTRAINT pk_tgb_counter PRIMARY KEY(tgb_id, utc_date_epoch))";
- String query = "SELECT tgb_id, tgb_name, (utc_date_epoch/10)*10 AS utc_epoch_hour,SUM(ack_success_count + ack_success_one_ack_count) AS ack_tx_sum" +
- " FROM " + tableName + " GROUP BY tgb_id, tgb_name, utc_epoch_hour";
-
- createTestTable(getUrl(), ddl);
- String dml = "UPSERT INTO " + tableName + " VALUES(?,?,?,?,?)";
- PreparedStatement stmt = conn.prepareStatement(dml);
- stmt.setInt(1, 1);
- stmt.setInt(2, 1000);
- stmt.setString(3, "aaa");
- stmt.setInt(4, 1);
- stmt.setInt(5, 1);
- stmt.execute();
- stmt.setInt(1, 2);
- stmt.setInt(2, 2000);
- stmt.setString(3, "bbb");
- stmt.setInt(4, 2);
- stmt.setInt(5, 2);
- stmt.execute();
- conn.commit();
-
- ResultSet rs = conn.createStatement().executeQuery(query);
- assertTrue(rs.next());
- assertEquals(1,rs.getInt(1));
- assertEquals("aaa",rs.getString(2));
- assertEquals(1000,rs.getInt(3));
- assertEquals(2,rs.getInt(4));
- assertTrue(rs.next());
- assertEquals(2,rs.getInt(1));
- assertEquals("bbb",rs.getString(2));
- assertEquals(2000,rs.getInt(3));
- assertEquals(4,rs.getInt(4));
- assertFalse(rs.next());
- rs.close();
- conn.close();
- }
-
- @Test
- public void testBooleanInGroupBy() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- String ddl = " create table " + tableName + "(id varchar primary key,v1 boolean, v2 integer, v3 integer)";
-
- createTestTable(getUrl(), ddl);
- PreparedStatement stmt = conn.prepareStatement("UPSERT INTO " + tableName + "(id,v2,v3) VALUES(?,?,?)");
- stmt.setString(1, "a");
- stmt.setInt(2, 1);
- stmt.setInt(3, 1);
- stmt.execute();
- stmt.close();
- stmt = conn.prepareStatement("UPSERT INTO " + tableName + " VALUES(?,?,?,?)");
- stmt.setString(1, "b");
- stmt.setBoolean(2, false);
- stmt.setInt(3, 2);
- stmt.setInt(4, 2);
- stmt.execute();
- stmt.setString(1, "c");
- stmt.setBoolean(2, true);
- stmt.setInt(3, 3);
- stmt.setInt(4, 3);
- stmt.execute();
- conn.commit();
-
- String[] gbs = {"v1,v2,v3","v1,v3,v2","v2,v1,v3"};
- for (String gb : gbs) {
- ResultSet rs = conn.createStatement().executeQuery("SELECT v1, v2, v3 from " + tableName + " group by " + gb);
- assertTrue(rs.next());
- assertEquals(false,rs.getBoolean("v1"));
- assertTrue(rs.wasNull());
- assertEquals(1,rs.getInt("v2"));
- assertEquals(1,rs.getInt("v3"));
- assertTrue(rs.next());
- assertEquals(false,rs.getBoolean("v1"));
- assertFalse(rs.wasNull());
- assertEquals(2,rs.getInt("v2"));
- assertEquals(2,rs.getInt("v3"));
- assertTrue(rs.next());
- assertEquals(true,rs.getBoolean("v1"));
- assertEquals(3,rs.getInt("v2"));
- assertEquals(3,rs.getInt("v3"));
- assertFalse(rs.next());
- rs.close();
- }
- conn.close();
- }
-
- @Test
- public void testScanUri() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- initData(conn, tableName);
- Statement stmt = conn.createStatement();
- ResultSet rs = stmt.executeQuery("select uri from " + tableName);
- assertTrue(rs.next());
- assertEquals("Report1", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("Report2", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("Report3", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("Report4", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("SOQL1", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("SOQL2", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("SOQL3", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("SOQL4", rs.getString(1));
- assertFalse(rs.next());
- conn.close();
- }
-
- @Test
- public void testCount() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- initData(conn, tableName);
- Statement stmt = conn.createStatement();
- ResultSet rs = stmt.executeQuery("select count(1) from " + tableName);
- assertTrue(rs.next());
- assertEquals(8, rs.getInt(1));
- assertFalse(rs.next());
- conn.close();
- }
-
- @Test
- public void testGroupByCase() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- String groupBy1 = "select " +
- "case when uri LIKE 'Report%' then 'Reports' else 'Other' END category" +
- ", avg(appcpu) from " + tableName +
- " group by category";
-
- String groupBy2 = "select " +
- "case uri when 'Report%' then 'Reports' else 'Other' END category" +
- ", avg(appcpu) from " + tableName +
- " group by appcpu, category";
-
- String groupBy3 = "select " +
- "case uri when 'Report%' then 'Reports' else 'Other' END category" +
- ", avg(appcpu) from " + tableName +
- " group by avg(appcpu), category";
- initData(conn, tableName);
- conn.createStatement().executeQuery(groupBy1);
- conn.createStatement().executeQuery(groupBy2);
- // TODO: validate query results
- try {
- conn.createStatement().executeQuery(groupBy3);
- fail();
- } catch (SQLException e) {
- assertTrue(e.getMessage().contains("Aggregate expressions may not be used in GROUP BY"));
- }
- conn.close();
- }
-
-
- @Test
- public void testGroupByArray() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
-
- String tableName = generateUniqueName();
- conn.createStatement().execute("CREATE TABLE " + tableName + "(\n" +
- " a VARCHAR NOT NULL,\n" +
- " b VARCHAR,\n" +
- " c INTEGER,\n" +
- " d VARCHAR,\n" +
- " e VARCHAR ARRAY,\n" +
- " f BIGINT,\n" +
- " g BIGINT,\n" +
- " CONSTRAINT pk PRIMARY KEY(a)\n" +
- ")");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('1', 'val', 100, 'a', ARRAY ['b'], 1, 2)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('2', 'val', 100, 'a', ARRAY ['b'], 3, 4)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('3', 'val', 100, 'a', ARRAY ['b','c'], 5, 6)");
- conn.commit();
-
- ResultSet rs = conn.createStatement().executeQuery("SELECT c, SUM(f + g) AS sumone, d, e\n" +
- "FROM " + tableName + "\n" +
- "WHERE b = 'val'\n" +
- " AND a IN ('1','2','3')\n" +
- "GROUP BY c, d, e\n" +
- "ORDER BY sumone DESC");
- assertTrue(rs.next());
- assertEquals(100, rs.getInt(1));
- assertEquals(11, rs.getLong(2));
- assertTrue(rs.next());
- assertEquals(100, rs.getInt(1));
- assertEquals(10, rs.getLong(2));
- assertFalse(rs.next());
- conn.close();
- }
-
- @Test
- public void testGroupByOrderPreserving() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
-
- conn.createStatement().execute("CREATE TABLE " + tableName + "(ORGANIZATION_ID char(15) not null, \n" +
- "JOURNEY_ID char(15) not null, \n" +
- "DATASOURCE SMALLINT not null, \n" +
- "MATCH_STATUS TINYINT not null, \n" +
- "EXTERNAL_DATASOURCE_KEY varchar(30), \n" +
- "ENTITY_ID char(15) not null, \n" +
- "CONSTRAINT PK PRIMARY KEY (\n" +
- " ORGANIZATION_ID, \n" +
- " JOURNEY_ID, \n" +
- " DATASOURCE, \n" +
- " MATCH_STATUS,\n" +
- " EXTERNAL_DATASOURCE_KEY,\n" +
- " ENTITY_ID))");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('000001111122222', '333334444455555', 0, 0, 'abc', '666667777788888')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('000001111122222', '333334444455555', 0, 0, 'abcd', '666667777788889')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('000001111122222', '333334444455555', 0, 0, 'abc', '666667777788899')");
- conn.commit();
- String query =
- "SELECT COUNT(1), EXTERNAL_DATASOURCE_KEY As DUP_COUNT\n" +
- " FROM " + tableName + " \n" +
- " WHERE JOURNEY_ID='333334444455555' AND \n" +
- " DATASOURCE=0 AND MATCH_STATUS <= 1 and \n" +
- " ORGANIZATION_ID='000001111122222' \n" +
- " GROUP BY MATCH_STATUS, EXTERNAL_DATASOURCE_KEY \n" +
- " HAVING COUNT(1) > 1";
- ResultSet rs = conn.createStatement().executeQuery(query);
- assertTrue(rs.next());
- assertEquals(2,rs.getInt(1));
- assertEquals("abc", rs.getString(2));
- assertFalse(rs.next());
-
- rs = conn.createStatement().executeQuery("EXPLAIN " + query);
- assertEquals(
- "CLIENT PARALLEL 1-WAY RANGE SCAN OVER " + tableName + " ['000001111122222','333334444455555',0,*] - ['000001111122222','333334444455555',0,1]\n" +
- " SERVER FILTER BY FIRST KEY ONLY\n" +
- " SERVER AGGREGATE INTO ORDERED DISTINCT ROWS BY [MATCH_STATUS, EXTERNAL_DATASOURCE_KEY]\n" +
- "CLIENT FILTER BY COUNT(1) > 1",QueryUtil.getExplainPlan(rs));
- }
-
- @Test
- public void testGroupByOrderPreservingDescSort() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- conn.createStatement().execute("CREATE TABLE " + tableName + " (k1 char(1) not null, k2 char(1) not null, constraint pk primary key (k1,k2)) split on ('ac','jc','nc')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 'a')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 'b')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 'c')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 'd')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 'a')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 'b')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 'c')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 'd')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 'a')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 'b')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 'c')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 'd')");
- conn.commit();
- String query = "SELECT k1,count(*) FROM " + tableName + " GROUP BY k1 ORDER BY k1 DESC";
- ResultSet rs = conn.createStatement().executeQuery(query);
- assertTrue(rs.next());
- assertEquals("n", rs.getString(1));
- assertEquals(4, rs.getInt(2));
- assertTrue(rs.next());
- assertEquals("j", rs.getString(1));
- assertEquals(4, rs.getInt(2));
- assertTrue(rs.next());
- assertEquals("a", rs.getString(1));
- assertEquals(4, rs.getInt(2));
- assertFalse(rs.next());
- rs = conn.createStatement().executeQuery("EXPLAIN " + query);
- assertEquals(
- "CLIENT PARALLEL 1-WAY REVERSE FULL SCAN OVER " + tableName + "\n" +
- " SERVER FILTER BY FIRST KEY ONLY\n" +
- " SERVER AGGREGATE INTO ORDERED DISTINCT ROWS BY [K1]", QueryUtil.getExplainPlan(rs));
- }
-
- @Test
- public void testSumGroupByOrderPreservingDesc() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
+import static org.apache.phoenix.util.TestUtil.assertResultSet;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
- PreparedStatement stmt = conn.prepareStatement("CREATE TABLE " + tableName + " (k1 char(1) not null, k2 integer not null, constraint pk primary key (k1,k2)) split on (?,?,?)");
- stmt.setBytes(1, ByteUtil.concat(PChar.INSTANCE.toBytes("a"), PInteger.INSTANCE.toBytes(3)));
- stmt.setBytes(2, ByteUtil.concat(PChar.INSTANCE.toBytes("j"), PInteger.INSTANCE.toBytes(3)));
- stmt.setBytes(3, ByteUtil.concat(PChar.INSTANCE.toBytes("n"), PInteger.INSTANCE.toBytes(3)));
- stmt.execute();
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 1)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 2)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 3)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 4)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('b', 5)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 1)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 2)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 3)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 4)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 1)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 2)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 3)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 4)");
- conn.commit();
- String query = "SELECT k1,sum(k2) FROM " + tableName + " GROUP BY k1 ORDER BY k1 DESC";
- ResultSet rs = conn.createStatement().executeQuery(query);
- assertTrue(rs.next());
- assertEquals("n", rs.getString(1));
- assertEquals(10, rs.getInt(2));
- assertTrue(rs.next());
- assertEquals("j", rs.getString(1));
- assertEquals(10, rs.getInt(2));
- assertTrue(rs.next());
- assertEquals("b", rs.getString(1));
- assertEquals(5, rs.getInt(2));
- assertTrue(rs.next());
- assertEquals("a", rs.getString(1));
- assertEquals(10, rs.getInt(2));
- assertFalse(rs.next());
- rs = conn.createStatement().executeQuery("EXPLAIN " + query);
- assertEquals(
- "CLIENT PARALLEL 1-WAY REVERSE FULL SCAN OVER " + tableName + "\n" +
- " SERVER FILTER BY FIRST KEY ONLY\n" +
- " SERVER AGGREGATE INTO ORDERED DISTINCT ROWS BY [K1]", QueryUtil.getExplainPlan(rs));
- }
+public class AggregateIT extends BaseAggregateIT {
- @Test
- public void testAvgGroupByOrderPreservingWithStats() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- ResultSet rs = conn.createStatement().executeQuery("SELECT COUNT(*) FROM " + "\"SYSTEM\".\"STATS\"" + " WHERE " + PhoenixDatabaseMetaData.PHYSICAL_NAME + " ='" + tableName + "'");
- assertTrue(rs.next());
- assertEquals(0,rs.getInt(1));
- initAvgGroupTable(conn, tableName, PhoenixDatabaseMetaData.GUIDE_POSTS_WIDTH + "=20 ");
- testAvgGroupByOrderPreserving(conn, tableName, 13);
- rs = conn.createStatement().executeQuery("SELECT COUNT(*) FROM " + "\"SYSTEM\".\"STATS\"" + " WHERE " + PhoenixDatabaseMetaData.PHYSICAL_NAME + " ='" + tableName + "'");
- assertTrue(rs.next());
- assertEquals(13,rs.getInt(1));
- conn.setAutoCommit(true);
- conn.createStatement().execute("DELETE FROM " + "\"SYSTEM\".\"STATS\"");
- rs = conn.createStatement().executeQuery("SELECT COUNT(*) FROM " + "\"SYSTEM\".\"STATS\"" + " WHERE " + PhoenixDatabaseMetaData.PHYSICAL_NAME + " ='" + tableName + "'");
- assertTrue(rs.next());
- assertEquals(0,rs.getInt(1));
- TestUtil.doMajorCompaction(conn, tableName);
- rs = conn.createStatement().executeQuery("SELECT COUNT(*) FROM " + "\"SYSTEM\".\"STATS\"" + " WHERE " + PhoenixDatabaseMetaData.PHYSICAL_NAME + " ='" + tableName + "'");
- assertTrue(rs.next());
- assertEquals(13,rs.getInt(1));
- testAvgGroupByOrderPreserving(conn, tableName, 13);
- conn.createStatement().execute("ALTER TABLE " + tableName + " SET " + PhoenixDatabaseMetaData.GUIDE_POSTS_WIDTH + "=100");
- testAvgGroupByOrderPreserving(conn, tableName, 6);
- conn.createStatement().execute("ALTER TABLE " + tableName + " SET " + PhoenixDatabaseMetaData.GUIDE_POSTS_WIDTH + "=null");
- testAvgGroupByOrderPreserving(conn, tableName, 4);
- }
-
- @Test
- public void testAvgGroupByOrderPreservingWithNoStats() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- initAvgGroupTable(conn, tableName, "");
- testAvgGroupByOrderPreserving(conn, tableName, 4);
- }
-
- private void initAvgGroupTable(Connection conn, String tableName, String tableProps) throws SQLException {
- PreparedStatement stmt = conn.prepareStatement("CREATE TABLE " + tableName + " (k1 char(1) not null, k2 integer not null, constraint pk primary key (k1,k2)) " + tableProps + " split on (?,?,?)");
- stmt.setBytes(1, ByteUtil.concat(PChar.INSTANCE.toBytes("a"), PInteger.INSTANCE.toBytes(3)));
- stmt.setBytes(2, ByteUtil.concat(PChar.INSTANCE.toBytes("j"), PInteger.INSTANCE.toBytes(3)));
- stmt.setBytes(3, ByteUtil.concat(PChar.INSTANCE.toBytes("n"), PInteger.INSTANCE.toBytes(3)));
- stmt.execute();
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 1)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 2)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 3)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 6)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('b', 5)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 1)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 2)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 3)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 10)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 1)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 2)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 3)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 2)");
- conn.commit();
- }
-
- private void testAvgGroupByOrderPreserving(Connection conn, String tableName, int nGuidePosts) throws SQLException, IOException {
- String query = "SELECT k1,avg(k2) FROM " + tableName + " GROUP BY k1";
- ResultSet rs = conn.createStatement().executeQuery(query);
- assertTrue(rs.next());
- assertEquals("a", rs.getString(1));
- assertEquals(3, rs.getInt(2));
- assertTrue(rs.next());
- assertEquals("b", rs.getString(1));
- assertEquals(5, rs.getInt(2));
- assertTrue(rs.next());
- assertEquals("j", rs.getString(1));
- assertEquals(4, rs.getInt(2));
- assertTrue(rs.next());
- assertEquals("n", rs.getString(1));
- assertEquals(2, rs.getInt(2));
- assertFalse(rs.next());
- rs = conn.createStatement().executeQuery("EXPLAIN " + query);
- assertEquals(
- "CLIENT PARALLEL 1-WAY FULL SCAN OVER " + tableName + "\n" +
- " SERVER FILTER BY FIRST KEY ONLY\n" +
- " SERVER AGGREGATE INTO ORDERED DISTINCT ROWS BY [K1]", QueryUtil.getExplainPlan(rs));
- TestUtil.analyzeTable(conn, tableName);
- List<KeyRange> splits = TestUtil.getAllSplits(conn, tableName);
- assertEquals(nGuidePosts, splits.size());
- }
-
@Test
public void testGroupByWithAliasWithSameColumnName() throws SQLException {
--- End diff --
The query test joins. QueryBuilder currently doesn't have support to generate a join query over two tables.
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by twdsilva <gi...@git.apache.org>.
Github user twdsilva commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r229889691
--- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/ParallelStatsDisabledIT.java ---
@@ -41,4 +53,29 @@ public static final void doSetup() throws Exception {
public static void tearDownMiniCluster() throws Exception {
BaseTest.tearDownMiniClusterIfBeyondThreshold();
}
+
+ protected ResultSet executeQuery(Connection conn, QueryBuilder queryBuilder) throws SQLException {
+ PreparedStatement statement = conn.prepareStatement(queryBuilder.build());
+ ResultSet rs = statement.executeQuery();
+ return rs;
+ }
+
+ protected ResultSet executeQueryThrowsException(Connection conn, QueryBuilder queryBuilder,
+ String expectedPhoenixExceptionMsg, String expectedSparkExceptionMsg) {
+ ResultSet rs = null;
+ try {
+ rs = executeQuery(conn, queryBuilder);
+ }
+ catch(Exception e) {
+ assertTrue(e.getMessage().contains(expectedPhoenixExceptionMsg));
+ }
+ return rs;
--- End diff --
we should fail if an exception is not thrown, I fixed this.
---
[GitHub] phoenix pull request #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and...
Posted by ChinmaySKulkarni <gi...@git.apache.org>.
Github user ChinmaySKulkarni commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/402#discussion_r229560408
--- Diff: phoenix-core/src/it/java/org/apache/phoenix/end2end/AggregateIT.java ---
@@ -17,507 +17,29 @@
*/
package org.apache.phoenix.end2end;
-import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertTrue;
-import static org.junit.Assert.fail;
-import static org.apache.phoenix.util.TestUtil.assertResultSet;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.schema.AmbiguousColumnException;
+import org.apache.phoenix.util.PropertiesUtil;
+import org.apache.phoenix.util.QueryBuilder;
+import org.apache.phoenix.util.TestUtil;
+import org.junit.Test;
-import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
-import java.sql.Statement;
-import java.util.List;
import java.util.Properties;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.phoenix.compile.QueryPlan;
-import org.apache.phoenix.jdbc.PhoenixConnection;
-import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
-import org.apache.phoenix.jdbc.PhoenixStatement;
-import org.apache.phoenix.query.KeyRange;
-import org.apache.phoenix.schema.AmbiguousColumnException;
-import org.apache.phoenix.schema.types.PChar;
-import org.apache.phoenix.schema.types.PInteger;
-import org.apache.phoenix.util.ByteUtil;
-import org.apache.phoenix.util.PropertiesUtil;
-import org.apache.phoenix.util.QueryUtil;
-import org.apache.phoenix.util.TestUtil;
-import org.junit.Test;
-
-
-public class AggregateIT extends ParallelStatsDisabledIT {
- private static void initData(Connection conn, String tableName) throws SQLException {
- conn.createStatement().execute("create table " + tableName +
- " (id varchar not null primary key,\n" +
- " uri varchar, appcpu integer)");
- insertRow(conn, tableName, "Report1", 10, 1);
- insertRow(conn, tableName, "Report2", 10, 2);
- insertRow(conn, tableName, "Report3", 30, 3);
- insertRow(conn, tableName, "Report4", 30, 4);
- insertRow(conn, tableName, "SOQL1", 10, 5);
- insertRow(conn, tableName, "SOQL2", 10, 6);
- insertRow(conn, tableName, "SOQL3", 30, 7);
- insertRow(conn, tableName, "SOQL4", 30, 8);
- conn.commit();
- }
-
- private static void insertRow(Connection conn, String tableName, String uri, int appcpu, int id) throws SQLException {
- PreparedStatement statement = conn.prepareStatement("UPSERT INTO " + tableName + "(id, uri, appcpu) values (?,?,?)");
- statement.setString(1, "id" + id);
- statement.setString(2, uri);
- statement.setInt(3, appcpu);
- statement.executeUpdate();
- }
-
- @Test
- public void testDuplicateTrailingAggExpr() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- conn.createStatement().execute("create table " + tableName +
- " (nam VARCHAR(20), address VARCHAR(20), id BIGINT "
- + "constraint my_pk primary key (id))");
- PreparedStatement statement = conn.prepareStatement("UPSERT INTO " + tableName + "(nam, address, id) values (?,?,?)");
- statement.setString(1, "pulkit");
- statement.setString(2, "badaun");
- statement.setInt(3, 1);
- statement.executeUpdate();
- conn.commit();
- Statement stmt = conn.createStatement();
- ResultSet rs = stmt.executeQuery("select distinct 'harshit' as \"test_column\", trim(nam), trim(nam) from " + tableName);
- assertTrue(rs.next());
- assertEquals("harshit", rs.getString(1));
- assertEquals("pulkit", rs.getString(2));
- assertEquals("pulkit", rs.getString(3));
- conn.close();
- }
-
- @Test
- public void testExpressionInGroupBy() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- String ddl = " create table " + tableName + "(tgb_id integer NOT NULL,utc_date_epoch integer NOT NULL,tgb_name varchar(40),ack_success_count integer" +
- ",ack_success_one_ack_count integer, CONSTRAINT pk_tgb_counter PRIMARY KEY(tgb_id, utc_date_epoch))";
- String query = "SELECT tgb_id, tgb_name, (utc_date_epoch/10)*10 AS utc_epoch_hour,SUM(ack_success_count + ack_success_one_ack_count) AS ack_tx_sum" +
- " FROM " + tableName + " GROUP BY tgb_id, tgb_name, utc_epoch_hour";
-
- createTestTable(getUrl(), ddl);
- String dml = "UPSERT INTO " + tableName + " VALUES(?,?,?,?,?)";
- PreparedStatement stmt = conn.prepareStatement(dml);
- stmt.setInt(1, 1);
- stmt.setInt(2, 1000);
- stmt.setString(3, "aaa");
- stmt.setInt(4, 1);
- stmt.setInt(5, 1);
- stmt.execute();
- stmt.setInt(1, 2);
- stmt.setInt(2, 2000);
- stmt.setString(3, "bbb");
- stmt.setInt(4, 2);
- stmt.setInt(5, 2);
- stmt.execute();
- conn.commit();
-
- ResultSet rs = conn.createStatement().executeQuery(query);
- assertTrue(rs.next());
- assertEquals(1,rs.getInt(1));
- assertEquals("aaa",rs.getString(2));
- assertEquals(1000,rs.getInt(3));
- assertEquals(2,rs.getInt(4));
- assertTrue(rs.next());
- assertEquals(2,rs.getInt(1));
- assertEquals("bbb",rs.getString(2));
- assertEquals(2000,rs.getInt(3));
- assertEquals(4,rs.getInt(4));
- assertFalse(rs.next());
- rs.close();
- conn.close();
- }
-
- @Test
- public void testBooleanInGroupBy() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- String ddl = " create table " + tableName + "(id varchar primary key,v1 boolean, v2 integer, v3 integer)";
-
- createTestTable(getUrl(), ddl);
- PreparedStatement stmt = conn.prepareStatement("UPSERT INTO " + tableName + "(id,v2,v3) VALUES(?,?,?)");
- stmt.setString(1, "a");
- stmt.setInt(2, 1);
- stmt.setInt(3, 1);
- stmt.execute();
- stmt.close();
- stmt = conn.prepareStatement("UPSERT INTO " + tableName + " VALUES(?,?,?,?)");
- stmt.setString(1, "b");
- stmt.setBoolean(2, false);
- stmt.setInt(3, 2);
- stmt.setInt(4, 2);
- stmt.execute();
- stmt.setString(1, "c");
- stmt.setBoolean(2, true);
- stmt.setInt(3, 3);
- stmt.setInt(4, 3);
- stmt.execute();
- conn.commit();
-
- String[] gbs = {"v1,v2,v3","v1,v3,v2","v2,v1,v3"};
- for (String gb : gbs) {
- ResultSet rs = conn.createStatement().executeQuery("SELECT v1, v2, v3 from " + tableName + " group by " + gb);
- assertTrue(rs.next());
- assertEquals(false,rs.getBoolean("v1"));
- assertTrue(rs.wasNull());
- assertEquals(1,rs.getInt("v2"));
- assertEquals(1,rs.getInt("v3"));
- assertTrue(rs.next());
- assertEquals(false,rs.getBoolean("v1"));
- assertFalse(rs.wasNull());
- assertEquals(2,rs.getInt("v2"));
- assertEquals(2,rs.getInt("v3"));
- assertTrue(rs.next());
- assertEquals(true,rs.getBoolean("v1"));
- assertEquals(3,rs.getInt("v2"));
- assertEquals(3,rs.getInt("v3"));
- assertFalse(rs.next());
- rs.close();
- }
- conn.close();
- }
-
- @Test
- public void testScanUri() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- initData(conn, tableName);
- Statement stmt = conn.createStatement();
- ResultSet rs = stmt.executeQuery("select uri from " + tableName);
- assertTrue(rs.next());
- assertEquals("Report1", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("Report2", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("Report3", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("Report4", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("SOQL1", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("SOQL2", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("SOQL3", rs.getString(1));
- assertTrue(rs.next());
- assertEquals("SOQL4", rs.getString(1));
- assertFalse(rs.next());
- conn.close();
- }
-
- @Test
- public void testCount() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- initData(conn, tableName);
- Statement stmt = conn.createStatement();
- ResultSet rs = stmt.executeQuery("select count(1) from " + tableName);
- assertTrue(rs.next());
- assertEquals(8, rs.getInt(1));
- assertFalse(rs.next());
- conn.close();
- }
-
- @Test
- public void testGroupByCase() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- String groupBy1 = "select " +
- "case when uri LIKE 'Report%' then 'Reports' else 'Other' END category" +
- ", avg(appcpu) from " + tableName +
- " group by category";
-
- String groupBy2 = "select " +
- "case uri when 'Report%' then 'Reports' else 'Other' END category" +
- ", avg(appcpu) from " + tableName +
- " group by appcpu, category";
-
- String groupBy3 = "select " +
- "case uri when 'Report%' then 'Reports' else 'Other' END category" +
- ", avg(appcpu) from " + tableName +
- " group by avg(appcpu), category";
- initData(conn, tableName);
- conn.createStatement().executeQuery(groupBy1);
- conn.createStatement().executeQuery(groupBy2);
- // TODO: validate query results
- try {
- conn.createStatement().executeQuery(groupBy3);
- fail();
- } catch (SQLException e) {
- assertTrue(e.getMessage().contains("Aggregate expressions may not be used in GROUP BY"));
- }
- conn.close();
- }
-
-
- @Test
- public void testGroupByArray() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
-
- String tableName = generateUniqueName();
- conn.createStatement().execute("CREATE TABLE " + tableName + "(\n" +
- " a VARCHAR NOT NULL,\n" +
- " b VARCHAR,\n" +
- " c INTEGER,\n" +
- " d VARCHAR,\n" +
- " e VARCHAR ARRAY,\n" +
- " f BIGINT,\n" +
- " g BIGINT,\n" +
- " CONSTRAINT pk PRIMARY KEY(a)\n" +
- ")");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('1', 'val', 100, 'a', ARRAY ['b'], 1, 2)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('2', 'val', 100, 'a', ARRAY ['b'], 3, 4)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('3', 'val', 100, 'a', ARRAY ['b','c'], 5, 6)");
- conn.commit();
-
- ResultSet rs = conn.createStatement().executeQuery("SELECT c, SUM(f + g) AS sumone, d, e\n" +
- "FROM " + tableName + "\n" +
- "WHERE b = 'val'\n" +
- " AND a IN ('1','2','3')\n" +
- "GROUP BY c, d, e\n" +
- "ORDER BY sumone DESC");
- assertTrue(rs.next());
- assertEquals(100, rs.getInt(1));
- assertEquals(11, rs.getLong(2));
- assertTrue(rs.next());
- assertEquals(100, rs.getInt(1));
- assertEquals(10, rs.getLong(2));
- assertFalse(rs.next());
- conn.close();
- }
-
- @Test
- public void testGroupByOrderPreserving() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
-
- conn.createStatement().execute("CREATE TABLE " + tableName + "(ORGANIZATION_ID char(15) not null, \n" +
- "JOURNEY_ID char(15) not null, \n" +
- "DATASOURCE SMALLINT not null, \n" +
- "MATCH_STATUS TINYINT not null, \n" +
- "EXTERNAL_DATASOURCE_KEY varchar(30), \n" +
- "ENTITY_ID char(15) not null, \n" +
- "CONSTRAINT PK PRIMARY KEY (\n" +
- " ORGANIZATION_ID, \n" +
- " JOURNEY_ID, \n" +
- " DATASOURCE, \n" +
- " MATCH_STATUS,\n" +
- " EXTERNAL_DATASOURCE_KEY,\n" +
- " ENTITY_ID))");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('000001111122222', '333334444455555', 0, 0, 'abc', '666667777788888')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('000001111122222', '333334444455555', 0, 0, 'abcd', '666667777788889')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('000001111122222', '333334444455555', 0, 0, 'abc', '666667777788899')");
- conn.commit();
- String query =
- "SELECT COUNT(1), EXTERNAL_DATASOURCE_KEY As DUP_COUNT\n" +
- " FROM " + tableName + " \n" +
- " WHERE JOURNEY_ID='333334444455555' AND \n" +
- " DATASOURCE=0 AND MATCH_STATUS <= 1 and \n" +
- " ORGANIZATION_ID='000001111122222' \n" +
- " GROUP BY MATCH_STATUS, EXTERNAL_DATASOURCE_KEY \n" +
- " HAVING COUNT(1) > 1";
- ResultSet rs = conn.createStatement().executeQuery(query);
- assertTrue(rs.next());
- assertEquals(2,rs.getInt(1));
- assertEquals("abc", rs.getString(2));
- assertFalse(rs.next());
-
- rs = conn.createStatement().executeQuery("EXPLAIN " + query);
- assertEquals(
- "CLIENT PARALLEL 1-WAY RANGE SCAN OVER " + tableName + " ['000001111122222','333334444455555',0,*] - ['000001111122222','333334444455555',0,1]\n" +
- " SERVER FILTER BY FIRST KEY ONLY\n" +
- " SERVER AGGREGATE INTO ORDERED DISTINCT ROWS BY [MATCH_STATUS, EXTERNAL_DATASOURCE_KEY]\n" +
- "CLIENT FILTER BY COUNT(1) > 1",QueryUtil.getExplainPlan(rs));
- }
-
- @Test
- public void testGroupByOrderPreservingDescSort() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- conn.createStatement().execute("CREATE TABLE " + tableName + " (k1 char(1) not null, k2 char(1) not null, constraint pk primary key (k1,k2)) split on ('ac','jc','nc')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 'a')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 'b')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 'c')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 'd')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 'a')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 'b')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 'c')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 'd')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 'a')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 'b')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 'c')");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 'd')");
- conn.commit();
- String query = "SELECT k1,count(*) FROM " + tableName + " GROUP BY k1 ORDER BY k1 DESC";
- ResultSet rs = conn.createStatement().executeQuery(query);
- assertTrue(rs.next());
- assertEquals("n", rs.getString(1));
- assertEquals(4, rs.getInt(2));
- assertTrue(rs.next());
- assertEquals("j", rs.getString(1));
- assertEquals(4, rs.getInt(2));
- assertTrue(rs.next());
- assertEquals("a", rs.getString(1));
- assertEquals(4, rs.getInt(2));
- assertFalse(rs.next());
- rs = conn.createStatement().executeQuery("EXPLAIN " + query);
- assertEquals(
- "CLIENT PARALLEL 1-WAY REVERSE FULL SCAN OVER " + tableName + "\n" +
- " SERVER FILTER BY FIRST KEY ONLY\n" +
- " SERVER AGGREGATE INTO ORDERED DISTINCT ROWS BY [K1]", QueryUtil.getExplainPlan(rs));
- }
-
- @Test
- public void testSumGroupByOrderPreservingDesc() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
+import static org.apache.phoenix.util.TestUtil.assertResultSet;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
- PreparedStatement stmt = conn.prepareStatement("CREATE TABLE " + tableName + " (k1 char(1) not null, k2 integer not null, constraint pk primary key (k1,k2)) split on (?,?,?)");
- stmt.setBytes(1, ByteUtil.concat(PChar.INSTANCE.toBytes("a"), PInteger.INSTANCE.toBytes(3)));
- stmt.setBytes(2, ByteUtil.concat(PChar.INSTANCE.toBytes("j"), PInteger.INSTANCE.toBytes(3)));
- stmt.setBytes(3, ByteUtil.concat(PChar.INSTANCE.toBytes("n"), PInteger.INSTANCE.toBytes(3)));
- stmt.execute();
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 1)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 2)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 3)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 4)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('b', 5)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 1)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 2)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 3)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 4)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 1)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 2)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 3)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 4)");
- conn.commit();
- String query = "SELECT k1,sum(k2) FROM " + tableName + " GROUP BY k1 ORDER BY k1 DESC";
- ResultSet rs = conn.createStatement().executeQuery(query);
- assertTrue(rs.next());
- assertEquals("n", rs.getString(1));
- assertEquals(10, rs.getInt(2));
- assertTrue(rs.next());
- assertEquals("j", rs.getString(1));
- assertEquals(10, rs.getInt(2));
- assertTrue(rs.next());
- assertEquals("b", rs.getString(1));
- assertEquals(5, rs.getInt(2));
- assertTrue(rs.next());
- assertEquals("a", rs.getString(1));
- assertEquals(10, rs.getInt(2));
- assertFalse(rs.next());
- rs = conn.createStatement().executeQuery("EXPLAIN " + query);
- assertEquals(
- "CLIENT PARALLEL 1-WAY REVERSE FULL SCAN OVER " + tableName + "\n" +
- " SERVER FILTER BY FIRST KEY ONLY\n" +
- " SERVER AGGREGATE INTO ORDERED DISTINCT ROWS BY [K1]", QueryUtil.getExplainPlan(rs));
- }
+public class AggregateIT extends BaseAggregateIT {
- @Test
- public void testAvgGroupByOrderPreservingWithStats() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- ResultSet rs = conn.createStatement().executeQuery("SELECT COUNT(*) FROM " + "\"SYSTEM\".\"STATS\"" + " WHERE " + PhoenixDatabaseMetaData.PHYSICAL_NAME + " ='" + tableName + "'");
- assertTrue(rs.next());
- assertEquals(0,rs.getInt(1));
- initAvgGroupTable(conn, tableName, PhoenixDatabaseMetaData.GUIDE_POSTS_WIDTH + "=20 ");
- testAvgGroupByOrderPreserving(conn, tableName, 13);
- rs = conn.createStatement().executeQuery("SELECT COUNT(*) FROM " + "\"SYSTEM\".\"STATS\"" + " WHERE " + PhoenixDatabaseMetaData.PHYSICAL_NAME + " ='" + tableName + "'");
- assertTrue(rs.next());
- assertEquals(13,rs.getInt(1));
- conn.setAutoCommit(true);
- conn.createStatement().execute("DELETE FROM " + "\"SYSTEM\".\"STATS\"");
- rs = conn.createStatement().executeQuery("SELECT COUNT(*) FROM " + "\"SYSTEM\".\"STATS\"" + " WHERE " + PhoenixDatabaseMetaData.PHYSICAL_NAME + " ='" + tableName + "'");
- assertTrue(rs.next());
- assertEquals(0,rs.getInt(1));
- TestUtil.doMajorCompaction(conn, tableName);
- rs = conn.createStatement().executeQuery("SELECT COUNT(*) FROM " + "\"SYSTEM\".\"STATS\"" + " WHERE " + PhoenixDatabaseMetaData.PHYSICAL_NAME + " ='" + tableName + "'");
- assertTrue(rs.next());
- assertEquals(13,rs.getInt(1));
- testAvgGroupByOrderPreserving(conn, tableName, 13);
- conn.createStatement().execute("ALTER TABLE " + tableName + " SET " + PhoenixDatabaseMetaData.GUIDE_POSTS_WIDTH + "=100");
- testAvgGroupByOrderPreserving(conn, tableName, 6);
- conn.createStatement().execute("ALTER TABLE " + tableName + " SET " + PhoenixDatabaseMetaData.GUIDE_POSTS_WIDTH + "=null");
- testAvgGroupByOrderPreserving(conn, tableName, 4);
- }
-
- @Test
- public void testAvgGroupByOrderPreservingWithNoStats() throws Exception {
- Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
- Connection conn = DriverManager.getConnection(getUrl(), props);
- String tableName = generateUniqueName();
- initAvgGroupTable(conn, tableName, "");
- testAvgGroupByOrderPreserving(conn, tableName, 4);
- }
-
- private void initAvgGroupTable(Connection conn, String tableName, String tableProps) throws SQLException {
- PreparedStatement stmt = conn.prepareStatement("CREATE TABLE " + tableName + " (k1 char(1) not null, k2 integer not null, constraint pk primary key (k1,k2)) " + tableProps + " split on (?,?,?)");
- stmt.setBytes(1, ByteUtil.concat(PChar.INSTANCE.toBytes("a"), PInteger.INSTANCE.toBytes(3)));
- stmt.setBytes(2, ByteUtil.concat(PChar.INSTANCE.toBytes("j"), PInteger.INSTANCE.toBytes(3)));
- stmt.setBytes(3, ByteUtil.concat(PChar.INSTANCE.toBytes("n"), PInteger.INSTANCE.toBytes(3)));
- stmt.execute();
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 1)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 2)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 3)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a', 6)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('b', 5)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 1)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 2)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 3)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('j', 10)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 1)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 2)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 3)");
- conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('n', 2)");
- conn.commit();
- }
-
- private void testAvgGroupByOrderPreserving(Connection conn, String tableName, int nGuidePosts) throws SQLException, IOException {
- String query = "SELECT k1,avg(k2) FROM " + tableName + " GROUP BY k1";
- ResultSet rs = conn.createStatement().executeQuery(query);
- assertTrue(rs.next());
- assertEquals("a", rs.getString(1));
- assertEquals(3, rs.getInt(2));
- assertTrue(rs.next());
- assertEquals("b", rs.getString(1));
- assertEquals(5, rs.getInt(2));
- assertTrue(rs.next());
- assertEquals("j", rs.getString(1));
- assertEquals(4, rs.getInt(2));
- assertTrue(rs.next());
- assertEquals("n", rs.getString(1));
- assertEquals(2, rs.getInt(2));
- assertFalse(rs.next());
- rs = conn.createStatement().executeQuery("EXPLAIN " + query);
- assertEquals(
- "CLIENT PARALLEL 1-WAY FULL SCAN OVER " + tableName + "\n" +
- " SERVER FILTER BY FIRST KEY ONLY\n" +
- " SERVER AGGREGATE INTO ORDERED DISTINCT ROWS BY [K1]", QueryUtil.getExplainPlan(rs));
- TestUtil.analyzeTable(conn, tableName);
- List<KeyRange> splits = TestUtil.getAllSplits(conn, tableName);
- assertEquals(nGuidePosts, splits.size());
- }
-
@Test
public void testGroupByWithAliasWithSameColumnName() throws SQLException {
--- End diff --
Why is this test case only specific to phoenix and not phoenix-spark?
---
[GitHub] phoenix issue #402: PHOENIX-4981 Add tests for ORDER BY, GROUP BY and salted...
Posted by ChinmaySKulkarni <gi...@git.apache.org>.
Github user ChinmaySKulkarni commented on the issue:
https://github.com/apache/phoenix/pull/402
@twdsilva changes look good. Do we want to continue using `SQLContext`? Documentation says that it is deprecated and that `SparkSession.builder` should be used instead (since spark 2.0.0).
---