You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tajo.apache.org by hyunsik <gi...@git.apache.org> on 2015/09/05 14:08:37 UTC
[GitHub] tajo pull request: TAJO-1816: Refactor SQL parser tests.
GitHub user hyunsik opened a pull request:
https://github.com/apache/tajo/pull/731
TAJO-1816: Refactor SQL parser tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hyunsik/tajo TAJO-1816
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/tajo/pull/731.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #731
----
commit 7fef106b0d8766708ed36d37f6ff1cf30dadc5b1
Author: Hyunsik Choi <hy...@apache.org>
Date: 2015-09-05T12:07:57Z
TAJO-1816: Refactor SQL parser tests.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1816: Refactor SQL parser tests.
Posted by blrunner <gi...@git.apache.org>.
Github user blrunner commented on the pull request:
https://github.com/apache/tajo/pull/731#issuecomment-138191057
+1
I left a few trivial comments. You can commit the patch after you reflect them if you agree with my comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1816: Refactor SQL parser tests.
Posted by blrunner <gi...@git.apache.org>.
Github user blrunner commented on a diff in the pull request:
https://github.com/apache/tajo/pull/731#discussion_r38834225
--- Diff: tajo-core-tests/src/test/java/org/apache/tajo/engine/parser/TestSQLAnalyzer.java ---
@@ -18,805 +18,192 @@
package org.apache.tajo.engine.parser;
+import com.google.common.base.Function;
+import com.google.common.base.Predicate;
+import com.google.common.collect.Lists;
import org.antlr.v4.runtime.ANTLRInputStream;
+import org.antlr.v4.runtime.BailErrorStrategy;
import org.antlr.v4.runtime.CommonTokenStream;
-import org.apache.tajo.algebra.*;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.tajo.algebra.Expr;
+import org.apache.tajo.annotation.Nullable;
+import org.apache.tajo.conf.TajoConf;
import org.apache.tajo.engine.parser.SQLParser.SqlContext;
+import org.apache.tajo.storage.StorageUtil;
import org.apache.tajo.util.FileUtil;
+import org.apache.tajo.util.Pair;
+import org.junit.Rule;
import org.junit.Test;
+import org.junit.rules.TestName;
import java.io.File;
+import java.io.FileNotFoundException;
import java.io.IOException;
-import java.util.Iterator;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.util.Collection;
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertTrue;
+import static com.google.common.collect.Collections2.filter;
+import static com.google.common.collect.Collections2.transform;
+import static org.junit.Assert.*;
/**
- * This unit tests uses a number of query files located in tajo/tajo-core/queries.
- * So, you must set tajo/tajo-core/ as the working directory.
+ * This class verifies SQLAnalyzer.
*/
public class TestSQLAnalyzer {
+ @Rule public TestName name = new TestName();
public static Expr parseQuery(String sql) {
ANTLRInputStream input = new ANTLRInputStream(sql);
SQLLexer lexer = new SQLLexer(input);
CommonTokenStream tokens = new CommonTokenStream(lexer);
+
SQLParser parser = new SQLParser(tokens);
+ parser.setErrorHandler(new BailErrorStrategy());
parser.setBuildParseTree(true);
+
SQLAnalyzer visitor = new SQLAnalyzer();
SqlContext context = parser.sql();
return visitor.visitSql(context);
}
- public void assertParseResult(String sqlFileName, String resultFileName) throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/TestSQLAnalyzer/" + sqlFileName);
- String result = FileUtil.readTextFileFromResource("results/TestSQLAnalyzer/" + resultFileName);
-
- Expr expr = parseQuery(sql);
- assertEquals(result.trim(), expr.toJson().trim());
- }
-
-
- @Test
- public void testSelect1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/select_1.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testSelect2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/select_2.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testSelect3() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/select_3.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testSelect4() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/select_4.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testSelect5() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/select_5.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testAsterisk1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/asterisk_1.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testAsterisk2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/asterisk_2.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testAsterisk3() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/asterisk_3.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testAsterisk4() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/asterisk_4.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testGroupby1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/groupby_1.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testJoin1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/join_1.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testJoin2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/join_2.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testJoin3() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/join_3.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testJoin4() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/join_4.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testJoin5() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/join_5.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testJoin6() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/join_6.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testJoin7() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/join_7.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testJoin8() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/join_8.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testJoin9() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/join_9.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testJoin10() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/join_10.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testJoin11() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/join_11.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testSet1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/set_1.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testSet2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/set_2.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testSet3() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/set_3.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testSet4() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/set_4.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testDropTable() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/drop_table.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testCreateTable1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_1.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testCreateTable2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_2.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testCreateTable3() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_3.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testCreateTable4() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_4.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testCreateTable5() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_5.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testCreateTable6() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_6.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testCreateTable7() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_7.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testCreateTable8() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_8.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testCreateTable9() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_9.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testCreateTable10() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_10.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testCreateTableLike1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_like_1.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.CreateTable, expr.getType());
- CreateTable createTable = (CreateTable) expr;
- assertEquals("orig_name", createTable.getLikeParentTableName());
- }
-
- @Test
- public void testCreateTablePartitionByHash1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_partition_by_hash_1.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.CreateTable, expr.getType());
- CreateTable createTable = (CreateTable) expr;
- assertTrue(createTable.hasPartition());
- assertEquals(CreateTable.PartitionType.HASH, createTable.getPartitionMethod().getPartitionType());
- CreateTable.HashPartition hashPartition = createTable.getPartitionMethod();
- assertEquals("col1", hashPartition.getColumns()[0].getCanonicalName());
- assertTrue(hashPartition.hasQuantifier());
- }
-
- @Test
- public void testCreateTablePartitionByHash2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_partition_by_hash_2.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.CreateTable, expr.getType());
- CreateTable createTable = (CreateTable) expr;
- assertTrue(createTable.hasPartition());
- assertEquals(CreateTable.PartitionType.HASH, createTable.getPartitionMethod().getPartitionType());
- CreateTable.HashPartition hashPartition = createTable.getPartitionMethod();
- assertEquals("col1", hashPartition.getColumns()[0].getCanonicalName());
- assertTrue(hashPartition.hasSpecifiers());
- assertEquals(3, hashPartition.getSpecifiers().size());
- }
-
- @Test
- public void testCreateTablePartitionByRange() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_partition_by_range.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.CreateTable, expr.getType());
- CreateTable createTable = (CreateTable) expr;
- assertTrue(createTable.hasPartition());
- assertEquals(CreateTable.PartitionType.RANGE, createTable.getPartitionMethod().getPartitionType());
- CreateTable.RangePartition rangePartition = createTable.getPartitionMethod();
- assertEquals("col1", rangePartition.getColumns()[0].getCanonicalName());
- assertEquals(3, rangePartition.getSpecifiers().size());
- }
-
- @Test
- public void testCreateTablePartitionByList() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_partition_by_list.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.CreateTable, expr.getType());
- CreateTable createTable = (CreateTable) expr;
- assertTrue(createTable.hasPartition());
- assertEquals(CreateTable.PartitionType.LIST, createTable.getPartitionMethod().getPartitionType());
- CreateTable.ListPartition listPartition = createTable.getPartitionMethod();
- assertEquals("col1", listPartition.getColumns()[0].getCanonicalName());
- assertEquals(2, listPartition.getSpecifiers().size());
- Iterator<CreateTable.ListPartitionSpecifier> iterator = listPartition.getSpecifiers().iterator();
- CreateTable.ListPartitionSpecifier specifier = iterator.next();
- LiteralValue value1 = (LiteralValue) specifier.getValueList().getValues()[0];
- LiteralValue value2 = (LiteralValue) specifier.getValueList().getValues()[1];
- assertEquals("Seoul", value1.getValue());
- assertEquals("서울", value2.getValue());
-
- specifier = iterator.next();
- value1 = (LiteralValue) specifier.getValueList().getValues()[0];
- value2 = (LiteralValue) specifier.getValueList().getValues()[1];
- assertEquals("Busan", value1.getValue());
- assertEquals("부산", value2.getValue());
- }
-
- @Test
- public void testCreateTablePartitionByColumn() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_partition_by_column.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.CreateTable, expr.getType());
- CreateTable createTable = (CreateTable) expr;
- assertTrue(createTable.hasPartition());
- assertEquals(CreateTable.PartitionType.COLUMN, createTable.getPartitionMethod().getPartitionType());
- CreateTable.ColumnPartition columnPartition = createTable.getPartitionMethod();
- assertEquals(3, columnPartition.getColumns().length);
- assertEquals("col3", columnPartition.getColumns()[0].getColumnName());
- assertEquals("col4", columnPartition.getColumns()[1].getColumnName());
- assertEquals("col5", columnPartition.getColumns()[2].getColumnName());
- }
-
- @Test
- public void testAlterTableAddPartition1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/alter_table_add_partition_1.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.AlterTable, expr.getType());
- AlterTable alterTable = (AlterTable)expr;
- assertEquals(alterTable.getAlterTableOpType(), AlterTableOpType.ADD_PARTITION);
- assertEquals(2, alterTable.getColumns().length);
- assertEquals(2, alterTable.getValues().length);
- assertEquals("col1", alterTable.getColumns()[0].getName());
- assertEquals("col2", alterTable.getColumns()[1].getName());
- LiteralValue value1 = (LiteralValue)alterTable.getValues()[0];
- assertEquals("1", value1.getValue());
- LiteralValue value2 = (LiteralValue)alterTable.getValues()[1];
- assertEquals("2", value2.getValue());
- assertFalse(alterTable.isIfNotExists());
- }
-
- @Test
- public void testAlterTableAddPartition2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/alter_table_add_partition_2.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.AlterTable, expr.getType());
- AlterTable alterTable = (AlterTable)expr;
- assertEquals(alterTable.getAlterTableOpType(), AlterTableOpType.ADD_PARTITION);
- assertEquals(2, alterTable.getColumns().length);
- assertEquals(2, alterTable.getValues().length);
- assertEquals("col1", alterTable.getColumns()[0].getName());
- assertEquals("col2", alterTable.getColumns()[1].getName());
- LiteralValue value1 = (LiteralValue)alterTable.getValues()[0];
- assertEquals("1", value1.getValue());
- LiteralValue value2 = (LiteralValue)alterTable.getValues()[1];
- assertEquals("2", value2.getValue());
- assertEquals(alterTable.getLocation(), "hdfs://xxx.com/warehouse/table1/col1=1/col2=2");
- assertFalse(alterTable.isIfNotExists());
- }
-
- @Test
- public void testAlterTableAddPartition3() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/alter_table_add_partition_3.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.AlterTable, expr.getType());
- AlterTable alterTable = (AlterTable)expr;
- assertEquals(alterTable.getAlterTableOpType(), AlterTableOpType.ADD_PARTITION);
- assertEquals(3, alterTable.getColumns().length);
- assertEquals(3, alterTable.getValues().length);
- assertEquals("col1", alterTable.getColumns()[0].getName());
- assertEquals("col2", alterTable.getColumns()[1].getName());
- assertEquals("col3", alterTable.getColumns()[2].getName());
- LiteralValue value1 = (LiteralValue)alterTable.getValues()[0];
- assertEquals("2015", value1.getValue());
- LiteralValue value2 = (LiteralValue)alterTable.getValues()[1];
- assertEquals("01", value2.getValue());
- LiteralValue value3 = (LiteralValue)alterTable.getValues()[2];
- assertEquals("11", value3.getValue());
- assertEquals(alterTable.getLocation(), "hdfs://xxx.com/warehouse/table1/col1=2015/col2=01/col3=11");
- assertFalse(alterTable.isIfNotExists());
- }
-
- @Test
- public void testAlterTableAddPartition4() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/alter_table_add_partition_4.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.AlterTable, expr.getType());
- AlterTable alterTable = (AlterTable)expr;
- assertEquals(alterTable.getAlterTableOpType(), AlterTableOpType.ADD_PARTITION);
- assertEquals(1, alterTable.getColumns().length);
- assertEquals(1, alterTable.getValues().length);
- assertEquals("col1", alterTable.getColumns()[0].getName());
- LiteralValue value1 = (LiteralValue)alterTable.getValues()[0];
- assertEquals("TAJO", value1.getValue());
- assertFalse(alterTable.isIfNotExists());
- }
-
- @Test
- public void testAlterTableAddPartition5() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/alter_table_add_partition_5.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.AlterTable, expr.getType());
- AlterTable alterTable = (AlterTable)expr;
- assertEquals(alterTable.getAlterTableOpType(), AlterTableOpType.ADD_PARTITION);
- assertEquals(1, alterTable.getColumns().length);
- assertEquals(1, alterTable.getValues().length);
- assertEquals("col1", alterTable.getColumns()[0].getName());
- LiteralValue value1 = (LiteralValue)alterTable.getValues()[0];
- assertEquals("TAJO", value1.getValue());
- assertTrue(alterTable.isIfNotExists());
- }
-
- @Test
- public void testAlterTableDropPartition1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/alter_table_drop_partition_1.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.AlterTable, expr.getType());
- AlterTable alterTable = (AlterTable)expr;
- assertEquals(alterTable.getAlterTableOpType(), AlterTableOpType.DROP_PARTITION);
- assertEquals(2, alterTable.getColumns().length);
- assertEquals(2, alterTable.getValues().length);
- assertEquals("col1", alterTable.getColumns()[0].getName());
- assertEquals("col2", alterTable.getColumns()[1].getName());
- LiteralValue value1 = (LiteralValue)alterTable.getValues()[0];
- assertEquals("1", value1.getValue());
- LiteralValue value2 = (LiteralValue)alterTable.getValues()[1];
- assertEquals("2", value2.getValue());
- assertFalse(alterTable.isPurge());
- assertFalse(alterTable.isIfExists());
- }
-
- @Test
- public void testAlterTableDropPartition2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/alter_table_drop_partition_2.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.AlterTable, expr.getType());
- AlterTable alterTable = (AlterTable)expr;
- assertEquals(alterTable.getAlterTableOpType(), AlterTableOpType.DROP_PARTITION);
- assertEquals(3, alterTable.getColumns().length);
- assertEquals(3, alterTable.getValues().length);
- assertEquals("col1", alterTable.getColumns()[0].getName());
- assertEquals("col2", alterTable.getColumns()[1].getName());
- assertEquals("col3", alterTable.getColumns()[2].getName());
- LiteralValue value1 = (LiteralValue)alterTable.getValues()[0];
- assertEquals("2015", value1.getValue());
- LiteralValue value2 = (LiteralValue)alterTable.getValues()[1];
- assertEquals("01", value2.getValue());
- LiteralValue value3 = (LiteralValue)alterTable.getValues()[2];
- assertEquals("11", value3.getValue());
- assertFalse(alterTable.isPurge());
- assertFalse(alterTable.isIfExists());
- }
-
- @Test
- public void testAlterTableDropPartition3() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/alter_table_drop_partition_3.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.AlterTable, expr.getType());
- AlterTable alterTable = (AlterTable)expr;
- assertEquals(alterTable.getAlterTableOpType(), AlterTableOpType.DROP_PARTITION);
- assertEquals(1, alterTable.getColumns().length);
- assertEquals(1, alterTable.getValues().length);
- assertEquals("col1", alterTable.getColumns()[0].getName());
- LiteralValue value1 = (LiteralValue)alterTable.getValues()[0];
- assertEquals("TAJO", value1.getValue());
- assertTrue(alterTable.isPurge());
- assertFalse(alterTable.isIfExists());
- }
-
- @Test
- public void testAlterTableDropPartition4() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/alter_table_drop_partition_4.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.AlterTable, expr.getType());
- AlterTable alterTable = (AlterTable)expr;
- assertEquals(alterTable.getAlterTableOpType(), AlterTableOpType.DROP_PARTITION);
- assertEquals(1, alterTable.getColumns().length);
- assertEquals(1, alterTable.getValues().length);
- assertEquals("col1", alterTable.getColumns()[0].getName());
- LiteralValue value1 = (LiteralValue)alterTable.getValues()[0];
- assertEquals("TAJO", value1.getValue());
- assertTrue(alterTable.isPurge());
- assertTrue(alterTable.isIfExists());
- }
-
- @Test
- public void testAlterTableSetProperty1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/alter_table_set_property_1.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.AlterTable, expr.getType());
- AlterTable alterTable = (AlterTable)expr;
- assertEquals(alterTable.getAlterTableOpType(), AlterTableOpType.SET_PROPERTY);
- assertTrue(alterTable.hasParams());
- assertTrue(alterTable.getParams().containsKey("timezone"));
- assertEquals("GMT-7", alterTable.getParams().get("timezone"));
- }
-
- @Test
- public void testAlterTableSetProperty2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/alter_table_set_property_2.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.AlterTable, expr.getType());
- AlterTable alterTable = (AlterTable)expr;
- assertEquals(alterTable.getAlterTableOpType(), AlterTableOpType.SET_PROPERTY);
- assertTrue(alterTable.hasParams());
- assertTrue(alterTable.getParams().containsKey("text.delimiter"));
- assertEquals("&", alterTable.getParams().get("text.delimiter"));
- }
-
- @Test
- public void testAlterTableSetProperty3() throws IOException {
- // update multiple table properties with a single 'SET PROPERTY' sql
- String sql = FileUtil.readTextFileFromResource("queries/default/alter_table_set_property_3.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.AlterTable, expr.getType());
- AlterTable alterTable = (AlterTable)expr;
- assertEquals(alterTable.getAlterTableOpType(), AlterTableOpType.SET_PROPERTY);
- assertTrue(alterTable.hasParams());
- assertTrue(alterTable.getParams().containsKey("compression.type"));
- assertEquals("RECORD", alterTable.getParams().get("compression.type"));
- assertTrue(alterTable.getParams().containsKey("compression.codec"));
- assertEquals("org.apache.hadoop.io.compress.SnappyCodec", alterTable.getParams().get("compression.codec"));
- }
-
- @Test
- public void testTableSubQuery1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/table_subquery1.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testTableSubQuery2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/table_subquery2.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testInSubquery1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/in_subquery_1.sql");
- parseQuery(sql);
- }
+ public Collection<File> getResourceFiles(String subdir) throws URISyntaxException, IOException {
+ URL uri = ClassLoader.getSystemResource("queries/TestSQLAnalyzer");
+ Path positiveQueryDir = StorageUtil.concatPath(new Path(uri.toURI()), subdir);
+ FileSystem fs = positiveQueryDir.getFileSystem(new TajoConf());
- @Test
- public void testInSubquery2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/in_subquery_2.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testExistsPredicate1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/exists_predicate_1.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testExistsPredicate2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/exists_predicate_2.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testInsertIntoTable() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/insert_into_select_1.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testInsertIntoLocation() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/insert_into_select_2.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testInsertIntoTable2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/insert_into_select_3.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testInsertOverwriteIntoTable() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/insert_overwrite_into_select_1.sql");
- parseQuery(sql);
- }
+ if (!fs.exists(positiveQueryDir)) {
+ throw new IOException("Cannot find " + positiveQueryDir);
+ }
- @Test
- public void testInsertOverwriteIntoLocation() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/insert_overwrite_into_select_2.sql");
- parseQuery(sql);
+ // get only files
+ Collection<FileStatus> files = filter(Lists.newArrayList(fs.listStatus(positiveQueryDir)),
+ new Predicate<FileStatus>() {
+ @Override
+ public boolean apply(@Nullable FileStatus input) {
+ return input.isFile();
+ }
+ }
+ );
+
+ // transform FileStatus into File
+ return transform(files, new Function<FileStatus, File>() {
+ @Override
+ public File apply(@Nullable FileStatus fileStatus) {
+ return new File(URI.create(fileStatus.getPath().toString()));
+ }
+ });
+ }
+
+ /**
+ * Return a pair of file name and SQL query
+ *
+ * @return a pair of file name and SQL query
+ * @throws IOException
+ * @throws URISyntaxException
+ */
+ public Collection<Pair<String, String>> getFileContents(String subdir) throws IOException, URISyntaxException {
+ return transform(getResourceFiles(subdir), new Function<File, Pair<String, String>>() {
+ @Override
+ public Pair<String, String> apply(@Nullable File file) {
+ try {
+ return new Pair<>(file.getName(), FileUtil.readTextFile(file));
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ });
+ }
+
+ /**
+ * In order to add more unit tests, please add SQL files into resources/results/TestSQLAnalyzer/positive.
+ * This test just checkes if SQL statements are parsed successfully.
+ *
+ * @throws IOException
+ * @throws URISyntaxException
+ */
+ @Test
+ public void testPositiveTests() throws IOException, URISyntaxException {
+ for (Pair<String, String> pair : getFileContents("positive")) {
+ try {
+ assertNotNull(parseQuery(pair.getSecond()));
+ System.out.println(pair.getFirst() + " test passed...");
--- End diff --
How about use log4j instead of PrintStream?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1816: Refactor SQL parser tests.
Posted by blrunner <gi...@git.apache.org>.
Github user blrunner commented on a diff in the pull request:
https://github.com/apache/tajo/pull/731#discussion_r38834296
--- Diff: tajo-core-tests/src/test/java/org/apache/tajo/engine/parser/TestSQLAnalyzer.java ---
@@ -18,805 +18,192 @@
package org.apache.tajo.engine.parser;
+import com.google.common.base.Function;
+import com.google.common.base.Predicate;
+import com.google.common.collect.Lists;
import org.antlr.v4.runtime.ANTLRInputStream;
+import org.antlr.v4.runtime.BailErrorStrategy;
import org.antlr.v4.runtime.CommonTokenStream;
-import org.apache.tajo.algebra.*;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.tajo.algebra.Expr;
+import org.apache.tajo.annotation.Nullable;
+import org.apache.tajo.conf.TajoConf;
import org.apache.tajo.engine.parser.SQLParser.SqlContext;
+import org.apache.tajo.storage.StorageUtil;
import org.apache.tajo.util.FileUtil;
+import org.apache.tajo.util.Pair;
+import org.junit.Rule;
import org.junit.Test;
+import org.junit.rules.TestName;
import java.io.File;
+import java.io.FileNotFoundException;
import java.io.IOException;
-import java.util.Iterator;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.util.Collection;
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertTrue;
+import static com.google.common.collect.Collections2.filter;
+import static com.google.common.collect.Collections2.transform;
+import static org.junit.Assert.*;
/**
- * This unit tests uses a number of query files located in tajo/tajo-core/queries.
- * So, you must set tajo/tajo-core/ as the working directory.
+ * This class verifies SQLAnalyzer.
*/
public class TestSQLAnalyzer {
+ @Rule public TestName name = new TestName();
public static Expr parseQuery(String sql) {
ANTLRInputStream input = new ANTLRInputStream(sql);
SQLLexer lexer = new SQLLexer(input);
CommonTokenStream tokens = new CommonTokenStream(lexer);
+
SQLParser parser = new SQLParser(tokens);
+ parser.setErrorHandler(new BailErrorStrategy());
parser.setBuildParseTree(true);
+
SQLAnalyzer visitor = new SQLAnalyzer();
SqlContext context = parser.sql();
return visitor.visitSql(context);
}
- public void assertParseResult(String sqlFileName, String resultFileName) throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/TestSQLAnalyzer/" + sqlFileName);
- String result = FileUtil.readTextFileFromResource("results/TestSQLAnalyzer/" + resultFileName);
-
- Expr expr = parseQuery(sql);
- assertEquals(result.trim(), expr.toJson().trim());
- }
-
-
- @Test
- public void testSelect1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/select_1.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testSelect2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/select_2.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testSelect3() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/select_3.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testSelect4() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/select_4.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testSelect5() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/select_5.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testAsterisk1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/asterisk_1.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testAsterisk2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/asterisk_2.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testAsterisk3() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/asterisk_3.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testAsterisk4() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/asterisk_4.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testGroupby1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/groupby_1.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testJoin1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/join_1.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testJoin2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/join_2.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testJoin3() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/join_3.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testJoin4() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/join_4.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testJoin5() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/join_5.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testJoin6() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/join_6.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testJoin7() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/join_7.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testJoin8() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/join_8.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testJoin9() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/join_9.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testJoin10() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/join_10.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testJoin11() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/join_11.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testSet1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/set_1.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testSet2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/set_2.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testSet3() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/set_3.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testSet4() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/set_4.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testDropTable() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/drop_table.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testCreateTable1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_1.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testCreateTable2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_2.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testCreateTable3() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_3.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testCreateTable4() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_4.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testCreateTable5() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_5.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testCreateTable6() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_6.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testCreateTable7() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_7.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testCreateTable8() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_8.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testCreateTable9() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_9.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testCreateTable10() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_10.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testCreateTableLike1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_like_1.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.CreateTable, expr.getType());
- CreateTable createTable = (CreateTable) expr;
- assertEquals("orig_name", createTable.getLikeParentTableName());
- }
-
- @Test
- public void testCreateTablePartitionByHash1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_partition_by_hash_1.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.CreateTable, expr.getType());
- CreateTable createTable = (CreateTable) expr;
- assertTrue(createTable.hasPartition());
- assertEquals(CreateTable.PartitionType.HASH, createTable.getPartitionMethod().getPartitionType());
- CreateTable.HashPartition hashPartition = createTable.getPartitionMethod();
- assertEquals("col1", hashPartition.getColumns()[0].getCanonicalName());
- assertTrue(hashPartition.hasQuantifier());
- }
-
- @Test
- public void testCreateTablePartitionByHash2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_partition_by_hash_2.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.CreateTable, expr.getType());
- CreateTable createTable = (CreateTable) expr;
- assertTrue(createTable.hasPartition());
- assertEquals(CreateTable.PartitionType.HASH, createTable.getPartitionMethod().getPartitionType());
- CreateTable.HashPartition hashPartition = createTable.getPartitionMethod();
- assertEquals("col1", hashPartition.getColumns()[0].getCanonicalName());
- assertTrue(hashPartition.hasSpecifiers());
- assertEquals(3, hashPartition.getSpecifiers().size());
- }
-
- @Test
- public void testCreateTablePartitionByRange() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_partition_by_range.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.CreateTable, expr.getType());
- CreateTable createTable = (CreateTable) expr;
- assertTrue(createTable.hasPartition());
- assertEquals(CreateTable.PartitionType.RANGE, createTable.getPartitionMethod().getPartitionType());
- CreateTable.RangePartition rangePartition = createTable.getPartitionMethod();
- assertEquals("col1", rangePartition.getColumns()[0].getCanonicalName());
- assertEquals(3, rangePartition.getSpecifiers().size());
- }
-
- @Test
- public void testCreateTablePartitionByList() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_partition_by_list.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.CreateTable, expr.getType());
- CreateTable createTable = (CreateTable) expr;
- assertTrue(createTable.hasPartition());
- assertEquals(CreateTable.PartitionType.LIST, createTable.getPartitionMethod().getPartitionType());
- CreateTable.ListPartition listPartition = createTable.getPartitionMethod();
- assertEquals("col1", listPartition.getColumns()[0].getCanonicalName());
- assertEquals(2, listPartition.getSpecifiers().size());
- Iterator<CreateTable.ListPartitionSpecifier> iterator = listPartition.getSpecifiers().iterator();
- CreateTable.ListPartitionSpecifier specifier = iterator.next();
- LiteralValue value1 = (LiteralValue) specifier.getValueList().getValues()[0];
- LiteralValue value2 = (LiteralValue) specifier.getValueList().getValues()[1];
- assertEquals("Seoul", value1.getValue());
- assertEquals("서울", value2.getValue());
-
- specifier = iterator.next();
- value1 = (LiteralValue) specifier.getValueList().getValues()[0];
- value2 = (LiteralValue) specifier.getValueList().getValues()[1];
- assertEquals("Busan", value1.getValue());
- assertEquals("부산", value2.getValue());
- }
-
- @Test
- public void testCreateTablePartitionByColumn() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/create_table_partition_by_column.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.CreateTable, expr.getType());
- CreateTable createTable = (CreateTable) expr;
- assertTrue(createTable.hasPartition());
- assertEquals(CreateTable.PartitionType.COLUMN, createTable.getPartitionMethod().getPartitionType());
- CreateTable.ColumnPartition columnPartition = createTable.getPartitionMethod();
- assertEquals(3, columnPartition.getColumns().length);
- assertEquals("col3", columnPartition.getColumns()[0].getColumnName());
- assertEquals("col4", columnPartition.getColumns()[1].getColumnName());
- assertEquals("col5", columnPartition.getColumns()[2].getColumnName());
- }
-
- @Test
- public void testAlterTableAddPartition1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/alter_table_add_partition_1.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.AlterTable, expr.getType());
- AlterTable alterTable = (AlterTable)expr;
- assertEquals(alterTable.getAlterTableOpType(), AlterTableOpType.ADD_PARTITION);
- assertEquals(2, alterTable.getColumns().length);
- assertEquals(2, alterTable.getValues().length);
- assertEquals("col1", alterTable.getColumns()[0].getName());
- assertEquals("col2", alterTable.getColumns()[1].getName());
- LiteralValue value1 = (LiteralValue)alterTable.getValues()[0];
- assertEquals("1", value1.getValue());
- LiteralValue value2 = (LiteralValue)alterTable.getValues()[1];
- assertEquals("2", value2.getValue());
- assertFalse(alterTable.isIfNotExists());
- }
-
- @Test
- public void testAlterTableAddPartition2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/alter_table_add_partition_2.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.AlterTable, expr.getType());
- AlterTable alterTable = (AlterTable)expr;
- assertEquals(alterTable.getAlterTableOpType(), AlterTableOpType.ADD_PARTITION);
- assertEquals(2, alterTable.getColumns().length);
- assertEquals(2, alterTable.getValues().length);
- assertEquals("col1", alterTable.getColumns()[0].getName());
- assertEquals("col2", alterTable.getColumns()[1].getName());
- LiteralValue value1 = (LiteralValue)alterTable.getValues()[0];
- assertEquals("1", value1.getValue());
- LiteralValue value2 = (LiteralValue)alterTable.getValues()[1];
- assertEquals("2", value2.getValue());
- assertEquals(alterTable.getLocation(), "hdfs://xxx.com/warehouse/table1/col1=1/col2=2");
- assertFalse(alterTable.isIfNotExists());
- }
-
- @Test
- public void testAlterTableAddPartition3() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/alter_table_add_partition_3.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.AlterTable, expr.getType());
- AlterTable alterTable = (AlterTable)expr;
- assertEquals(alterTable.getAlterTableOpType(), AlterTableOpType.ADD_PARTITION);
- assertEquals(3, alterTable.getColumns().length);
- assertEquals(3, alterTable.getValues().length);
- assertEquals("col1", alterTable.getColumns()[0].getName());
- assertEquals("col2", alterTable.getColumns()[1].getName());
- assertEquals("col3", alterTable.getColumns()[2].getName());
- LiteralValue value1 = (LiteralValue)alterTable.getValues()[0];
- assertEquals("2015", value1.getValue());
- LiteralValue value2 = (LiteralValue)alterTable.getValues()[1];
- assertEquals("01", value2.getValue());
- LiteralValue value3 = (LiteralValue)alterTable.getValues()[2];
- assertEquals("11", value3.getValue());
- assertEquals(alterTable.getLocation(), "hdfs://xxx.com/warehouse/table1/col1=2015/col2=01/col3=11");
- assertFalse(alterTable.isIfNotExists());
- }
-
- @Test
- public void testAlterTableAddPartition4() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/alter_table_add_partition_4.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.AlterTable, expr.getType());
- AlterTable alterTable = (AlterTable)expr;
- assertEquals(alterTable.getAlterTableOpType(), AlterTableOpType.ADD_PARTITION);
- assertEquals(1, alterTable.getColumns().length);
- assertEquals(1, alterTable.getValues().length);
- assertEquals("col1", alterTable.getColumns()[0].getName());
- LiteralValue value1 = (LiteralValue)alterTable.getValues()[0];
- assertEquals("TAJO", value1.getValue());
- assertFalse(alterTable.isIfNotExists());
- }
-
- @Test
- public void testAlterTableAddPartition5() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/alter_table_add_partition_5.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.AlterTable, expr.getType());
- AlterTable alterTable = (AlterTable)expr;
- assertEquals(alterTable.getAlterTableOpType(), AlterTableOpType.ADD_PARTITION);
- assertEquals(1, alterTable.getColumns().length);
- assertEquals(1, alterTable.getValues().length);
- assertEquals("col1", alterTable.getColumns()[0].getName());
- LiteralValue value1 = (LiteralValue)alterTable.getValues()[0];
- assertEquals("TAJO", value1.getValue());
- assertTrue(alterTable.isIfNotExists());
- }
-
- @Test
- public void testAlterTableDropPartition1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/alter_table_drop_partition_1.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.AlterTable, expr.getType());
- AlterTable alterTable = (AlterTable)expr;
- assertEquals(alterTable.getAlterTableOpType(), AlterTableOpType.DROP_PARTITION);
- assertEquals(2, alterTable.getColumns().length);
- assertEquals(2, alterTable.getValues().length);
- assertEquals("col1", alterTable.getColumns()[0].getName());
- assertEquals("col2", alterTable.getColumns()[1].getName());
- LiteralValue value1 = (LiteralValue)alterTable.getValues()[0];
- assertEquals("1", value1.getValue());
- LiteralValue value2 = (LiteralValue)alterTable.getValues()[1];
- assertEquals("2", value2.getValue());
- assertFalse(alterTable.isPurge());
- assertFalse(alterTable.isIfExists());
- }
-
- @Test
- public void testAlterTableDropPartition2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/alter_table_drop_partition_2.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.AlterTable, expr.getType());
- AlterTable alterTable = (AlterTable)expr;
- assertEquals(alterTable.getAlterTableOpType(), AlterTableOpType.DROP_PARTITION);
- assertEquals(3, alterTable.getColumns().length);
- assertEquals(3, alterTable.getValues().length);
- assertEquals("col1", alterTable.getColumns()[0].getName());
- assertEquals("col2", alterTable.getColumns()[1].getName());
- assertEquals("col3", alterTable.getColumns()[2].getName());
- LiteralValue value1 = (LiteralValue)alterTable.getValues()[0];
- assertEquals("2015", value1.getValue());
- LiteralValue value2 = (LiteralValue)alterTable.getValues()[1];
- assertEquals("01", value2.getValue());
- LiteralValue value3 = (LiteralValue)alterTable.getValues()[2];
- assertEquals("11", value3.getValue());
- assertFalse(alterTable.isPurge());
- assertFalse(alterTable.isIfExists());
- }
-
- @Test
- public void testAlterTableDropPartition3() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/alter_table_drop_partition_3.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.AlterTable, expr.getType());
- AlterTable alterTable = (AlterTable)expr;
- assertEquals(alterTable.getAlterTableOpType(), AlterTableOpType.DROP_PARTITION);
- assertEquals(1, alterTable.getColumns().length);
- assertEquals(1, alterTable.getValues().length);
- assertEquals("col1", alterTable.getColumns()[0].getName());
- LiteralValue value1 = (LiteralValue)alterTable.getValues()[0];
- assertEquals("TAJO", value1.getValue());
- assertTrue(alterTable.isPurge());
- assertFalse(alterTable.isIfExists());
- }
-
- @Test
- public void testAlterTableDropPartition4() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/alter_table_drop_partition_4.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.AlterTable, expr.getType());
- AlterTable alterTable = (AlterTable)expr;
- assertEquals(alterTable.getAlterTableOpType(), AlterTableOpType.DROP_PARTITION);
- assertEquals(1, alterTable.getColumns().length);
- assertEquals(1, alterTable.getValues().length);
- assertEquals("col1", alterTable.getColumns()[0].getName());
- LiteralValue value1 = (LiteralValue)alterTable.getValues()[0];
- assertEquals("TAJO", value1.getValue());
- assertTrue(alterTable.isPurge());
- assertTrue(alterTable.isIfExists());
- }
-
- @Test
- public void testAlterTableSetProperty1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/alter_table_set_property_1.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.AlterTable, expr.getType());
- AlterTable alterTable = (AlterTable)expr;
- assertEquals(alterTable.getAlterTableOpType(), AlterTableOpType.SET_PROPERTY);
- assertTrue(alterTable.hasParams());
- assertTrue(alterTable.getParams().containsKey("timezone"));
- assertEquals("GMT-7", alterTable.getParams().get("timezone"));
- }
-
- @Test
- public void testAlterTableSetProperty2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/alter_table_set_property_2.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.AlterTable, expr.getType());
- AlterTable alterTable = (AlterTable)expr;
- assertEquals(alterTable.getAlterTableOpType(), AlterTableOpType.SET_PROPERTY);
- assertTrue(alterTable.hasParams());
- assertTrue(alterTable.getParams().containsKey("text.delimiter"));
- assertEquals("&", alterTable.getParams().get("text.delimiter"));
- }
-
- @Test
- public void testAlterTableSetProperty3() throws IOException {
- // update multiple table properties with a single 'SET PROPERTY' sql
- String sql = FileUtil.readTextFileFromResource("queries/default/alter_table_set_property_3.sql");
- Expr expr = parseQuery(sql);
- assertEquals(OpType.AlterTable, expr.getType());
- AlterTable alterTable = (AlterTable)expr;
- assertEquals(alterTable.getAlterTableOpType(), AlterTableOpType.SET_PROPERTY);
- assertTrue(alterTable.hasParams());
- assertTrue(alterTable.getParams().containsKey("compression.type"));
- assertEquals("RECORD", alterTable.getParams().get("compression.type"));
- assertTrue(alterTable.getParams().containsKey("compression.codec"));
- assertEquals("org.apache.hadoop.io.compress.SnappyCodec", alterTable.getParams().get("compression.codec"));
- }
-
- @Test
- public void testTableSubQuery1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/table_subquery1.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testTableSubQuery2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/table_subquery2.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testInSubquery1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/in_subquery_1.sql");
- parseQuery(sql);
- }
+ public Collection<File> getResourceFiles(String subdir) throws URISyntaxException, IOException {
+ URL uri = ClassLoader.getSystemResource("queries/TestSQLAnalyzer");
+ Path positiveQueryDir = StorageUtil.concatPath(new Path(uri.toURI()), subdir);
+ FileSystem fs = positiveQueryDir.getFileSystem(new TajoConf());
- @Test
- public void testInSubquery2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/in_subquery_2.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testExistsPredicate1() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/exists_predicate_1.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testExistsPredicate2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/exists_predicate_2.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testInsertIntoTable() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/insert_into_select_1.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testInsertIntoLocation() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/insert_into_select_2.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testInsertIntoTable2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/insert_into_select_3.sql");
- parseQuery(sql);
- }
-
- @Test
- public void testInsertOverwriteIntoTable() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/insert_overwrite_into_select_1.sql");
- parseQuery(sql);
- }
+ if (!fs.exists(positiveQueryDir)) {
+ throw new IOException("Cannot find " + positiveQueryDir);
+ }
- @Test
- public void testInsertOverwriteIntoLocation() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/insert_overwrite_into_select_2.sql");
- parseQuery(sql);
+ // get only files
+ Collection<FileStatus> files = filter(Lists.newArrayList(fs.listStatus(positiveQueryDir)),
+ new Predicate<FileStatus>() {
+ @Override
+ public boolean apply(@Nullable FileStatus input) {
+ return input.isFile();
+ }
+ }
+ );
+
+ // transform FileStatus into File
+ return transform(files, new Function<FileStatus, File>() {
+ @Override
+ public File apply(@Nullable FileStatus fileStatus) {
+ return new File(URI.create(fileStatus.getPath().toString()));
+ }
+ });
+ }
+
+ /**
+ * Return a pair of file name and SQL query
+ *
+ * @return a pair of file name and SQL query
+ * @throws IOException
+ * @throws URISyntaxException
+ */
+ public Collection<Pair<String, String>> getFileContents(String subdir) throws IOException, URISyntaxException {
+ return transform(getResourceFiles(subdir), new Function<File, Pair<String, String>>() {
+ @Override
+ public Pair<String, String> apply(@Nullable File file) {
+ try {
+ return new Pair<>(file.getName(), FileUtil.readTextFile(file));
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ });
+ }
+
+ /**
+ * In order to add more unit tests, please add SQL files into resources/results/TestSQLAnalyzer/positive.
+ * This test just checkes if SQL statements are parsed successfully.
+ *
+ * @throws IOException
+ * @throws URISyntaxException
+ */
+ @Test
+ public void testPositiveTests() throws IOException, URISyntaxException {
+ for (Pair<String, String> pair : getFileContents("positive")) {
+ try {
+ assertNotNull(parseQuery(pair.getSecond()));
+ System.out.println(pair.getFirst() + " test passed...");
+ } catch (Throwable t) {
+ fail("Parsing '" + pair.getFirst() + "' failed, its cause: " + t.getMessage());
+ }
+ }
}
- @Test
- public void testInsertOverwriteIntoTable2() throws IOException {
- String sql = FileUtil.readTextFileFromResource("queries/default/insert_overwrite_into_select_3.sql");
- parseQuery(sql);
+ /**
+ * In order to add more unit tests, add a SQL file including a SQL statement
+ * into the directory resources/queries/TestSQLAnalyzer
+ * and its generated algebraic expression formatted by json into resources/results/TestSQLAnalyzer.
+ * Each result file name must be the same to its SQL file.
+ *
+ * @throws IOException
+ * @throws URISyntaxException
+ */
+ @Test
+ public void testGeneratedAlgebras() throws IOException, URISyntaxException {
+ for (Pair<String, String> pair : getFileContents(".")) {
+ Expr expr = parseQuery(pair.getSecond());
+
+ String expectedResult = null;
+ String fileName = null;
+ try {
+ fileName = pair.getFirst().split("\\.")[0];
+ expectedResult = FileUtil.readTextFileFromResource("results/TestSQLAnalyzer/" + fileName + ".result");
+ } catch (FileNotFoundException ioe) {
+ expectedResult = "";
+ } catch (Throwable t) {
+ fail(t.getMessage());
+ }
+
+ assertEquals(pair.getFirst() + " is different from " + fileName + ".result",
+ expectedResult.trim(), expr.toJson().trim());
+ System.out.println(pair.getFirst() + " test passed..");
--- End diff --
It would be better if log4j is used instead of PrintStream.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1816: Refactor SQL parser tests.
Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on the pull request:
https://github.com/apache/tajo/pull/731#issuecomment-138196834
Thank you for the quick review. In my opinion, these prints will not be problematic because there are no concerns about log level or buffer flush problem of System.out. I'll commit it soon. Thank you for the quick review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1816: Refactor SQL parser tests.
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/tajo/pull/731
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---