You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tajo.apache.org by bl...@apache.org on 2015/04/17 06:26:03 UTC
[1/4] tajo git commit: TAJO-1442: Improve Hive Compatibility
Repository: tajo
Updated Branches:
refs/heads/master 7b78668b7 -> 955a7bf84
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-docs/src/main/sphinx/configuration/catalog_configuration.rst
----------------------------------------------------------------------
diff --git a/tajo-docs/src/main/sphinx/configuration/catalog_configuration.rst b/tajo-docs/src/main/sphinx/configuration/catalog_configuration.rst
index c629d9e..bb88720 100644
--- a/tajo-docs/src/main/sphinx/configuration/catalog_configuration.rst
+++ b/tajo-docs/src/main/sphinx/configuration/catalog_configuration.rst
@@ -7,21 +7,21 @@ If you want to customize the catalog service, copy ``$TAJO_HOME/conf/catalog-sit
* tajo.catalog.master.addr - If you want to launch a Tajo cluster in distributed mode, you must specify this address. For more detail information, see [Default Ports](#DefaultPorts).
* tajo.catalog.store.class - If you want to change the persistent storage of the catalog server, specify the class name. Its default value is tajo.catalog.store.DerbyStore. In the current version, Tajo provides three persistent storage classes as follows:
-+-----------------------------------+------------------------------------------------+
-| Driver Class | Descriptions |
-+===================================+================================================+
-| tajo.catalog.store.DerbyStore | this storage class uses Apache Derby. |
-+-----------------------------------+------------------------------------------------+
-| tajo.catalog.store.MySQLStore | this storage class uses MySQL. |
-+-----------------------------------+------------------------------------------------+
-| tajo.catalog.store.MariaDBStore | this storage class uses MariaDB. |
-+-----------------------------------+------------------------------------------------+
-| tajo.catalog.store.MemStore | this is the in-memory storage. It is only used |
-| | in unit tests to shorten the duration of unit |
-| | tests. |
-+-----------------------------------+------------------------------------------------+
-| tajo.catalog.store.HCatalogStore | this storage class uses HiveMetaStore. |
-+-----------------------------------+------------------------------------------------+
++--------------------------------------+------------------------------------------------+
+| Driver Class | Descriptions |
++======================================+================================================+
+| tajo.catalog.store.DerbyStore | this storage class uses Apache Derby. |
++--------------------------------------+------------------------------------------------+
+| tajo.catalog.store.MySQLStore | this storage class uses MySQL. |
++--------------------------------------+------------------------------------------------+
+| tajo.catalog.store.MariaDBStore | this storage class uses MariaDB. |
++--------------------------------------+------------------------------------------------+
+| tajo.catalog.store.MemStore | this is the in-memory storage. It is only used |
+| | in unit tests to shorten the duration of unit |
+| | tests. |
++--------------------------------------+------------------------------------------------+
+| tajo.catalog.store.HiveCatalogStore | this storage class uses HiveMetaStore. |
++--------------------------------------+------------------------------------------------+
=========================
Derby Configuration
@@ -148,21 +148,19 @@ Finally, you must add the following configurations to `conf/catalog-site.xml` :
</property>
==================================
-HCatalogStore Configuration
+HiveCatalogStore Configuration
==================================
-Tajo support HCatalogStore to integrate with hive. If you want to use HCatalogStore, you just do as follows.
+Tajo support HiveCatalogStore to integrate with hive. If you want to use HiveCatalogStore, you just do as follows.
First, you must compile source code and get a binary archive as follows:
.. code-block:: sh
$ git clone https://git-wip-us.apache.org/repos/asf/tajo.git tajo
- $ mvn clean install -DskipTests -Pdist -Dtar -Phcatalog-0.1x.0
+ $ mvn clean install -DskipTests -Pdist -Dtar
$ ls tajo-dist/target/tajo-x.y.z-SNAPSHOT.tar.gz
-Currently Tajo supports hive 0.12.0, hive 0.13.0, hive 0.13.1. If you enables HCatalogStore, you set the maven profile as ``-Phcatalog-0.12.0``.
-
Second, you must set your hive home directory to HIVE_HOME variable in ``conf/tajo-env.sh`` with it as follows:
.. code-block:: sh
@@ -182,5 +180,5 @@ Lastly, you should add the following config to ``conf/catalog-site.xml`` :
<property>
<name>tajo.catalog.store.class</name>
- <value>org.apache.tajo.catalog.store.HCatalogStore</value>
+ <value>org.apache.tajo.catalog.store.HiveCatalogStore</value>
</property>
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-docs/src/main/sphinx/hcatalog_integration.rst
----------------------------------------------------------------------
diff --git a/tajo-docs/src/main/sphinx/hcatalog_integration.rst b/tajo-docs/src/main/sphinx/hcatalog_integration.rst
deleted file mode 100644
index d81975d..0000000
--- a/tajo-docs/src/main/sphinx/hcatalog_integration.rst
+++ /dev/null
@@ -1,52 +0,0 @@
-*************************************
-HCatalog Integration
-*************************************
-
-Apache Tajo™ catalog supports HCatalogStore driver to integrate with Apache Hive™.
-This integration allows Tajo to access all tables used in Apache Hive.
-Depending on your purpose, you can execute either SQL queries or HiveQL queries on the
-same tables managed in Apache Hive.
-
-In order to use this feature, you need to build Tajo with a specified maven profile
-and then add some configs into ``conf/tajo-env.sh`` and ``conf/catalog-site.xml``.
-This section describes how to setup HCatalog integration.
-This instruction would take no more than ten minutes.
-
-First, you need to compile the source code with hcatalog profile.
-Currently, Tajo supports hcatalog-0.11.0 and hcatalog-0.12.0 profile.
-So, if you want to use Hive 0.11.0, you need to set ``-Phcatalog-0.11.0`` as the maven profile ::
-
- $ mvn clean package -DskipTests -Pdist -Dtar -Phcatalog-0.11.0
-
-Or, if you want to use Hive 0.12.0, you need to set ``-Phcatalog-0.12.0`` as the maven profile ::
-
- $ mvn clean package -DskipTests -Pdist -Dtar -Phcatalog-0.12.0
-
-Then, you need to set your Hive home directory to the environment variable ``HIVE_HOME`` in conf/tajo-env.sh as follows: ::
-
- export HIVE_HOME=/path/to/your/hive/directory
-
-If you need to use jdbc to connect HiveMetaStore, you have to prepare MySQL jdbc driver.
-Next, you should set the path of MySQL JDBC driver jar file to the environment variable HIVE_JDBC_DRIVER_DIR in conf/tajo-env.sh as follows: ::
-
- export HIVE_JDBC_DRIVER_DIR==/path/to/your/mysql_jdbc_driver/mysql-connector-java-x.x.x-bin.jar
-
-Finally, you should specify HCatalogStore as Tajo catalog driver class in ``conf/catalog-site.xml`` as follows: ::
-
- <property>
- <name>tajo.catalog.store.class</name>
- <value>org.apache.tajo.catalog.store.HCatalogStore</value>
- </property>
-
-.. note::
-
- Hive stores a list of partitions for each table in its metastore. If new partitions are
- directly added to HDFS, HiveMetastore will not able aware of these partitions unless the user
- ``ALTER TABLE table_name ADD PARTITION`` commands on each of the newly added partitions or
- ``MSCK REPAIR TABLE table_name`` command.
-
- But current tajo doesn't provide ``ADD PARTITION`` command and hive doesn't provide an api for
- responding to ``MSK REPAIR TABLE`` command. Thus, if you insert data to hive partitioned
- table and you want to scan the updated partitions through Tajo, you must run following command on hive ::
-
- $ MSCK REPAIR TABLE [table_name];
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-docs/src/main/sphinx/hive_integration.rst
----------------------------------------------------------------------
diff --git a/tajo-docs/src/main/sphinx/hive_integration.rst b/tajo-docs/src/main/sphinx/hive_integration.rst
new file mode 100644
index 0000000..4c1d8d4
--- /dev/null
+++ b/tajo-docs/src/main/sphinx/hive_integration.rst
@@ -0,0 +1,42 @@
+*************************************
+Hive Integration
+*************************************
+
+Apache Tajo™ catalog supports HiveCatalogStore to integrate with Apache Hive™.
+This integration allows Tajo to access all tables used in Apache Hive.
+Depending on your purpose, you can execute either SQL queries or HiveQL queries on the
+same tables managed in Apache Hive.
+
+In order to use this feature, you need to build Tajo with a specified maven profile
+and then add some configs into ``conf/tajo-env.sh`` and ``conf/catalog-site.xml``.
+This section describes how to setup HiveMetaStore integration.
+This instruction would take no more than five minutes.
+
+You need to set your Hive home directory to the environment variable ``HIVE_HOME`` in conf/tajo-env.sh as follows: ::
+
+ export HIVE_HOME=/path/to/your/hive/directory
+
+If you need to use jdbc to connect HiveMetaStore, you have to prepare MySQL jdbc driver.
+Next, you should set the path of MySQL JDBC driver jar file to the environment variable HIVE_JDBC_DRIVER_DIR in conf/tajo-env.sh as follows: ::
+
+ export HIVE_JDBC_DRIVER_DIR==/path/to/your/mysql_jdbc_driver/mysql-connector-java-x.x.x-bin.jar
+
+Finally, you should specify HiveCatalogStore as Tajo catalog driver class in ``conf/catalog-site.xml`` as follows: ::
+
+ <property>
+ <name>tajo.catalog.store.class</name>
+ <value>org.apache.tajo.catalog.store.HiveCatalogStore</value>
+ </property>
+
+.. note::
+
+ Hive stores a list of partitions for each table in its metastore. If new partitions are
+ directly added to HDFS, HiveMetastore will not able aware of these partitions unless the user
+ ``ALTER TABLE table_name ADD PARTITION`` commands on each of the newly added partitions or
+ ``MSCK REPAIR TABLE table_name`` command.
+
+ But current tajo doesn't provide ``ADD PARTITION`` command and hive doesn't provide an api for
+ responding to ``MSK REPAIR TABLE`` command. Thus, if you insert data to hive partitioned
+ table and you want to scan the updated partitions through Tajo, you must run following command on hive ::
+
+ $ MSCK REPAIR TABLE [table_name];
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-docs/src/main/sphinx/index.rst
----------------------------------------------------------------------
diff --git a/tajo-docs/src/main/sphinx/index.rst b/tajo-docs/src/main/sphinx/index.rst
index 0ab50b6..730bed4 100644
--- a/tajo-docs/src/main/sphinx/index.rst
+++ b/tajo-docs/src/main/sphinx/index.rst
@@ -39,7 +39,7 @@ Table of Contents:
table_partitioning
index_overview
backup_and_restore
- hcatalog_integration
+ hive_integration
hbase_integration
swift_integration
jdbc_driver
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/FilterPushDownRule.java
----------------------------------------------------------------------
diff --git a/tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/FilterPushDownRule.java b/tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/FilterPushDownRule.java
index a6a7c78..dc6b8ef 100644
--- a/tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/FilterPushDownRule.java
+++ b/tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/FilterPushDownRule.java
@@ -856,7 +856,7 @@ public class FilterPushDownRule extends BasicLogicalPlanVisitor<FilterPushDownCo
}
Column column = columns.iterator().next();
- // If catalog runs with HCatalog, partition column is a qualified name
+ // If catalog runs with HiveCatalogStore, partition column is a qualified name
// Else partition column is a simple name
boolean isPartitionColumn = false;
if (hasQualifiedName) {
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-project/pom.xml
----------------------------------------------------------------------
diff --git a/tajo-project/pom.xml b/tajo-project/pom.xml
index 65fbaa3..6fa995d 100644
--- a/tajo-project/pom.xml
+++ b/tajo-project/pom.xml
@@ -37,6 +37,7 @@
<protobuf.version>2.5.0</protobuf.version>
<tajo.version>0.11.0-SNAPSHOT</tajo.version>
<hbase.version>0.98.7-hadoop2</hbase.version>
+ <hive.version>1.1.0</hive.version>
<netty.version>4.0.25.Final</netty.version>
<jersey.version>2.6</jersey.version>
<tajo.root>${project.parent.relativePath}/..</tajo.root>
[3/4] tajo git commit: TAJO-1442: Improve Hive Compatibility
Posted by bl...@apache.org.
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java
----------------------------------------------------------------------
diff --git a/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java b/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java
deleted file mode 100644
index 32ab674..0000000
--- a/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java
+++ /dev/null
@@ -1,467 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.tajo.catalog.store;
-
-
-import org.apache.commons.lang.StringEscapeUtils;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.hive.conf.HiveConf;
-import org.apache.tajo.catalog.*;
-import org.apache.tajo.catalog.partition.PartitionDesc;
-import org.apache.tajo.catalog.partition.PartitionKey;
-import org.apache.tajo.catalog.partition.PartitionMethodDesc;
-import org.apache.tajo.catalog.proto.CatalogProtos;
-import org.apache.tajo.common.TajoDataTypes;
-import org.apache.tajo.conf.TajoConf;
-import org.apache.tajo.storage.StorageConstants;
-import org.apache.tajo.util.CommonTestingUtil;
-import org.apache.tajo.util.KeyValueSet;
-import org.junit.AfterClass;
-import org.junit.BeforeClass;
-import org.junit.Test;
-
-import java.io.IOException;
-import java.net.URI;
-import java.util.ArrayList;
-import java.util.List;
-
-import static org.apache.tajo.TajoConstants.DEFAULT_DATABASE_NAME;
-import static org.junit.Assert.*;
-
-/**
- * TestHCatalogStore. Test case for
- * {@link org.apache.tajo.catalog.store.HCatalogStore}
- */
-
-public class TestHCatalogStore {
- private static final String DB_NAME = "test_hive";
- private static final String CUSTOMER = "customer";
- private static final String NATION = "nation";
- private static final String REGION = "region";
- private static final String SUPPLIER = "supplier";
-
- private static HCatalogStore store;
- private static Path warehousePath;
-
- @BeforeClass
- public static void setUp() throws Exception {
- Path testPath = CommonTestingUtil.getTestDir();
- warehousePath = new Path(testPath, "warehouse");
-
- //create local hiveMeta
- HiveConf conf = new HiveConf();
- String jdbcUri = "jdbc:derby:;databaseName="+testPath.toUri().getPath()+"metastore_db;create=true";
- conf.set(HiveConf.ConfVars.METASTOREWAREHOUSE.varname, warehousePath.toUri().toString());
- conf.set(HiveConf.ConfVars.METASTORECONNECTURLKEY.varname, jdbcUri);
- conf.set(TajoConf.ConfVars.WAREHOUSE_DIR.varname, warehousePath.toUri().toString());
-
- // create local HCatalogStore.
- TajoConf tajoConf = new TajoConf(conf);
- store = new HCatalogStore(tajoConf);
- store.createDatabase(DB_NAME, null);
- }
-
- @AfterClass
- public static void tearDown() throws IOException {
- store.close();
- }
-
- @Test
- public void testTableUsingTextFile() throws Exception {
- TableMeta meta = new TableMeta(CatalogProtos.StoreType.CSV, new KeyValueSet());
-
- org.apache.tajo.catalog.Schema schema = new org.apache.tajo.catalog.Schema();
- schema.addColumn("c_custkey", TajoDataTypes.Type.INT4);
- schema.addColumn("c_name", TajoDataTypes.Type.TEXT);
- schema.addColumn("c_address", TajoDataTypes.Type.TEXT);
- schema.addColumn("c_nationkey", TajoDataTypes.Type.INT4);
- schema.addColumn("c_phone", TajoDataTypes.Type.TEXT);
- schema.addColumn("c_acctbal", TajoDataTypes.Type.FLOAT8);
- schema.addColumn("c_mktsegment", TajoDataTypes.Type.TEXT);
- schema.addColumn("c_comment", TajoDataTypes.Type.TEXT);
-
- TableDesc table = new TableDesc(CatalogUtil.buildFQName(DB_NAME, CUSTOMER), schema, meta,
- new Path(warehousePath, new Path(DB_NAME, CUSTOMER)).toUri());
- store.createTable(table.getProto());
- assertTrue(store.existTable(DB_NAME, CUSTOMER));
-
- TableDesc table1 = new TableDesc(store.getTable(DB_NAME, CUSTOMER));
- assertEquals(table.getName(), table1.getName());
- assertEquals(table.getPath(), table1.getPath());
- assertEquals(table.getSchema().size(), table1.getSchema().size());
- for (int i = 0; i < table.getSchema().size(); i++) {
- assertEquals(table.getSchema().getColumn(i).getSimpleName(), table1.getSchema().getColumn(i).getSimpleName());
- }
-
- assertEquals(StringEscapeUtils.escapeJava(StorageConstants.DEFAULT_FIELD_DELIMITER),
- table1.getMeta().getOption(StorageConstants.TEXT_DELIMITER));
- store.dropTable(DB_NAME, CUSTOMER);
- }
-
- @Test
- public void testTableUsingRCFileWithBinarySerde() throws Exception {
- KeyValueSet options = new KeyValueSet();
- options.set(StorageConstants.RCFILE_SERDE, StorageConstants.DEFAULT_BINARY_SERDE);
- TableMeta meta = new TableMeta(CatalogProtos.StoreType.RCFILE, options);
-
- org.apache.tajo.catalog.Schema schema = new org.apache.tajo.catalog.Schema();
- schema.addColumn("r_regionkey", TajoDataTypes.Type.INT4);
- schema.addColumn("r_name", TajoDataTypes.Type.TEXT);
- schema.addColumn("r_comment", TajoDataTypes.Type.TEXT);
-
- TableDesc table = new TableDesc(CatalogUtil.buildFQName(DB_NAME, REGION), schema, meta,
- new Path(warehousePath, new Path(DB_NAME, REGION)).toUri());
- store.createTable(table.getProto());
- assertTrue(store.existTable(DB_NAME, REGION));
-
- TableDesc table1 = new TableDesc(store.getTable(DB_NAME, REGION));
- assertEquals(table.getName(), table1.getName());
- assertEquals(table.getPath(), table1.getPath());
- assertEquals(table.getSchema().size(), table1.getSchema().size());
- for (int i = 0; i < table.getSchema().size(); i++) {
- assertEquals(table.getSchema().getColumn(i).getSimpleName(), table1.getSchema().getColumn(i).getSimpleName());
- }
-
- assertEquals(StorageConstants.DEFAULT_BINARY_SERDE,
- table1.getMeta().getOption(StorageConstants.RCFILE_SERDE));
- store.dropTable(DB_NAME, REGION);
- }
-
- @Test
- public void testTableUsingRCFileWithTextSerde() throws Exception {
- KeyValueSet options = new KeyValueSet();
- options.set(StorageConstants.RCFILE_SERDE, StorageConstants.DEFAULT_TEXT_SERDE);
- TableMeta meta = new TableMeta(CatalogProtos.StoreType.RCFILE, options);
-
- org.apache.tajo.catalog.Schema schema = new org.apache.tajo.catalog.Schema();
- schema.addColumn("r_regionkey", TajoDataTypes.Type.INT4);
- schema.addColumn("r_name", TajoDataTypes.Type.TEXT);
- schema.addColumn("r_comment", TajoDataTypes.Type.TEXT);
-
- TableDesc table = new TableDesc(CatalogUtil.buildFQName(DB_NAME, REGION), schema, meta,
- new Path(warehousePath, new Path(DB_NAME, REGION)).toUri());
- store.createTable(table.getProto());
- assertTrue(store.existTable(DB_NAME, REGION));
-
- TableDesc table1 = new TableDesc(store.getTable(DB_NAME, REGION));
- assertEquals(table.getName(), table1.getName());
- assertEquals(table.getPath(), table1.getPath());
- assertEquals(table.getSchema().size(), table1.getSchema().size());
- for (int i = 0; i < table.getSchema().size(); i++) {
- assertEquals(table.getSchema().getColumn(i).getSimpleName(), table1.getSchema().getColumn(i).getSimpleName());
- }
-
- assertEquals(StorageConstants.DEFAULT_TEXT_SERDE, table1.getMeta().getOption(StorageConstants.RCFILE_SERDE));
- store.dropTable(DB_NAME, REGION);
- }
-
- @Test
- public void testTableWithNullValue() throws Exception {
- KeyValueSet options = new KeyValueSet();
- options.set(StorageConstants.TEXT_DELIMITER, StringEscapeUtils.escapeJava("\u0002"));
- options.set(StorageConstants.TEXT_NULL, StringEscapeUtils.escapeJava("\u0003"));
- TableMeta meta = new TableMeta(CatalogProtos.StoreType.CSV, options);
-
- org.apache.tajo.catalog.Schema schema = new org.apache.tajo.catalog.Schema();
- schema.addColumn("s_suppkey", TajoDataTypes.Type.INT4);
- schema.addColumn("s_name", TajoDataTypes.Type.TEXT);
- schema.addColumn("s_address", TajoDataTypes.Type.TEXT);
- schema.addColumn("s_nationkey", TajoDataTypes.Type.INT4);
- schema.addColumn("s_phone", TajoDataTypes.Type.TEXT);
- schema.addColumn("s_acctbal", TajoDataTypes.Type.FLOAT8);
- schema.addColumn("s_comment", TajoDataTypes.Type.TEXT);
-
- TableDesc table = new TableDesc(CatalogUtil.buildFQName(DB_NAME, SUPPLIER), schema, meta,
- new Path(warehousePath, new Path(DB_NAME, SUPPLIER)).toUri());
-
- store.createTable(table.getProto());
- assertTrue(store.existTable(DB_NAME, SUPPLIER));
-
- TableDesc table1 = new TableDesc(store.getTable(DB_NAME, SUPPLIER));
- assertEquals(table.getName(), table1.getName());
- assertEquals(table.getPath(), table1.getPath());
- assertEquals(table.getSchema().size(), table1.getSchema().size());
- for (int i = 0; i < table.getSchema().size(); i++) {
- assertEquals(table.getSchema().getColumn(i).getSimpleName(), table1.getSchema().getColumn(i).getSimpleName());
- }
-
- assertEquals(table.getMeta().getOption(StorageConstants.TEXT_DELIMITER),
- table1.getMeta().getOption(StorageConstants.TEXT_DELIMITER));
-
- assertEquals(table.getMeta().getOption(StorageConstants.TEXT_NULL),
- table1.getMeta().getOption(StorageConstants.TEXT_NULL));
-
- assertEquals(table1.getMeta().getOption(StorageConstants.TEXT_DELIMITER),
- StringEscapeUtils.escapeJava("\u0002"));
-
- assertEquals(table1.getMeta().getOption(StorageConstants.TEXT_NULL),
- StringEscapeUtils.escapeJava("\u0003"));
-
- store.dropTable(DB_NAME, SUPPLIER);
-
- }
-
- @Test
- public void testAddTableByPartition() throws Exception {
- TableMeta meta = new TableMeta(CatalogProtos.StoreType.CSV, new KeyValueSet());
-
- org.apache.tajo.catalog.Schema schema = new org.apache.tajo.catalog.Schema();
- schema.addColumn("n_name", TajoDataTypes.Type.TEXT);
- schema.addColumn("n_regionkey", TajoDataTypes.Type.INT4);
- schema.addColumn("n_comment", TajoDataTypes.Type.TEXT);
-
-
- TableDesc table = new TableDesc(CatalogUtil.buildFQName(DB_NAME, NATION), schema, meta,
- new Path(warehousePath, new Path(DB_NAME, NATION)).toUri());
-
- org.apache.tajo.catalog.Schema expressionSchema = new org.apache.tajo.catalog.Schema();
- expressionSchema.addColumn("n_nationkey", TajoDataTypes.Type.INT4);
- expressionSchema.addColumn("n_date", TajoDataTypes.Type.TEXT);
-
- PartitionMethodDesc partitions = new PartitionMethodDesc(
- DB_NAME,
- NATION,
- CatalogProtos.PartitionType.COLUMN, "n_nationkey,n_date", expressionSchema);
- table.setPartitionMethod(partitions);
-
- store.createTable(table.getProto());
- assertTrue(store.existTable(DB_NAME, NATION));
-
- TableDesc table1 = new TableDesc(store.getTable(DB_NAME, NATION));
- assertEquals(table.getName(), table1.getName());
- assertEquals(table.getPath(), table1.getPath());
- assertEquals(table.getSchema().size(), table1.getSchema().size());
- for (int i = 0; i < table.getSchema().size(); i++) {
- assertEquals(table.getSchema().getColumn(i).getSimpleName(), table1.getSchema().getColumn(i).getSimpleName());
- }
-
- Schema partitionSchema = table.getPartitionMethod().getExpressionSchema();
- Schema partitionSchema1 = table1.getPartitionMethod().getExpressionSchema();
- assertEquals(partitionSchema.size(), partitionSchema1.size());
-
- for (int i = 0; i < partitionSchema.size(); i++) {
- assertEquals(partitionSchema.getColumn(i).getSimpleName(), partitionSchema1.getColumn(i).getSimpleName());
- }
-
- testAddPartition(table1.getPath(), NATION, "n_nationkey=10/n_date=20150101");
- testAddPartition(table1.getPath(), NATION, "n_nationkey=20/n_date=20150102");
-
- testDropPartition(NATION, "n_nationkey=10/n_date=20150101");
- testDropPartition(NATION, "n_nationkey=20/n_date=20150102");
-
- CatalogProtos.PartitionDescProto partition = store.getPartition(DB_NAME, NATION, "n_nationkey=10/n_date=20150101");
- assertNull(partition);
-
- partition = store.getPartition(DB_NAME, NATION, "n_nationkey=20/n_date=20150102");
- assertNull(partition);
-
- store.dropTable(DB_NAME, NATION);
- }
-
-
- private void testAddPartition(URI uri, String tableName, String partitionName) throws Exception {
- AlterTableDesc alterTableDesc = new AlterTableDesc();
- alterTableDesc.setTableName(DB_NAME + "." + tableName);
- alterTableDesc.setAlterTableType(AlterTableType.ADD_PARTITION);
-
- Path path = new Path(uri.getPath(), partitionName);
-
- PartitionDesc partitionDesc = new PartitionDesc();
- partitionDesc.setPartitionName(partitionName);
-
- List<PartitionKey> partitionKeyList = new ArrayList<PartitionKey>();
- String[] partitionNames = partitionName.split("/");
- for(int i = 0; i < partitionNames.length; i++) {
- String[] eachPartitionName = partitionNames[i].split("=");
- partitionKeyList.add(new PartitionKey(eachPartitionName[0], eachPartitionName[1]));
- }
- partitionDesc.setPartitionKeys(partitionKeyList);
- partitionDesc.setPath(path.toString());
-
- alterTableDesc.setPartitionDesc(partitionDesc);
-
- store.alterTable(alterTableDesc.getProto());
-
- CatalogProtos.PartitionDescProto resultDesc = store.getPartition(DB_NAME, NATION, partitionName);
- assertNotNull(resultDesc);
- assertEquals(resultDesc.getPartitionName(), partitionName);
- assertEquals(resultDesc.getPath(), uri.toString() + "/" + partitionName);
- assertEquals(resultDesc.getPartitionKeysCount(), 2);
-
- for (int i = 0; i < resultDesc.getPartitionKeysCount(); i++) {
- CatalogProtos.PartitionKeyProto keyProto = resultDesc.getPartitionKeys(i);
- String[] eachName = partitionNames[i].split("=");
- assertEquals(keyProto.getPartitionValue(), eachName[1]);
- }
- }
-
-
- private void testDropPartition(String tableName, String partitionName) throws Exception {
- AlterTableDesc alterTableDesc = new AlterTableDesc();
- alterTableDesc.setTableName(DB_NAME + "." + tableName);
- alterTableDesc.setAlterTableType(AlterTableType.DROP_PARTITION);
-
- PartitionDesc partitionDesc = new PartitionDesc();
- partitionDesc.setPartitionName(partitionName);
-
- alterTableDesc.setPartitionDesc(partitionDesc);
-
- store.alterTable(alterTableDesc.getProto());
- }
-
- @Test
- public void testGetAllTableNames() throws Exception{
- TableMeta meta = new TableMeta(CatalogProtos.StoreType.CSV, new KeyValueSet());
- org.apache.tajo.catalog.Schema schema = new org.apache.tajo.catalog.Schema();
- schema.addColumn("n_name", TajoDataTypes.Type.TEXT);
- schema.addColumn("n_regionkey", TajoDataTypes.Type.INT4);
- schema.addColumn("n_comment", TajoDataTypes.Type.TEXT);
-
- String[] tableNames = new String[]{"table1", "table2", "table3"};
-
- for(String tableName : tableNames){
- TableDesc table = new TableDesc(CatalogUtil.buildFQName("default", tableName), schema, meta,
- new Path(warehousePath, new Path(DB_NAME, tableName)).toUri());
- store.createTable(table.getProto());
- }
-
- List<String> tables = store.getAllTableNames("default");
- assertEquals(tableNames.length, tables.size());
-
- for(String tableName : tableNames){
- assertTrue(tables.contains(tableName));
- }
-
- for(String tableName : tableNames){
- store.dropTable("default", tableName);
- }
- }
-
- @Test
- public void testDeleteTable() throws Exception {
- TableMeta meta = new TableMeta(CatalogProtos.StoreType.CSV, new KeyValueSet());
- org.apache.tajo.catalog.Schema schema = new org.apache.tajo.catalog.Schema();
- schema.addColumn("n_name", TajoDataTypes.Type.TEXT);
- schema.addColumn("n_regionkey", TajoDataTypes.Type.INT4);
- schema.addColumn("n_comment", TajoDataTypes.Type.TEXT);
-
- String tableName = "table1";
- TableDesc table = new TableDesc(DB_NAME + "." + tableName, schema, meta, warehousePath.toUri());
- store.createTable(table.getProto());
- assertTrue(store.existTable(DB_NAME, tableName));
-
- TableDesc table1 = new TableDesc(store.getTable(DB_NAME, tableName));
- FileSystem fs = FileSystem.getLocal(new Configuration());
- assertTrue(fs.exists(new Path(table1.getPath())));
-
- store.dropTable(DB_NAME, tableName);
- assertFalse(store.existTable(DB_NAME, tableName));
- fs.close();
- }
-
- @Test
- public void testTableUsingSequenceFileWithBinarySerde() throws Exception {
- KeyValueSet options = new KeyValueSet();
- options.set(StorageConstants.SEQUENCEFILE_SERDE, StorageConstants.DEFAULT_BINARY_SERDE);
- TableMeta meta = new TableMeta(CatalogProtos.StoreType.SEQUENCEFILE, options);
-
- org.apache.tajo.catalog.Schema schema = new org.apache.tajo.catalog.Schema();
- schema.addColumn("r_regionkey", TajoDataTypes.Type.INT4);
- schema.addColumn("r_name", TajoDataTypes.Type.TEXT);
- schema.addColumn("r_comment", TajoDataTypes.Type.TEXT);
-
- TableDesc table = new TableDesc(CatalogUtil.buildFQName(DB_NAME, REGION), schema, meta,
- new Path(warehousePath, new Path(DB_NAME, REGION)).toUri());
- store.createTable(table.getProto());
- assertTrue(store.existTable(DB_NAME, REGION));
-
- TableDesc table1 = new TableDesc(store.getTable(DB_NAME, REGION));
- assertEquals(table.getName(), table1.getName());
- assertEquals(table.getPath(), table1.getPath());
- assertEquals(table.getSchema().size(), table1.getSchema().size());
- for (int i = 0; i < table.getSchema().size(); i++) {
- assertEquals(table.getSchema().getColumn(i).getSimpleName(), table1.getSchema().getColumn(i).getSimpleName());
- }
-
- assertEquals(StorageConstants.DEFAULT_BINARY_SERDE,
- table1.getMeta().getOption(StorageConstants.SEQUENCEFILE_SERDE));
- store.dropTable(DB_NAME, REGION);
- }
-
- @Test
- public void testTableUsingSequenceFileWithTextSerde() throws Exception {
- KeyValueSet options = new KeyValueSet();
- options.set(StorageConstants.SEQUENCEFILE_SERDE, StorageConstants.DEFAULT_TEXT_SERDE);
- TableMeta meta = new TableMeta(CatalogProtos.StoreType.SEQUENCEFILE, options);
-
- org.apache.tajo.catalog.Schema schema = new org.apache.tajo.catalog.Schema();
- schema.addColumn("r_regionkey", TajoDataTypes.Type.INT4);
- schema.addColumn("r_name", TajoDataTypes.Type.TEXT);
- schema.addColumn("r_comment", TajoDataTypes.Type.TEXT);
-
- TableDesc table = new TableDesc(CatalogUtil.buildFQName(DB_NAME, REGION), schema, meta,
- new Path(warehousePath, new Path(DB_NAME, REGION)).toUri());
- store.createTable(table.getProto());
- assertTrue(store.existTable(DB_NAME, REGION));
-
- TableDesc table1 = new TableDesc(store.getTable(DB_NAME, REGION));
- assertEquals(table.getName(), table1.getName());
- assertEquals(table.getPath(), table1.getPath());
- assertEquals(table.getSchema().size(), table1.getSchema().size());
- for (int i = 0; i < table.getSchema().size(); i++) {
- assertEquals(table.getSchema().getColumn(i).getSimpleName(), table1.getSchema().getColumn(i).getSimpleName());
- }
-
- assertEquals(StorageConstants.DEFAULT_TEXT_SERDE, table1.getMeta().getOption(StorageConstants.SEQUENCEFILE_SERDE));
- store.dropTable(DB_NAME, REGION);
- }
-
-
- @Test
- public void testTableUsingParquet() throws Exception {
- TableMeta meta = new TableMeta(CatalogProtos.StoreType.PARQUET, new KeyValueSet());
-
- org.apache.tajo.catalog.Schema schema = new org.apache.tajo.catalog.Schema();
- schema.addColumn("c_custkey", TajoDataTypes.Type.INT4);
- schema.addColumn("c_name", TajoDataTypes.Type.TEXT);
- schema.addColumn("c_address", TajoDataTypes.Type.TEXT);
- schema.addColumn("c_nationkey", TajoDataTypes.Type.INT4);
- schema.addColumn("c_phone", TajoDataTypes.Type.TEXT);
- schema.addColumn("c_acctbal", TajoDataTypes.Type.FLOAT8);
- schema.addColumn("c_mktsegment", TajoDataTypes.Type.TEXT);
- schema.addColumn("c_comment", TajoDataTypes.Type.TEXT);
-
- TableDesc table = new TableDesc(CatalogUtil.buildFQName(DB_NAME, CUSTOMER), schema, meta,
- new Path(warehousePath, new Path(DB_NAME, CUSTOMER)).toUri());
- store.createTable(table.getProto());
- assertTrue(store.existTable(DB_NAME, CUSTOMER));
-
- TableDesc table1 = new TableDesc(store.getTable(DB_NAME, CUSTOMER));
- assertEquals(table.getName(), table1.getName());
- assertEquals(table.getPath(), table1.getPath());
- assertEquals(table.getSchema().size(), table1.getSchema().size());
- for (int i = 0; i < table.getSchema().size(); i++) {
- assertEquals(table.getSchema().getColumn(i).getSimpleName(), table1.getSchema().getColumn(i).getSimpleName());
- }
-
- store.dropTable(DB_NAME, CUSTOMER);
- }
-}
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-catalog/tajo-catalog-drivers/tajo-hive/pom.xml
----------------------------------------------------------------------
diff --git a/tajo-catalog/tajo-catalog-drivers/tajo-hive/pom.xml b/tajo-catalog/tajo-catalog-drivers/tajo-hive/pom.xml
new file mode 100644
index 0000000..16cb170
--- /dev/null
+++ b/tajo-catalog/tajo-catalog-drivers/tajo-hive/pom.xml
@@ -0,0 +1,351 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>tajo-project</artifactId>
+ <groupId>org.apache.tajo</groupId>
+ <version>0.11.0-SNAPSHOT</version>
+ <relativePath>../../../tajo-project</relativePath>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+ <artifactId>tajo-hive</artifactId>
+ <packaging>jar</packaging>
+ <name>Tajo Catalog Drivers Hive</name>
+ <properties>
+ <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+ <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
+ <parquet.version>1.5.0</parquet.version>
+ <parquet.format.version>2.1.0</parquet.format.version>
+ </properties>
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-compiler-plugin</artifactId>
+ <configuration>
+ <source>1.6</source>
+ <target>1.6</target>
+ <encoding>${project.build.sourceEncoding}</encoding>
+ </configuration>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.rat</groupId>
+ <artifactId>apache-rat-plugin</artifactId>
+ <executions>
+ <execution>
+ <phase>verify</phase>
+ <goals>
+ <goal>check</goal>
+ </goals>
+ </execution>
+ </executions>
+ <configuration>
+ <excludes>
+ <exclude>derby.log</exclude>
+ </excludes>
+ </configuration>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>copy-dependencies</id>
+ <phase>prepare-package</phase>
+ <goals>
+ <goal>copy-dependencies</goal>
+ </goals>
+ <configuration>
+ <includeScope>runtime</includeScope>
+ <outputDirectory>${project.build.directory}/lib</outputDirectory>
+ <overWriteReleases>false</overWriteReleases>
+ <overWriteSnapshots>false</overWriteSnapshots>
+ <overWriteIfNewer>true</overWriteIfNewer>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-report-plugin</artifactId>
+ </plugin>
+ </plugins>
+ </build>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.tajo</groupId>
+ <artifactId>tajo-common</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.tajo</groupId>
+ <artifactId>tajo-catalog-common</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.tajo</groupId>
+ <artifactId>tajo-catalog-client</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.tajo</groupId>
+ <artifactId>tajo-catalog-server</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.tajo</groupId>
+ <artifactId>tajo-rpc-protobuf</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.tajo</groupId>
+ <artifactId>tajo-storage-common</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.thrift</groupId>
+ <artifactId>libfb303</artifactId>
+ <version>0.9.0</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.thrift</groupId>
+ <artifactId>libthrift</artifactId>
+ <version>0.9.0</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-mapreduce-client-core</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-exec</artifactId>
+ <version>${hive.version}</version>
+ <scope>provided</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-common</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-contrib</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-hbase-handler</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-metastore</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-serde</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-shims</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-testutils</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.thrift</groupId>
+ <artifactId>libfb303</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.thrift</groupId>
+ <artifactId>libthrift</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.jolbox</groupId>
+ <artifactId>bonecp</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.google.protobuf</groupId>
+ <artifactId>protobuf-java</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.calcite</groupId>
+ <artifactId>calcite-core</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.calcite</groupId>
+ <artifactId>calcite-avatica</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-metastore</artifactId>
+ <version>${hive.version}</version>
+ <scope>provided</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-common</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-serde</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-shimss</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.thrift</groupId>
+ <artifactId>libfb303</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.thrift</groupId>
+ <artifactId>libthrift</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.jolbox</groupId>
+ <artifactId>bonecp</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-cli</artifactId>
+ <version>${hive.version}</version>
+ <scope>provided</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-common</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-exec</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-metastore</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-serde</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-service</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-shims</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.jolbox</groupId>
+ <artifactId>bonecp</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>jline</groupId>
+ <artifactId>jline</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>com.twitter</groupId>
+ <artifactId>parquet-hive-bundle</artifactId>
+ <version>${parquet.version}</version>
+ </dependency>
+ </dependencies>
+
+ <profiles>
+ <profile>
+ <id>docs</id>
+ <activation>
+ <activeByDefault>false</activeByDefault>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-javadoc-plugin</artifactId>
+ <executions>
+ <execution>
+ <!-- build javadoc jars per jar for publishing to maven -->
+ <id>module-javadocs</id>
+ <phase>package</phase>
+ <goals>
+ <goal>jar</goal>
+ </goals>
+ <configuration>
+ <destDir>${project.build.directory}</destDir>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ <profile>
+ <id>src</id>
+ <activation>
+ <activeByDefault>false</activeByDefault>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-source-plugin</artifactId>
+ <executions>
+ <execution>
+ <!-- builds source jars and attaches them to the project for publishing -->
+ <id>hadoop-java-sources</id>
+ <phase>package</phase>
+ <goals>
+ <goal>jar-no-fork</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
+
+ <reporting>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-report-plugin</artifactId>
+ </plugin>
+ </plugins>
+ </reporting>
+
+</project>
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java
----------------------------------------------------------------------
diff --git a/tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java b/tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java
new file mode 100644
index 0000000..5b1a996
--- /dev/null
+++ b/tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStore.java
@@ -0,0 +1,964 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tajo.catalog.store;
+
+import com.google.common.collect.Lists;
+
+import org.apache.commons.lang.StringEscapeUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.metastore.TableType;
+import org.apache.hadoop.hive.metastore.api.*;
+import org.apache.hadoop.hive.serde.serdeConstants;
+import org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe;
+import org.apache.hadoop.hive.serde2.columnar.LazyBinaryColumnarSerDe;
+import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe;
+import org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe;
+import org.apache.tajo.TajoConstants;
+import org.apache.tajo.catalog.*;
+import org.apache.tajo.catalog.exception.*;
+import org.apache.tajo.catalog.partition.PartitionMethodDesc;
+import org.apache.tajo.catalog.proto.CatalogProtos;
+import org.apache.tajo.catalog.proto.CatalogProtos.ColumnProto;
+import org.apache.tajo.catalog.proto.CatalogProtos.DatabaseProto;
+import org.apache.tajo.catalog.proto.CatalogProtos.IndexProto;
+import org.apache.tajo.catalog.proto.CatalogProtos.TableDescriptorProto;
+import org.apache.tajo.catalog.proto.CatalogProtos.TableOptionProto;
+import org.apache.tajo.catalog.proto.CatalogProtos.TablePartitionProto;
+import org.apache.tajo.catalog.proto.CatalogProtos.TableStatsProto;
+import org.apache.tajo.catalog.proto.CatalogProtos.TablespaceProto;
+import org.apache.tajo.catalog.statistics.TableStats;
+import org.apache.tajo.common.TajoDataTypes;
+import org.apache.tajo.common.exception.NotImplementedException;
+import org.apache.tajo.conf.TajoConf;
+import org.apache.tajo.exception.InternalException;
+import org.apache.tajo.storage.StorageConstants;
+import org.apache.tajo.util.KeyValueSet;
+import org.apache.thrift.TException;
+
+import java.io.IOException;
+import java.util.*;
+
+import static org.apache.tajo.catalog.proto.CatalogProtos.PartitionType;
+
+public class HiveCatalogStore extends CatalogConstants implements CatalogStore {
+ protected final Log LOG = LogFactory.getLog(getClass());
+
+ private static String HIVE_WAREHOUSE_DIR_CONF_KEY = "hive.metastore.warehouse.dir";
+
+ protected Configuration conf;
+ private static final int CLIENT_POOL_SIZE = 2;
+ private final HiveCatalogStoreClientPool clientPool;
+ private final String defaultTableSpaceUri;
+
+ public HiveCatalogStore(final Configuration conf) throws InternalException {
+ if (!(conf instanceof TajoConf)) {
+ throw new CatalogException("Invalid Configuration Type:" + conf.getClass().getSimpleName());
+ }
+ this.conf = conf;
+ this.defaultTableSpaceUri = TajoConf.getWarehouseDir((TajoConf) conf).toString();
+ this.clientPool = new HiveCatalogStoreClientPool(CLIENT_POOL_SIZE, conf);
+ }
+
+ @Override
+ public boolean existTable(final String databaseName, final String tableName) throws CatalogException {
+ boolean exist = false;
+ org.apache.hadoop.hive.ql.metadata.Table table;
+ HiveCatalogStoreClientPool.HiveCatalogStoreClient client = null;
+
+ // get table
+ try {
+ client = clientPool.getClient();
+ table = HiveCatalogUtil.getTable(client.getHiveClient(), databaseName, tableName);
+ if (table != null) {
+ exist = true;
+ }
+ } catch (NoSuchObjectException nsoe) {
+ exist = false;
+ } catch (Exception e) {
+ throw new CatalogException(e);
+ } finally {
+ if (client != null) {
+ client.release();
+ }
+ }
+
+ return exist;
+ }
+
+ @Override
+ public final CatalogProtos.TableDescProto getTable(String databaseName, final String tableName) throws CatalogException {
+ org.apache.hadoop.hive.ql.metadata.Table table = null;
+ HiveCatalogStoreClientPool.HiveCatalogStoreClient client = null;
+ Path path = null;
+ CatalogProtos.StoreType storeType = null;
+ org.apache.tajo.catalog.Schema schema = null;
+ KeyValueSet options = null;
+ TableStats stats = null;
+ PartitionMethodDesc partitions = null;
+
+ //////////////////////////////////
+ // set tajo table schema.
+ //////////////////////////////////
+ try {
+ // get hive table schema
+ try {
+ client = clientPool.getClient();
+ table = HiveCatalogUtil.getTable(client.getHiveClient(), databaseName, tableName);
+ path = table.getPath();
+ } catch (NoSuchObjectException nsoe) {
+ throw new CatalogException("Table not found. - tableName:" + tableName, nsoe);
+ } catch (Exception e) {
+ throw new CatalogException(e);
+ }
+
+ // convert HiveCatalogStore field schema into tajo field schema.
+ schema = new org.apache.tajo.catalog.Schema();
+
+ List<FieldSchema> fieldSchemaList = table.getCols();
+ boolean isPartitionKey = false;
+ for (FieldSchema eachField : fieldSchemaList) {
+ isPartitionKey = false;
+
+ if (table.getPartitionKeys() != null) {
+ for (FieldSchema partitionKey : table.getPartitionKeys()) {
+ if (partitionKey.getName().equals(eachField.getName())) {
+ isPartitionKey = true;
+ }
+ }
+ }
+
+ if (!isPartitionKey) {
+ String fieldName = databaseName + CatalogConstants.IDENTIFIER_DELIMITER + tableName +
+ CatalogConstants.IDENTIFIER_DELIMITER + eachField.getName();
+ TajoDataTypes.Type dataType = HiveCatalogUtil.getTajoFieldType(eachField.getType().toString());
+ schema.addColumn(fieldName, dataType);
+ }
+ }
+
+ // validate field schema.
+ HiveCatalogUtil.validateSchema(table);
+
+ stats = new TableStats();
+ options = new KeyValueSet();
+ options.putAll(table.getParameters());
+ options.remove("EXTERNAL");
+
+ Properties properties = table.getMetadata();
+ if (properties != null) {
+ // set field delimiter
+ String fieldDelimiter = "", nullFormat = "";
+ if (properties.getProperty(serdeConstants.FIELD_DELIM) != null) {
+ fieldDelimiter = properties.getProperty(serdeConstants.FIELD_DELIM);
+ } else {
+ // if hive table used default row format delimiter, Properties doesn't have it.
+ // So, Tajo must set as follows:
+ fieldDelimiter = "\u0001";
+ }
+
+ // set null format
+ if (properties.getProperty(serdeConstants.SERIALIZATION_NULL_FORMAT) != null) {
+ nullFormat = properties.getProperty(serdeConstants.SERIALIZATION_NULL_FORMAT);
+ } else {
+ nullFormat = "\\N";
+ }
+ options.remove(serdeConstants.SERIALIZATION_NULL_FORMAT);
+
+ // set file output format
+ String fileOutputformat = properties.getProperty(hive_metastoreConstants.FILE_OUTPUT_FORMAT);
+ storeType = CatalogUtil.getStoreType(HiveCatalogUtil.getStoreType(fileOutputformat));
+
+ if (storeType.equals(CatalogProtos.StoreType.TEXTFILE)) {
+ options.set(StorageConstants.TEXT_DELIMITER, StringEscapeUtils.escapeJava(fieldDelimiter));
+ options.set(StorageConstants.TEXT_NULL, StringEscapeUtils.escapeJava(nullFormat));
+ } else if (storeType.equals(CatalogProtos.StoreType.RCFILE)) {
+ options.set(StorageConstants.RCFILE_NULL, StringEscapeUtils.escapeJava(nullFormat));
+ String serde = properties.getProperty(serdeConstants.SERIALIZATION_LIB);
+ if (LazyBinaryColumnarSerDe.class.getName().equals(serde)) {
+ options.set(StorageConstants.RCFILE_SERDE, StorageConstants.DEFAULT_BINARY_SERDE);
+ } else if (ColumnarSerDe.class.getName().equals(serde)) {
+ options.set(StorageConstants.RCFILE_SERDE, StorageConstants.DEFAULT_TEXT_SERDE);
+ }
+ } else if (storeType.equals(CatalogProtos.StoreType.SEQUENCEFILE) ) {
+ options.set(StorageConstants.SEQUENCEFILE_DELIMITER, StringEscapeUtils.escapeJava(fieldDelimiter));
+ options.set(StorageConstants.SEQUENCEFILE_NULL, StringEscapeUtils.escapeJava(nullFormat));
+ String serde = properties.getProperty(serdeConstants.SERIALIZATION_LIB);
+ if (LazyBinarySerDe.class.getName().equals(serde)) {
+ options.set(StorageConstants.SEQUENCEFILE_SERDE, StorageConstants.DEFAULT_BINARY_SERDE);
+ } else if (LazySimpleSerDe.class.getName().equals(serde)) {
+ options.set(StorageConstants.SEQUENCEFILE_SERDE, StorageConstants.DEFAULT_TEXT_SERDE);
+ }
+ }
+
+ // set data size
+ long totalSize = 0;
+ if (properties.getProperty("totalSize") != null) {
+ totalSize = Long.parseLong(properties.getProperty("totalSize"));
+ } else {
+ try {
+ FileSystem fs = path.getFileSystem(conf);
+ if (fs.exists(path)) {
+ totalSize = fs.getContentSummary(path).getLength();
+ }
+ } catch (IOException ioe) {
+ throw new CatalogException("Fail to get path. - path:" + path.toString(), ioe);
+ }
+ }
+ stats.setNumBytes(totalSize);
+ }
+
+ // set partition keys
+ List<FieldSchema> partitionKeys = table.getPartitionKeys();
+
+ if (null != partitionKeys) {
+ org.apache.tajo.catalog.Schema expressionSchema = new org.apache.tajo.catalog.Schema();
+ StringBuilder sb = new StringBuilder();
+ if (partitionKeys.size() > 0) {
+ for (int i = 0; i < partitionKeys.size(); i++) {
+ FieldSchema fieldSchema = partitionKeys.get(i);
+ TajoDataTypes.Type dataType = HiveCatalogUtil.getTajoFieldType(fieldSchema.getType().toString());
+ String fieldName = databaseName + CatalogConstants.IDENTIFIER_DELIMITER + tableName +
+ CatalogConstants.IDENTIFIER_DELIMITER + fieldSchema.getName();
+ expressionSchema.addColumn(new Column(fieldName, dataType));
+ if (i > 0) {
+ sb.append(",");
+ }
+ sb.append(fieldSchema.getName());
+ }
+ partitions = new PartitionMethodDesc(
+ databaseName,
+ tableName,
+ PartitionType.COLUMN,
+ sb.toString(),
+ expressionSchema);
+ }
+ }
+ } finally {
+ if(client != null) client.release();
+ }
+ TableMeta meta = new TableMeta(storeType, options);
+ TableDesc tableDesc = new TableDesc(databaseName + "." + tableName, schema, meta, path.toUri());
+ if (table.getTableType().equals(TableType.EXTERNAL_TABLE)) {
+ tableDesc.setExternal(true);
+ }
+ if (stats != null) {
+ tableDesc.setStats(stats);
+ }
+ if (partitions != null) {
+ tableDesc.setPartitionMethod(partitions);
+ }
+ return tableDesc.getProto();
+ }
+
+
+ private TajoDataTypes.Type getDataType(final String typeStr) {
+ try {
+ return Enum.valueOf(TajoDataTypes.Type.class, typeStr);
+ } catch (IllegalArgumentException iae) {
+ LOG.error("Cannot find a matched type against from '" + typeStr + "'");
+ return null;
+ }
+ }
+
+ @Override
+ public final List<String> getAllTableNames(String databaseName) throws CatalogException {
+ HiveCatalogStoreClientPool.HiveCatalogStoreClient client = null;
+
+ try {
+ client = clientPool.getClient();
+ return client.getHiveClient().getAllTables(databaseName);
+ } catch (TException e) {
+ throw new CatalogException(e);
+ } finally {
+ if(client != null) client.release();
+ }
+ }
+
+ @Override
+ public void createTablespace(String spaceName, String spaceUri) throws CatalogException {
+ // SKIP
+ }
+
+ @Override
+ public boolean existTablespace(String spaceName) throws CatalogException {
+ // SKIP
+ return spaceName.equals(TajoConstants.DEFAULT_TABLESPACE_NAME);
+ }
+
+ @Override
+ public void dropTablespace(String spaceName) throws CatalogException {
+ // SKIP
+ }
+
+ @Override
+ public Collection<String> getAllTablespaceNames() throws CatalogException {
+ return Lists.newArrayList(TajoConstants.DEFAULT_TABLESPACE_NAME);
+ }
+
+ @Override
+ public TablespaceProto getTablespace(String spaceName) throws CatalogException {
+ if (spaceName.equals(TajoConstants.DEFAULT_TABLESPACE_NAME)) {
+ TablespaceProto.Builder builder = TablespaceProto.newBuilder();
+ builder.setSpaceName(TajoConstants.DEFAULT_TABLESPACE_NAME);
+ builder.setUri(defaultTableSpaceUri);
+ return builder.build();
+ } else {
+ throw new CatalogException("tablespace concept is not supported in HiveCatalogStore");
+ }
+ }
+
+ @Override
+ public void updateTableStats(CatalogProtos.UpdateTableStatsProto statsProto) throws
+ CatalogException {
+ // TODO - not implemented yet
+ }
+
+ @Override
+ public void alterTablespace(CatalogProtos.AlterTablespaceProto alterProto) throws CatalogException {
+ throw new CatalogException("tablespace concept is not supported in HiveCatalogStore");
+ }
+
+ @Override
+ public void createDatabase(String databaseName, String tablespaceName) throws CatalogException {
+ HiveCatalogStoreClientPool.HiveCatalogStoreClient client = null;
+
+ try {
+ Database database = new Database(
+ databaseName,
+ "",
+ defaultTableSpaceUri + "/" + databaseName,
+ new HashMap<String, String>());
+ client = clientPool.getClient();
+ client.getHiveClient().createDatabase(database);
+ } catch (AlreadyExistsException e) {
+ throw new AlreadyExistsDatabaseException(databaseName);
+ } catch (Throwable t) {
+ throw new CatalogException(t);
+ } finally {
+ if (client != null) {
+ client.release();
+ }
+ }
+ }
+
+ @Override
+ public boolean existDatabase(String databaseName) throws CatalogException {
+ HiveCatalogStoreClientPool.HiveCatalogStoreClient client = null;
+
+ try {
+ client = clientPool.getClient();
+ List<String> databaseNames = client.getHiveClient().getAllDatabases();
+ return databaseNames.contains(databaseName);
+ } catch (Throwable t) {
+ throw new CatalogException(t);
+ } finally {
+ if (client != null) {
+ client.release();
+ }
+ }
+ }
+
+ @Override
+ public void dropDatabase(String databaseName) throws CatalogException {
+ HiveCatalogStoreClientPool.HiveCatalogStoreClient client = null;
+
+ try {
+ client = clientPool.getClient();
+ client.getHiveClient().dropDatabase(databaseName);
+ } catch (NoSuchObjectException e) {
+ throw new NoSuchDatabaseException(databaseName);
+ } catch (Throwable t) {
+ throw new CatalogException(databaseName);
+ } finally {
+ if (client != null) {
+ client.release();
+ }
+ }
+ }
+
+ @Override
+ public Collection<String> getAllDatabaseNames() throws CatalogException {
+ HiveCatalogStoreClientPool.HiveCatalogStoreClient client = null;
+
+ try {
+ client = clientPool.getClient();
+ return client.getHiveClient().getAllDatabases();
+ } catch (TException e) {
+ throw new CatalogException(e);
+ } finally {
+ if (client != null) {
+ client.release();
+ }
+ }
+ }
+
+ @Override
+ public final void createTable(final CatalogProtos.TableDescProto tableDescProto) throws CatalogException {
+ HiveCatalogStoreClientPool.HiveCatalogStoreClient client = null;
+
+ TableDesc tableDesc = new TableDesc(tableDescProto);
+ String[] splitted = CatalogUtil.splitFQTableName(tableDesc.getName());
+ String databaseName = splitted[0];
+ String tableName = splitted[1];
+
+ try {
+ client = clientPool.getClient();
+
+ org.apache.hadoop.hive.metastore.api.Table table = new org.apache.hadoop.hive.metastore.api.Table();
+ table.setDbName(databaseName);
+ table.setTableName(tableName);
+ table.setParameters(new HashMap<String, String>(tableDesc.getMeta().getOptions().getAllKeyValus()));
+ // TODO: set owner
+ //table.setOwner();
+
+ StorageDescriptor sd = new StorageDescriptor();
+ sd.setSerdeInfo(new SerDeInfo());
+ sd.getSerdeInfo().setParameters(new HashMap<String, String>());
+ sd.getSerdeInfo().setName(table.getTableName());
+
+ // if tajo set location method, thrift client make exception as follows:
+ // Caused by: MetaException(message:java.lang.NullPointerException)
+ // If you want to modify table path, you have to modify on Hive cli.
+ if (tableDesc.isExternal()) {
+ table.setTableType(TableType.EXTERNAL_TABLE.name());
+ table.putToParameters("EXTERNAL", "TRUE");
+
+ Path tablePath = new Path(tableDesc.getPath());
+ FileSystem fs = tablePath.getFileSystem(conf);
+ if (fs.isFile(tablePath)) {
+ LOG.warn("A table path is a file, but HiveCatalogStore does not allow a file path.");
+ sd.setLocation(tablePath.getParent().toString());
+ } else {
+ sd.setLocation(tablePath.toString());
+ }
+ }
+
+ // set column information
+ List<Column> columns = tableDesc.getSchema().getColumns();
+ ArrayList<FieldSchema> cols = new ArrayList<FieldSchema>(columns.size());
+
+ for (Column eachField : columns) {
+ cols.add(new FieldSchema(eachField.getSimpleName(),
+ HiveCatalogUtil.getHiveFieldType(eachField.getDataType()), ""));
+ }
+ sd.setCols(cols);
+
+ // set partition keys
+ if (tableDesc.hasPartition() && tableDesc.getPartitionMethod().getPartitionType().equals(PartitionType.COLUMN)) {
+ List<FieldSchema> partitionKeys = new ArrayList<FieldSchema>();
+ for (Column eachPartitionKey : tableDesc.getPartitionMethod().getExpressionSchema().getColumns()) {
+ partitionKeys.add(new FieldSchema(eachPartitionKey.getSimpleName(),
+ HiveCatalogUtil.getHiveFieldType(eachPartitionKey.getDataType()), ""));
+ }
+ table.setPartitionKeys(partitionKeys);
+ }
+
+ if (tableDesc.getMeta().getStoreType().equals(CatalogProtos.StoreType.RCFILE)) {
+ String serde = tableDesc.getMeta().getOption(StorageConstants.RCFILE_SERDE);
+ sd.setInputFormat(org.apache.hadoop.hive.ql.io.RCFileInputFormat.class.getName());
+ sd.setOutputFormat(org.apache.hadoop.hive.ql.io.RCFileOutputFormat.class.getName());
+ if (StorageConstants.DEFAULT_TEXT_SERDE.equals(serde)) {
+ sd.getSerdeInfo().setSerializationLib(org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe.class.getName());
+ } else {
+ sd.getSerdeInfo().setSerializationLib(
+ org.apache.hadoop.hive.serde2.columnar.LazyBinaryColumnarSerDe.class.getName());
+ }
+
+ if (tableDesc.getMeta().getOptions().containsKey(StorageConstants.RCFILE_NULL)) {
+ table.putToParameters(serdeConstants.SERIALIZATION_NULL_FORMAT,
+ StringEscapeUtils.unescapeJava(tableDesc.getMeta().getOption(StorageConstants.RCFILE_NULL)));
+ }
+ } else if (tableDesc.getMeta().getStoreType().equals(CatalogProtos.StoreType.CSV)
+ || tableDesc.getMeta().getStoreType().equals(CatalogProtos.StoreType.TEXTFILE)) {
+ sd.getSerdeInfo().setSerializationLib(org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.class.getName());
+ sd.setInputFormat(org.apache.hadoop.mapred.TextInputFormat.class.getName());
+ sd.setOutputFormat(org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.class.getName());
+
+ String fieldDelimiter = tableDesc.getMeta().getOption(StorageConstants.TEXT_DELIMITER,
+ StorageConstants.DEFAULT_FIELD_DELIMITER);
+
+ // User can use an unicode for filed delimiter such as \u0001, \001.
+ // In this case, java console will convert this value into "\\u001".
+ // And hive will un-espace this value again.
+ // As a result, user can use right field delimiter.
+ // So, we have to un-escape this value.
+ sd.getSerdeInfo().putToParameters(serdeConstants.SERIALIZATION_FORMAT,
+ StringEscapeUtils.unescapeJava(fieldDelimiter));
+ sd.getSerdeInfo().putToParameters(serdeConstants.FIELD_DELIM,
+ StringEscapeUtils.unescapeJava(fieldDelimiter));
+ table.getParameters().remove(StorageConstants.TEXT_DELIMITER);
+
+ if (tableDesc.getMeta().containsOption(StorageConstants.TEXT_NULL)) {
+ table.putToParameters(serdeConstants.SERIALIZATION_NULL_FORMAT,
+ StringEscapeUtils.unescapeJava(tableDesc.getMeta().getOption(StorageConstants.TEXT_NULL)));
+ table.getParameters().remove(StorageConstants.TEXT_NULL);
+ }
+ } else if (tableDesc.getMeta().getStoreType().equals(CatalogProtos.StoreType.SEQUENCEFILE)) {
+ String serde = tableDesc.getMeta().getOption(StorageConstants.SEQUENCEFILE_SERDE);
+ sd.setInputFormat(org.apache.hadoop.mapred.SequenceFileInputFormat.class.getName());
+ sd.setOutputFormat(org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat.class.getName());
+
+ if (StorageConstants.DEFAULT_TEXT_SERDE.equals(serde)) {
+ sd.getSerdeInfo().setSerializationLib(org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.class.getName());
+
+ String fieldDelimiter = tableDesc.getMeta().getOption(StorageConstants.SEQUENCEFILE_DELIMITER,
+ StorageConstants.DEFAULT_FIELD_DELIMITER);
+
+ // User can use an unicode for filed delimiter such as \u0001, \001.
+ // In this case, java console will convert this value into "\\u001".
+ // And hive will un-espace this value again.
+ // As a result, user can use right field delimiter.
+ // So, we have to un-escape this value.
+ sd.getSerdeInfo().putToParameters(serdeConstants.SERIALIZATION_FORMAT,
+ StringEscapeUtils.unescapeJava(fieldDelimiter));
+ sd.getSerdeInfo().putToParameters(serdeConstants.FIELD_DELIM,
+ StringEscapeUtils.unescapeJava(fieldDelimiter));
+ table.getParameters().remove(StorageConstants.SEQUENCEFILE_DELIMITER);
+ } else {
+ sd.getSerdeInfo().setSerializationLib(org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.class.getName());
+ }
+
+ if (tableDesc.getMeta().containsOption(StorageConstants.SEQUENCEFILE_NULL)) {
+ table.putToParameters(serdeConstants.SERIALIZATION_NULL_FORMAT,
+ StringEscapeUtils.unescapeJava(tableDesc.getMeta().getOption(StorageConstants.SEQUENCEFILE_NULL)));
+ table.getParameters().remove(StorageConstants.SEQUENCEFILE_NULL);
+ }
+ } else {
+ if (tableDesc.getMeta().getStoreType().equals(CatalogProtos.StoreType.PARQUET)) {
+ sd.setInputFormat(parquet.hive.DeprecatedParquetInputFormat.class.getName());
+ sd.setOutputFormat(parquet.hive.DeprecatedParquetOutputFormat.class.getName());
+ sd.getSerdeInfo().setSerializationLib(parquet.hive.serde.ParquetHiveSerDe.class.getName());
+ } else {
+ throw new CatalogException(new NotImplementedException(tableDesc.getMeta().getStoreType
+ ().name()));
+ }
+ }
+
+ sd.setSortCols(new ArrayList<Order>());
+
+ table.setSd(sd);
+ client.getHiveClient().createTable(table);
+ } catch (RuntimeException e) {
+ throw e;
+ } catch (Exception e) {
+ throw new CatalogException(e);
+ } finally {
+ if(client != null) client.release();
+ }
+ }
+
+ @Override
+ public final void dropTable(String databaseName, final String tableName) throws CatalogException {
+ HiveCatalogStoreClientPool.HiveCatalogStoreClient client = null;
+
+ try {
+ client = clientPool.getClient();
+ client.getHiveClient().dropTable(databaseName, tableName, false, false);
+ } catch (NoSuchObjectException nsoe) {
+ } catch (Exception e) {
+ throw new CatalogException(e);
+ } finally {
+ if (client != null) {
+ client.release();
+ }
+ }
+ }
+
+
+ @Override
+ public void alterTable(final CatalogProtos.AlterTableDescProto alterTableDescProto) throws CatalogException {
+ final String[] split = CatalogUtil.splitFQTableName(alterTableDescProto.getTableName());
+
+ if (split.length == 1) {
+ throw new IllegalArgumentException("alterTable() requires a qualified table name, but it is \""
+ + alterTableDescProto.getTableName() + "\".");
+ }
+
+ final String databaseName = split[0];
+ final String tableName = split[1];
+ String partitionName = null;
+ CatalogProtos.PartitionDescProto partitionDesc = null;
+
+ switch (alterTableDescProto.getAlterTableType()) {
+ case RENAME_TABLE:
+ if (existTable(databaseName,alterTableDescProto.getNewTableName().toLowerCase())) {
+ throw new AlreadyExistsTableException(alterTableDescProto.getNewTableName());
+ }
+ renameTable(databaseName, tableName, alterTableDescProto.getNewTableName().toLowerCase());
+ break;
+ case RENAME_COLUMN:
+ if (existColumn(databaseName,tableName, alterTableDescProto.getAlterColumnName().getNewColumnName())) {
+ throw new ColumnNameAlreadyExistException(alterTableDescProto.getAlterColumnName().getNewColumnName());
+ }
+ renameColumn(databaseName, tableName, alterTableDescProto.getAlterColumnName());
+ break;
+ case ADD_COLUMN:
+ if (existColumn(databaseName,tableName, alterTableDescProto.getAddColumn().getName())) {
+ throw new ColumnNameAlreadyExistException(alterTableDescProto.getAddColumn().getName());
+ }
+ addNewColumn(databaseName, tableName, alterTableDescProto.getAddColumn());
+ break;
+ case ADD_PARTITION:
+ partitionName = alterTableDescProto.getPartitionDesc().getPartitionName();
+ partitionDesc = getPartition(databaseName, tableName, partitionName);
+ if(partitionDesc != null) {
+ throw new AlreadyExistsPartitionException(databaseName, tableName, partitionName);
+ }
+ addPartition(databaseName, tableName, alterTableDescProto.getPartitionDesc());
+ break;
+ case DROP_PARTITION:
+ partitionName = alterTableDescProto.getPartitionDesc().getPartitionName();
+ partitionDesc = getPartition(databaseName, tableName, partitionName);
+ if(partitionDesc == null) {
+ throw new NoSuchPartitionException(databaseName, tableName, partitionName);
+ }
+ dropPartition(databaseName, tableName, partitionDesc);
+ break;
+ case SET_PROPERTY:
+ // TODO - not implemented yet
+ break;
+ default:
+ //TODO
+ }
+ }
+
+
+ private void renameTable(String databaseName, String tableName, String newTableName) {
+ HiveCatalogStoreClientPool.HiveCatalogStoreClient client = null;
+ try {
+ client = clientPool.getClient();
+ Table newTable = client.getHiveClient().getTable(databaseName, tableName);
+ newTable.setTableName(newTableName);
+ client.getHiveClient().alter_table(databaseName, tableName, newTable);
+
+ } catch (NoSuchObjectException nsoe) {
+ } catch (Exception e) {
+ throw new CatalogException(e);
+ } finally {
+ if (client != null) {
+ client.release();
+ }
+ }
+ }
+
+ private void renameColumn(String databaseName, String tableName, CatalogProtos.AlterColumnProto alterColumnProto) {
+ HiveCatalogStoreClientPool.HiveCatalogStoreClient client = null;
+ try {
+
+ client = clientPool.getClient();
+ Table table = client.getHiveClient().getTable(databaseName, tableName);
+ List<FieldSchema> columns = table.getSd().getCols();
+
+ for (final FieldSchema currentColumn : columns) {
+ if (currentColumn.getName().equalsIgnoreCase(alterColumnProto.getOldColumnName())) {
+ currentColumn.setName(alterColumnProto.getNewColumnName());
+ }
+ }
+ client.getHiveClient().alter_table(databaseName, tableName, table);
+
+ } catch (NoSuchObjectException nsoe) {
+ } catch (Exception e) {
+ throw new CatalogException(e);
+ } finally {
+ if (client != null) {
+ client.release();
+ }
+ }
+ }
+
+
+ private void addNewColumn(String databaseName, String tableName, CatalogProtos.ColumnProto columnProto) {
+ HiveCatalogStoreClientPool.HiveCatalogStoreClient client = null;
+ try {
+
+ client = clientPool.getClient();
+ Table table = client.getHiveClient().getTable(databaseName, tableName);
+ List<FieldSchema> columns = table.getSd().getCols();
+ columns.add(new FieldSchema(columnProto.getName(),
+ HiveCatalogUtil.getHiveFieldType(columnProto.getDataType()), ""));
+ client.getHiveClient().alter_table(databaseName, tableName, table);
+
+
+ } catch (NoSuchObjectException nsoe) {
+ } catch (Exception e) {
+ throw new CatalogException(e);
+ } finally {
+ if (client != null) {
+ client.release();
+ }
+ }
+ }
+
+ private void addPartition(String databaseName, String tableName, CatalogProtos.PartitionDescProto
+ partitionDescProto) {
+ HiveCatalogStoreClientPool.HiveCatalogStoreClient client = null;
+ try {
+
+ client = clientPool.getClient();
+
+ Partition partition = new Partition();
+ partition.setDbName(databaseName);
+ partition.setTableName(tableName);
+
+ List<String> values = Lists.newArrayList();
+ for(CatalogProtos.PartitionKeyProto keyProto : partitionDescProto.getPartitionKeysList()) {
+ values.add(keyProto.getPartitionValue());
+ }
+ partition.setValues(values);
+
+ Table table = client.getHiveClient().getTable(databaseName, tableName);
+ StorageDescriptor sd = table.getSd();
+ sd.setLocation(partitionDescProto.getPath());
+ partition.setSd(sd);
+
+ client.getHiveClient().add_partition(partition);
+ } catch (Exception e) {
+ throw new CatalogException(e);
+ } finally {
+ if (client != null) {
+ client.release();
+ }
+ }
+ }
+
+ private void dropPartition(String databaseName, String tableName, CatalogProtos.PartitionDescProto
+ partitionDescProto) {
+ HiveCatalogStoreClientPool.HiveCatalogStoreClient client = null;
+ try {
+
+ client = clientPool.getClient();
+
+ List<String> values = Lists.newArrayList();
+ for(CatalogProtos.PartitionKeyProto keyProto : partitionDescProto.getPartitionKeysList()) {
+ values.add(keyProto.getPartitionValue());
+ }
+ client.getHiveClient().dropPartition(databaseName, tableName, values, true);
+ } catch (Exception e) {
+ throw new CatalogException(e);
+ } finally {
+ if (client != null) {
+ client.release();
+ }
+ }
+ }
+
+ @Override
+ public void addPartitionMethod(CatalogProtos.PartitionMethodProto partitionMethodProto) throws CatalogException {
+ // TODO - not implemented yet
+ }
+
+ @Override
+ public CatalogProtos.PartitionMethodProto getPartitionMethod(String databaseName, String tableName)
+ throws CatalogException {
+ return null; // TODO - not implemented yet
+ }
+
+ @Override
+ public boolean existPartitionMethod(String databaseName, String tableName) throws CatalogException {
+ return false; // TODO - not implemented yet
+ }
+
+ @Override
+ public void dropPartitionMethod(String databaseName, String tableName) throws CatalogException {
+ // TODO - not implemented yet
+ }
+
+ @Override
+ public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName,
+ String tableName) throws CatalogException {
+ throw new UnsupportedOperationException();
+ }
+
+
+ @Override
+ public CatalogProtos.PartitionDescProto getPartition(String databaseName, String tableName,
+ String partitionName) throws CatalogException {
+ HiveCatalogStoreClientPool.HiveCatalogStoreClient client = null;
+ CatalogProtos.PartitionDescProto.Builder builder = null;
+
+ try {
+ client = clientPool.getClient();
+
+ Partition partition = client.getHiveClient().getPartition(databaseName, tableName, partitionName);
+ builder = CatalogProtos.PartitionDescProto.newBuilder();
+ builder.setPartitionName(partitionName);
+ builder.setPath(partition.getSd().getLocation());
+
+ String[] partitionNames = partitionName.split("/");
+
+ for (int i = 0; i < partition.getValues().size(); i++) {
+ String value = partition.getValues().get(i);
+ CatalogProtos.PartitionKeyProto.Builder keyBuilder = CatalogProtos.PartitionKeyProto.newBuilder();
+
+ String columnName = partitionNames[i].split("=")[0];
+ keyBuilder.setColumnName(columnName);
+ keyBuilder.setPartitionValue(value);
+ builder.addPartitionKeys(keyBuilder);
+ }
+ } catch (NoSuchObjectException e) {
+ return null;
+ } catch (Exception e) {
+ throw new CatalogException(e);
+ } finally {
+ if (client != null) {
+ client.release();
+ }
+ }
+ return builder.build();
+ }
+
+ @Override
+ public final void addFunction(final FunctionDesc func) throws CatalogException {
+ // TODO - not implemented yet
+ }
+
+ @Override
+ public final void deleteFunction(final FunctionDesc func) throws CatalogException {
+ // TODO - not implemented yet
+ }
+
+ @Override
+ public final void existFunction(final FunctionDesc func) throws CatalogException {
+ // TODO - not implemented yet
+ }
+
+ @Override
+ public final List<String> getAllFunctionNames() throws CatalogException {
+ // TODO - not implemented yet
+ return null;
+ }
+
+ @Override
+ public void dropIndex(String databaseName, String indexName) throws CatalogException {
+ // TODO - not implemented yet
+ }
+
+ @Override
+ public boolean existIndexByName(String databaseName, String indexName) throws CatalogException {
+ // TODO - not implemented yet
+ return false;
+ }
+
+ @Override
+ public CatalogProtos.IndexDescProto[] getIndexes(String databaseName, String tableName) throws CatalogException {
+ // TODO - not implemented yet
+ return null;
+ }
+
+ @Override
+ public void createIndex(CatalogProtos.IndexDescProto proto) throws CatalogException {
+ // TODO - not implemented yet
+ }
+
+ @Override
+ public CatalogProtos.IndexDescProto getIndexByName(String databaseName, String indexName) throws CatalogException {
+ // TODO - not implemented yet
+ return null;
+ }
+
+ @Override
+ public CatalogProtos.IndexDescProto getIndexByColumn(String databaseName, String tableName, String columnName)
+ throws CatalogException {
+ // TODO - not implemented yet
+ return null;
+ }
+
+ @Override
+ public boolean existIndexByColumn(String databaseName, String tableName, String columnName) throws CatalogException {
+ // TODO - not implemented yet
+ return false;
+ }
+
+ @Override
+ public final void close() {
+ clientPool.close();
+ }
+
+ private boolean existColumn(final String databaseName ,final String tableName , final String columnName) throws CatalogException {
+ boolean exist = false;
+ HiveCatalogStoreClientPool.HiveCatalogStoreClient client = null;
+
+ try {
+
+ client = clientPool.getClient();
+ Table table = client.getHiveClient().getTable(databaseName, tableName);
+ List<FieldSchema> columns = table.getSd().getCols();
+
+ for (final FieldSchema currentColumn : columns) {
+ if (currentColumn.getName().equalsIgnoreCase(columnName)) {
+ exist = true;
+ }
+ }
+ client.getHiveClient().alter_table(databaseName, tableName, table);
+
+ } catch (NoSuchObjectException nsoe) {
+ } catch (Exception e) {
+ throw new CatalogException(e);
+ } finally {
+ if (client != null) {
+ client.release();
+ }
+ }
+
+ return exist;
+ }
+
+ @Override
+ public List<ColumnProto> getAllColumns() throws CatalogException {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public List<DatabaseProto> getAllDatabases() throws CatalogException {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public List<IndexProto> getAllIndexes() throws CatalogException {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public List<TablePartitionProto> getAllPartitions() throws CatalogException {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public List<TableOptionProto> getAllTableOptions() throws CatalogException {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public List<TableStatsProto> getAllTableStats() throws CatalogException {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public List<TableDescriptorProto> getAllTables() throws CatalogException {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public List<TablespaceProto> getTablespaces() throws CatalogException {
+ throw new UnsupportedOperationException();
+ }
+}
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStoreClientPool.java
----------------------------------------------------------------------
diff --git a/tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStoreClientPool.java b/tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStoreClientPool.java
new file mode 100644
index 0000000..9053c56
--- /dev/null
+++ b/tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogStoreClientPool.java
@@ -0,0 +1,170 @@
+// Copyright 2012 Cloudera Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+package org.apache.tajo.catalog.store;
+
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.*;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.log4j.Logger;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+/**
+ * Manages a pool of HiveMetaStoreClient connections. If the connection pool is empty
+ * a new client is created and added to the pool. There is no size limit.
+ */
+public class HiveCatalogStoreClientPool {
+ private static final Logger LOG = Logger.getLogger(HiveCatalogStoreClientPool.class);
+ private final ConcurrentLinkedQueue<HiveCatalogStoreClient> clientPool =
+ new ConcurrentLinkedQueue<HiveCatalogStoreClient>();
+ private AtomicBoolean poolClosed = new AtomicBoolean(false);
+ private HiveConf hiveConf;
+
+ /**
+ * A wrapper around the HiveMetaStoreClient that manages interactions with the
+ * connection pool.
+ */
+ public class HiveCatalogStoreClient {
+ private final IMetaStoreClient hiveClient;
+ public AtomicBoolean isInUse = new AtomicBoolean(false);
+
+ private HiveCatalogStoreClient(HiveConf hiveConf) {
+ try {
+ HiveMetaHookLoader hookLoader = new HiveMetaHookLoader() {
+ @Override
+ public HiveMetaHook getHook(Table table) throws MetaException {
+ /* metadata hook implementation, or null if this
+ * storage handler does not need any metadata notifications
+ */
+ return null;
+ }
+ };
+
+ this.hiveClient = RetryingMetaStoreClient.getProxy(hiveConf, hookLoader, HiveMetaStoreClient.class.getName());
+ clientPool.add(this);
+ LOG.info("MetaStoreClient created (size = " + clientPool.size() + ")");
+ } catch (Exception e) {
+ // Turn in to an unchecked exception
+ throw new IllegalStateException(e);
+ }
+ }
+
+ /**
+ * Returns the internal HiveMetaStoreClient object.
+ */
+ public IMetaStoreClient getHiveClient() {
+ return hiveClient;
+ }
+
+ /**
+ * Returns this client back to the connection pool. If the connection pool has been
+ * closed, just close the Hive client connection.
+ */
+ public synchronized void release() {
+ if(!this.isInUse.getAndSet(false)){
+ return;
+ }
+ // Ensure the connection isn't returned to the pool if the pool has been closed.
+ // This lock is needed to ensure proper behavior when a thread reads poolClosed
+ // is false, but a call to pool.close() comes in immediately afterward.
+ if (poolClosed.get()) {
+ this.getHiveClient().close();
+ } else {
+ clientPool.add(this);
+ }
+ }
+
+ // Marks this client as in use
+ private void markInUse() {
+ isInUse.set(true);
+ }
+ }
+
+ public HiveCatalogStoreClientPool(int initialSize) {
+ this(initialSize, new HiveConf(HiveCatalogStoreClientPool.class));
+ }
+
+ public HiveCatalogStoreClientPool(int initialSize, HiveConf hiveConf) {
+ this.hiveConf = hiveConf;
+ addClients(initialSize);
+ }
+
+ public HiveCatalogStoreClientPool(int initialSize, Configuration conf) {
+ this.hiveConf = new HiveConf();
+ setParameters(conf);
+ addClients(initialSize);
+ }
+
+ public void setParameters(Configuration conf) {
+ for( Iterator<Entry<String, String>> iter = conf.iterator(); iter.hasNext();) {
+ Map.Entry<String, String> entry = iter.next();
+ this.hiveConf.set(entry.getKey(), entry.getValue());
+ }
+ }
+
+ /**
+ * Add numClients to the client pool.
+ */
+ public void addClients(int numClients) {
+ for (int i = 0; i < numClients; ++i) {
+ clientPool.add(new HiveCatalogStoreClient(hiveConf));
+ }
+ }
+
+ /**
+ * Gets a client from the pool. If the pool is empty a new client is created.
+ */
+ public synchronized HiveCatalogStoreClient getClient() {
+ // The MetaStoreClient c'tor relies on knowing the Hadoop version by asking
+ // org.apache.hadoop.util.VersionInfo. The VersionInfo class relies on opening
+ // the 'common-version-info.properties' file as a resource from hadoop-common*.jar
+ // using the Thread's context classloader. If necessary, set the Thread's context
+ // classloader, otherwise VersionInfo will fail in it's c'tor.
+ if (Thread.currentThread().getContextClassLoader() == null) {
+ Thread.currentThread().setContextClassLoader(ClassLoader.getSystemClassLoader());
+ }
+
+ HiveCatalogStoreClient client = clientPool.poll();
+ // The pool was empty so create a new client and return that.
+ if (client == null) {
+ client = new HiveCatalogStoreClient(hiveConf);
+ }
+ client.markInUse();
+
+ return client;
+ }
+
+ /**
+ * Removes all items from the connection pool and closes all Hive Meta Store client
+ * connections. Can be called multiple times.
+ */
+ public void close() {
+ // Ensure no more items get added to the pool once close is called.
+ if (poolClosed.getAndSet(true)) {
+ return;
+ }
+
+ HiveCatalogStoreClient client = null;
+ while ((client = clientPool.poll()) != null) {
+ client.getHiveClient().close();
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogUtil.java
----------------------------------------------------------------------
diff --git a/tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogUtil.java b/tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogUtil.java
new file mode 100644
index 0000000..59910b8
--- /dev/null
+++ b/tajo-catalog/tajo-catalog-drivers/tajo-hive/src/main/java/org/apache/tajo/catalog/store/HiveCatalogUtil.java
@@ -0,0 +1,127 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tajo.catalog.store;
+
+import com.google.common.base.Preconditions;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat;
+import org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat;
+import org.apache.hadoop.hive.ql.io.RCFileOutputFormat;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.serde.serdeConstants;
+import org.apache.tajo.catalog.exception.CatalogException;
+import org.apache.tajo.catalog.proto.CatalogProtos;
+import org.apache.tajo.catalog.CatalogUtil;
+import org.apache.tajo.common.TajoDataTypes;
+import org.apache.thrift.TException;
+import parquet.hadoop.mapred.DeprecatedParquetOutputFormat;
+
+public class HiveCatalogUtil {
+ public static void validateSchema(Table tblSchema) throws CatalogException {
+ for (FieldSchema fieldSchema : tblSchema.getCols()) {
+ String fieldType = fieldSchema.getType();
+ if (fieldType.equalsIgnoreCase("ARRAY") || fieldType.equalsIgnoreCase("STRUCT")
+ || fieldType.equalsIgnoreCase("MAP")) {
+ throw new CatalogException("Unsupported field type :" + fieldType.toUpperCase());
+ }
+ }
+ }
+
+ public static TajoDataTypes.Type getTajoFieldType(String fieldType) {
+ Preconditions.checkNotNull(fieldType);
+
+ if(fieldType.equalsIgnoreCase(serdeConstants.INT_TYPE_NAME)) {
+ return TajoDataTypes.Type.INT4;
+ } else if(fieldType.equalsIgnoreCase(serdeConstants.TINYINT_TYPE_NAME)) {
+ return TajoDataTypes.Type.INT1;
+ } else if(fieldType.equalsIgnoreCase(serdeConstants.SMALLINT_TYPE_NAME)) {
+ return TajoDataTypes.Type.INT2;
+ } else if(fieldType.equalsIgnoreCase(serdeConstants.BIGINT_TYPE_NAME)) {
+ return TajoDataTypes.Type.INT8;
+ } else if(fieldType.equalsIgnoreCase(serdeConstants.BOOLEAN_TYPE_NAME)) {
+ return TajoDataTypes.Type.BOOLEAN;
+ } else if(fieldType.equalsIgnoreCase(serdeConstants.FLOAT_TYPE_NAME)) {
+ return TajoDataTypes.Type.FLOAT4;
+ } else if(fieldType.equalsIgnoreCase(serdeConstants.DOUBLE_TYPE_NAME)) {
+ return TajoDataTypes.Type.FLOAT8;
+ } else if(fieldType.equalsIgnoreCase(serdeConstants.STRING_TYPE_NAME)) {
+ return TajoDataTypes.Type.TEXT;
+ } else if(fieldType.equalsIgnoreCase(serdeConstants.BINARY_TYPE_NAME)) {
+ return TajoDataTypes.Type.BLOB;
+ } else if(fieldType.equalsIgnoreCase(serdeConstants.TIMESTAMP_TYPE_NAME)) {
+ return TajoDataTypes.Type.TIMESTAMP;
+ } else if(fieldType.equalsIgnoreCase(serdeConstants.DATE_TYPE_NAME)) {
+ return TajoDataTypes.Type.DATE;
+ } else {
+ throw new CatalogException("Cannot find a matched type against from '" + fieldType + "'");
+ }
+ }
+
+ public static String getHiveFieldType(TajoDataTypes.DataType dataType) {
+ Preconditions.checkNotNull(dataType);
+
+ switch (dataType.getType()) {
+ case CHAR: return serdeConstants.CHAR_TYPE_NAME;
+ case BOOLEAN: return serdeConstants.BOOLEAN_TYPE_NAME;
+ case INT1: return serdeConstants.TINYINT_TYPE_NAME;
+ case INT2: return serdeConstants.SMALLINT_TYPE_NAME;
+ case INT4: return serdeConstants.INT_TYPE_NAME;
+ case INT8: return serdeConstants.BIGINT_TYPE_NAME;
+ case FLOAT4: return serdeConstants.FLOAT_TYPE_NAME;
+ case FLOAT8: return serdeConstants.DOUBLE_TYPE_NAME;
+ case TEXT: return serdeConstants.STRING_TYPE_NAME;
+ case VARCHAR: return serdeConstants.VARCHAR_TYPE_NAME;
+ case NCHAR: return serdeConstants.VARCHAR_TYPE_NAME;
+ case NVARCHAR: return serdeConstants.VARCHAR_TYPE_NAME;
+ case BINARY: return serdeConstants.BINARY_TYPE_NAME;
+ case VARBINARY: return serdeConstants.BINARY_TYPE_NAME;
+ case BLOB: return serdeConstants.BINARY_TYPE_NAME;
+ case DATE: return serdeConstants.DATE_TYPE_NAME;
+ case TIMESTAMP: return serdeConstants.TIMESTAMP_TYPE_NAME;
+ default:
+ throw new CatalogException(dataType + " is not supported.");
+ }
+ }
+
+ public static String getStoreType(String fileFormat) {
+ Preconditions.checkNotNull(fileFormat);
+
+ String[] fileFormatArrary = fileFormat.split("\\.");
+ if(fileFormatArrary.length < 1) {
+ throw new CatalogException("Hive file output format is wrong. - file output format:" + fileFormat);
+ }
+
+ String outputFormatClass = fileFormatArrary[fileFormatArrary.length-1];
+ if(outputFormatClass.equals(HiveIgnoreKeyTextOutputFormat.class.getSimpleName())) {
+ return CatalogUtil.TEXTFILE_NAME;
+ } else if(outputFormatClass.equals(HiveSequenceFileOutputFormat.class.getSimpleName())) {
+ return CatalogProtos.StoreType.SEQUENCEFILE.name();
+ } else if(outputFormatClass.equals(RCFileOutputFormat.class.getSimpleName())) {
+ return CatalogProtos.StoreType.RCFILE.name();
+ } else if(outputFormatClass.equals(DeprecatedParquetOutputFormat.class.getSimpleName())) {
+ return CatalogProtos.StoreType.PARQUET.name();
+ } else {
+ throw new CatalogException("Not supported file output format. - file output format:" + fileFormat);
+ }
+ }
+
+ public static Table getTable(IMetaStoreClient client, String dbName, String tableName) throws TException {
+ return new Table(client.getTable(dbName, tableName));
+ }
+}
[4/4] tajo git commit: TAJO-1442: Improve Hive Compatibility
Posted by bl...@apache.org.
TAJO-1442: Improve Hive Compatibility
Closes #453
Project: http://git-wip-us.apache.org/repos/asf/tajo/repo
Commit: http://git-wip-us.apache.org/repos/asf/tajo/commit/955a7bf8
Tree: http://git-wip-us.apache.org/repos/asf/tajo/tree/955a7bf8
Diff: http://git-wip-us.apache.org/repos/asf/tajo/diff/955a7bf8
Branch: refs/heads/master
Commit: 955a7bf84ec44e2ae301cd5775760166da5fbcc2
Parents: 7b78668
Author: JaeHwa Jung <bl...@apache.org>
Authored: Fri Apr 17 12:01:57 2015 +0900
Committer: JaeHwa Jung <bl...@apache.org>
Committed: Fri Apr 17 12:01:57 2015 +0900
----------------------------------------------------------------------
.travis.yml | 2 +-
tajo-catalog/pom.xml | 4 +-
tajo-catalog/tajo-catalog-drivers/pom.xml | 34 +-
.../tajo-catalog-drivers/tajo-hcatalog/pom.xml | 739 --------------
.../tajo/catalog/store/HCatalogStore.java | 977 -------------------
.../catalog/store/HCatalogStoreClientPool.java | 170 ----
.../apache/tajo/catalog/store/HCatalogUtil.java | 147 ---
.../tajo/catalog/store/TestHCatalogStore.java | 467 ---------
.../tajo-catalog-drivers/tajo-hive/pom.xml | 351 +++++++
.../tajo/catalog/store/HiveCatalogStore.java | 964 ++++++++++++++++++
.../store/HiveCatalogStoreClientPool.java | 170 ++++
.../tajo/catalog/store/HiveCatalogUtil.java | 127 +++
.../catalog/store/TestHiveCatalogStore.java | 504 ++++++++++
.../org/apache/tajo/catalog/TestCatalog.java | 6 +-
tajo-core/pom.xml | 212 ----
.../java/org/apache/tajo/QueryTestCaseBase.java | 4 +-
.../org/apache/tajo/TajoTestingCluster.java | 28 +-
.../org/apache/tajo/cli/tools/TestTajoDump.java | 4 +-
.../org/apache/tajo/cli/tsql/TestTajoCli.java | 56 +-
.../tajo/engine/query/TestAlterTablespace.java | 2 +-
.../apache/tajo/engine/query/TestCTASQuery.java | 6 +-
.../tajo/engine/query/TestCreateTable.java | 14 +-
.../tajo/engine/query/TestInsertQuery.java | 40 +-
.../apache/tajo/engine/query/TestNetTypes.java | 26 +-
.../tajo/engine/query/TestSelectQuery.java | 4 +-
.../apache/tajo/engine/query/TestSortQuery.java | 6 +-
.../tajo/engine/query/TestTablePartitions.java | 22 +-
.../org/apache/tajo/jdbc/TestResultSet.java | 4 +-
.../tajo/jdbc/TestTajoDatabaseMetaData.java | 22 +-
.../java/org/apache/tajo/jdbc/TestTajoJdbc.java | 12 +-
.../create_table_various_types_for_hcatalog.sql | 50 -
...ate_table_various_types_for_hive_catalog.sql | 50 +
tajo-dist/pom.xml | 4 +-
tajo-dist/src/main/bin/tajo | 3 -
.../src/main/conf/catalog-site.xml.template | 6 +-
tajo-dist/src/main/conf/tajo-env.sh | 2 +-
.../configuration/catalog_configuration.rst | 40 +-
.../src/main/sphinx/hcatalog_integration.rst | 52 -
tajo-docs/src/main/sphinx/hive_integration.rst | 42 +
tajo-docs/src/main/sphinx/index.rst | 2 +-
.../plan/rewrite/rules/FilterPushDownRule.java | 2 +-
tajo-project/pom.xml | 1 +
42 files changed, 2371 insertions(+), 3007 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/.travis.yml
----------------------------------------------------------------------
diff --git a/.travis.yml b/.travis.yml
index ab16f1d..4b3459a 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -37,4 +37,4 @@ notifications:
install: ./dev-support/travis-install-dependencies.sh
script:
- mvn clean install -Pparallel-test,hcatalog-0.12.0 -DLOG_LEVEL=WARN -Dmaven.fork.count=2
+ mvn clean install -Pparallel-test -DLOG_LEVEL=WARN -Dmaven.fork.count=2
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-catalog/pom.xml
----------------------------------------------------------------------
diff --git a/tajo-catalog/pom.xml b/tajo-catalog/pom.xml
index 9cfdd5d..864e1fe 100644
--- a/tajo-catalog/pom.xml
+++ b/tajo-catalog/pom.xml
@@ -132,9 +132,9 @@
run cp -r ${basedir}/tajo-catalog-common/target/tajo-catalog-common-${project.version}*.jar .
run cp -r ${basedir}/tajo-catalog-client/target/tajo-catalog-client-${project.version}*.jar .
run cp -r ${basedir}/tajo-catalog-server/target/tajo-catalog-server-${project.version}*.jar .
- if [ -f ${basedir}/tajo-catalog-drivers/tajo-hcatalog/target/tajo-hcatalog-${project.version}.jar ]
+ if [ -f ${basedir}/tajo-catalog-drivers/tajo-hive/target/tajo-hive-${project.version}.jar ]
then
- run cp -r ${basedir}/tajo-catalog-drivers/tajo-hcatalog/target/tajo-hcatalog-${project.version}*.jar .
+ run cp -r ${basedir}/tajo-catalog-drivers/tajo-hive/target/tajo-hive-${project.version}*.jar .
fi
echo
echo "Tajo Catalog dist layout available at: ${project.build.directory}/tajo-catalog-${project.version}"
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-catalog/tajo-catalog-drivers/pom.xml
----------------------------------------------------------------------
diff --git a/tajo-catalog/tajo-catalog-drivers/pom.xml b/tajo-catalog/tajo-catalog-drivers/pom.xml
index 89f3061..5a0e327 100644
--- a/tajo-catalog/tajo-catalog-drivers/pom.xml
+++ b/tajo-catalog/tajo-catalog-drivers/pom.xml
@@ -40,7 +40,7 @@
<artifactId>apache-rat-plugin</artifactId>
<configuration>
<excludes>
- <exclude>tajo-hcatalog/**</exclude>
+ <exclude>tajo-hive/**</exclude>
</excludes>
</configuration>
</plugin>
@@ -57,35 +57,9 @@
</plugins>
</build>
- <profiles>
- <profile>
- <id>hcatalog-0.12.0</id>
- <activation>
- <activeByDefault>false</activeByDefault>
- </activation>
- <modules>
- <module>tajo-hcatalog</module>
- </modules>
- </profile>
- <profile>
- <id>hcatalog-0.13.0</id>
- <activation>
- <activeByDefault>false</activeByDefault>
- </activation>
- <modules>
- <module>tajo-hcatalog</module>
- </modules>
- </profile>
- <profile>
- <id>hcatalog-0.13.1</id>
- <activation>
- <activeByDefault>false</activeByDefault>
- </activation>
- <modules>
- <module>tajo-hcatalog</module>
- </modules>
- </profile>
- </profiles>
+ <modules>
+ <module>tajo-hive</module>
+ </modules>
<reporting>
<plugins>
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/pom.xml
----------------------------------------------------------------------
diff --git a/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/pom.xml b/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/pom.xml
deleted file mode 100644
index 7c3efdd..0000000
--- a/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/pom.xml
+++ /dev/null
@@ -1,739 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
- -->
-
-<project xmlns="http://maven.apache.org/POM/4.0.0"
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
- <parent>
- <artifactId>tajo-project</artifactId>
- <groupId>org.apache.tajo</groupId>
- <version>0.11.0-SNAPSHOT</version>
- <relativePath>../../../tajo-project</relativePath>
- </parent>
- <modelVersion>4.0.0</modelVersion>
- <artifactId>tajo-hcatalog</artifactId>
- <packaging>jar</packaging>
- <name>Tajo Catalog Drivers HCatalog</name>
- <properties>
- <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
- <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
- </properties>
-
- <build>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-compiler-plugin</artifactId>
- <configuration>
- <source>1.6</source>
- <target>1.6</target>
- <encoding>${project.build.sourceEncoding}</encoding>
- </configuration>
- </plugin>
- <plugin>
- <groupId>org.apache.rat</groupId>
- <artifactId>apache-rat-plugin</artifactId>
- <executions>
- <execution>
- <phase>verify</phase>
- <goals>
- <goal>check</goal>
- </goals>
- </execution>
- </executions>
- <configuration>
- <excludes>
- <exclude>derby.log</exclude>
- </excludes>
- </configuration>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-dependency-plugin</artifactId>
- <executions>
- <execution>
- <id>copy-dependencies</id>
- <phase>prepare-package</phase>
- <goals>
- <goal>copy-dependencies</goal>
- </goals>
- <configuration>
- <includeScope>runtime</includeScope>
- <outputDirectory>${project.build.directory}/lib</outputDirectory>
- <overWriteReleases>false</overWriteReleases>
- <overWriteSnapshots>false</overWriteSnapshots>
- <overWriteIfNewer>true</overWriteIfNewer>
- </configuration>
- </execution>
- </executions>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-surefire-report-plugin</artifactId>
- </plugin>
- </plugins>
- </build>
-
- <dependencies>
- <dependency>
- <groupId>org.apache.tajo</groupId>
- <artifactId>tajo-common</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.tajo</groupId>
- <artifactId>tajo-catalog-common</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.tajo</groupId>
- <artifactId>tajo-catalog-client</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.tajo</groupId>
- <artifactId>tajo-catalog-server</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.tajo</groupId>
- <artifactId>tajo-rpc-protobuf</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.tajo</groupId>
- <artifactId>tajo-storage-common</artifactId>
- </dependency>
- <dependency>
- <groupId>junit</groupId>
- <artifactId>junit</artifactId>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.thrift</groupId>
- <artifactId>libfb303</artifactId>
- <version>0.9.0</version>
- <scope>provided</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.thrift</groupId>
- <artifactId>libthrift</artifactId>
- <version>0.9.0</version>
- <scope>provided</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-mapreduce-client-core</artifactId>
- <version>${hadoop.version}</version>
- <scope>provided</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- <version>${hadoop.version}</version>
- <scope>provided</scope>
- </dependency>
- </dependencies>
-
- <profiles>
- <profile>
- <id>hcatalog-0.12.0</id>
- <activation>
- <activeByDefault>false</activeByDefault>
- </activation>
- <properties>
- <hive.version>0.12.0</hive.version>
- <parquet.version>1.5.0</parquet.version>
- <parquet.format.version>2.1.0</parquet.format.version>
- </properties>
- <dependencies>
- <dependency>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-exec</artifactId>
- <version>${hive.version}</version>
- <scope>provided</scope>
- <exclusions>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-common</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-contrib</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-hbase-handler</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-metastore</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-serde</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-shims</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-testutils</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.thrift</groupId>
- <artifactId>libfb303</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.thrift</groupId>
- <artifactId>libthrift</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.jolbox</groupId>
- <artifactId>bonecp</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.google.protobuf</groupId>
- <artifactId>protobuf-java</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- <dependency>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-metastore</artifactId>
- <version>${hive.version}</version>
- <scope>provided</scope>
- <exclusions>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-common</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-serde</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-shimss</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.thrift</groupId>
- <artifactId>libfb303</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.thrift</groupId>
- <artifactId>libthrift</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.jolbox</groupId>
- <artifactId>bonecp</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- <dependency>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-cli</artifactId>
- <version>${hive.version}</version>
- <scope>provided</scope>
- <exclusions>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-common</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-exec</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-metastore</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-serde</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-service</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-shims</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.jolbox</groupId>
- <artifactId>bonecp</artifactId>
- </exclusion>
- <exclusion>
- <groupId>jline</groupId>
- <artifactId>jline</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- <dependency>
- <groupId>org.apache.hive.hcatalog</groupId>
- <artifactId>hcatalog-core</artifactId>
- <version>${hive.version}</version>
- <exclusions>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-cli</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-common</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-exec</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-metastore</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-serde</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-service</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-shims</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.jolbox</groupId>
- <artifactId>bonecp</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- <dependency>
- <groupId>com.twitter</groupId>
- <artifactId>parquet-hive-bundle</artifactId>
- <version>${parquet.version}</version>
- </dependency>
- </dependencies>
- </profile>
- <profile>
- <id>hcatalog-0.13.0</id>
- <activation>
- <activeByDefault>false</activeByDefault>
- </activation>
- <properties>
- <hive.version>0.13.0</hive.version>
- <parquet.version>1.5.0</parquet.version>
- <parquet.format.version>2.1.0</parquet.format.version>
- </properties>
- <dependencies>
- <dependency>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-exec</artifactId>
- <version>${hive.version}</version>
- <scope>provided</scope>
- <exclusions>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-common</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-contrib</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-hbase-handler</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-metastore</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-serde</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-shims</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-testutils</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.thrift</groupId>
- <artifactId>libfb303</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.thrift</groupId>
- <artifactId>libthrift</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.jolbox</groupId>
- <artifactId>bonecp</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.google.protobuf</groupId>
- <artifactId>protobuf-java</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- <dependency>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-metastore</artifactId>
- <version>${hive.version}</version>
- <scope>provided</scope>
- <exclusions>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-common</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-serde</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-shimss</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.thrift</groupId>
- <artifactId>libfb303</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.thrift</groupId>
- <artifactId>libthrift</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.jolbox</groupId>
- <artifactId>bonecp</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- <dependency>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-cli</artifactId>
- <version>${hive.version}</version>
- <scope>provided</scope>
- <exclusions>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-common</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-exec</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-metastore</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-serde</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-service</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-shims</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.jolbox</groupId>
- <artifactId>bonecp</artifactId>
- </exclusion>
- <exclusion>
- <groupId>jline</groupId>
- <artifactId>jline</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- <dependency>
- <groupId>org.apache.hive.hcatalog</groupId>
- <artifactId>hive-hcatalog-core</artifactId>
- <version>${hive.version}</version>
- <exclusions>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-cli</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-common</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-exec</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-metastore</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.google.guava</groupId>
- <artifactId>guava</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.codehaus.jackson</groupId>
- <artifactId>jackson-mapper-asl</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- <dependency>
- <groupId>com.twitter</groupId>
- <artifactId>parquet-hive-bundle</artifactId>
- <version>${parquet.version}</version>
- </dependency>
- </dependencies>
- </profile>
- <profile>
- <id>hcatalog-0.13.1</id>
- <activation>
- <activeByDefault>false</activeByDefault>
- </activation>
- <properties>
- <hive.version>0.13.1</hive.version>
- <parquet.version>1.5.0</parquet.version>
- <parquet.format.version>2.1.0</parquet.format.version>
- </properties>
- <dependencies>
- <dependency>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-exec</artifactId>
- <version>${hive.version}</version>
- <scope>provided</scope>
- <exclusions>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-common</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-contrib</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-hbase-handler</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-metastore</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-serde</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-shims</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-testutils</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.thrift</groupId>
- <artifactId>libfb303</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.thrift</groupId>
- <artifactId>libthrift</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.jolbox</groupId>
- <artifactId>bonecp</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.google.protobuf</groupId>
- <artifactId>protobuf-java</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- <dependency>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-metastore</artifactId>
- <version>${hive.version}</version>
- <scope>provided</scope>
- <exclusions>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-common</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-serde</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-shimss</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.thrift</groupId>
- <artifactId>libfb303</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.thrift</groupId>
- <artifactId>libthrift</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.jolbox</groupId>
- <artifactId>bonecp</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- <dependency>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-cli</artifactId>
- <version>${hive.version}</version>
- <scope>provided</scope>
- <exclusions>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-common</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-exec</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-metastore</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-serde</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-service</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-shims</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.jolbox</groupId>
- <artifactId>bonecp</artifactId>
- </exclusion>
- <exclusion>
- <groupId>jline</groupId>
- <artifactId>jline</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- <dependency>
- <groupId>org.apache.hive.hcatalog</groupId>
- <artifactId>hive-hcatalog-core</artifactId>
- <version>${hive.version}</version>
- <exclusions>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-cli</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-common</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-exec</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-metastore</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.google.guava</groupId>
- <artifactId>guava</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.codehaus.jackson</groupId>
- <artifactId>jackson-mapper-asl</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- <dependency>
- <groupId>com.twitter</groupId>
- <artifactId>parquet-hive-bundle</artifactId>
- <version>${parquet.version}</version>
- </dependency>
- </dependencies>
- </profile>
- <profile>
- <id>docs</id>
- <activation>
- <activeByDefault>false</activeByDefault>
- </activation>
- <build>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-javadoc-plugin</artifactId>
- <executions>
- <execution>
- <!-- build javadoc jars per jar for publishing to maven -->
- <id>module-javadocs</id>
- <phase>package</phase>
- <goals>
- <goal>jar</goal>
- </goals>
- <configuration>
- <destDir>${project.build.directory}</destDir>
- </configuration>
- </execution>
- </executions>
- </plugin>
- </plugins>
- </build>
- </profile>
- <profile>
- <id>src</id>
- <activation>
- <activeByDefault>false</activeByDefault>
- </activation>
- <build>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-source-plugin</artifactId>
- <executions>
- <execution>
- <!-- builds source jars and attaches them to the project for publishing -->
- <id>hadoop-java-sources</id>
- <phase>package</phase>
- <goals>
- <goal>jar-no-fork</goal>
- </goals>
- </execution>
- </executions>
- </plugin>
- </plugins>
- </build>
- </profile>
- </profiles>
-
- <reporting>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-surefire-report-plugin</artifactId>
- </plugin>
- </plugins>
- </reporting>
-
-</project>
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java
----------------------------------------------------------------------
diff --git a/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java b/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java
deleted file mode 100644
index e829b94..0000000
--- a/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java
+++ /dev/null
@@ -1,977 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.tajo.catalog.store;
-
-import com.google.common.collect.Lists;
-
-import org.apache.commons.lang.StringEscapeUtils;
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.hive.metastore.TableType;
-import org.apache.hadoop.hive.metastore.api.*;
-import org.apache.hadoop.hive.serde.serdeConstants;
-import org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe;
-import org.apache.hadoop.hive.serde2.columnar.LazyBinaryColumnarSerDe;
-import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe;
-import org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe;
-import org.apache.hcatalog.common.HCatUtil;
-import org.apache.hcatalog.data.schema.HCatFieldSchema;
-import org.apache.hcatalog.data.schema.HCatSchema;
-import org.apache.tajo.TajoConstants;
-import org.apache.tajo.catalog.*;
-import org.apache.tajo.catalog.exception.*;
-import org.apache.tajo.catalog.partition.PartitionMethodDesc;
-import org.apache.tajo.catalog.proto.CatalogProtos;
-import org.apache.tajo.catalog.proto.CatalogProtos.ColumnProto;
-import org.apache.tajo.catalog.proto.CatalogProtos.DatabaseProto;
-import org.apache.tajo.catalog.proto.CatalogProtos.IndexProto;
-import org.apache.tajo.catalog.proto.CatalogProtos.TableDescriptorProto;
-import org.apache.tajo.catalog.proto.CatalogProtos.TableOptionProto;
-import org.apache.tajo.catalog.proto.CatalogProtos.TablePartitionProto;
-import org.apache.tajo.catalog.proto.CatalogProtos.TableStatsProto;
-import org.apache.tajo.catalog.proto.CatalogProtos.TablespaceProto;
-import org.apache.tajo.catalog.statistics.TableStats;
-import org.apache.tajo.common.TajoDataTypes;
-import org.apache.tajo.common.exception.NotImplementedException;
-import org.apache.tajo.conf.TajoConf;
-import org.apache.tajo.exception.InternalException;
-import org.apache.tajo.storage.StorageConstants;
-import org.apache.tajo.util.KeyValueSet;
-import org.apache.thrift.TException;
-
-import java.io.IOException;
-import java.util.*;
-
-import static org.apache.tajo.catalog.proto.CatalogProtos.PartitionType;
-
-public class HCatalogStore extends CatalogConstants implements CatalogStore {
- protected final Log LOG = LogFactory.getLog(getClass());
-
- private static String HIVE_WAREHOUSE_DIR_CONF_KEY = "hive.metastore.warehouse.dir";
-
- protected Configuration conf;
- private static final int CLIENT_POOL_SIZE = 2;
- private final HCatalogStoreClientPool clientPool;
- private final String defaultTableSpaceUri;
-
- public HCatalogStore(final Configuration conf) throws InternalException {
- if (!(conf instanceof TajoConf)) {
- throw new CatalogException("Invalid Configuration Type:" + conf.getClass().getSimpleName());
- }
- this.conf = conf;
- this.defaultTableSpaceUri = TajoConf.getWarehouseDir((TajoConf) conf).toString();
- this.clientPool = new HCatalogStoreClientPool(CLIENT_POOL_SIZE, conf);
- }
-
- @Override
- public boolean existTable(final String databaseName, final String tableName) throws CatalogException {
- boolean exist = false;
- org.apache.hadoop.hive.ql.metadata.Table table;
- HCatalogStoreClientPool.HCatalogStoreClient client = null;
-
- // get table
- try {
- client = clientPool.getClient();
- table = HCatalogUtil.getTable(client.getHiveClient(), databaseName, tableName);
- if (table != null) {
- exist = true;
- }
- } catch (NoSuchObjectException nsoe) {
- exist = false;
- } catch (Exception e) {
- throw new CatalogException(e);
- } finally {
- if (client != null) {
- client.release();
- }
- }
-
- return exist;
- }
-
- @Override
- public final CatalogProtos.TableDescProto getTable(String databaseName, final String tableName) throws CatalogException {
- org.apache.hadoop.hive.ql.metadata.Table table = null;
- HCatalogStoreClientPool.HCatalogStoreClient client = null;
- Path path = null;
- CatalogProtos.StoreType storeType = null;
- org.apache.tajo.catalog.Schema schema = null;
- KeyValueSet options = null;
- TableStats stats = null;
- PartitionMethodDesc partitions = null;
-
- //////////////////////////////////
- // set tajo table schema.
- //////////////////////////////////
- try {
- // get hive table schema
- try {
- client = clientPool.getClient();
- table = HCatalogUtil.getTable(client.getHiveClient(), databaseName, tableName);
- path = table.getPath();
- } catch (NoSuchObjectException nsoe) {
- throw new CatalogException("Table not found. - tableName:" + tableName, nsoe);
- } catch (Exception e) {
- throw new CatalogException(e);
- }
-
- // convert hcatalog field schema into tajo field schema.
- schema = new org.apache.tajo.catalog.Schema();
- HCatSchema tableSchema = null;
-
- try {
- tableSchema = HCatUtil.getTableSchemaWithPtnCols(table);
- } catch (IOException ioe) {
- throw new CatalogException("Fail to get table schema. - tableName:" + tableName, ioe);
- }
- List<HCatFieldSchema> fieldSchemaList = tableSchema.getFields();
- boolean isPartitionKey = false;
- for (HCatFieldSchema eachField : fieldSchemaList) {
- isPartitionKey = false;
-
- if (table.getPartitionKeys() != null) {
- for (FieldSchema partitionKey : table.getPartitionKeys()) {
- if (partitionKey.getName().equals(eachField.getName())) {
- isPartitionKey = true;
- }
- }
- }
-
- if (!isPartitionKey) {
- String fieldName = databaseName + CatalogConstants.IDENTIFIER_DELIMITER + tableName +
- CatalogConstants.IDENTIFIER_DELIMITER + eachField.getName();
- TajoDataTypes.Type dataType = HCatalogUtil.getTajoFieldType(eachField.getType().toString());
- schema.addColumn(fieldName, dataType);
- }
- }
-
- // validate field schema.
- try {
- HCatalogUtil.validateHCatTableAndTajoSchema(tableSchema);
- } catch (Exception e) {
- throw new CatalogException("HCatalog cannot support schema. - schema:" + tableSchema.toString(), e);
- }
-
- stats = new TableStats();
- options = new KeyValueSet();
- options.putAll(table.getParameters());
- options.remove("EXTERNAL");
-
- Properties properties = table.getMetadata();
- if (properties != null) {
- // set field delimiter
- String fieldDelimiter = "", nullFormat = "";
- if (properties.getProperty(serdeConstants.FIELD_DELIM) != null) {
- fieldDelimiter = properties.getProperty(serdeConstants.FIELD_DELIM);
- } else {
- // if hive table used default row format delimiter, Properties doesn't have it.
- // So, Tajo must set as follows:
- fieldDelimiter = "\u0001";
- }
-
- // set null format
- if (properties.getProperty(serdeConstants.SERIALIZATION_NULL_FORMAT) != null) {
- nullFormat = properties.getProperty(serdeConstants.SERIALIZATION_NULL_FORMAT);
- } else {
- nullFormat = "\\N";
- }
- options.remove(serdeConstants.SERIALIZATION_NULL_FORMAT);
-
- // set file output format
- String fileOutputformat = properties.getProperty(hive_metastoreConstants.FILE_OUTPUT_FORMAT);
- storeType = CatalogUtil.getStoreType(HCatalogUtil.getStoreType(fileOutputformat));
-
- if (storeType.equals(CatalogProtos.StoreType.TEXTFILE)) {
- options.set(StorageConstants.TEXT_DELIMITER, StringEscapeUtils.escapeJava(fieldDelimiter));
- options.set(StorageConstants.TEXT_NULL, StringEscapeUtils.escapeJava(nullFormat));
- } else if (storeType.equals(CatalogProtos.StoreType.RCFILE)) {
- options.set(StorageConstants.RCFILE_NULL, StringEscapeUtils.escapeJava(nullFormat));
- String serde = properties.getProperty(serdeConstants.SERIALIZATION_LIB);
- if (LazyBinaryColumnarSerDe.class.getName().equals(serde)) {
- options.set(StorageConstants.RCFILE_SERDE, StorageConstants.DEFAULT_BINARY_SERDE);
- } else if (ColumnarSerDe.class.getName().equals(serde)) {
- options.set(StorageConstants.RCFILE_SERDE, StorageConstants.DEFAULT_TEXT_SERDE);
- }
- } else if (storeType.equals(CatalogProtos.StoreType.SEQUENCEFILE) ) {
- options.set(StorageConstants.SEQUENCEFILE_DELIMITER, StringEscapeUtils.escapeJava(fieldDelimiter));
- options.set(StorageConstants.SEQUENCEFILE_NULL, StringEscapeUtils.escapeJava(nullFormat));
- String serde = properties.getProperty(serdeConstants.SERIALIZATION_LIB);
- if (LazyBinarySerDe.class.getName().equals(serde)) {
- options.set(StorageConstants.SEQUENCEFILE_SERDE, StorageConstants.DEFAULT_BINARY_SERDE);
- } else if (LazySimpleSerDe.class.getName().equals(serde)) {
- options.set(StorageConstants.SEQUENCEFILE_SERDE, StorageConstants.DEFAULT_TEXT_SERDE);
- }
- }
-
- // set data size
- long totalSize = 0;
- if (properties.getProperty("totalSize") != null) {
- totalSize = Long.parseLong(properties.getProperty("totalSize"));
- } else {
- try {
- FileSystem fs = path.getFileSystem(conf);
- if (fs.exists(path)) {
- totalSize = fs.getContentSummary(path).getLength();
- }
- } catch (IOException ioe) {
- throw new CatalogException("Fail to get path. - path:" + path.toString(), ioe);
- }
- }
- stats.setNumBytes(totalSize);
- }
-
- // set partition keys
- List<FieldSchema> partitionKeys = table.getPartitionKeys();
-
- if (null != partitionKeys) {
- org.apache.tajo.catalog.Schema expressionSchema = new org.apache.tajo.catalog.Schema();
- StringBuilder sb = new StringBuilder();
- if (partitionKeys.size() > 0) {
- for (int i = 0; i < partitionKeys.size(); i++) {
- FieldSchema fieldSchema = partitionKeys.get(i);
- TajoDataTypes.Type dataType = HCatalogUtil.getTajoFieldType(fieldSchema.getType().toString());
- String fieldName = databaseName + CatalogConstants.IDENTIFIER_DELIMITER + tableName +
- CatalogConstants.IDENTIFIER_DELIMITER + fieldSchema.getName();
- expressionSchema.addColumn(new Column(fieldName, dataType));
- if (i > 0) {
- sb.append(",");
- }
- sb.append(fieldSchema.getName());
- }
- partitions = new PartitionMethodDesc(
- databaseName,
- tableName,
- PartitionType.COLUMN,
- sb.toString(),
- expressionSchema);
- }
- }
- } finally {
- if(client != null) client.release();
- }
- TableMeta meta = new TableMeta(storeType, options);
- TableDesc tableDesc = new TableDesc(databaseName + "." + tableName, schema, meta, path.toUri());
- if (table.getTableType().equals(TableType.EXTERNAL_TABLE)) {
- tableDesc.setExternal(true);
- }
- if (stats != null) {
- tableDesc.setStats(stats);
- }
- if (partitions != null) {
- tableDesc.setPartitionMethod(partitions);
- }
- return tableDesc.getProto();
- }
-
-
- private TajoDataTypes.Type getDataType(final String typeStr) {
- try {
- return Enum.valueOf(TajoDataTypes.Type.class, typeStr);
- } catch (IllegalArgumentException iae) {
- LOG.error("Cannot find a matched type against from '" + typeStr + "'");
- return null;
- }
- }
-
- @Override
- public final List<String> getAllTableNames(String databaseName) throws CatalogException {
- HCatalogStoreClientPool.HCatalogStoreClient client = null;
-
- try {
- client = clientPool.getClient();
- return client.getHiveClient().getAllTables(databaseName);
- } catch (TException e) {
- throw new CatalogException(e);
- } finally {
- if(client != null) client.release();
- }
- }
-
- @Override
- public void createTablespace(String spaceName, String spaceUri) throws CatalogException {
- // SKIP
- }
-
- @Override
- public boolean existTablespace(String spaceName) throws CatalogException {
- // SKIP
- return spaceName.equals(TajoConstants.DEFAULT_TABLESPACE_NAME);
- }
-
- @Override
- public void dropTablespace(String spaceName) throws CatalogException {
- // SKIP
- }
-
- @Override
- public Collection<String> getAllTablespaceNames() throws CatalogException {
- return Lists.newArrayList(TajoConstants.DEFAULT_TABLESPACE_NAME);
- }
-
- @Override
- public TablespaceProto getTablespace(String spaceName) throws CatalogException {
- if (spaceName.equals(TajoConstants.DEFAULT_TABLESPACE_NAME)) {
- TablespaceProto.Builder builder = TablespaceProto.newBuilder();
- builder.setSpaceName(TajoConstants.DEFAULT_TABLESPACE_NAME);
- builder.setUri(defaultTableSpaceUri);
- return builder.build();
- } else {
- throw new CatalogException("tablespace concept is not supported in HCatalogStore");
- }
- }
-
- @Override
- public void updateTableStats(CatalogProtos.UpdateTableStatsProto statsProto) throws
- CatalogException {
- // TODO - not implemented yet
- }
-
- @Override
- public void alterTablespace(CatalogProtos.AlterTablespaceProto alterProto) throws CatalogException {
- throw new CatalogException("tablespace concept is not supported in HCatalogStore");
- }
-
- @Override
- public void createDatabase(String databaseName, String tablespaceName) throws CatalogException {
- HCatalogStoreClientPool.HCatalogStoreClient client = null;
-
- try {
- Database database = new Database(
- databaseName,
- "",
- defaultTableSpaceUri + "/" + databaseName,
- new HashMap<String, String>());
- client = clientPool.getClient();
- client.getHiveClient().createDatabase(database);
- } catch (AlreadyExistsException e) {
- throw new AlreadyExistsDatabaseException(databaseName);
- } catch (Throwable t) {
- throw new CatalogException(t);
- } finally {
- if (client != null) {
- client.release();
- }
- }
- }
-
- @Override
- public boolean existDatabase(String databaseName) throws CatalogException {
- HCatalogStoreClientPool.HCatalogStoreClient client = null;
-
- try {
- client = clientPool.getClient();
- List<String> databaseNames = client.getHiveClient().getAllDatabases();
- return databaseNames.contains(databaseName);
- } catch (Throwable t) {
- throw new CatalogException(t);
- } finally {
- if (client != null) {
- client.release();
- }
- }
- }
-
- @Override
- public void dropDatabase(String databaseName) throws CatalogException {
- HCatalogStoreClientPool.HCatalogStoreClient client = null;
-
- try {
- client = clientPool.getClient();
- client.getHiveClient().dropDatabase(databaseName);
- } catch (NoSuchObjectException e) {
- throw new NoSuchDatabaseException(databaseName);
- } catch (Throwable t) {
- throw new CatalogException(databaseName);
- } finally {
- if (client != null) {
- client.release();
- }
- }
- }
-
- @Override
- public Collection<String> getAllDatabaseNames() throws CatalogException {
- HCatalogStoreClientPool.HCatalogStoreClient client = null;
-
- try {
- client = clientPool.getClient();
- return client.getHiveClient().getAllDatabases();
- } catch (TException e) {
- throw new CatalogException(e);
- } finally {
- if (client != null) {
- client.release();
- }
- }
- }
-
- @Override
- public final void createTable(final CatalogProtos.TableDescProto tableDescProto) throws CatalogException {
- HCatalogStoreClientPool.HCatalogStoreClient client = null;
-
- TableDesc tableDesc = new TableDesc(tableDescProto);
- String[] splitted = CatalogUtil.splitFQTableName(tableDesc.getName());
- String databaseName = splitted[0];
- String tableName = splitted[1];
-
- try {
- client = clientPool.getClient();
-
- org.apache.hadoop.hive.metastore.api.Table table = new org.apache.hadoop.hive.metastore.api.Table();
- table.setDbName(databaseName);
- table.setTableName(tableName);
- table.setParameters(new HashMap<String, String>(tableDesc.getMeta().getOptions().getAllKeyValus()));
- // TODO: set owner
- //table.setOwner();
-
- StorageDescriptor sd = new StorageDescriptor();
- sd.setSerdeInfo(new SerDeInfo());
- sd.getSerdeInfo().setParameters(new HashMap<String, String>());
- sd.getSerdeInfo().setName(table.getTableName());
-
- // if tajo set location method, thrift client make exception as follows:
- // Caused by: MetaException(message:java.lang.NullPointerException)
- // If you want to modify table path, you have to modify on Hive cli.
- if (tableDesc.isExternal()) {
- table.setTableType(TableType.EXTERNAL_TABLE.name());
- table.putToParameters("EXTERNAL", "TRUE");
-
- Path tablePath = new Path(tableDesc.getPath());
- FileSystem fs = tablePath.getFileSystem(conf);
- if (fs.isFile(tablePath)) {
- LOG.warn("A table path is a file, but HCatalog does not allow a file path.");
- sd.setLocation(tablePath.getParent().toString());
- } else {
- sd.setLocation(tablePath.toString());
- }
- }
-
- // set column information
- List<Column> columns = tableDesc.getSchema().getColumns();
- ArrayList<FieldSchema> cols = new ArrayList<FieldSchema>(columns.size());
-
- for (Column eachField : columns) {
- cols.add(new FieldSchema(eachField.getSimpleName(),
- HCatalogUtil.getHiveFieldType(eachField.getDataType()), ""));
- }
- sd.setCols(cols);
-
- // set partition keys
- if (tableDesc.hasPartition() && tableDesc.getPartitionMethod().getPartitionType().equals(PartitionType.COLUMN)) {
- List<FieldSchema> partitionKeys = new ArrayList<FieldSchema>();
- for (Column eachPartitionKey : tableDesc.getPartitionMethod().getExpressionSchema().getColumns()) {
- partitionKeys.add(new FieldSchema(eachPartitionKey.getSimpleName(),
- HCatalogUtil.getHiveFieldType(eachPartitionKey.getDataType()), ""));
- }
- table.setPartitionKeys(partitionKeys);
- }
-
- if (tableDesc.getMeta().getStoreType().equals(CatalogProtos.StoreType.RCFILE)) {
- String serde = tableDesc.getMeta().getOption(StorageConstants.RCFILE_SERDE);
- sd.setInputFormat(org.apache.hadoop.hive.ql.io.RCFileInputFormat.class.getName());
- sd.setOutputFormat(org.apache.hadoop.hive.ql.io.RCFileOutputFormat.class.getName());
- if (StorageConstants.DEFAULT_TEXT_SERDE.equals(serde)) {
- sd.getSerdeInfo().setSerializationLib(org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe.class.getName());
- } else {
- sd.getSerdeInfo().setSerializationLib(
- org.apache.hadoop.hive.serde2.columnar.LazyBinaryColumnarSerDe.class.getName());
- }
-
- if (tableDesc.getMeta().getOptions().containsKey(StorageConstants.RCFILE_NULL)) {
- table.putToParameters(serdeConstants.SERIALIZATION_NULL_FORMAT,
- StringEscapeUtils.unescapeJava(tableDesc.getMeta().getOption(StorageConstants.RCFILE_NULL)));
- }
- } else if (tableDesc.getMeta().getStoreType().equals(CatalogProtos.StoreType.CSV)
- || tableDesc.getMeta().getStoreType().equals(CatalogProtos.StoreType.TEXTFILE)) {
- sd.getSerdeInfo().setSerializationLib(org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.class.getName());
- sd.setInputFormat(org.apache.hadoop.mapred.TextInputFormat.class.getName());
- sd.setOutputFormat(org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.class.getName());
-
- String fieldDelimiter = tableDesc.getMeta().getOption(StorageConstants.TEXT_DELIMITER,
- StorageConstants.DEFAULT_FIELD_DELIMITER);
-
- // User can use an unicode for filed delimiter such as \u0001, \001.
- // In this case, java console will convert this value into "\\u001".
- // And hive will un-espace this value again.
- // As a result, user can use right field delimiter.
- // So, we have to un-escape this value.
- sd.getSerdeInfo().putToParameters(serdeConstants.SERIALIZATION_FORMAT,
- StringEscapeUtils.unescapeJava(fieldDelimiter));
- sd.getSerdeInfo().putToParameters(serdeConstants.FIELD_DELIM,
- StringEscapeUtils.unescapeJava(fieldDelimiter));
- table.getParameters().remove(StorageConstants.TEXT_DELIMITER);
-
- if (tableDesc.getMeta().containsOption(StorageConstants.TEXT_NULL)) {
- table.putToParameters(serdeConstants.SERIALIZATION_NULL_FORMAT,
- StringEscapeUtils.unescapeJava(tableDesc.getMeta().getOption(StorageConstants.TEXT_NULL)));
- table.getParameters().remove(StorageConstants.TEXT_NULL);
- }
- } else if (tableDesc.getMeta().getStoreType().equals(CatalogProtos.StoreType.SEQUENCEFILE)) {
- String serde = tableDesc.getMeta().getOption(StorageConstants.SEQUENCEFILE_SERDE);
- sd.setInputFormat(org.apache.hadoop.mapred.SequenceFileInputFormat.class.getName());
- sd.setOutputFormat(org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat.class.getName());
-
- if (StorageConstants.DEFAULT_TEXT_SERDE.equals(serde)) {
- sd.getSerdeInfo().setSerializationLib(org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.class.getName());
-
- String fieldDelimiter = tableDesc.getMeta().getOption(StorageConstants.SEQUENCEFILE_DELIMITER,
- StorageConstants.DEFAULT_FIELD_DELIMITER);
-
- // User can use an unicode for filed delimiter such as \u0001, \001.
- // In this case, java console will convert this value into "\\u001".
- // And hive will un-espace this value again.
- // As a result, user can use right field delimiter.
- // So, we have to un-escape this value.
- sd.getSerdeInfo().putToParameters(serdeConstants.SERIALIZATION_FORMAT,
- StringEscapeUtils.unescapeJava(fieldDelimiter));
- sd.getSerdeInfo().putToParameters(serdeConstants.FIELD_DELIM,
- StringEscapeUtils.unescapeJava(fieldDelimiter));
- table.getParameters().remove(StorageConstants.SEQUENCEFILE_DELIMITER);
- } else {
- sd.getSerdeInfo().setSerializationLib(org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.class.getName());
- }
-
- if (tableDesc.getMeta().containsOption(StorageConstants.SEQUENCEFILE_NULL)) {
- table.putToParameters(serdeConstants.SERIALIZATION_NULL_FORMAT,
- StringEscapeUtils.unescapeJava(tableDesc.getMeta().getOption(StorageConstants.SEQUENCEFILE_NULL)));
- table.getParameters().remove(StorageConstants.SEQUENCEFILE_NULL);
- }
- } else {
- if (tableDesc.getMeta().getStoreType().equals(CatalogProtos.StoreType.PARQUET)) {
- sd.setInputFormat(parquet.hive.DeprecatedParquetInputFormat.class.getName());
- sd.setOutputFormat(parquet.hive.DeprecatedParquetOutputFormat.class.getName());
- sd.getSerdeInfo().setSerializationLib(parquet.hive.serde.ParquetHiveSerDe.class.getName());
- } else {
- throw new CatalogException(new NotImplementedException(tableDesc.getMeta().getStoreType
- ().name()));
- }
- }
-
- sd.setSortCols(new ArrayList<Order>());
-
- table.setSd(sd);
- client.getHiveClient().createTable(table);
- } catch (RuntimeException e) {
- throw e;
- } catch (Exception e) {
- throw new CatalogException(e);
- } finally {
- if(client != null) client.release();
- }
- }
-
- @Override
- public final void dropTable(String databaseName, final String tableName) throws CatalogException {
- HCatalogStoreClientPool.HCatalogStoreClient client = null;
-
- try {
- client = clientPool.getClient();
- client.getHiveClient().dropTable(databaseName, tableName, false, false);
- } catch (NoSuchObjectException nsoe) {
- } catch (Exception e) {
- throw new CatalogException(e);
- } finally {
- if (client != null) {
- client.release();
- }
- }
- }
-
-
- @Override
- public void alterTable(final CatalogProtos.AlterTableDescProto alterTableDescProto) throws CatalogException {
- final String[] split = CatalogUtil.splitFQTableName(alterTableDescProto.getTableName());
-
- if (split.length == 1) {
- throw new IllegalArgumentException("alterTable() requires a qualified table name, but it is \""
- + alterTableDescProto.getTableName() + "\".");
- }
-
- final String databaseName = split[0];
- final String tableName = split[1];
- String partitionName = null;
- CatalogProtos.PartitionDescProto partitionDesc = null;
-
- switch (alterTableDescProto.getAlterTableType()) {
- case RENAME_TABLE:
- if (existTable(databaseName,alterTableDescProto.getNewTableName().toLowerCase())) {
- throw new AlreadyExistsTableException(alterTableDescProto.getNewTableName());
- }
- renameTable(databaseName, tableName, alterTableDescProto.getNewTableName().toLowerCase());
- break;
- case RENAME_COLUMN:
- if (existColumn(databaseName,tableName, alterTableDescProto.getAlterColumnName().getNewColumnName())) {
- throw new ColumnNameAlreadyExistException(alterTableDescProto.getAlterColumnName().getNewColumnName());
- }
- renameColumn(databaseName, tableName, alterTableDescProto.getAlterColumnName());
- break;
- case ADD_COLUMN:
- if (existColumn(databaseName,tableName, alterTableDescProto.getAddColumn().getName())) {
- throw new ColumnNameAlreadyExistException(alterTableDescProto.getAddColumn().getName());
- }
- addNewColumn(databaseName, tableName, alterTableDescProto.getAddColumn());
- break;
- case ADD_PARTITION:
- partitionName = alterTableDescProto.getPartitionDesc().getPartitionName();
- partitionDesc = getPartition(databaseName, tableName, partitionName);
- if(partitionDesc != null) {
- throw new AlreadyExistsPartitionException(databaseName, tableName, partitionName);
- }
- addPartition(databaseName, tableName, alterTableDescProto.getPartitionDesc());
- break;
- case DROP_PARTITION:
- partitionName = alterTableDescProto.getPartitionDesc().getPartitionName();
- partitionDesc = getPartition(databaseName, tableName, partitionName);
- if(partitionDesc == null) {
- throw new NoSuchPartitionException(databaseName, tableName, partitionName);
- }
- dropPartition(databaseName, tableName, partitionDesc);
- break;
- case SET_PROPERTY:
- // TODO - not implemented yet
- break;
- default:
- //TODO
- }
- }
-
-
- private void renameTable(String databaseName, String tableName, String newTableName) {
- HCatalogStoreClientPool.HCatalogStoreClient client = null;
- try {
- client = clientPool.getClient();
- Table newTable = client.getHiveClient().getTable(databaseName, tableName);
- newTable.setTableName(newTableName);
- client.getHiveClient().alter_table(databaseName, tableName, newTable);
-
- } catch (NoSuchObjectException nsoe) {
- } catch (Exception e) {
- throw new CatalogException(e);
- } finally {
- if (client != null) {
- client.release();
- }
- }
- }
-
- private void renameColumn(String databaseName, String tableName, CatalogProtos.AlterColumnProto alterColumnProto) {
- HCatalogStoreClientPool.HCatalogStoreClient client = null;
- try {
-
- client = clientPool.getClient();
- Table table = client.getHiveClient().getTable(databaseName, tableName);
- List<FieldSchema> columns = table.getSd().getCols();
-
- for (final FieldSchema currentColumn : columns) {
- if (currentColumn.getName().equalsIgnoreCase(alterColumnProto.getOldColumnName())) {
- currentColumn.setName(alterColumnProto.getNewColumnName());
- }
- }
- client.getHiveClient().alter_table(databaseName, tableName, table);
-
- } catch (NoSuchObjectException nsoe) {
- } catch (Exception e) {
- throw new CatalogException(e);
- } finally {
- if (client != null) {
- client.release();
- }
- }
- }
-
-
- private void addNewColumn(String databaseName, String tableName, CatalogProtos.ColumnProto columnProto) {
- HCatalogStoreClientPool.HCatalogStoreClient client = null;
- try {
-
- client = clientPool.getClient();
- Table table = client.getHiveClient().getTable(databaseName, tableName);
- List<FieldSchema> columns = table.getSd().getCols();
- columns.add(new FieldSchema(columnProto.getName(),
- HCatalogUtil.getHiveFieldType(columnProto.getDataType()), ""));
- client.getHiveClient().alter_table(databaseName, tableName, table);
-
-
- } catch (NoSuchObjectException nsoe) {
- } catch (Exception e) {
- throw new CatalogException(e);
- } finally {
- if (client != null) {
- client.release();
- }
- }
- }
-
- private void addPartition(String databaseName, String tableName, CatalogProtos.PartitionDescProto
- partitionDescProto) {
- HCatalogStoreClientPool.HCatalogStoreClient client = null;
- try {
-
- client = clientPool.getClient();
-
- Partition partition = new Partition();
- partition.setDbName(databaseName);
- partition.setTableName(tableName);
-
- List<String> values = Lists.newArrayList();
- for(CatalogProtos.PartitionKeyProto keyProto : partitionDescProto.getPartitionKeysList()) {
- values.add(keyProto.getPartitionValue());
- }
- partition.setValues(values);
-
- Table table = client.getHiveClient().getTable(databaseName, tableName);
- StorageDescriptor sd = table.getSd();
- sd.setLocation(partitionDescProto.getPath());
- partition.setSd(sd);
-
- client.getHiveClient().add_partition(partition);
- } catch (Exception e) {
- throw new CatalogException(e);
- } finally {
- if (client != null) {
- client.release();
- }
- }
- }
-
- private void dropPartition(String databaseName, String tableName, CatalogProtos.PartitionDescProto
- partitionDescProto) {
- HCatalogStoreClientPool.HCatalogStoreClient client = null;
- try {
-
- client = clientPool.getClient();
-
- List<String> values = Lists.newArrayList();
- for(CatalogProtos.PartitionKeyProto keyProto : partitionDescProto.getPartitionKeysList()) {
- values.add(keyProto.getPartitionValue());
- }
- client.getHiveClient().dropPartition(databaseName, tableName, values, true);
- } catch (Exception e) {
- throw new CatalogException(e);
- } finally {
- if (client != null) {
- client.release();
- }
- }
- }
-
- @Override
- public void addPartitionMethod(CatalogProtos.PartitionMethodProto partitionMethodProto) throws CatalogException {
- // TODO - not implemented yet
- }
-
- @Override
- public CatalogProtos.PartitionMethodProto getPartitionMethod(String databaseName, String tableName)
- throws CatalogException {
- return null; // TODO - not implemented yet
- }
-
- @Override
- public boolean existPartitionMethod(String databaseName, String tableName) throws CatalogException {
- return false; // TODO - not implemented yet
- }
-
- @Override
- public void dropPartitionMethod(String databaseName, String tableName) throws CatalogException {
- // TODO - not implemented yet
- }
-
- @Override
- public List<CatalogProtos.PartitionDescProto> getPartitions(String databaseName,
- String tableName) throws CatalogException {
- throw new UnsupportedOperationException();
- }
-
-
- @Override
- public CatalogProtos.PartitionDescProto getPartition(String databaseName, String tableName,
- String partitionName) throws CatalogException {
- HCatalogStoreClientPool.HCatalogStoreClient client = null;
- CatalogProtos.PartitionDescProto.Builder builder = null;
-
- try {
- client = clientPool.getClient();
-
- Partition partition = client.getHiveClient().getPartition(databaseName, tableName, partitionName);
- builder = CatalogProtos.PartitionDescProto.newBuilder();
- builder.setPartitionName(partitionName);
- builder.setPath(partition.getSd().getLocation());
-
- String[] partitionNames = partitionName.split("/");
-
- for (int i = 0; i < partition.getValues().size(); i++) {
- String value = partition.getValues().get(i);
- CatalogProtos.PartitionKeyProto.Builder keyBuilder = CatalogProtos.PartitionKeyProto.newBuilder();
-
- String columnName = partitionNames[i].split("=")[0];
- keyBuilder.setColumnName(columnName);
- keyBuilder.setPartitionValue(value);
- builder.addPartitionKeys(keyBuilder);
- }
- } catch (NoSuchObjectException e) {
- return null;
- } catch (Exception e) {
- throw new CatalogException(e);
- } finally {
- if (client != null) {
- client.release();
- }
- }
- return builder.build();
- }
-
- @Override
- public final void addFunction(final FunctionDesc func) throws CatalogException {
- // TODO - not implemented yet
- }
-
- @Override
- public final void deleteFunction(final FunctionDesc func) throws CatalogException {
- // TODO - not implemented yet
- }
-
- @Override
- public final void existFunction(final FunctionDesc func) throws CatalogException {
- // TODO - not implemented yet
- }
-
- @Override
- public final List<String> getAllFunctionNames() throws CatalogException {
- // TODO - not implemented yet
- return null;
- }
-
- @Override
- public void dropIndex(String databaseName, String indexName) throws CatalogException {
- // TODO - not implemented yet
- }
-
- @Override
- public boolean existIndexByName(String databaseName, String indexName) throws CatalogException {
- // TODO - not implemented yet
- return false;
- }
-
- @Override
- public CatalogProtos.IndexDescProto[] getIndexes(String databaseName, String tableName) throws CatalogException {
- // TODO - not implemented yet
- return null;
- }
-
- @Override
- public void createIndex(CatalogProtos.IndexDescProto proto) throws CatalogException {
- // TODO - not implemented yet
- }
-
- @Override
- public CatalogProtos.IndexDescProto getIndexByName(String databaseName, String indexName) throws CatalogException {
- // TODO - not implemented yet
- return null;
- }
-
- @Override
- public CatalogProtos.IndexDescProto getIndexByColumn(String databaseName, String tableName, String columnName)
- throws CatalogException {
- // TODO - not implemented yet
- return null;
- }
-
- @Override
- public boolean existIndexByColumn(String databaseName, String tableName, String columnName) throws CatalogException {
- // TODO - not implemented yet
- return false;
- }
-
- @Override
- public final void close() {
- clientPool.close();
- }
-
- private boolean existColumn(final String databaseName ,final String tableName , final String columnName) throws CatalogException {
- boolean exist = false;
- HCatalogStoreClientPool.HCatalogStoreClient client = null;
-
- try {
-
- client = clientPool.getClient();
- Table table = client.getHiveClient().getTable(databaseName, tableName);
- List<FieldSchema> columns = table.getSd().getCols();
-
- for (final FieldSchema currentColumn : columns) {
- if (currentColumn.getName().equalsIgnoreCase(columnName)) {
- exist = true;
- }
- }
- client.getHiveClient().alter_table(databaseName, tableName, table);
-
- } catch (NoSuchObjectException nsoe) {
- } catch (Exception e) {
- throw new CatalogException(e);
- } finally {
- if (client != null) {
- client.release();
- }
- }
-
- return exist;
- }
-
- @Override
- public List<ColumnProto> getAllColumns() throws CatalogException {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public List<DatabaseProto> getAllDatabases() throws CatalogException {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public List<IndexProto> getAllIndexes() throws CatalogException {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public List<TablePartitionProto> getAllPartitions() throws CatalogException {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public List<TableOptionProto> getAllTableOptions() throws CatalogException {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public List<TableStatsProto> getAllTableStats() throws CatalogException {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public List<TableDescriptorProto> getAllTables() throws CatalogException {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public List<TablespaceProto> getTablespaces() throws CatalogException {
- throw new UnsupportedOperationException();
- }
-}
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStoreClientPool.java
----------------------------------------------------------------------
diff --git a/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStoreClientPool.java b/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStoreClientPool.java
deleted file mode 100644
index 8ccb100..0000000
--- a/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStoreClientPool.java
+++ /dev/null
@@ -1,170 +0,0 @@
-// Copyright 2012 Cloudera Inc.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-package org.apache.tajo.catalog.store;
-
-
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.hive.conf.HiveConf;
-import org.apache.hadoop.hive.metastore.*;
-import org.apache.hadoop.hive.metastore.api.MetaException;
-import org.apache.hadoop.hive.metastore.api.Table;
-import org.apache.log4j.Logger;
-
-import java.util.Iterator;
-import java.util.Map;
-import java.util.Map.Entry;
-import java.util.concurrent.ConcurrentLinkedQueue;
-import java.util.concurrent.atomic.AtomicBoolean;
-
-/**
- * Manages a pool of HiveMetaStoreClient connections. If the connection pool is empty
- * a new client is created and added to the pool. There is no size limit.
- */
-public class HCatalogStoreClientPool {
- private static final Logger LOG = Logger.getLogger(HCatalogStoreClientPool.class);
- private final ConcurrentLinkedQueue<HCatalogStoreClient> clientPool =
- new ConcurrentLinkedQueue<HCatalogStoreClient>();
- private AtomicBoolean poolClosed = new AtomicBoolean(false);
- private HiveConf hiveConf;
-
- /**
- * A wrapper around the HiveMetaStoreClient that manages interactions with the
- * connection pool.
- */
- public class HCatalogStoreClient {
- private final IMetaStoreClient hiveClient;
- public AtomicBoolean isInUse = new AtomicBoolean(false);
-
- private HCatalogStoreClient(HiveConf hiveConf) {
- try {
- HiveMetaHookLoader hookLoader = new HiveMetaHookLoader() {
- @Override
- public HiveMetaHook getHook(Table table) throws MetaException {
- /* metadata hook implementation, or null if this
- * storage handler does not need any metadata notifications
- */
- return null;
- }
- };
-
- this.hiveClient = RetryingMetaStoreClient.getProxy(hiveConf, hookLoader, HiveMetaStoreClient.class.getName());
- clientPool.add(this);
- LOG.info("MetaStoreClient created (size = " + clientPool.size() + ")");
- } catch (Exception e) {
- // Turn in to an unchecked exception
- throw new IllegalStateException(e);
- }
- }
-
- /**
- * Returns the internal HiveMetaStoreClient object.
- */
- public IMetaStoreClient getHiveClient() {
- return hiveClient;
- }
-
- /**
- * Returns this client back to the connection pool. If the connection pool has been
- * closed, just close the Hive client connection.
- */
- public synchronized void release() {
- if(!this.isInUse.getAndSet(false)){
- return;
- }
- // Ensure the connection isn't returned to the pool if the pool has been closed.
- // This lock is needed to ensure proper behavior when a thread reads poolClosed
- // is false, but a call to pool.close() comes in immediately afterward.
- if (poolClosed.get()) {
- this.getHiveClient().close();
- } else {
- clientPool.add(this);
- }
- }
-
- // Marks this client as in use
- private void markInUse() {
- isInUse.set(true);
- }
- }
-
- public HCatalogStoreClientPool(int initialSize) {
- this(initialSize, new HiveConf(HCatalogStoreClientPool.class));
- }
-
- public HCatalogStoreClientPool(int initialSize, HiveConf hiveConf) {
- this.hiveConf = hiveConf;
- addClients(initialSize);
- }
-
- public HCatalogStoreClientPool(int initialSize, Configuration conf) {
- this.hiveConf = new HiveConf();
- setParameters(conf);
- addClients(initialSize);
- }
-
- public void setParameters(Configuration conf) {
- for( Iterator<Entry<String, String>> iter = conf.iterator(); iter.hasNext();) {
- Map.Entry<String, String> entry = iter.next();
- this.hiveConf.set(entry.getKey(), entry.getValue());
- }
- }
-
- /**
- * Add numClients to the client pool.
- */
- public void addClients(int numClients) {
- for (int i = 0; i < numClients; ++i) {
- clientPool.add(new HCatalogStoreClient(hiveConf));
- }
- }
-
- /**
- * Gets a client from the pool. If the pool is empty a new client is created.
- */
- public synchronized HCatalogStoreClient getClient() {
- // The MetaStoreClient c'tor relies on knowing the Hadoop version by asking
- // org.apache.hadoop.util.VersionInfo. The VersionInfo class relies on opening
- // the 'common-version-info.properties' file as a resource from hadoop-common*.jar
- // using the Thread's context classloader. If necessary, set the Thread's context
- // classloader, otherwise VersionInfo will fail in it's c'tor.
- if (Thread.currentThread().getContextClassLoader() == null) {
- Thread.currentThread().setContextClassLoader(ClassLoader.getSystemClassLoader());
- }
-
- HCatalogStoreClient client = clientPool.poll();
- // The pool was empty so create a new client and return that.
- if (client == null) {
- client = new HCatalogStoreClient(hiveConf);
- }
- client.markInUse();
-
- return client;
- }
-
- /**
- * Removes all items from the connection pool and closes all Hive Meta Store client
- * connections. Can be called multiple times.
- */
- public void close() {
- // Ensure no more items get added to the pool once close is called.
- if (poolClosed.getAndSet(true)) {
- return;
- }
-
- HCatalogStoreClient client = null;
- while ((client = clientPool.poll()) != null) {
- client.getHiveClient().close();
- }
- }
-}
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java
----------------------------------------------------------------------
diff --git a/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java b/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java
deleted file mode 100644
index 8e8e58c..0000000
--- a/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java
+++ /dev/null
@@ -1,147 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.tajo.catalog.store;
-
-import com.google.common.base.Preconditions;
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
-import org.apache.hadoop.hive.metastore.IMetaStoreClient;
-import org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat;
-import org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat;
-import org.apache.hadoop.hive.ql.io.RCFileOutputFormat;
-import org.apache.hadoop.hive.ql.metadata.Table;
-import org.apache.hadoop.hive.serde.serdeConstants;
-import org.apache.hcatalog.common.HCatException;
-import org.apache.hcatalog.data.schema.HCatFieldSchema;
-import org.apache.hcatalog.data.schema.HCatSchema;
-import org.apache.tajo.catalog.exception.CatalogException;
-import org.apache.tajo.catalog.proto.CatalogProtos;
-import org.apache.tajo.catalog.CatalogUtil;
-import org.apache.tajo.common.TajoDataTypes;
-import org.apache.thrift.TException;
-import parquet.hadoop.mapred.DeprecatedParquetOutputFormat;
-
-public class HCatalogUtil {
- protected final Log LOG = LogFactory.getLog(getClass());
-
- public static void validateHCatTableAndTajoSchema(HCatSchema tblSchema) throws CatalogException {
- for (HCatFieldSchema hcatField : tblSchema.getFields()) {
- validateHCatFieldAndTajoSchema(hcatField);
- }
- }
-
- private static void validateHCatFieldAndTajoSchema(HCatFieldSchema fieldSchema) throws CatalogException {
- try {
- HCatFieldSchema.Type fieldType = fieldSchema.getType();
- switch (fieldType) {
- case ARRAY:
- throw new HCatException("Tajo cannot support array field type.");
- case STRUCT:
- throw new HCatException("Tajo cannot support struct field type.");
- case MAP:
- throw new HCatException("Tajo cannot support map field type.");
- }
- } catch (HCatException e) {
- throw new CatalogException("incompatible hcatalog types when assigning to tajo type. - " +
- "HCatFieldSchema:" + fieldSchema);
- }
- }
-
- public static TajoDataTypes.Type getTajoFieldType(String fieldType) {
- Preconditions.checkNotNull(fieldType);
-
- String typeStr = null;
-
- if(fieldType.equalsIgnoreCase(serdeConstants.INT_TYPE_NAME))
- typeStr = "INT4";
- else if(fieldType.equalsIgnoreCase(serdeConstants.TINYINT_TYPE_NAME))
- typeStr = "INT1";
- else if(fieldType.equalsIgnoreCase(serdeConstants.SMALLINT_TYPE_NAME))
- typeStr = "INT2";
- else if(fieldType.equalsIgnoreCase(serdeConstants.BIGINT_TYPE_NAME))
- typeStr = "INT8";
- else if(fieldType.equalsIgnoreCase(serdeConstants.BOOLEAN_TYPE_NAME))
- typeStr = "BOOLEAN";
- else if(fieldType.equalsIgnoreCase(serdeConstants.FLOAT_TYPE_NAME))
- typeStr = "FLOAT4";
- else if(fieldType.equalsIgnoreCase(serdeConstants.DOUBLE_TYPE_NAME))
- typeStr = "FLOAT8";
- else if(fieldType.equalsIgnoreCase(serdeConstants.STRING_TYPE_NAME))
- typeStr = "TEXT";
- else if(fieldType.equalsIgnoreCase(serdeConstants.BINARY_TYPE_NAME))
- typeStr = "BLOB";
-
- try {
- return Enum.valueOf(TajoDataTypes.Type.class, typeStr);
- } catch (IllegalArgumentException iae) {
- throw new CatalogException("Cannot find a matched type against from '" + typeStr + "'");
- }
- }
-
- public static String getHiveFieldType(TajoDataTypes.DataType dataType) {
- Preconditions.checkNotNull(dataType);
-
- switch (dataType.getType()) {
- case CHAR: return serdeConstants.CHAR_TYPE_NAME;
- case BOOLEAN: return serdeConstants.BOOLEAN_TYPE_NAME;
- case INT1: return serdeConstants.TINYINT_TYPE_NAME;
- case INT2: return serdeConstants.SMALLINT_TYPE_NAME;
- case INT4: return serdeConstants.INT_TYPE_NAME;
- case INT8: return serdeConstants.BIGINT_TYPE_NAME;
- case FLOAT4: return serdeConstants.FLOAT_TYPE_NAME;
- case FLOAT8: return serdeConstants.DOUBLE_TYPE_NAME;
- case TEXT: return serdeConstants.STRING_TYPE_NAME;
- case VARCHAR: return serdeConstants.VARCHAR_TYPE_NAME;
- case NCHAR: return serdeConstants.VARCHAR_TYPE_NAME;
- case NVARCHAR: return serdeConstants.VARCHAR_TYPE_NAME;
- case BINARY: return serdeConstants.BINARY_TYPE_NAME;
- case VARBINARY: return serdeConstants.BINARY_TYPE_NAME;
- case BLOB: return serdeConstants.BINARY_TYPE_NAME;
- case DATE: return serdeConstants.DATE_TYPE_NAME;
- case TIMESTAMP: return serdeConstants.TIMESTAMP_TYPE_NAME;
- default:
- throw new CatalogException(dataType + " is not supported.");
- }
- }
-
- public static String getStoreType(String fileFormat) {
- Preconditions.checkNotNull(fileFormat);
-
- String[] fileFormatArrary = fileFormat.split("\\.");
- if(fileFormatArrary.length < 1) {
- throw new CatalogException("Hive file output format is wrong. - file output format:" + fileFormat);
- }
-
- String outputFormatClass = fileFormatArrary[fileFormatArrary.length-1];
- if(outputFormatClass.equals(HiveIgnoreKeyTextOutputFormat.class.getSimpleName())) {
- return CatalogUtil.TEXTFILE_NAME;
- } else if(outputFormatClass.equals(HiveSequenceFileOutputFormat.class.getSimpleName())) {
- return CatalogProtos.StoreType.SEQUENCEFILE.name();
- } else if(outputFormatClass.equals(RCFileOutputFormat.class.getSimpleName())) {
- return CatalogProtos.StoreType.RCFILE.name();
- } else if(outputFormatClass.equals(DeprecatedParquetOutputFormat.class.getSimpleName())) {
- return CatalogProtos.StoreType.PARQUET.name();
- } else {
- throw new CatalogException("Not supported file output format. - file output format:" + fileFormat);
- }
- }
-
- public static Table getTable(IMetaStoreClient client, String dbName, String tableName) throws TException {
- return new Table(client.getTable(dbName, tableName));
- }
-}
[2/4] tajo git commit: TAJO-1442: Improve Hive Compatibility
Posted by bl...@apache.org.
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-catalog/tajo-catalog-drivers/tajo-hive/src/test/java/org/apache/tajo/catalog/store/TestHiveCatalogStore.java
----------------------------------------------------------------------
diff --git a/tajo-catalog/tajo-catalog-drivers/tajo-hive/src/test/java/org/apache/tajo/catalog/store/TestHiveCatalogStore.java b/tajo-catalog/tajo-catalog-drivers/tajo-hive/src/test/java/org/apache/tajo/catalog/store/TestHiveCatalogStore.java
new file mode 100644
index 0000000..05e4e0a
--- /dev/null
+++ b/tajo-catalog/tajo-catalog-drivers/tajo-hive/src/test/java/org/apache/tajo/catalog/store/TestHiveCatalogStore.java
@@ -0,0 +1,504 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tajo.catalog.store;
+
+
+import org.apache.commons.lang.StringEscapeUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.tajo.catalog.*;
+import org.apache.tajo.catalog.partition.PartitionDesc;
+import org.apache.tajo.catalog.partition.PartitionKey;
+import org.apache.tajo.catalog.partition.PartitionMethodDesc;
+import org.apache.tajo.catalog.proto.CatalogProtos;
+import org.apache.tajo.common.TajoDataTypes;
+import org.apache.tajo.conf.TajoConf;
+import org.apache.tajo.storage.StorageConstants;
+import org.apache.tajo.util.CommonTestingUtil;
+import org.apache.tajo.util.KeyValueSet;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.tajo.TajoConstants.DEFAULT_DATABASE_NAME;
+import static org.junit.Assert.*;
+
+/**
+ * TestHiveCatalogStore. Test case for
+ * {@link org.apache.tajo.catalog.store.HiveCatalogStore}
+ */
+
+public class TestHiveCatalogStore {
+ private static final String DB_NAME = "test_hive";
+ private static final String CUSTOMER = "customer";
+ private static final String NATION = "nation";
+ private static final String REGION = "region";
+ private static final String SUPPLIER = "supplier";
+
+ private static HiveCatalogStore store;
+ private static Path warehousePath;
+
+ @BeforeClass
+ public static void setUp() throws Exception {
+ Path testPath = CommonTestingUtil.getTestDir();
+ warehousePath = new Path(testPath, "warehouse");
+
+ //create local hiveMeta
+ HiveConf conf = new HiveConf();
+ String jdbcUri = "jdbc:derby:;databaseName="+testPath.toUri().getPath()+"metastore_db;create=true";
+ conf.set(HiveConf.ConfVars.METASTOREWAREHOUSE.varname, warehousePath.toUri().toString());
+ conf.set(HiveConf.ConfVars.METASTORECONNECTURLKEY.varname, jdbcUri);
+ conf.set(TajoConf.ConfVars.WAREHOUSE_DIR.varname, warehousePath.toUri().toString());
+
+ // create local HiveCatalogStore.
+ TajoConf tajoConf = new TajoConf(conf);
+ store = new HiveCatalogStore(tajoConf);
+ store.createDatabase(DB_NAME, null);
+ }
+
+ @AfterClass
+ public static void tearDown() throws IOException {
+ store.close();
+ }
+
+ @Test
+ public void testTableUsingTextFile() throws Exception {
+ TableMeta meta = new TableMeta(CatalogProtos.StoreType.CSV, new KeyValueSet());
+
+ org.apache.tajo.catalog.Schema schema = new org.apache.tajo.catalog.Schema();
+ schema.addColumn("c_custkey", TajoDataTypes.Type.INT4);
+ schema.addColumn("c_name", TajoDataTypes.Type.TEXT);
+ schema.addColumn("c_address", TajoDataTypes.Type.TEXT);
+ schema.addColumn("c_nationkey", TajoDataTypes.Type.INT4);
+ schema.addColumn("c_phone", TajoDataTypes.Type.TEXT);
+ schema.addColumn("c_acctbal", TajoDataTypes.Type.FLOAT8);
+ schema.addColumn("c_mktsegment", TajoDataTypes.Type.TEXT);
+ schema.addColumn("c_comment", TajoDataTypes.Type.TEXT);
+
+ TableDesc table = new TableDesc(CatalogUtil.buildFQName(DB_NAME, CUSTOMER), schema, meta,
+ new Path(warehousePath, new Path(DB_NAME, CUSTOMER)).toUri());
+ store.createTable(table.getProto());
+ assertTrue(store.existTable(DB_NAME, CUSTOMER));
+
+ TableDesc table1 = new TableDesc(store.getTable(DB_NAME, CUSTOMER));
+ assertEquals(table.getName(), table1.getName());
+ assertEquals(table.getPath(), table1.getPath());
+ assertEquals(table.getSchema().size(), table1.getSchema().size());
+ for (int i = 0; i < table.getSchema().size(); i++) {
+ assertEquals(table.getSchema().getColumn(i).getSimpleName(), table1.getSchema().getColumn(i).getSimpleName());
+ }
+
+ assertEquals(StringEscapeUtils.escapeJava(StorageConstants.DEFAULT_FIELD_DELIMITER),
+ table1.getMeta().getOption(StorageConstants.TEXT_DELIMITER));
+ store.dropTable(DB_NAME, CUSTOMER);
+ }
+
+ @Test
+ public void testTableUsingRCFileWithBinarySerde() throws Exception {
+ KeyValueSet options = new KeyValueSet();
+ options.set(StorageConstants.RCFILE_SERDE, StorageConstants.DEFAULT_BINARY_SERDE);
+ TableMeta meta = new TableMeta(CatalogProtos.StoreType.RCFILE, options);
+
+ org.apache.tajo.catalog.Schema schema = new org.apache.tajo.catalog.Schema();
+ schema.addColumn("r_regionkey", TajoDataTypes.Type.INT4);
+ schema.addColumn("r_name", TajoDataTypes.Type.TEXT);
+ schema.addColumn("r_comment", TajoDataTypes.Type.TEXT);
+
+ TableDesc table = new TableDesc(CatalogUtil.buildFQName(DB_NAME, REGION), schema, meta,
+ new Path(warehousePath, new Path(DB_NAME, REGION)).toUri());
+ store.createTable(table.getProto());
+ assertTrue(store.existTable(DB_NAME, REGION));
+
+ TableDesc table1 = new TableDesc(store.getTable(DB_NAME, REGION));
+ assertEquals(table.getName(), table1.getName());
+ assertEquals(table.getPath(), table1.getPath());
+ assertEquals(table.getSchema().size(), table1.getSchema().size());
+ for (int i = 0; i < table.getSchema().size(); i++) {
+ assertEquals(table.getSchema().getColumn(i).getSimpleName(), table1.getSchema().getColumn(i).getSimpleName());
+ }
+
+ assertEquals(StorageConstants.DEFAULT_BINARY_SERDE,
+ table1.getMeta().getOption(StorageConstants.RCFILE_SERDE));
+ store.dropTable(DB_NAME, REGION);
+ }
+
+ @Test
+ public void testTableUsingRCFileWithTextSerde() throws Exception {
+ KeyValueSet options = new KeyValueSet();
+ options.set(StorageConstants.RCFILE_SERDE, StorageConstants.DEFAULT_TEXT_SERDE);
+ TableMeta meta = new TableMeta(CatalogProtos.StoreType.RCFILE, options);
+
+ org.apache.tajo.catalog.Schema schema = new org.apache.tajo.catalog.Schema();
+ schema.addColumn("r_regionkey", TajoDataTypes.Type.INT4);
+ schema.addColumn("r_name", TajoDataTypes.Type.TEXT);
+ schema.addColumn("r_comment", TajoDataTypes.Type.TEXT);
+
+ TableDesc table = new TableDesc(CatalogUtil.buildFQName(DB_NAME, REGION), schema, meta,
+ new Path(warehousePath, new Path(DB_NAME, REGION)).toUri());
+ store.createTable(table.getProto());
+ assertTrue(store.existTable(DB_NAME, REGION));
+
+ TableDesc table1 = new TableDesc(store.getTable(DB_NAME, REGION));
+ assertEquals(table.getName(), table1.getName());
+ assertEquals(table.getPath(), table1.getPath());
+ assertEquals(table.getSchema().size(), table1.getSchema().size());
+ for (int i = 0; i < table.getSchema().size(); i++) {
+ assertEquals(table.getSchema().getColumn(i).getSimpleName(), table1.getSchema().getColumn(i).getSimpleName());
+ }
+
+ assertEquals(StorageConstants.DEFAULT_TEXT_SERDE, table1.getMeta().getOption(StorageConstants.RCFILE_SERDE));
+ store.dropTable(DB_NAME, REGION);
+ }
+
+ @Test
+ public void testTableWithNullValue() throws Exception {
+ KeyValueSet options = new KeyValueSet();
+ options.set(StorageConstants.TEXT_DELIMITER, StringEscapeUtils.escapeJava("\u0002"));
+ options.set(StorageConstants.TEXT_NULL, StringEscapeUtils.escapeJava("\u0003"));
+ TableMeta meta = new TableMeta(CatalogProtos.StoreType.CSV, options);
+
+ org.apache.tajo.catalog.Schema schema = new org.apache.tajo.catalog.Schema();
+ schema.addColumn("s_suppkey", TajoDataTypes.Type.INT4);
+ schema.addColumn("s_name", TajoDataTypes.Type.TEXT);
+ schema.addColumn("s_address", TajoDataTypes.Type.TEXT);
+ schema.addColumn("s_nationkey", TajoDataTypes.Type.INT4);
+ schema.addColumn("s_phone", TajoDataTypes.Type.TEXT);
+ schema.addColumn("s_acctbal", TajoDataTypes.Type.FLOAT8);
+ schema.addColumn("s_comment", TajoDataTypes.Type.TEXT);
+
+ TableDesc table = new TableDesc(CatalogUtil.buildFQName(DB_NAME, SUPPLIER), schema, meta,
+ new Path(warehousePath, new Path(DB_NAME, SUPPLIER)).toUri());
+
+ store.createTable(table.getProto());
+ assertTrue(store.existTable(DB_NAME, SUPPLIER));
+
+ TableDesc table1 = new TableDesc(store.getTable(DB_NAME, SUPPLIER));
+ assertEquals(table.getName(), table1.getName());
+ assertEquals(table.getPath(), table1.getPath());
+ assertEquals(table.getSchema().size(), table1.getSchema().size());
+ for (int i = 0; i < table.getSchema().size(); i++) {
+ assertEquals(table.getSchema().getColumn(i).getSimpleName(), table1.getSchema().getColumn(i).getSimpleName());
+ }
+
+ assertEquals(table.getMeta().getOption(StorageConstants.TEXT_DELIMITER),
+ table1.getMeta().getOption(StorageConstants.TEXT_DELIMITER));
+
+ assertEquals(table.getMeta().getOption(StorageConstants.TEXT_NULL),
+ table1.getMeta().getOption(StorageConstants.TEXT_NULL));
+
+ assertEquals(table1.getMeta().getOption(StorageConstants.TEXT_DELIMITER),
+ StringEscapeUtils.escapeJava("\u0002"));
+
+ assertEquals(table1.getMeta().getOption(StorageConstants.TEXT_NULL),
+ StringEscapeUtils.escapeJava("\u0003"));
+
+ store.dropTable(DB_NAME, SUPPLIER);
+
+ }
+
+ @Test
+ public void testAddTableByPartition() throws Exception {
+ TableMeta meta = new TableMeta(CatalogProtos.StoreType.CSV, new KeyValueSet());
+
+ org.apache.tajo.catalog.Schema schema = new org.apache.tajo.catalog.Schema();
+ schema.addColumn("n_name", TajoDataTypes.Type.TEXT);
+ schema.addColumn("n_regionkey", TajoDataTypes.Type.INT4);
+ schema.addColumn("n_comment", TajoDataTypes.Type.TEXT);
+
+
+ TableDesc table = new TableDesc(CatalogUtil.buildFQName(DB_NAME, NATION), schema, meta,
+ new Path(warehousePath, new Path(DB_NAME, NATION)).toUri());
+
+ org.apache.tajo.catalog.Schema expressionSchema = new org.apache.tajo.catalog.Schema();
+ expressionSchema.addColumn("n_nationkey", TajoDataTypes.Type.INT4);
+ expressionSchema.addColumn("n_date", TajoDataTypes.Type.TEXT);
+
+ PartitionMethodDesc partitions = new PartitionMethodDesc(
+ DB_NAME,
+ NATION,
+ CatalogProtos.PartitionType.COLUMN, "n_nationkey,n_date", expressionSchema);
+ table.setPartitionMethod(partitions);
+
+ store.createTable(table.getProto());
+ assertTrue(store.existTable(DB_NAME, NATION));
+
+ TableDesc table1 = new TableDesc(store.getTable(DB_NAME, NATION));
+ assertEquals(table.getName(), table1.getName());
+ assertEquals(table.getPath(), table1.getPath());
+ assertEquals(table.getSchema().size(), table1.getSchema().size());
+ for (int i = 0; i < table.getSchema().size(); i++) {
+ assertEquals(table.getSchema().getColumn(i).getSimpleName(), table1.getSchema().getColumn(i).getSimpleName());
+ }
+
+ Schema partitionSchema = table.getPartitionMethod().getExpressionSchema();
+ Schema partitionSchema1 = table1.getPartitionMethod().getExpressionSchema();
+ assertEquals(partitionSchema.size(), partitionSchema1.size());
+
+ for (int i = 0; i < partitionSchema.size(); i++) {
+ assertEquals(partitionSchema.getColumn(i).getSimpleName(), partitionSchema1.getColumn(i).getSimpleName());
+ }
+
+ testAddPartition(table1.getPath(), NATION, "n_nationkey=10/n_date=20150101");
+ testAddPartition(table1.getPath(), NATION, "n_nationkey=20/n_date=20150102");
+
+ testDropPartition(NATION, "n_nationkey=10/n_date=20150101");
+ testDropPartition(NATION, "n_nationkey=20/n_date=20150102");
+
+ CatalogProtos.PartitionDescProto partition = store.getPartition(DB_NAME, NATION, "n_nationkey=10/n_date=20150101");
+ assertNull(partition);
+
+ partition = store.getPartition(DB_NAME, NATION, "n_nationkey=20/n_date=20150102");
+ assertNull(partition);
+
+ store.dropTable(DB_NAME, NATION);
+ }
+
+
+ private void testAddPartition(URI uri, String tableName, String partitionName) throws Exception {
+ AlterTableDesc alterTableDesc = new AlterTableDesc();
+ alterTableDesc.setTableName(DB_NAME + "." + tableName);
+ alterTableDesc.setAlterTableType(AlterTableType.ADD_PARTITION);
+
+ Path path = new Path(uri.getPath(), partitionName);
+
+ PartitionDesc partitionDesc = new PartitionDesc();
+ partitionDesc.setPartitionName(partitionName);
+
+ List<PartitionKey> partitionKeyList = new ArrayList<PartitionKey>();
+ String[] partitionNames = partitionName.split("/");
+ for(int i = 0; i < partitionNames.length; i++) {
+ String[] eachPartitionName = partitionNames[i].split("=");
+ partitionKeyList.add(new PartitionKey(eachPartitionName[0], eachPartitionName[1]));
+ }
+ partitionDesc.setPartitionKeys(partitionKeyList);
+ partitionDesc.setPath(path.toString());
+
+ alterTableDesc.setPartitionDesc(partitionDesc);
+
+ store.alterTable(alterTableDesc.getProto());
+
+ CatalogProtos.PartitionDescProto resultDesc = store.getPartition(DB_NAME, NATION, partitionName);
+ assertNotNull(resultDesc);
+ assertEquals(resultDesc.getPartitionName(), partitionName);
+ assertEquals(resultDesc.getPath(), uri.toString() + "/" + partitionName);
+ assertEquals(resultDesc.getPartitionKeysCount(), 2);
+
+ for (int i = 0; i < resultDesc.getPartitionKeysCount(); i++) {
+ CatalogProtos.PartitionKeyProto keyProto = resultDesc.getPartitionKeys(i);
+ String[] eachName = partitionNames[i].split("=");
+ assertEquals(keyProto.getPartitionValue(), eachName[1]);
+ }
+ }
+
+
+ private void testDropPartition(String tableName, String partitionName) throws Exception {
+ AlterTableDesc alterTableDesc = new AlterTableDesc();
+ alterTableDesc.setTableName(DB_NAME + "." + tableName);
+ alterTableDesc.setAlterTableType(AlterTableType.DROP_PARTITION);
+
+ PartitionDesc partitionDesc = new PartitionDesc();
+ partitionDesc.setPartitionName(partitionName);
+
+ alterTableDesc.setPartitionDesc(partitionDesc);
+
+ store.alterTable(alterTableDesc.getProto());
+ }
+
+ @Test
+ public void testGetAllTableNames() throws Exception{
+ TableMeta meta = new TableMeta(CatalogProtos.StoreType.CSV, new KeyValueSet());
+ org.apache.tajo.catalog.Schema schema = new org.apache.tajo.catalog.Schema();
+ schema.addColumn("n_name", TajoDataTypes.Type.TEXT);
+ schema.addColumn("n_regionkey", TajoDataTypes.Type.INT4);
+ schema.addColumn("n_comment", TajoDataTypes.Type.TEXT);
+
+ String[] tableNames = new String[]{"table1", "table2", "table3"};
+
+ for(String tableName : tableNames){
+ TableDesc table = new TableDesc(CatalogUtil.buildFQName("default", tableName), schema, meta,
+ new Path(warehousePath, new Path(DB_NAME, tableName)).toUri());
+ store.createTable(table.getProto());
+ }
+
+ List<String> tables = store.getAllTableNames("default");
+ assertEquals(tableNames.length, tables.size());
+
+ for(String tableName : tableNames){
+ assertTrue(tables.contains(tableName));
+ }
+
+ for(String tableName : tableNames){
+ store.dropTable("default", tableName);
+ }
+ }
+
+ @Test
+ public void testDeleteTable() throws Exception {
+ TableMeta meta = new TableMeta(CatalogProtos.StoreType.CSV, new KeyValueSet());
+ org.apache.tajo.catalog.Schema schema = new org.apache.tajo.catalog.Schema();
+ schema.addColumn("n_name", TajoDataTypes.Type.TEXT);
+ schema.addColumn("n_regionkey", TajoDataTypes.Type.INT4);
+ schema.addColumn("n_comment", TajoDataTypes.Type.TEXT);
+
+ String tableName = "table1";
+ TableDesc table = new TableDesc(DB_NAME + "." + tableName, schema, meta, warehousePath.toUri());
+ store.createTable(table.getProto());
+ assertTrue(store.existTable(DB_NAME, tableName));
+
+ TableDesc table1 = new TableDesc(store.getTable(DB_NAME, tableName));
+ FileSystem fs = FileSystem.getLocal(new Configuration());
+ assertTrue(fs.exists(new Path(table1.getPath())));
+
+ store.dropTable(DB_NAME, tableName);
+ assertFalse(store.existTable(DB_NAME, tableName));
+ fs.close();
+ }
+
+ @Test
+ public void testTableUsingSequenceFileWithBinarySerde() throws Exception {
+ KeyValueSet options = new KeyValueSet();
+ options.set(StorageConstants.SEQUENCEFILE_SERDE, StorageConstants.DEFAULT_BINARY_SERDE);
+ TableMeta meta = new TableMeta(CatalogProtos.StoreType.SEQUENCEFILE, options);
+
+ org.apache.tajo.catalog.Schema schema = new org.apache.tajo.catalog.Schema();
+ schema.addColumn("r_regionkey", TajoDataTypes.Type.INT4);
+ schema.addColumn("r_name", TajoDataTypes.Type.TEXT);
+ schema.addColumn("r_comment", TajoDataTypes.Type.TEXT);
+
+ TableDesc table = new TableDesc(CatalogUtil.buildFQName(DB_NAME, REGION), schema, meta,
+ new Path(warehousePath, new Path(DB_NAME, REGION)).toUri());
+ store.createTable(table.getProto());
+ assertTrue(store.existTable(DB_NAME, REGION));
+
+ TableDesc table1 = new TableDesc(store.getTable(DB_NAME, REGION));
+ assertEquals(table.getName(), table1.getName());
+ assertEquals(table.getPath(), table1.getPath());
+ assertEquals(table.getSchema().size(), table1.getSchema().size());
+ for (int i = 0; i < table.getSchema().size(); i++) {
+ assertEquals(table.getSchema().getColumn(i).getSimpleName(), table1.getSchema().getColumn(i).getSimpleName());
+ }
+
+ assertEquals(StorageConstants.DEFAULT_BINARY_SERDE,
+ table1.getMeta().getOption(StorageConstants.SEQUENCEFILE_SERDE));
+ store.dropTable(DB_NAME, REGION);
+ }
+
+ @Test
+ public void testTableUsingSequenceFileWithTextSerde() throws Exception {
+ KeyValueSet options = new KeyValueSet();
+ options.set(StorageConstants.SEQUENCEFILE_SERDE, StorageConstants.DEFAULT_TEXT_SERDE);
+ TableMeta meta = new TableMeta(CatalogProtos.StoreType.SEQUENCEFILE, options);
+
+ org.apache.tajo.catalog.Schema schema = new org.apache.tajo.catalog.Schema();
+ schema.addColumn("r_regionkey", TajoDataTypes.Type.INT4);
+ schema.addColumn("r_name", TajoDataTypes.Type.TEXT);
+ schema.addColumn("r_comment", TajoDataTypes.Type.TEXT);
+
+ TableDesc table = new TableDesc(CatalogUtil.buildFQName(DB_NAME, REGION), schema, meta,
+ new Path(warehousePath, new Path(DB_NAME, REGION)).toUri());
+ store.createTable(table.getProto());
+ assertTrue(store.existTable(DB_NAME, REGION));
+
+ TableDesc table1 = new TableDesc(store.getTable(DB_NAME, REGION));
+ assertEquals(table.getName(), table1.getName());
+ assertEquals(table.getPath(), table1.getPath());
+ assertEquals(table.getSchema().size(), table1.getSchema().size());
+ for (int i = 0; i < table.getSchema().size(); i++) {
+ assertEquals(table.getSchema().getColumn(i).getSimpleName(), table1.getSchema().getColumn(i).getSimpleName());
+ }
+
+ assertEquals(StorageConstants.DEFAULT_TEXT_SERDE, table1.getMeta().getOption(StorageConstants.SEQUENCEFILE_SERDE));
+ store.dropTable(DB_NAME, REGION);
+ }
+
+
+ @Test
+ public void testTableUsingParquet() throws Exception {
+ TableMeta meta = new TableMeta(CatalogProtos.StoreType.PARQUET, new KeyValueSet());
+
+ org.apache.tajo.catalog.Schema schema = new org.apache.tajo.catalog.Schema();
+ schema.addColumn("c_custkey", TajoDataTypes.Type.INT4);
+ schema.addColumn("c_name", TajoDataTypes.Type.TEXT);
+ schema.addColumn("c_address", TajoDataTypes.Type.TEXT);
+ schema.addColumn("c_nationkey", TajoDataTypes.Type.INT4);
+ schema.addColumn("c_phone", TajoDataTypes.Type.TEXT);
+ schema.addColumn("c_acctbal", TajoDataTypes.Type.FLOAT8);
+ schema.addColumn("c_mktsegment", TajoDataTypes.Type.TEXT);
+ schema.addColumn("c_comment", TajoDataTypes.Type.TEXT);
+
+ TableDesc table = new TableDesc(CatalogUtil.buildFQName(DB_NAME, CUSTOMER), schema, meta,
+ new Path(warehousePath, new Path(DB_NAME, CUSTOMER)).toUri());
+ store.createTable(table.getProto());
+ assertTrue(store.existTable(DB_NAME, CUSTOMER));
+
+ TableDesc table1 = new TableDesc(store.getTable(DB_NAME, CUSTOMER));
+ assertEquals(table.getName(), table1.getName());
+ assertEquals(table.getPath(), table1.getPath());
+ assertEquals(table.getSchema().size(), table1.getSchema().size());
+ for (int i = 0; i < table.getSchema().size(); i++) {
+ assertEquals(table.getSchema().getColumn(i).getSimpleName(), table1.getSchema().getColumn(i).getSimpleName());
+ }
+
+ store.dropTable(DB_NAME, CUSTOMER);
+ }
+
+ @Test
+ public void testDataTypeCompatibility() throws Exception {
+ String tableName = CatalogUtil.normalizeIdentifier("testDataTypeCompatibility");
+
+ TableMeta meta = new TableMeta(CatalogProtos.StoreType.CSV, new KeyValueSet());
+
+ org.apache.tajo.catalog.Schema schema = new org.apache.tajo.catalog.Schema();
+ schema.addColumn("col1", TajoDataTypes.Type.INT4);
+ schema.addColumn("col2", TajoDataTypes.Type.INT1);
+ schema.addColumn("col3", TajoDataTypes.Type.INT2);
+ schema.addColumn("col4", TajoDataTypes.Type.INT8);
+ schema.addColumn("col5", TajoDataTypes.Type.BOOLEAN);
+ schema.addColumn("col6", TajoDataTypes.Type.FLOAT4);
+ schema.addColumn("col7", TajoDataTypes.Type.FLOAT8);
+ schema.addColumn("col8", TajoDataTypes.Type.TEXT);
+ schema.addColumn("col9", TajoDataTypes.Type.BLOB);
+ schema.addColumn("col10", TajoDataTypes.Type.TIMESTAMP);
+ schema.addColumn("col11", TajoDataTypes.Type.DATE);
+
+ TableDesc table = new TableDesc(CatalogUtil.buildFQName(DB_NAME, tableName), schema, meta,
+ new Path(warehousePath, new Path(DB_NAME, tableName)).toUri());
+ store.createTable(table.getProto());
+ assertTrue(store.existTable(DB_NAME, tableName));
+
+ TableDesc table1 = new TableDesc(store.getTable(DB_NAME, tableName));
+ assertEquals(table.getName(), table1.getName());
+ assertEquals(table.getPath(), table1.getPath());
+ assertEquals(table.getSchema().size(), table1.getSchema().size());
+ for (int i = 0; i < table.getSchema().size(); i++) {
+ assertEquals(table.getSchema().getColumn(i).getSimpleName(), table1.getSchema().getColumn(i).getSimpleName());
+ }
+
+ assertEquals(StringEscapeUtils.escapeJava(StorageConstants.DEFAULT_FIELD_DELIMITER),
+ table1.getMeta().getOption(StorageConstants.TEXT_DELIMITER));
+ store.dropTable(DB_NAME, tableName);
+ }
+}
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java
----------------------------------------------------------------------
diff --git a/tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java b/tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java
index cef4027..0a2a8cc 100644
--- a/tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java
+++ b/tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java
@@ -70,12 +70,12 @@ public class TestCatalog {
@BeforeClass
public static void setUp() throws Exception {
- final String HCATALOG_CLASS_NAME = "org.apache.tajo.catalog.store.HCatalogStore";
+ final String HIVE_CATALOG_CLASS_NAME = "org.apache.tajo.catalog.store.HiveCatalogStore";
String driverClass = System.getProperty(CatalogConstants.STORE_CLASS);
- // here, we don't choose HCatalogStore due to some dependency problems.
- if (driverClass == null || driverClass.equals(HCATALOG_CLASS_NAME)) {
+ // here, we don't choose HiveCatalogStore due to some dependency problems.
+ if (driverClass == null || driverClass.equals(HIVE_CATALOG_CLASS_NAME)) {
driverClass = DerbyStore.class.getCanonicalName();
}
String catalogURI = System.getProperty(CatalogConstants.CATALOG_URI);
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-core/pom.xml
----------------------------------------------------------------------
diff --git a/tajo-core/pom.xml b/tajo-core/pom.xml
index a9403d5..c9367b2 100644
--- a/tajo-core/pom.xml
+++ b/tajo-core/pom.xml
@@ -565,218 +565,6 @@
</profile>
<profile>
- <id>hcatalog-0.12.0</id>
- <activation>
- <activeByDefault>false</activeByDefault>
- </activation>
- <dependencies>
- <dependency>
- <groupId>org.apache.thrift</groupId>
- <artifactId>libfb303</artifactId>
- <version>0.9.0</version>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.thrift</groupId>
- <artifactId>libthrift</artifactId>
- <version>0.9.0</version>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.tajo</groupId>
- <artifactId>tajo-hcatalog</artifactId>
- <version>${tajo.version}</version>
- <scope>test</scope>
- <exclusions>
- <exclusion>
- <groupId>com.google.protobuf</groupId>
- <artifactId>protobuf-java</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- <dependency>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-exec</artifactId>
- <version>0.12.0</version>
- <scope>provided</scope>
- <exclusions>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-common</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-contrib</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-hbase-handler</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-metastore</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-serde</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-shims</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-testutils</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.thrift</groupId>
- <artifactId>libfb303</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.thrift</groupId>
- <artifactId>libthrift</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.jolbox</groupId>
- <artifactId>bonecp</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hbase</groupId>
- <artifactId>hbase</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.google.protobuf</groupId>
- <artifactId>protobuf-java</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- <dependency>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-metastore</artifactId>
- <version>0.12.0</version>
- <scope>provided</scope>
- <exclusions>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-common</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-serde</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-shimss</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.thrift</groupId>
- <artifactId>libfb303</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.thrift</groupId>
- <artifactId>libthrift</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.jolbox</groupId>
- <artifactId>bonecp</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.google.protobuf</groupId>
- <artifactId>protobuf-java</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- <dependency>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-cli</artifactId>
- <version>0.12.0</version>
- <scope>provided</scope>
- <exclusions>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-common</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-exec</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-metastore</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-serde</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-service</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-shims</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.jolbox</groupId>
- <artifactId>bonecp</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.google.protobuf</groupId>
- <artifactId>protobuf-java</artifactId>
- </exclusion>
- <exclusion>
- <groupId>jline</groupId>
- <artifactId>jline</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- <dependency>
- <groupId>org.apache.hive.hcatalog</groupId>
- <artifactId>hcatalog-core</artifactId>
- <version>0.12.0</version>
- <scope>provided</scope>
- <exclusions>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-cli</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-common</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-exec</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-metastore</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-serde</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-service</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-shims</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.jolbox</groupId>
- <artifactId>bonecp</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.google.protobuf</groupId>
- <artifactId>protobuf-java</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- </dependencies>
- </profile>
- <profile>
<id>parallel-test</id>
<activation>
<activeByDefault>false</activeByDefault>
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-core/src/test/java/org/apache/tajo/QueryTestCaseBase.java
----------------------------------------------------------------------
diff --git a/tajo-core/src/test/java/org/apache/tajo/QueryTestCaseBase.java b/tajo-core/src/test/java/org/apache/tajo/QueryTestCaseBase.java
index 727192a..ede667e 100644
--- a/tajo-core/src/test/java/org/apache/tajo/QueryTestCaseBase.java
+++ b/tajo-core/src/test/java/org/apache/tajo/QueryTestCaseBase.java
@@ -231,8 +231,8 @@ public class QueryTestCaseBase {
public QueryTestCaseBase() {
// hive 0.12 does not support quoted identifier.
- // So, we use lower case database names when Tajo uses HCatalogStore.
- if (testingCluster.isHCatalogStoreRunning()) {
+ // So, we use lower case database names when Tajo uses HiveCatalogStore.
+ if (testingCluster.isHiveCatalogStoreRunning()) {
this.currentDatabase = getClass().getSimpleName().toLowerCase();
} else {
this.currentDatabase = getClass().getSimpleName();
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-core/src/test/java/org/apache/tajo/TajoTestingCluster.java
----------------------------------------------------------------------
diff --git a/tajo-core/src/test/java/org/apache/tajo/TajoTestingCluster.java b/tajo-core/src/test/java/org/apache/tajo/TajoTestingCluster.java
index f7fb2f2..e680736 100644
--- a/tajo-core/src/test/java/org/apache/tajo/TajoTestingCluster.java
+++ b/tajo-core/src/test/java/org/apache/tajo/TajoTestingCluster.java
@@ -86,9 +86,9 @@ public class TajoTestingCluster {
System.getProperty("tajo.test.data.dir", "test-data");
/**
- * True If HCatalogStore is used. Otherwise, it is FALSE.
+ * True If HiveCatalogStore is used. Otherwise, it is FALSE.
*/
- public Boolean isHCatalogStoreUse = false;
+ public Boolean isHiveCatalogStoreUse = false;
private static final String LOG_LEVEL;
@@ -320,8 +320,8 @@ public class TajoTestingCluster {
return this.catalogServer;
}
- public boolean isHCatalogStoreRunning() {
- return isHCatalogStoreUse;
+ public boolean isHiveCatalogStoreRunning() {
+ return isHiveCatalogStoreUse;
}
////////////////////////////////////////////////////////
@@ -383,31 +383,31 @@ public class TajoTestingCluster {
}
private void setupCatalogForTesting(TajoConf c, File testBuildDir) throws IOException {
- final String HCATALOG_CLASS_NAME = "org.apache.tajo.catalog.store.HCatalogStore";
- boolean hcatalogClassExists = false;
+ final String HIVE_CATALOG_CLASS_NAME = "org.apache.tajo.catalog.store.HiveCatalogStore";
+ boolean hiveCatalogClassExists = false;
try {
- getClass().getClassLoader().loadClass(HCATALOG_CLASS_NAME);
- hcatalogClassExists = true;
+ getClass().getClassLoader().loadClass(HIVE_CATALOG_CLASS_NAME);
+ hiveCatalogClassExists = true;
} catch (ClassNotFoundException e) {
- LOG.info("HCatalogStore is not available.");
+ LOG.info("HiveCatalogStore is not available.");
}
String driverClass = System.getProperty(CatalogConstants.STORE_CLASS);
- if (hcatalogClassExists &&
- driverClass != null && driverClass.equals(HCATALOG_CLASS_NAME)) {
+ if (hiveCatalogClassExists &&
+ driverClass != null && driverClass.equals(HIVE_CATALOG_CLASS_NAME)) {
try {
- getClass().getClassLoader().loadClass(HCATALOG_CLASS_NAME);
+ getClass().getClassLoader().loadClass(HIVE_CATALOG_CLASS_NAME);
String jdbcUri = "jdbc:derby:;databaseName="+ testBuildDir.toURI().getPath() + "/metastore_db;create=true";
c.set("hive.metastore.warehouse.dir", TajoConf.getWarehouseDir(c).toString() + "/default");
c.set("javax.jdo.option.ConnectionURL", jdbcUri);
c.set(TajoConf.ConfVars.WAREHOUSE_DIR.varname, conf.getVar(ConfVars.WAREHOUSE_DIR));
- c.set(CatalogConstants.STORE_CLASS, HCATALOG_CLASS_NAME);
+ c.set(CatalogConstants.STORE_CLASS, HIVE_CATALOG_CLASS_NAME);
Path defaultDatabasePath = new Path(TajoConf.getWarehouseDir(c).toString() + "/default");
FileSystem fs = defaultDatabasePath.getFileSystem(c);
if (!fs.exists(defaultDatabasePath)) {
fs.mkdirs(defaultDatabasePath);
}
- isHCatalogStoreUse = true;
+ isHiveCatalogStoreUse = true;
} catch (ClassNotFoundException cnfe) {
throw new IOException(cnfe);
}
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-core/src/test/java/org/apache/tajo/cli/tools/TestTajoDump.java
----------------------------------------------------------------------
diff --git a/tajo-core/src/test/java/org/apache/tajo/cli/tools/TestTajoDump.java b/tajo-core/src/test/java/org/apache/tajo/cli/tools/TestTajoDump.java
index 5819bd2..b371be2 100644
--- a/tajo-core/src/test/java/org/apache/tajo/cli/tools/TestTajoDump.java
+++ b/tajo-core/src/test/java/org/apache/tajo/cli/tools/TestTajoDump.java
@@ -29,7 +29,7 @@ public class TestTajoDump extends QueryTestCaseBase {
@Test
public void testDump1() throws Exception {
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
executeString("CREATE TABLE \"" + getCurrentDatabase() +
"\".\"TableName1\" (\"Age\" int, \"FirstName\" TEXT, lastname TEXT)");
@@ -48,7 +48,7 @@ public class TestTajoDump extends QueryTestCaseBase {
@Test
public void testDump2() throws Exception {
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
executeString("CREATE TABLE \"" + getCurrentDatabase() +
"\".\"TableName2\" (\"Age\" int, \"Name\" Record (\"FirstName\" TEXT, lastname TEXT))");
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-core/src/test/java/org/apache/tajo/cli/tsql/TestTajoCli.java
----------------------------------------------------------------------
diff --git a/tajo-core/src/test/java/org/apache/tajo/cli/tsql/TestTajoCli.java b/tajo-core/src/test/java/org/apache/tajo/cli/tsql/TestTajoCli.java
index aee5a02..fdf9ca4 100644
--- a/tajo-core/src/test/java/org/apache/tajo/cli/tsql/TestTajoCli.java
+++ b/tajo-core/src/test/java/org/apache/tajo/cli/tsql/TestTajoCli.java
@@ -100,7 +100,7 @@ public class TestTajoCli {
}
private void assertOutputResult(String expectedResultFile, String actual, String[] paramKeys, String[] paramValues)
- throws Exception {
+ throws Exception {
FileSystem fs = currentResultPath.getFileSystem(testBase.getTestingCluster().getConfiguration());
Path resultFile = StorageUtil.concatPath(currentResultPath, expectedResultFile);
assertTrue(resultFile.toString() + " existence check", fs.exists(resultFile));
@@ -140,7 +140,7 @@ public class TestTajoCli {
@Test
public void testParseConf() throws Exception {
String[] args = new String[]{"--conf", "tajo.cli.print.pause=false",
- "--conf", "tajo.executor.join.inner.in-memory-table-num=256"};
+ "--conf", "tajo.executor.join.inner.in-memory-table-num=256"};
CommandLineParser parser = new PosixParser();
CommandLine cmd = parser.parse(TajoCli.options, args);
@@ -186,7 +186,7 @@ public class TestTajoCli {
public void testConnectDatabase() throws Exception {
String databaseName;
- if (cluster.isHCatalogStoreRunning()) {
+ if (cluster.isHiveCatalogStoreRunning()) {
databaseName = "TEST_CONNECTION_DATABASE".toLowerCase();
} else {
databaseName = "TEST_CONNECTION_DATABASE";
@@ -215,16 +215,16 @@ public class TestTajoCli {
String consoleResult = new String(out.toByteArray());
FileSystem fs = FileSystem.get(testBase.getTestingCluster().getConfiguration());
- if (!cluster.isHCatalogStoreRunning()) {
+ if (!cluster.isHiveCatalogStoreRunning()) {
assertOutputResult(resultFileName, consoleResult, new String[]{"${table.path}"},
- new String[]{fs.getUri() + "/tajo/warehouse/default/" + tableName});
+ new String[]{fs.getUri() + "/tajo/warehouse/default/" + tableName});
}
}
@Test
public void testDescTable() throws Exception {
String tableName;
- if (cluster.isHCatalogStoreRunning()) {
+ if (cluster.isHiveCatalogStoreRunning()) {
tableName = "TEST_DESC_TABLE".toLowerCase();
} else {
tableName = "TEST_DESC_TABLE";
@@ -237,7 +237,7 @@ public class TestTajoCli {
@Test
public void testDescTableForNestedSchema() throws Exception {
String tableName;
- if (cluster.isHCatalogStoreRunning()) {
+ if (cluster.isHiveCatalogStoreRunning()) {
tableName = "TEST_DESC_TABLE_NESTED".toLowerCase();
} else {
tableName = "TEST_DESC_TABLE_NESTED";
@@ -250,15 +250,15 @@ public class TestTajoCli {
@Test
public void testSelectResultWithNullFalse() throws Exception {
String sql =
- "select\n" +
- " c_custkey,\n" +
- " orders.o_orderkey,\n" +
- " orders.o_orderstatus \n" +
- "from\n" +
- " orders full outer join customer on c_custkey = o_orderkey\n" +
- "order by\n" +
- " c_custkey,\n" +
- " orders.o_orderkey;\n";
+ "select\n" +
+ " c_custkey,\n" +
+ " orders.o_orderkey,\n" +
+ " orders.o_orderstatus \n" +
+ "from\n" +
+ " orders full outer join customer on c_custkey = o_orderkey\n" +
+ "order by\n" +
+ " c_custkey,\n" +
+ " orders.o_orderkey;\n";
setVar(tajoCli, SessionVars.CLI_FORMATTER_CLASS, TajoCliOutputTestFormatter.class.getName());
tajoCli.executeScript(sql);
@@ -269,15 +269,15 @@ public class TestTajoCli {
private void verifySelectResultWithNullTrue() throws Exception {
String sql =
- "select\n" +
- " c_custkey,\n" +
- " orders.o_orderkey,\n" +
- " orders.o_orderstatus \n" +
- "from\n" +
- " orders full outer join customer on c_custkey = o_orderkey\n" +
- "order by\n" +
- " c_custkey,\n" +
- " orders.o_orderkey;\n";
+ "select\n" +
+ " c_custkey,\n" +
+ " orders.o_orderkey,\n" +
+ " orders.o_orderstatus \n" +
+ "from\n" +
+ " orders full outer join customer on c_custkey = o_orderkey\n" +
+ "order by\n" +
+ " c_custkey,\n" +
+ " orders.o_orderkey;\n";
setVar(tajoCli, SessionVars.CLI_FORMATTER_CLASS, TajoCliOutputTestFormatter.class.getName());
@@ -307,8 +307,8 @@ public class TestTajoCli {
assertSessionVar(tajoCli, SessionVars.ON_ERROR_STOP.keyname(), "true");
tajoCli.executeScript("select count(*) from lineitem; " +
- "select count(*) from lineitem2; " +
- "select count(*) from orders");
+ "select count(*) from lineitem2; " +
+ "select count(*) from orders");
String consoleResult = new String(out.toByteArray());
assertOutputResult(consoleResult);
@@ -391,7 +391,7 @@ public class TestTajoCli {
}
@Test
- public void testAlterTableAddPartition() throws Exception {
+ public void testAlterTableAddPartition() throws Exception {
String tableName = CatalogUtil.normalizeIdentifier("testAlterTableAddPartition");
tajoCli.executeScript("create table " + tableName + " (col1 int4, col2 int4) partition by column(key float8)");
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-core/src/test/java/org/apache/tajo/engine/query/TestAlterTablespace.java
----------------------------------------------------------------------
diff --git a/tajo-core/src/test/java/org/apache/tajo/engine/query/TestAlterTablespace.java b/tajo-core/src/test/java/org/apache/tajo/engine/query/TestAlterTablespace.java
index 47e98a9..b0ec92d 100644
--- a/tajo-core/src/test/java/org/apache/tajo/engine/query/TestAlterTablespace.java
+++ b/tajo-core/src/test/java/org/apache/tajo/engine/query/TestAlterTablespace.java
@@ -31,7 +31,7 @@ public class TestAlterTablespace extends QueryTestCaseBase {
@Test
public final void testAlterLocation() throws Exception {
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
//////////////////////////////////////////////////////////////////////////////
// Create two table spaces
//////////////////////////////////////////////////////////////////////////////
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-core/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java
----------------------------------------------------------------------
diff --git a/tajo-core/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java b/tajo-core/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java
index 18c9fbc..f79f703 100644
--- a/tajo-core/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java
+++ b/tajo-core/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java
@@ -78,7 +78,7 @@ public class TestCTASQuery extends QueryTestCaseBase {
assertTrue(fs.isDirectory(new Path(path.toUri() + "/key=38.0")));
assertTrue(fs.isDirectory(new Path(path.toUri() + "/key=45.0")));
assertTrue(fs.isDirectory(new Path(path.toUri() + "/key=49.0")));
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(5, desc.getStats().getNumRows().intValue());
}
@@ -121,7 +121,7 @@ public class TestCTASQuery extends QueryTestCaseBase {
assertTrue(fs.isDirectory(new Path(path.toUri() + "/key=38.0")));
assertTrue(fs.isDirectory(new Path(path.toUri() + "/key=45.0")));
assertTrue(fs.isDirectory(new Path(path.toUri() + "/key=49.0")));
- if (!cluster.isHCatalogStoreRunning()) {
+ if (!cluster.isHiveCatalogStoreRunning()) {
assertEquals(5, desc.getStats().getNumRows().intValue());
}
@@ -261,7 +261,7 @@ public class TestCTASQuery extends QueryTestCaseBase {
ResultSet res = executeFile("CtasWithManagedTable.sql");
res.close();
- if (testingCluster.isHCatalogStoreRunning()) {
+ if (testingCluster.isHiveCatalogStoreRunning()) {
assertTrue(client.existTable("managed_table1"));
TableDesc desc = client.getTableDesc("managed_table1");
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-core/src/test/java/org/apache/tajo/engine/query/TestCreateTable.java
----------------------------------------------------------------------
diff --git a/tajo-core/src/test/java/org/apache/tajo/engine/query/TestCreateTable.java b/tajo-core/src/test/java/org/apache/tajo/engine/query/TestCreateTable.java
index 1fbe7c5..f1a3ddc 100644
--- a/tajo-core/src/test/java/org/apache/tajo/engine/query/TestCreateTable.java
+++ b/tajo-core/src/test/java/org/apache/tajo/engine/query/TestCreateTable.java
@@ -41,8 +41,8 @@ public class TestCreateTable extends QueryTestCaseBase {
@Test
public final void testVariousTypes() throws Exception {
List<String> createdNames;
- if (testingCluster.isHCatalogStoreRunning()) {
- createdNames = executeDDL("create_table_various_types_for_hcatalog.sql", null);
+ if (testingCluster.isHiveCatalogStoreRunning()) {
+ createdNames = executeDDL("create_table_various_types_for_hive_catalog.sql", null);
} else {
createdNames = executeDDL("create_table_various_types.sql", null);
}
@@ -135,7 +135,7 @@ public class TestCreateTable extends QueryTestCaseBase {
@Test
public final void testCreatedTableWithQuotedIdentifierAndVerifyPath() throws Exception {
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertPathOfCreatedTable("D6", "OldTable", "NewMgmtTable", "CREATE TABLE \"D6\".\"OldTable\" (age integer);");
}
}
@@ -174,7 +174,7 @@ public class TestCreateTable extends QueryTestCaseBase {
@Test
public final void testDelimitedIdentifierWithNonAsciiCharacters() throws Exception {
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
ResultSet res = null;
try {
List<String> tableNames = executeDDL("quoted_identifier_non_ascii_ddl.sql", "table1", "\"테이블1\"");
@@ -207,7 +207,7 @@ public class TestCreateTable extends QueryTestCaseBase {
@Test
public final void testDelimitedIdentifierWithMixedCharacters() throws Exception {
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
ResultSet res = null;
try {
@@ -442,8 +442,8 @@ public class TestCreateTable extends QueryTestCaseBase {
@Test
public final void testCreateTableLike1() throws Exception {
- // Hcatalog does not support varchar type in hive-0.12.0
- if (testingCluster.isHCatalogStoreRunning()) {
+ // //HiveCatalogStore does not support varchar type in hive-0.12.0
+ if (testingCluster.isHiveCatalogStoreRunning()) {
// Basic create table with default database
executeString("CREATE TABLE table1 (c1 int, c2 text);").close();
executeString("CREATE TABLE table2 LIKE table1");
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-core/src/test/java/org/apache/tajo/engine/query/TestInsertQuery.java
----------------------------------------------------------------------
diff --git a/tajo-core/src/test/java/org/apache/tajo/engine/query/TestInsertQuery.java b/tajo-core/src/test/java/org/apache/tajo/engine/query/TestInsertQuery.java
index 0799d22..4a1f601 100644
--- a/tajo-core/src/test/java/org/apache/tajo/engine/query/TestInsertQuery.java
+++ b/tajo-core/src/test/java/org/apache/tajo/engine/query/TestInsertQuery.java
@@ -56,7 +56,7 @@ public class TestInsertQuery extends QueryTestCaseBase {
res.close();
TableDesc desc = catalog.getTableDesc(getCurrentDatabase(), "table1");
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(5, desc.getStats().getNumRows().intValue());
}
@@ -76,7 +76,7 @@ public class TestInsertQuery extends QueryTestCaseBase {
res.close();
TableDesc desc = catalog.getTableDesc(getCurrentDatabase(), "table1");
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(5, desc.getStats().getNumRows().intValue());
}
@@ -311,7 +311,7 @@ public class TestInsertQuery extends QueryTestCaseBase {
res = executeFile("testInsertOverwriteSmallerColumns.sql");
res.close();
TableDesc desc = catalog.getTableDesc(getCurrentDatabase(), "table1");
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(5, desc.getStats().getNumRows().intValue());
}
assertEquals(originalDesc.getSchema(), desc.getSchema());
@@ -331,7 +331,7 @@ public class TestInsertQuery extends QueryTestCaseBase {
res = executeFile("testInsertOverwriteWithTargetColumns.sql");
res.close();
TableDesc desc = catalog.getTableDesc(getCurrentDatabase(), "table1");
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(5, desc.getStats().getNumRows().intValue());
}
@@ -385,7 +385,7 @@ public class TestInsertQuery extends QueryTestCaseBase {
res = executeString("insert overwrite into full_table_csv select * from default.lineitem where l_orderkey = 3");
res.close();
TableDesc desc = catalog.getTableDesc(getCurrentDatabase(), "full_table_csv");
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(2, desc.getStats().getNumRows().intValue());
}
executeString("DROP TABLE full_table_csv PURGE");
@@ -402,7 +402,7 @@ public class TestInsertQuery extends QueryTestCaseBase {
res = executeFile("load_to_lineitem_year_month.sql");
res.close();
TableDesc desc = catalog.getTableDesc(getCurrentDatabase(), "lineitem_year_month");
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(5, desc.getStats().getNumRows().intValue());
}
@@ -424,7 +424,7 @@ public class TestInsertQuery extends QueryTestCaseBase {
CatalogService catalog = testingCluster.getMaster().getCatalog();
assertTrue(catalog.existsTable(getCurrentDatabase(), tableName));
TableDesc orderKeys = catalog.getTableDesc(getCurrentDatabase(), tableName);
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(5, orderKeys.getStats().getNumRows().intValue());
}
@@ -435,7 +435,7 @@ public class TestInsertQuery extends QueryTestCaseBase {
assertTrue(catalog.existsTable(getCurrentDatabase(), tableName));
orderKeys = catalog.getTableDesc(getCurrentDatabase(), tableName);
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(2, orderKeys.getStats().getNumRows().intValue());
}
executeString("DROP TABLE " + tableName + " PURGE");
@@ -453,7 +453,7 @@ public class TestInsertQuery extends QueryTestCaseBase {
res = executeString("insert overwrite into " + tableName + " select * from default.lineitem where l_orderkey = 3");
res.close();
TableDesc desc = catalog.getTableDesc(getCurrentDatabase(), tableName);
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(2, desc.getStats().getNumRows().intValue());
}
executeString("DROP TABLE " + tableName + " PURGE");
@@ -480,7 +480,7 @@ public class TestInsertQuery extends QueryTestCaseBase {
res = executeQuery();
res.close();
TableDesc desc = catalog.getTableDesc(getCurrentDatabase(), tableName);
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(2, desc.getStats().getNumRows().intValue());
}
@@ -497,7 +497,7 @@ public class TestInsertQuery extends QueryTestCaseBase {
@Test
public final void testInsertOverwriteLocationWithCompression() throws Exception {
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
ResultSet res = executeQuery();
res.close();
FileSystem fs = FileSystem.get(testingCluster.getConfiguration());
@@ -515,7 +515,7 @@ public class TestInsertQuery extends QueryTestCaseBase {
@Test
public final void testInsertOverwriteWithAsteriskUsingParquet() throws Exception {
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
ResultSet res = executeFile("full_table_parquet_ddl.sql");
res.close();
@@ -526,7 +526,7 @@ public class TestInsertQuery extends QueryTestCaseBase {
"insert overwrite into full_table_parquet select * from default.lineitem where l_orderkey = 3");
res.close();
TableDesc desc = catalog.getTableDesc(getCurrentDatabase(), "full_table_parquet");
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(2, desc.getStats().getNumRows().intValue());
}
@@ -542,7 +542,7 @@ public class TestInsertQuery extends QueryTestCaseBase {
@Test
public final void testInsertOverwriteIntoParquet() throws Exception {
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
executeString("create table parquet_table " +
"(l_orderkey int4, l_shipdate text, l_shipdate_function text) using parquet").close();
@@ -554,7 +554,7 @@ public class TestInsertQuery extends QueryTestCaseBase {
"select l_orderkey, l_shipdate, substr(l_shipdate, 1, 10) from default.lineitem").close();
TableDesc desc = catalog.getTableDesc(getCurrentDatabase(), "parquet_table");
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(5, desc.getStats().getNumRows().intValue());
}
@@ -577,7 +577,7 @@ public class TestInsertQuery extends QueryTestCaseBase {
@Test
public final void testInsertOverwriteIntoPartitionedParquet() throws Exception {
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
executeString("create table parquet_table " +
"(l_orderkey int4, l_shipdate_function text) using parquet partition by column (l_shipdate text)").close();
@@ -589,7 +589,7 @@ public class TestInsertQuery extends QueryTestCaseBase {
"select l_orderkey, substr(l_shipdate, 1, 10), l_shipdate from default.lineitem").close();
TableDesc desc = catalog.getTableDesc(getCurrentDatabase(), "parquet_table");
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(5, desc.getStats().getNumRows().intValue());
}
@@ -622,7 +622,7 @@ public class TestInsertQuery extends QueryTestCaseBase {
res.close();
TableDesc desc = catalog.getTableDesc(getCurrentDatabase(), "table1");
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(5, desc.getStats().getNumRows().intValue());
}
executeString("DROP TABLE table1 PURGE");
@@ -642,7 +642,7 @@ public class TestInsertQuery extends QueryTestCaseBase {
res.close();
TableDesc desc = catalog.getTableDesc(getCurrentDatabase(), tableName);
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(1, desc.getStats().getNumRows().intValue());
}
@@ -669,7 +669,7 @@ public class TestInsertQuery extends QueryTestCaseBase {
res.close();
TableDesc desc = catalog.getTableDesc(getCurrentDatabase(), tableName);
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(1, desc.getStats().getNumRows().intValue());
}
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-core/src/test/java/org/apache/tajo/engine/query/TestNetTypes.java
----------------------------------------------------------------------
diff --git a/tajo-core/src/test/java/org/apache/tajo/engine/query/TestNetTypes.java b/tajo-core/src/test/java/org/apache/tajo/engine/query/TestNetTypes.java
index 182bf5b..bd8f830 100644
--- a/tajo-core/src/test/java/org/apache/tajo/engine/query/TestNetTypes.java
+++ b/tajo-core/src/test/java/org/apache/tajo/engine/query/TestNetTypes.java
@@ -28,7 +28,7 @@ public class TestNetTypes extends QueryTestCaseBase {
@Before
public final void setUp() throws Exception {
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
executeDDL("table1_ddl.sql", "table1");
executeDDL("table2_ddl.sql", "table2");
}
@@ -36,8 +36,8 @@ public class TestNetTypes extends QueryTestCaseBase {
@Test
public final void testSelect() throws Exception {
- // Skip all tests when HCatalogStore is used.
- if (!testingCluster.isHCatalogStoreRunning()) {
+ // Skip all tests when HiveCatalogStore is used.
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
// select name, addr from table1;
ResultSet res = executeQuery();
assertResultSet(res);
@@ -47,8 +47,8 @@ public class TestNetTypes extends QueryTestCaseBase {
@Test
public final void testGroupby() throws Exception {
- // Skip all tests when HCatalogStore is used.
- if (!testingCluster.isHCatalogStoreRunning()) {
+ // Skip all tests when HiveCatalogStore is used.
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
// select name, addr, count(1) from table1 group by name, addr;
ResultSet res = executeQuery();
assertResultSet(res);
@@ -58,8 +58,8 @@ public class TestNetTypes extends QueryTestCaseBase {
@Test
public final void testGroupby2() throws Exception {
- // Skip all tests when HCatalogStore is used.
- if (!testingCluster.isHCatalogStoreRunning()) {
+ // Skip all tests when HiveCatalogStore is used.
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
// select addr, count(*) from table1 group by addr;
ResultSet res = executeQuery();
assertResultSet(res);
@@ -69,8 +69,8 @@ public class TestNetTypes extends QueryTestCaseBase {
@Test
public final void testSort() throws Exception {
- // Skip all tests when HCatalogStore is used.
- if (!testingCluster.isHCatalogStoreRunning()) {
+ // Skip all tests when HiveCatalogStore is used.
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
// select * from table1 order by addr;
ResultSet res = executeQuery();
assertResultSet(res);
@@ -80,8 +80,8 @@ public class TestNetTypes extends QueryTestCaseBase {
@Test
public final void testSort2() throws Exception {
- // Skip all tests when HCatalogStore is used.
- if (!testingCluster.isHCatalogStoreRunning()) {
+ // Skip all tests when HiveCatalogStore is used.
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
// select addr from table2 order by addr;
ResultSet res = executeQuery();
assertResultSet(res);
@@ -91,8 +91,8 @@ public class TestNetTypes extends QueryTestCaseBase {
@Test
public final void testJoin() throws Exception {
- // Skip all tests when HCatalogStore is used.
- if (!testingCluster.isHCatalogStoreRunning()) {
+ // Skip all tests when HiveCatalogStore is used.
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
// select * from table1 as t1, table2 as t2 where t1.addr = t2.addr;
ResultSet res = executeQuery();
assertResultSet(res);
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-core/src/test/java/org/apache/tajo/engine/query/TestSelectQuery.java
----------------------------------------------------------------------
diff --git a/tajo-core/src/test/java/org/apache/tajo/engine/query/TestSelectQuery.java b/tajo-core/src/test/java/org/apache/tajo/engine/query/TestSelectQuery.java
index 48a3343..513868b 100644
--- a/tajo-core/src/test/java/org/apache/tajo/engine/query/TestSelectQuery.java
+++ b/tajo-core/src/test/java/org/apache/tajo/engine/query/TestSelectQuery.java
@@ -377,7 +377,7 @@ public class TestSelectQuery extends QueryTestCaseBase {
CatalogService catalog = cluster.getMaster().getCatalog();
assertTrue(catalog.existsTable(DEFAULT_DATABASE_NAME, "orderkeys"));
TableDesc orderKeys = catalog.getTableDesc(DEFAULT_DATABASE_NAME, "orderkeys");
- if (!cluster.isHCatalogStoreRunning()) {
+ if (!cluster.isHiveCatalogStoreRunning()) {
assertEquals(5, orderKeys.getStats().getNumRows().intValue());
}
}
@@ -399,7 +399,7 @@ public class TestSelectQuery extends QueryTestCaseBase {
@Test
public final void testDatabaseRef() throws Exception {
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
executeString("CREATE DATABASE \"TestSelectQuery\"").close();
executeString("CREATE TABLE \"TestSelectQuery\".\"LineItem\" AS SELECT * FROM default.lineitem" ).close();
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-core/src/test/java/org/apache/tajo/engine/query/TestSortQuery.java
----------------------------------------------------------------------
diff --git a/tajo-core/src/test/java/org/apache/tajo/engine/query/TestSortQuery.java b/tajo-core/src/test/java/org/apache/tajo/engine/query/TestSortQuery.java
index ff91177..1aee961 100644
--- a/tajo-core/src/test/java/org/apache/tajo/engine/query/TestSortQuery.java
+++ b/tajo-core/src/test/java/org/apache/tajo/engine/query/TestSortQuery.java
@@ -159,10 +159,10 @@ public class TestSortQuery extends QueryTestCaseBase {
@Test
public final void testSortWithDate() throws Exception {
- // skip this test if catalog uses HCatalogStore.
- // It is because HCatalogStore does not support Time data type.
+ // skip this test if catalog uses HiveCatalogStore.
+ // It is because HiveCatalogStore does not support Time data type.
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
// create external table table1 (col1 timestamp, col2 date, col3 time) ...
executeDDL("create_table_with_date_ddl.sql", "table1");
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-core/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
----------------------------------------------------------------------
diff --git a/tajo-core/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java b/tajo-core/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
index b48720a..bb15bfc 100644
--- a/tajo-core/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
+++ b/tajo-core/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
@@ -209,7 +209,7 @@ public class TestTablePartitions extends QueryTestCaseBase {
assertTrue(fs.isDirectory(new Path(path.toUri() + "/key=38.0")));
assertTrue(fs.isDirectory(new Path(path.toUri() + "/key=45.0")));
assertTrue(fs.isDirectory(new Path(path.toUri() + "/key=49.0")));
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(5, desc.getStats().getNumRows().intValue());
}
}
@@ -324,7 +324,7 @@ public class TestTablePartitions extends QueryTestCaseBase {
assertTrue(fs.isDirectory(new Path(path.toUri() + "/col1=3/col2=3")));
assertTrue(fs.isDirectory(new Path(path.toUri() + "/col1=3/col2=2/col3=45.0")));
assertTrue(fs.isDirectory(new Path(path.toUri() + "/col1=3/col2=3/col3=49.0")));
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(5, desc.getStats().getNumRows().intValue());
}
@@ -388,7 +388,7 @@ public class TestTablePartitions extends QueryTestCaseBase {
assertTrue(fs.isDirectory(new Path(path.toUri() + "/col1=3/col2=3")));
assertTrue(fs.isDirectory(new Path(path.toUri() + "/col1=3/col2=2/col3=45.0")));
assertTrue(fs.isDirectory(new Path(path.toUri() + "/col1=3/col2=3/col3=49.0")));
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(5, desc.getStats().getNumRows().intValue());
}
@@ -441,7 +441,7 @@ public class TestTablePartitions extends QueryTestCaseBase {
assertTrue(fs.isDirectory(new Path(path.toUri() + "/col1=3/col2=2/col3=45.0")));
assertTrue(fs.isDirectory(new Path(path.toUri() + "/col1=3/col2=3/col3=49.0")));
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(5, desc.getStats().getNumRows().intValue());
}
String expected = "N\n" +
@@ -503,7 +503,7 @@ public class TestTablePartitions extends QueryTestCaseBase {
assertTrue(fs.isDirectory(new Path(path.toUri() + "/col1=3/col2=2/col3=45.0")));
assertTrue(fs.isDirectory(new Path(path.toUri() + "/col1=3/col2=3/col3=49.0")));
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
// TODO: If there is existing another partition directory, we must add its rows number to result row numbers.
// assertEquals(6, desc.getStats().getNumRows().intValue());
}
@@ -550,7 +550,7 @@ public class TestTablePartitions extends QueryTestCaseBase {
"insert overwrite into " + tableName + " select l_partkey, l_quantity, l_orderkey from lineitem");
res.close();
TableDesc desc = catalog.getTableDesc(DEFAULT_DATABASE_NAME, tableName);
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(5, desc.getStats().getNumRows().intValue());
}
@@ -587,7 +587,7 @@ public class TestTablePartitions extends QueryTestCaseBase {
" select l_quantity, l_returnflag, l_orderkey, l_partkey from lineitem");
res.close();
TableDesc desc = catalog.getTableDesc(DEFAULT_DATABASE_NAME, tableName);
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(5, desc.getStats().getNumRows().intValue());
}
@@ -632,7 +632,7 @@ public class TestTablePartitions extends QueryTestCaseBase {
" select l_returnflag, l_orderkey, l_partkey, l_quantity from lineitem");
res.close();
TableDesc desc = catalog.getTableDesc(DEFAULT_DATABASE_NAME, tableName);
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(5, desc.getStats().getNumRows().intValue());
}
@@ -715,7 +715,7 @@ public class TestTablePartitions extends QueryTestCaseBase {
" select l_returnflag , l_orderkey, l_partkey, l_quantity from lineitem");
res.close();
TableDesc desc = catalog.getTableDesc(DEFAULT_DATABASE_NAME, tableName);
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(5, desc.getStats().getNumRows().intValue());
}
@@ -820,7 +820,7 @@ public class TestTablePartitions extends QueryTestCaseBase {
res.close();
TableDesc desc = catalog.getTableDesc("testinsertquery1", "table1");
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(5, desc.getStats().getNumRows().intValue());
}
@@ -829,7 +829,7 @@ public class TestTablePartitions extends QueryTestCaseBase {
res.close();
desc = catalog.getTableDesc("testinsertquery2", "table1");
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(5, desc.getStats().getNumRows().intValue());
}
}
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-core/src/test/java/org/apache/tajo/jdbc/TestResultSet.java
----------------------------------------------------------------------
diff --git a/tajo-core/src/test/java/org/apache/tajo/jdbc/TestResultSet.java b/tajo-core/src/test/java/org/apache/tajo/jdbc/TestResultSet.java
index 524c6e8..98e5ff9 100644
--- a/tajo-core/src/test/java/org/apache/tajo/jdbc/TestResultSet.java
+++ b/tajo-core/src/test/java/org/apache/tajo/jdbc/TestResultSet.java
@@ -139,8 +139,8 @@ public class TestResultSet {
@Test
public void testDateTimeType() throws Exception {
- // Hcatalog does not support date type, time type in hive-0.12.0
- if(util.isHCatalogStoreRunning()) return;
+ // HiveCatalog does not support date type, time type in hive-0.12.0
+ if(util.isHiveCatalogStoreRunning()) return;
ResultSet res = null;
TajoClient client = util.newTajoClient();
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-core/src/test/java/org/apache/tajo/jdbc/TestTajoDatabaseMetaData.java
----------------------------------------------------------------------
diff --git a/tajo-core/src/test/java/org/apache/tajo/jdbc/TestTajoDatabaseMetaData.java b/tajo-core/src/test/java/org/apache/tajo/jdbc/TestTajoDatabaseMetaData.java
index dfb0269..8ee6755 100644
--- a/tajo-core/src/test/java/org/apache/tajo/jdbc/TestTajoDatabaseMetaData.java
+++ b/tajo-core/src/test/java/org/apache/tajo/jdbc/TestTajoDatabaseMetaData.java
@@ -62,7 +62,7 @@ public class TestTajoDatabaseMetaData extends QueryTestCaseBase {
assertDatabaseExists("jdbc_test1");
pstmt.close();
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertDatabaseNotExists("Jdbc_Test2");
pstmt = conn.prepareStatement("CREATE DATABASE \"Jdbc_Test2\"");
pstmt.executeUpdate();
@@ -72,7 +72,7 @@ public class TestTajoDatabaseMetaData extends QueryTestCaseBase {
conn.setCatalog("jdbc_test1");
assertEquals("jdbc_test1", conn.getCatalog());
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
conn.setCatalog("Jdbc_Test2");
assertEquals("Jdbc_Test2", conn.getCatalog());
}
@@ -95,7 +95,7 @@ public class TestTajoDatabaseMetaData extends QueryTestCaseBase {
pstmt = conn.prepareStatement("DROP DATABASE jdbc_test1");
pstmt.executeUpdate();
pstmt.close();
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
pstmt = conn.prepareStatement("DROP DATABASE \"Jdbc_Test2\"");
pstmt.executeUpdate();
pstmt.close();
@@ -126,7 +126,7 @@ public class TestTajoDatabaseMetaData extends QueryTestCaseBase {
pstmt.executeUpdate();
pstmt.close();
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
// create database "jdbc_test2" and its tables
assertDatabaseNotExists("Jdbc_Test4");
pstmt = defaultConnect.prepareStatement("CREATE DATABASE \"Jdbc_Test4\"");
@@ -147,13 +147,13 @@ public class TestTajoDatabaseMetaData extends QueryTestCaseBase {
List<String> newDatabases = getListFromResultSet(dbmd.getCatalogs(), "TABLE_CAT");
newDatabases.removeAll(existingDatabases);
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertEquals(2, newDatabases.size());
} else {
assertEquals(1, newDatabases.size());
}
assertTrue(newDatabases.contains("jdbc_test3"));
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
assertTrue(newDatabases.contains("Jdbc_Test4"));
}
@@ -162,7 +162,7 @@ public class TestTajoDatabaseMetaData extends QueryTestCaseBase {
assertResultSet(res, "getTables1.result");
res.close();
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
res = defaultConnect.getMetaData().getTables("Jdbc_Test4", null, null, null);
assertResultSet(res, "getTables2.result");
res.close();
@@ -182,7 +182,7 @@ public class TestTajoDatabaseMetaData extends QueryTestCaseBase {
executeString("DROP TABLE jdbc_test3.table2");
executeString("DROP DATABASE jdbc_test3");
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
String jdbcTest2ConnUri =
TestTajoJdbc.buildConnectionUri(tajoMasterAddress.getHostName(), tajoMasterAddress.getPort(), "Jdbc_Test4");
Connection jdbcTest2Conn = DriverManager.getConnection(jdbcTest2ConnUri);
@@ -268,7 +268,7 @@ public class TestTajoDatabaseMetaData extends QueryTestCaseBase {
@Test
public void testGetColumnsWithPattern() throws Exception {
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
String connUri = TestTajoJdbc.buildConnectionUri(tajoMasterAddress.getHostName(), tajoMasterAddress.getPort(),
TajoConstants.DEFAULT_DATABASE_NAME);
Connection conn = DriverManager.getConnection(connUri);
@@ -397,7 +397,7 @@ public class TestTajoDatabaseMetaData extends QueryTestCaseBase {
@Test
public void testEmptyMetaInfo() throws Exception {
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
String connUri = TestTajoJdbc.buildConnectionUri(tajoMasterAddress.getHostName(), tajoMasterAddress.getPort(),
TajoConstants.DEFAULT_DATABASE_NAME);
Connection conn = DriverManager.getConnection(connUri);
@@ -464,7 +464,7 @@ public class TestTajoDatabaseMetaData extends QueryTestCaseBase {
@Test
public void testGetTypeInfo() throws Exception {
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
String connUri = TestTajoJdbc.buildConnectionUri(tajoMasterAddress.getHostName(), tajoMasterAddress.getPort(),
TajoConstants.DEFAULT_DATABASE_NAME);
Connection conn = DriverManager.getConnection(connUri);
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-core/src/test/java/org/apache/tajo/jdbc/TestTajoJdbc.java
----------------------------------------------------------------------
diff --git a/tajo-core/src/test/java/org/apache/tajo/jdbc/TestTajoJdbc.java b/tajo-core/src/test/java/org/apache/tajo/jdbc/TestTajoJdbc.java
index 36bbd94..0dbb8e5 100644
--- a/tajo-core/src/test/java/org/apache/tajo/jdbc/TestTajoJdbc.java
+++ b/tajo-core/src/test/java/org/apache/tajo/jdbc/TestTajoJdbc.java
@@ -553,10 +553,10 @@ public class TestTajoJdbc extends QueryTestCaseBase {
Connection conn = null;
int result;
- // skip this test if catalog uses HCatalogStore.
- // It is because HCatalogStore does not support Time data type.
+ // skip this test if catalog uses HiveCatalogStore.
+ // It is because HiveCatalogStore does not support Time data type.
try {
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
executeDDL("create_table_with_date_ddl.sql", "table1");
String connUri = buildConnectionUri(tajoMasterAddress.getHostName(),
@@ -604,10 +604,10 @@ public class TestTajoJdbc extends QueryTestCaseBase {
int result;
String errorMessage = null;
- // skip this test if catalog uses HCatalogStore.
- // It is because HCatalogStore does not support Time data type.
+ // skip this test if catalog uses HiveCatalogStore.
+ // It is because HiveCatalogStore does not support Time data type.
try {
- if (!testingCluster.isHCatalogStoreRunning()) {
+ if (!testingCluster.isHiveCatalogStoreRunning()) {
String connUri = buildConnectionUri(tajoMasterAddress.getHostName(), tajoMasterAddress.getPort(),
DEFAULT_DATABASE_NAME);
conn = DriverManager.getConnection(connUri);
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-core/src/test/resources/queries/TestCreateTable/create_table_various_types_for_hcatalog.sql
----------------------------------------------------------------------
diff --git a/tajo-core/src/test/resources/queries/TestCreateTable/create_table_various_types_for_hcatalog.sql b/tajo-core/src/test/resources/queries/TestCreateTable/create_table_various_types_for_hcatalog.sql
deleted file mode 100644
index 3eda994..0000000
--- a/tajo-core/src/test/resources/queries/TestCreateTable/create_table_various_types_for_hcatalog.sql
+++ /dev/null
@@ -1,50 +0,0 @@
--- Some types were commented out due to Hive meta test.
-
-create table various_types (
--- col0 bit,
--- col1 BIT(10),
--- col2 bit varying,
--- col3 bit VARYING(10),
- col4 tinyint,
- col5 smallInt,
- col6 integer,
- col7 biginT,
- col8 real,
- col9 float,
- col10 float(53),
- col11 double,
- col12 doublE precision,
--- col13 numeric,
--- col14 numeric(10),
--- col15 numeric(10,2),
--- col16 decimal,
--- col17 decimal(10),
--- col18 decimal(10,2),
--- col19 char,
--- col20 character,
--- col21 chaR(10),
--- col22 character(10),
--- col23 varchar,
--- col24 character varying,
--- col25 varchar(255),
--- col26 character varying (255),
--- col27 nchar,
--- col28 nchar(255),
--- col29 national character,
--- col30 national character(255),
--- col31 nvarchar,
--- col32 nvarchar(255),
--- col33 natIonal character varying,
--- col34 national character varying (255),
--- col35 date,
--- col36 time,
--- col37 timetz,
--- col38 time With time zone,
--- col39 timesTamptz,
--- col40 timestamp with time zone,
- col41 binary,
--- col42 binary(10),
--- col43 varbinary(10),
--- col44 binary Varying(10),
- col45 blOb
-);
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-core/src/test/resources/queries/TestCreateTable/create_table_various_types_for_hive_catalog.sql
----------------------------------------------------------------------
diff --git a/tajo-core/src/test/resources/queries/TestCreateTable/create_table_various_types_for_hive_catalog.sql b/tajo-core/src/test/resources/queries/TestCreateTable/create_table_various_types_for_hive_catalog.sql
new file mode 100644
index 0000000..3eda994
--- /dev/null
+++ b/tajo-core/src/test/resources/queries/TestCreateTable/create_table_various_types_for_hive_catalog.sql
@@ -0,0 +1,50 @@
+-- Some types were commented out due to Hive meta test.
+
+create table various_types (
+-- col0 bit,
+-- col1 BIT(10),
+-- col2 bit varying,
+-- col3 bit VARYING(10),
+ col4 tinyint,
+ col5 smallInt,
+ col6 integer,
+ col7 biginT,
+ col8 real,
+ col9 float,
+ col10 float(53),
+ col11 double,
+ col12 doublE precision,
+-- col13 numeric,
+-- col14 numeric(10),
+-- col15 numeric(10,2),
+-- col16 decimal,
+-- col17 decimal(10),
+-- col18 decimal(10,2),
+-- col19 char,
+-- col20 character,
+-- col21 chaR(10),
+-- col22 character(10),
+-- col23 varchar,
+-- col24 character varying,
+-- col25 varchar(255),
+-- col26 character varying (255),
+-- col27 nchar,
+-- col28 nchar(255),
+-- col29 national character,
+-- col30 national character(255),
+-- col31 nvarchar,
+-- col32 nvarchar(255),
+-- col33 natIonal character varying,
+-- col34 national character varying (255),
+-- col35 date,
+-- col36 time,
+-- col37 timetz,
+-- col38 time With time zone,
+-- col39 timesTamptz,
+-- col40 timestamp with time zone,
+ col41 binary,
+-- col42 binary(10),
+-- col43 varbinary(10),
+-- col44 binary Varying(10),
+ col45 blOb
+);
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-dist/pom.xml
----------------------------------------------------------------------
diff --git a/tajo-dist/pom.xml b/tajo-dist/pom.xml
index da5f48f..0cc61aa 100644
--- a/tajo-dist/pom.xml
+++ b/tajo-dist/pom.xml
@@ -151,9 +151,9 @@
run mkdir -p share/jdbc-dist
run cp -r $ROOT/tajo-jdbc/target/tajo-jdbc-${project.version}-jar-with-dependencies.jar ./share/jdbc-dist/tajo-jdbc-${project.version}.jar
- if [ -f $ROOT/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/target/lib/parquet-hive-bundle-*.jar ]
+ if [ -f $ROOT/tajo-catalog/tajo-catalog-drivers/tajo-hive/target/lib/parquet-hive-bundle-*.jar ]
then
- run cp -r $ROOT/tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/target/lib/parquet-hive-bundle-*.jar lib/
+ run cp -r $ROOT/tajo-catalog/tajo-catalog-drivers/tajo-hive/target/lib/parquet-hive-bundle-*.jar lib/
echo
echo "Tajo installed parquet-hive-bundle library at: ${project.build.directory}/tajo-${project.version}"
echo
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-dist/src/main/bin/tajo
----------------------------------------------------------------------
diff --git a/tajo-dist/src/main/bin/tajo b/tajo-dist/src/main/bin/tajo
index a92d1a9..4383f3c 100755
--- a/tajo-dist/src/main/bin/tajo
+++ b/tajo-dist/src/main/bin/tajo
@@ -295,9 +295,6 @@ if [ -d ${HIVE_LIB} ]; then
CLASSPATH=${CLASSPATH}:$f;
done
- for f in $HIVE_HOME/hcatalog/share/hcatalog/*hcatalog-core-*.jar; do
- CLASSPATH=${CLASSPATH}:$f;
- done
fi
if [ "${HIVE_JDBC_DRIVER_DIR}" != "" ]; then
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-dist/src/main/conf/catalog-site.xml.template
----------------------------------------------------------------------
diff --git a/tajo-dist/src/main/conf/catalog-site.xml.template b/tajo-dist/src/main/conf/catalog-site.xml.template
index dddcc89..365de6b 100644
--- a/tajo-dist/src/main/conf/catalog-site.xml.template
+++ b/tajo-dist/src/main/conf/catalog-site.xml.template
@@ -86,12 +86,12 @@
-->
-<!-- HCatalog Store Driver -->
-<!-- Please remove comment if you want to use HCatalog. -->
+<!-- HiveCatalog Store Driver -->
+<!-- Please remove comment if you want to use HiveCatalog. -->
<!--
<property>
<name>tajo.catalog.store.class</name>
- <value>org.apache.tajo.catalog.store.HCatalogStore</value>
+ <value>org.apache.tajo.catalog.store.HiveCatalogStore</value>
</property>
-->
http://git-wip-us.apache.org/repos/asf/tajo/blob/955a7bf8/tajo-dist/src/main/conf/tajo-env.sh
----------------------------------------------------------------------
diff --git a/tajo-dist/src/main/conf/tajo-env.sh b/tajo-dist/src/main/conf/tajo-env.sh
index d52a827..bb8465f 100755
--- a/tajo-dist/src/main/conf/tajo-env.sh
+++ b/tajo-dist/src/main/conf/tajo-env.sh
@@ -72,7 +72,7 @@
# The scheduling priority for daemon processes. See 'man nice'.
# export TAJO_NICENESS=10
-# It must be required to use HCatalogStore
+# It must be required to use HiveCatalogStore
# export HIVE_HOME=
# export HIVE_JDBC_DRIVER_DIR=