You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by mohammadshahidkhan <gi...@git.apache.org> on 2017/10/17 17:08:03 UTC
[GitHub] carbondata pull request #1418: Support db location
GitHub user mohammadshahidkhan opened a pull request:
https://github.com/apache/carbondata/pull/1418
Support db location
**Problem:**
Currently carbon stores the database and tables at the fixed configured location “carbon.storeLocation”. So even though in case of federated cluster i.e. multiple independent Namespaces/ Namenodes only one Namespace/ Namenode can be used as storage for the database and tables.
The requirement is to support when create database can specify the LOCATION; the location can be viewfs path, name service path.
Using database location attribute configuration different name nodes can be used to store different databases. Having different name nodes for different databases will scale the read/write throughput.
**Solution:**
Setting the LOCATION attribute for a new database is a way to work with sets of files in an HDFS directory structure outside the fixed carbon store location.
In Create database command user can specify the location, the location where database and table under that database namespace could be stored.
1. The database at specified location could be created at defined location using create database command with location attribute.
Syntax:
CREATE DATABASE [IF NOT EXISTS] database_name[COMMENT 'database_comment'] [LOCATION hdfs_path];
The HDFS path can be viewfs path, nameservice path.
2. The database name and database location will be retrieved from the HiveSessionCatalog. The tables will be retrieved by scanning the table folders.
3. The read/write operation on old databases/ new databases created without specifying the LOCATION attribute should be performed the fixed carbon store location.
4. The table path formation will be same as the hive. The table path should be formed from database location or fixed Carbon store location and table name as given below.
The will be three possible scenarios:
I. Table path for the databases defined with location attribute.
tablePath = databaseLocation +”/” + tableName
II. Table path for the databases defined without location attribute.
tablePath = carbon.storeLocation + “/” + database_Name+”.db” +”/” + tableName
III. New table path for the default database.
tablePath = carbon.storeLocation +”/” + tableName
5. The carbon should support hybrid Nameservice path i.e. different operation could be performed in different name service. For example in load command the raw CSV files could be read from the nameservice ns1 i.e. “hdfs://ns1/user/data.csv” and database location could point to viewfs path “viewfs://federation_cluster/dir1/carbonstore”.
Example:
CREATE DATABASE carbon LOCATION “viewfs://federation_cluster/dir1/carbonstore”;
CREATE TABLE carbon.carbon_table (c1 string, c2 int) stored by ‘carbondata’;
LOAD DATA INPATH ‘hdfs://ns1/user/data.csv’ into table carbon.carbon_table;
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mohammadshahidkhan/incubator-carbondata support_db_location
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1418.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1418
----
commit b92be927761391f37018f5018521813f588825b8
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-10T05:20:11Z
Support database location:
The database name and database location will be retrieved from the HiveSessionCatalog. The tables will be retrieved by scanning the table folders.
The read/write operation on old databases/ new databases created without specifying the LOCATION attribute should be performed the fixed carbon store location.
commit 2a88536a19f483b8aee833039ba364ec85a78ee4
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-10T06:00:43Z
Support configurable path for the store modification info file (modifiedTime.mdt).
commit 7ab2f29b38d35e2590a40f5e2f0939b5033715dc
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-10T10:17:00Z
Fixed drop database issue
commit 524f4532e8fcaf0c40a884c7c5b51fd0d887066e
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-11T10:00:51Z
Corrected lock file location
commit dbb3160919a28c9b0eb93ab9ed8dfecfe93129b4
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-16T08:58:07Z
Making carbon table path same as hive.
----
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/513/
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r150173260
--- Diff: core/src/main/java/org/apache/carbondata/core/locks/LocalFileLock.java ---
@@ -90,8 +90,9 @@ public LocalFileLock(String lockFileLocation, String lockFile) {
* @param tableIdentifier
* @param lockFile
*/
- public LocalFileLock(CarbonTableIdentifier tableIdentifier, String lockFile) {
- this(tableIdentifier.getDatabaseName() + CarbonCommonConstants.FILE_SEPARATOR + tableIdentifier
+ public LocalFileLock(AbsoluteTableIdentifier tableIdentifier, String lockFile) {
+ this(tableIdentifier.getCarbonTableIdentifier().getDatabaseName()
--- End diff --
Fixed
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/920/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on the issue:
https://github.com/apache/carbondata/pull/1418
retest this please
---
[GitHub] carbondata pull request #1418: [WIP] Support db location
Posted by mohammadshahidkhan <gi...@git.apache.org>.
GitHub user mohammadshahidkhan reopened a pull request:
https://github.com/apache/carbondata/pull/1418
[WIP] Support db location
**Problem:**
Currently carbon stores the database and tables at the fixed configured location “carbon.storeLocation”. So even though in case of federated cluster i.e. multiple independent Namespaces/ Namenodes only one Namespace/ Namenode can be used as storage for the database and tables.
The requirement is to support when create database can specify the LOCATION; the location can be viewfs path, name service path.
Using database location attribute configuration different name nodes can be used to store different databases. Having different name nodes for different databases will scale the read/write throughput.
**Solution:**
Setting the LOCATION attribute for a new database is a way to work with sets of files in an HDFS directory structure outside the fixed carbon store location.
In Create database command user can specify the location, the location where database and table under that database namespace could be stored.
1. The database at specified location could be created at defined location using create database command with location attribute.
Syntax:
CREATE DATABASE [IF NOT EXISTS] database_name[COMMENT 'database_comment'] [LOCATION hdfs_path];
The HDFS path can be viewfs path, nameservice path.
2. The database name and database location will be retrieved from the HiveSessionCatalog. The tables will be retrieved by scanning the table folders.
3. The read/write operation on old databases/ new databases created without specifying the LOCATION attribute should be performed the fixed carbon store location.
4. The table path formation will be same as the hive. The table path should be formed from database location or fixed Carbon store location and table name as given below.
The will be three possible scenarios:
I. Table path for the databases defined with location attribute.
tablePath = databaseLocation +”/” + tableName
II. Table path for the databases defined without location attribute.
tablePath = carbon.storeLocation + “/” + database_Name+”.db” +”/” + tableName
III. New table path for the default database.
tablePath = carbon.storeLocation +”/” + tableName
5. The carbon should support hybrid Nameservice path i.e. different operation could be performed in different name service. For example in load command the raw CSV files could be read from the nameservice ns1 i.e. “hdfs://ns1/user/data.csv” and database location could point to viewfs path “viewfs://federation_cluster/dir1/carbonstore”.
Example:
CREATE DATABASE carbon LOCATION “viewfs://federation_cluster/dir1/carbonstore”;
CREATE TABLE carbon.carbon_table (c1 string, c2 int) stored by ‘carbondata’;
LOAD DATA INPATH ‘hdfs://ns1/user/data.csv’ into table carbon.carbon_table;
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mohammadshahidkhan/incubator-carbondata support_db_location
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1418.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1418
----
commit efc5edb8f721bc22edcaa8af16179b5ed734d32b
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-10T05:20:11Z
Support database location:
The database name and database location will be retrieved from the HiveSessionCatalog. The tables will be retrieved by scanning the table folders.
The read/write operation on old databases/ new databases created without specifying the LOCATION attribute should be performed the fixed carbon store location.
commit bdb11c9a57bb5b7adc77a7d73ff242ef438be7cc
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-10T06:00:43Z
Support configurable path for the store modification info file (modifiedTime.mdt).
commit 8bf6ac4b37d40bde6c9223e2b06609e2c771c004
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-10T10:17:00Z
Fixed drop database issue
commit 6a336153297c6edc38010f6d871511a85b916531
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-11T10:00:51Z
Corrected lock file location
commit b60a874ca6ba60f71ec030c96b397d0d20562fbe
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-16T08:58:07Z
Making carbon table path same as hive.
commit d0d15148c0c4cc3f9a7c53c725138bb64def3e61
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-18T15:31:13Z
fixed failing test cases
commit 3a32d6ba03b1a813cf348db21c9e5190ba442c7f
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-19T10:04:04Z
fixed failing test cases
commit 151f25264e3cc1ecd1a78b051b5b9dc408aa3d90
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-19T14:02:53Z
Added database location test and fixed existing failing test cases
commit 92599d21f06085299aa3357d64c6cc5efd075de9
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-19T18:11:32Z
Fixed alter rename table + Corrected test cases
Fixed drop table issue
----
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
retest this please
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on the issue:
https://github.com/apache/carbondata/pull/1418
retest this please
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r149966044
--- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java ---
@@ -1377,6 +1377,10 @@
public static final String CARBON_USE_BLOCKLET_DISTRIBUTION_DEFAULT = "true";
+ public static final String CARBON_UPDATE_SYNC_FOLDER = "carbon.update.sync.folder";
--- End diff --
Add comment here for usage.
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] Support Database Location Configur...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/879/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1131/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/680/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1154/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/982/
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r149996665
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/CarbonCreateTableCommand.scala ---
@@ -37,11 +37,13 @@ case class CarbonCreateTableCommand(
}
override def processSchema(sparkSession: SparkSession): Seq[Row] = {
- val storePath = CarbonEnv.getInstance(sparkSession).storePath
+ var storePath = CarbonEnv.getInstance(sparkSession).storePath
CarbonEnv.getInstance(sparkSession).carbonMetastore.
- checkSchemasModifiedTimeAndReloadTables(storePath)
+ checkSchemasModifiedTimeAndReloadTables()
val LOGGER = LogServiceFactory.getLogService(this.getClass.getCanonicalName)
cm.databaseName = GetDB.getDatabaseName(cm.databaseNameOp, sparkSession)
+ storePath = GetDB.getDatabaseLocation(cm.databaseName, sparkSession, storePath)
--- End diff --
assign to another variable with proper name
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1576/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1583/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1313/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/502/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1535/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1124/
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r149979665
--- Diff: integration/presto/src/test/scala/org/apache/carbondata/presto/integrationtest/PrestoAllDataTypeTest.scala ---
@@ -31,7 +31,7 @@ class PrestoAllDataTypeTest extends FunSuiteLike with BeforeAndAfterAll {
.getLogService(classOf[PrestoAllDataTypeTest].getCanonicalName)
private val rootPath = new File(this.getClass.getResource("/").getPath
- + "../../../..").getCanonicalPath
+ + "../../../..").getCanonicalPath.replaceAll("\\\\", "/")
--- End diff --
please remove if not required
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/660/
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r149977104
--- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableInputFormat.java ---
@@ -885,4 +882,24 @@ public static DataTypeConverter getDataTypeConverter(Configuration configuration
}
return (DataTypeConverter) ObjectSerializationUtil.convertStringToObject(converter);
}
+
+ public static void setDatabaseName(Configuration configuration, String databaseName) {
+ if (null != databaseName) {
+ configuration.set(DATABASE_NAME, databaseName);
+ }
+ }
+
+ public static String getDatabaseName(Configuration configuration) {
+ return configuration.get(DATABASE_NAME);
+ }
+
+ public static void setTableName(Configuration configuration, String tableName) {
+ if (null != tableName) {
+ configuration.set(TABLE_NAME, tableName);
+ }
+ }
+
+ public static String getTableName(Configuration configuration) {
+ return configuration.get(TABLE_NAME);
--- End diff --
Throw exception if not set for dbname and tablename
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/989/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1097/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1141/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] Support Database Location Configur...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1397/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1588/
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r149998981
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableRenameCommand.scala ---
@@ -93,6 +96,22 @@ private[sql] case class CarbonAlterTableRenameCommand(
timeStamp = System.currentTimeMillis()
schemaEvolutionEntry.setTime_stamp(timeStamp)
renameBadRecords(oldTableName, newTableName, oldDatabaseName)
+
--- End diff --
Why the order changed?
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] Support Database Location Configur...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/886/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/363/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] Support Database Location Configur...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on the issue:
https://github.com/apache/carbondata/pull/1418
retest this please
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1161/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/669/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1048/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/500/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1129/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1135/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/952/
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r149993636
--- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CommonUtil.scala ---
@@ -839,14 +841,56 @@ object CommonUtil {
*/
def mergeIndexFiles(sparkContext: SparkContext,
segmentIds: Seq[String],
- storePath: String,
+ tablePath: String,
carbonTable: CarbonTable): Unit = {
if (CarbonProperties.getInstance().getProperty(
CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT_DEFAULT).toBoolean) {
- new CarbonMergeFilesRDD(sparkContext, AbsoluteTableIdentifier.from(storePath,
+ new CarbonMergeFilesRDD(sparkContext, AbsoluteTableIdentifier.from(tablePath,
carbonTable.getDatabaseName, carbonTable.getFactTableName).getTablePath,
segmentIds).collect()
}
}
+
+ /**
+ * The default database and databases ending with .db will point to the
+ * either either "carbon.storeLocation" or "spark.sql.warehouse.dir".
+ *
+ * @param storeLocation
+ * @param databaseName
+ * @param fixedStorePath
+ * @return
+ */
+ def getValidStorePath(storeLocation: String, databaseName: String,
+ fixedStorePath: String): String = {
--- End diff --
Use ends with .db to decide whether to use dblocation or storepath
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r150248529
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/CarbonCreateTableCommand.scala ---
@@ -37,11 +37,13 @@ case class CarbonCreateTableCommand(
}
override def processSchema(sparkSession: SparkSession): Seq[Row] = {
- val storePath = CarbonEnv.getInstance(sparkSession).storePath
+ var storePath = CarbonEnv.getInstance(sparkSession).storePath
CarbonEnv.getInstance(sparkSession).carbonMetastore.
- checkSchemasModifiedTimeAndReloadTables(storePath)
+ checkSchemasModifiedTimeAndReloadTables()
val LOGGER = LogServiceFactory.getLogService(this.getClass.getCanonicalName)
cm.databaseName = GetDB.getDatabaseName(cm.databaseNameOp, sparkSession)
+ storePath = GetDB.getDatabaseLocation(cm.databaseName, sparkSession, storePath)
--- End diff --
fixed
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r150171431
--- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java ---
@@ -1377,6 +1377,10 @@
public static final String CARBON_USE_BLOCKLET_DISTRIBUTION_DEFAULT = "true";
+ public static final String CARBON_UPDATE_SYNC_FOLDER = "carbon.update.sync.folder";
+
+ public static final String CARBON_UPDATE_SYNC_FOLDER_DEFAULT = "/tmp/carbondata";
--- End diff --
fixed
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1520/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/488/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1707/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1016/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/506/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1068/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/469/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/508/
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r149981553
--- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/CarbonMergerRDD.scala ---
@@ -167,7 +167,7 @@ class CarbonMergerRDD[K, V](
val dataFileMetadataSegMapping: java.util.Map[String, List[DataFileFooter]] =
CarbonCompactionUtil.createDataFileFooterMappingForSegments(tableBlockInfoList)
- carbonLoadModel.setStorePath(hdfsStoreLocation)
+ carbonLoadModel.setTablePath(hdfsStoreLocation)
--- End diff --
change name to hdfstablepath
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1567/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/987/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1018/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/524/
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/1418
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1108/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1059/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1525/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1122/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1041/
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r149997215
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/CarbonDropTableCommand.scala ---
@@ -43,24 +44,31 @@ case class CarbonDropTableCommand(
override def processSchema(sparkSession: SparkSession): Seq[Row] = {
val LOGGER: LogService = LogServiceFactory.getLogService(this.getClass.getCanonicalName)
val dbName = GetDB.getDatabaseName(databaseNameOp, sparkSession)
- val identifier = TableIdentifier(tableName, Option(dbName))
val carbonTableIdentifier = new CarbonTableIdentifier(dbName, tableName, "")
val locksToBeAcquired = List(LockUsage.METADATA_LOCK, LockUsage.DROP_TABLE_LOCK)
val carbonEnv = CarbonEnv.getInstance(sparkSession)
val catalog = carbonEnv.carbonMetastore
- val tableIdentifier =
- AbsoluteTableIdentifier.from(CarbonEnv.getInstance(sparkSession).storePath,
- dbName.toLowerCase, tableName.toLowerCase)
- catalog.checkSchemasModifiedTimeAndReloadTables(tableIdentifier.getStorePath)
+ // get the absolute table identifier to drop the table.
+ val metadataCache = catalog.getTableFromMetadataCache(dbName, tableName)
+ val absoluteTableIdentifier = metadataCache match {
+ case Some(tableMeta) =>
+ tableMeta.carbonTable.getAbsoluteTableIdentifier
+ case None =>
+ val storePath = GetDB.getDatabaseLocation(dbName, sparkSession,
--- End diff --
rename to dbpath
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/496/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1136/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1128/
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r150442218
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableRenameCommand.scala ---
@@ -93,6 +96,22 @@ private[sql] case class CarbonAlterTableRenameCommand(
timeStamp = System.currentTimeMillis()
schemaEvolutionEntry.setTime_stamp(timeStamp)
renameBadRecords(oldTableName, newTableName, oldDatabaseName)
+
--- End diff --
if hive and carbon both the tables points to the same location, and we do folder rename first then then hive alter command will fail.
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r149993949
--- Diff: integration/spark-common/src/main/scala/org/apache/spark/sql/test/TestQueryExecutor.scala ---
@@ -47,10 +47,11 @@ object TestQueryExecutor {
private val LOGGER = LogServiceFactory.getLogService(this.getClass.getCanonicalName)
val projectPath = new File(this.getClass.getResource("/").getPath + "../../../..")
- .getCanonicalPath
+ .getCanonicalPath.replaceAll("\\\\", "/")
--- End diff --
Don't change if not required
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1108/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1533/
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r150172471
--- Diff: core/src/main/java/org/apache/carbondata/core/locks/HdfsFileLock.java ---
@@ -47,26 +44,12 @@
private static String tmpPath;
- static {
--- End diff --
fixed
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1130/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/344/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/649/
---
[GitHub] carbondata pull request #1418: [WIP] Support db location
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan closed the pull request at:
https://github.com/apache/carbondata/pull/1418
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1163/
---
[GitHub] carbondata pull request #1418: [WIP] Support db location
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan closed the pull request at:
https://github.com/apache/carbondata/pull/1418
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r149966152
--- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java ---
@@ -1377,6 +1377,10 @@
public static final String CARBON_USE_BLOCKLET_DISTRIBUTION_DEFAULT = "true";
+ public static final String CARBON_UPDATE_SYNC_FOLDER = "carbon.update.sync.folder";
+
+ public static final String CARBON_UPDATE_SYNC_FOLDER_DEFAULT = "/tmp/carbondata";
--- End diff --
Add annotation
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r149995554
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDatasourceHadoopRelation.scala ---
@@ -48,7 +48,8 @@ case class CarbonDatasourceHadoopRelation(
isSubquery: ArrayBuffer[Boolean] = new ArrayBuffer[Boolean]())
extends BaseRelation with InsertableRelation {
- lazy val identifier: AbsoluteTableIdentifier = AbsoluteTableIdentifier.fromTablePath(paths.head)
+ lazy val identifier: AbsoluteTableIdentifier = AbsoluteTableIdentifier.from(paths.head,
+ parameters("dbname"), parameters("tablename"))
--- End diff --
Check the camel case of parameters, please debug and verify oce
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/542/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1594/
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r149979447
--- Diff: integration/presto/src/main/java/org/apache/carbondata/presto/impl/CarbonTableReader.java ---
@@ -271,14 +271,20 @@ private CarbonTable parseCarbonMetadata(SchemaTableName table) {
// If table is not previously cached, then:
// Step 1: get store path of the table and cache it.
- String storePath = config.getStorePath();
// create table identifier. the table id is randomly generated.
cache.carbonTableIdentifier =
new CarbonTableIdentifier(table.getSchemaName(), table.getTableName(),
UUID.randomUUID().toString());
+ String storePath = config.getStorePath();
--- End diff --
Add TODO saying that developer should pass table path not store path
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1666/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1540/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] Support Database Location Configur...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/905/
---
[GitHub] carbondata pull request #1418: [WIP] Support db location
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan closed the pull request at:
https://github.com/apache/carbondata/pull/1418
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r150004203
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala ---
@@ -399,16 +396,31 @@ class CarbonFileMetastore extends CarbonMetaStore {
FileFactory.isFileExist(tablePath, fileType)
} catch {
case e: Exception =>
- false
+ // this is identify the stale tables, table physically present but removed from hive
+ try {
--- End diff --
remove and try whether all tests passes
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1711/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] Support Database Location Configur...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1338/
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r149997376
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/CarbonDropTableCommand.scala ---
@@ -80,10 +88,21 @@ case class CarbonDropTableCommand(
override def processData(sparkSession: SparkSession): Seq[Row] = {
// delete the table folder
val dbName = GetDB.getDatabaseName(databaseNameOp, sparkSession)
- val tableIdentifier =
- AbsoluteTableIdentifier.from(CarbonEnv.getInstance(sparkSession).storePath, dbName, tableName)
+ // get the absolute table identifier to drop the table.
+ val carbonEnv = CarbonEnv.getInstance(sparkSession)
+ val catalog = carbonEnv.carbonMetastore
+ val metadataCache = catalog.getTableFromMetadataCache(dbName, tableName)
+ val absoluteTableIdentifier = metadataCache match {
+ case Some(tableMeta) =>
+ tableMeta.carbonTable.getAbsoluteTableIdentifier
+ case None =>
+ val storePath = GetDB.getDatabaseLocation(dbName, sparkSession,
--- End diff --
Change name to dbpath in all places
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1701/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/962/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] Support Database Location Configur...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1322/
---
[GitHub] carbondata pull request #1418: [WIP] Support db location
Posted by mohammadshahidkhan <gi...@git.apache.org>.
GitHub user mohammadshahidkhan reopened a pull request:
https://github.com/apache/carbondata/pull/1418
[WIP] Support db location
**Problem:**
Currently carbon stores the database and tables at the fixed configured location “carbon.storeLocation”. So even though in case of federated cluster i.e. multiple independent Namespaces/ Namenodes only one Namespace/ Namenode can be used as storage for the database and tables.
The requirement is to support when create database can specify the LOCATION; the location can be viewfs path, name service path.
Using database location attribute configuration different name nodes can be used to store different databases. Having different name nodes for different databases will scale the read/write throughput.
**Solution:**
Setting the LOCATION attribute for a new database is a way to work with sets of files in an HDFS directory structure outside the fixed carbon store location.
In Create database command user can specify the location, the location where database and table under that database namespace could be stored.
1. The database at specified location could be created at defined location using create database command with location attribute.
Syntax:
CREATE DATABASE [IF NOT EXISTS] database_name[COMMENT 'database_comment'] [LOCATION hdfs_path];
The HDFS path can be viewfs path, nameservice path.
2. The database name and database location will be retrieved from the HiveSessionCatalog. The tables will be retrieved by scanning the table folders.
3. The read/write operation on old databases/ new databases created without specifying the LOCATION attribute should be performed the fixed carbon store location.
4. The table path formation will be same as the hive. The table path should be formed from database location or fixed Carbon store location and table name as given below.
The will be three possible scenarios:
I. Table path for the databases defined with location attribute.
tablePath = databaseLocation +”/” + tableName
II. Table path for the databases defined without location attribute.
tablePath = carbon.storeLocation + “/” + database_Name+”.db” +”/” + tableName
III. New table path for the default database.
tablePath = carbon.storeLocation +”/” + tableName
5. The carbon should support hybrid Nameservice path i.e. different operation could be performed in different name service. For example in load command the raw CSV files could be read from the nameservice ns1 i.e. “hdfs://ns1/user/data.csv” and database location could point to viewfs path “viewfs://federation_cluster/dir1/carbonstore”.
Example:
CREATE DATABASE carbon LOCATION “viewfs://federation_cluster/dir1/carbonstore”;
CREATE TABLE carbon.carbon_table (c1 string, c2 int) stored by ‘carbondata’;
LOAD DATA INPATH ‘hdfs://ns1/user/data.csv’ into table carbon.carbon_table;
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mohammadshahidkhan/incubator-carbondata support_db_location
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1418.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1418
----
commit efc5edb8f721bc22edcaa8af16179b5ed734d32b
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-10T05:20:11Z
Support database location:
The database name and database location will be retrieved from the HiveSessionCatalog. The tables will be retrieved by scanning the table folders.
The read/write operation on old databases/ new databases created without specifying the LOCATION attribute should be performed the fixed carbon store location.
commit bdb11c9a57bb5b7adc77a7d73ff242ef438be7cc
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-10T06:00:43Z
Support configurable path for the store modification info file (modifiedTime.mdt).
commit 8bf6ac4b37d40bde6c9223e2b06609e2c771c004
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-10T10:17:00Z
Fixed drop database issue
commit 6a336153297c6edc38010f6d871511a85b916531
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-11T10:00:51Z
Corrected lock file location
commit b60a874ca6ba60f71ec030c96b397d0d20562fbe
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-16T08:58:07Z
Making carbon table path same as hive.
commit d0d15148c0c4cc3f9a7c53c725138bb64def3e61
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-18T15:31:13Z
fixed failing test cases
commit 3a32d6ba03b1a813cf348db21c9e5190ba442c7f
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-19T10:04:04Z
fixed failing test cases
commit 151f25264e3cc1ecd1a78b051b5b9dc408aa3d90
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-19T14:02:53Z
Added database location test and fixed existing failing test cases
commit ba728d356d35816d5406b71a566088085e80362f
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-19T18:11:32Z
Fixed alter rename table + Corrected test cases
Fixed drop table issue
commit 6005c9397fd5380f0ba9b76c8dafb0ba7848a010
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-21T09:47:11Z
Fixed presto compilation issue
commit 38383327ce7cd1b6b78e5fbc01cfb2b0303d09a3
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-23T12:39:01Z
added log to debug the code
----
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1635/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/372/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/658/
---
[GitHub] carbondata pull request #1418: [WIP] Support db location
Posted by mohammadshahidkhan <gi...@git.apache.org>.
GitHub user mohammadshahidkhan reopened a pull request:
https://github.com/apache/carbondata/pull/1418
[WIP] Support db location
**Problem:**
Currently carbon stores the database and tables at the fixed configured location “carbon.storeLocation”. So even though in case of federated cluster i.e. multiple independent Namespaces/ Namenodes only one Namespace/ Namenode can be used as storage for the database and tables.
The requirement is to support when create database can specify the LOCATION; the location can be viewfs path, name service path.
Using database location attribute configuration different name nodes can be used to store different databases. Having different name nodes for different databases will scale the read/write throughput.
**Solution:**
Setting the LOCATION attribute for a new database is a way to work with sets of files in an HDFS directory structure outside the fixed carbon store location.
In Create database command user can specify the location, the location where database and table under that database namespace could be stored.
1. The database at specified location could be created at defined location using create database command with location attribute.
Syntax:
CREATE DATABASE [IF NOT EXISTS] database_name[COMMENT 'database_comment'] [LOCATION hdfs_path];
The HDFS path can be viewfs path, nameservice path.
2. The database name and database location will be retrieved from the HiveSessionCatalog. The tables will be retrieved by scanning the table folders.
3. The read/write operation on old databases/ new databases created without specifying the LOCATION attribute should be performed the fixed carbon store location.
4. The table path formation will be same as the hive. The table path should be formed from database location or fixed Carbon store location and table name as given below.
The will be three possible scenarios:
I. Table path for the databases defined with location attribute.
tablePath = databaseLocation +”/” + tableName
II. Table path for the databases defined without location attribute.
tablePath = carbon.storeLocation + “/” + database_Name+”.db” +”/” + tableName
III. New table path for the default database.
tablePath = carbon.storeLocation +”/” + tableName
5. The carbon should support hybrid Nameservice path i.e. different operation could be performed in different name service. For example in load command the raw CSV files could be read from the nameservice ns1 i.e. “hdfs://ns1/user/data.csv” and database location could point to viewfs path “viewfs://federation_cluster/dir1/carbonstore”.
Example:
CREATE DATABASE carbon LOCATION “viewfs://federation_cluster/dir1/carbonstore”;
CREATE TABLE carbon.carbon_table (c1 string, c2 int) stored by ‘carbondata’;
LOAD DATA INPATH ‘hdfs://ns1/user/data.csv’ into table carbon.carbon_table;
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mohammadshahidkhan/incubator-carbondata support_db_location
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1418.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1418
----
commit efc5edb8f721bc22edcaa8af16179b5ed734d32b
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-10T05:20:11Z
Support database location:
The database name and database location will be retrieved from the HiveSessionCatalog. The tables will be retrieved by scanning the table folders.
The read/write operation on old databases/ new databases created without specifying the LOCATION attribute should be performed the fixed carbon store location.
commit bdb11c9a57bb5b7adc77a7d73ff242ef438be7cc
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-10T06:00:43Z
Support configurable path for the store modification info file (modifiedTime.mdt).
commit 8bf6ac4b37d40bde6c9223e2b06609e2c771c004
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-10T10:17:00Z
Fixed drop database issue
commit 6a336153297c6edc38010f6d871511a85b916531
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-11T10:00:51Z
Corrected lock file location
commit b60a874ca6ba60f71ec030c96b397d0d20562fbe
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-16T08:58:07Z
Making carbon table path same as hive.
commit d0d15148c0c4cc3f9a7c53c725138bb64def3e61
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-18T15:31:13Z
fixed failing test cases
commit 3a32d6ba03b1a813cf348db21c9e5190ba442c7f
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-19T10:04:04Z
fixed failing test cases
commit 151f25264e3cc1ecd1a78b051b5b9dc408aa3d90
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-19T14:02:53Z
Added database location test and fixed existing failing test cases
commit fbdb7685e4822322d232ff8c255b82afd19f3d0f
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-19T18:11:32Z
Fixed alter rename table + Corrected test cases
Fixed drop table issue
----
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1116/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] Support Database Location Configur...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1493/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on the issue:
https://github.com/apache/carbondata/pull/1418
retest this please
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1105/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1598/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] Support Database Location Configur...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/685/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/921/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1043/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] Support Database Location Configur...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on the issue:
https://github.com/apache/carbondata/pull/1418
retest this please
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1636/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] Support Database Location Configur...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1499/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1661/
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r150248692
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/CarbonDropTableCommand.scala ---
@@ -80,10 +88,21 @@ case class CarbonDropTableCommand(
override def processData(sparkSession: SparkSession): Seq[Row] = {
// delete the table folder
val dbName = GetDB.getDatabaseName(databaseNameOp, sparkSession)
- val tableIdentifier =
- AbsoluteTableIdentifier.from(CarbonEnv.getInstance(sparkSession).storePath, dbName, tableName)
+ // get the absolute table identifier to drop the table.
+ val carbonEnv = CarbonEnv.getInstance(sparkSession)
+ val catalog = carbonEnv.carbonMetastore
+ val metadataCache = catalog.getTableFromMetadataCache(dbName, tableName)
+ val absoluteTableIdentifier = metadataCache match {
+ case Some(tableMeta) =>
+ tableMeta.carbonTable.getAbsoluteTableIdentifier
+ case None =>
+ val storePath = GetDB.getDatabaseLocation(dbName, sparkSession,
--- End diff --
Fixed
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/672/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1309/
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r149971236
--- Diff: core/src/main/java/org/apache/carbondata/core/locks/HdfsFileLock.java ---
@@ -47,26 +44,12 @@
private static String tmpPath;
- static {
--- End diff --
Use the mdtfile path to use systemlocking
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1715/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/531/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1131/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] Support Database Location Configur...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1501/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1112/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1126/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] Support Database Location Configur...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1496/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/910/
---
[GitHub] carbondata pull request #1418: [WIP] Support db location
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan closed the pull request at:
https://github.com/apache/carbondata/pull/1418
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1306/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/974/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1536/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/918/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1295/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/503/
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r150442280
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDatasourceHadoopRelation.scala ---
@@ -48,7 +48,8 @@ case class CarbonDatasourceHadoopRelation(
isSubquery: ArrayBuffer[Boolean] = new ArrayBuffer[Boolean]())
extends BaseRelation with InsertableRelation {
- lazy val identifier: AbsoluteTableIdentifier = AbsoluteTableIdentifier.fromTablePath(paths.head)
+ lazy val identifier: AbsoluteTableIdentifier = AbsoluteTableIdentifier.from(paths.head,
+ parameters("dbname"), parameters("tablename"))
--- End diff --
the map parameters is case insensitive
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/514/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/480/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
LGTM
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1285/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1142/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1173/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1634/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1017/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/355/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1275/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1317/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/533/
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r150442899
--- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CommonUtil.scala ---
@@ -839,14 +841,56 @@ object CommonUtil {
*/
def mergeIndexFiles(sparkContext: SparkContext,
segmentIds: Seq[String],
- storePath: String,
+ tablePath: String,
carbonTable: CarbonTable): Unit = {
if (CarbonProperties.getInstance().getProperty(
CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT_DEFAULT).toBoolean) {
- new CarbonMergeFilesRDD(sparkContext, AbsoluteTableIdentifier.from(storePath,
+ new CarbonMergeFilesRDD(sparkContext, AbsoluteTableIdentifier.from(tablePath,
carbonTable.getDatabaseName, carbonTable.getFactTableName).getTablePath,
segmentIds).collect()
}
}
+
+ /**
+ * The default database and databases ending with .db will point to the
+ * either either "carbon.storeLocation" or "spark.sql.warehouse.dir".
+ *
+ * @param storeLocation
+ * @param databaseName
+ * @param fixedStorePath
+ * @return
+ */
+ def getValidStorePath(storeLocation: String, databaseName: String,
+ fixedStorePath: String): String = {
--- End diff --
fixed
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1116/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/925/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/969/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1134/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/507/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1299/
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r149994202
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonCatalystOperators.scala ---
@@ -70,6 +71,14 @@ object GetDB {
dbName.getOrElse(
sparkSession.sessionState.catalog.asInstanceOf[HiveSessionCatalog].getCurrentDatabase)
}
+
+ def getDatabaseLocation(dbName: String, sparkSession: SparkSession,
--- End diff --
Add some comment
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/529/
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r150171389
--- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java ---
@@ -1377,6 +1377,10 @@
public static final String CARBON_USE_BLOCKLET_DISTRIBUTION_DEFAULT = "true";
+ public static final String CARBON_UPDATE_SYNC_FOLDER = "carbon.update.sync.folder";
--- End diff --
Fixed
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1297/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/676/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] Support Database Location Configur...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/760/
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/662/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] Support Database Location Configur...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/702/
---
[GitHub] carbondata pull request #1418: [WIP] Support db location
Posted by mohammadshahidkhan <gi...@git.apache.org>.
GitHub user mohammadshahidkhan reopened a pull request:
https://github.com/apache/carbondata/pull/1418
[WIP] Support db location
**Problem:**
Currently carbon stores the database and tables at the fixed configured location “carbon.storeLocation”. So even though in case of federated cluster i.e. multiple independent Namespaces/ Namenodes only one Namespace/ Namenode can be used as storage for the database and tables.
The requirement is to support when create database can specify the LOCATION; the location can be viewfs path, name service path.
Using database location attribute configuration different name nodes can be used to store different databases. Having different name nodes for different databases will scale the read/write throughput.
**Solution:**
Setting the LOCATION attribute for a new database is a way to work with sets of files in an HDFS directory structure outside the fixed carbon store location.
In Create database command user can specify the location, the location where database and table under that database namespace could be stored.
1. The database at specified location could be created at defined location using create database command with location attribute.
Syntax:
CREATE DATABASE [IF NOT EXISTS] database_name[COMMENT 'database_comment'] [LOCATION hdfs_path];
The HDFS path can be viewfs path, nameservice path.
2. The database name and database location will be retrieved from the HiveSessionCatalog. The tables will be retrieved by scanning the table folders.
3. The read/write operation on old databases/ new databases created without specifying the LOCATION attribute should be performed the fixed carbon store location.
4. The table path formation will be same as the hive. The table path should be formed from database location or fixed Carbon store location and table name as given below.
The will be three possible scenarios:
I. Table path for the databases defined with location attribute.
tablePath = databaseLocation +”/” + tableName
II. Table path for the databases defined without location attribute.
tablePath = carbon.storeLocation + “/” + database_Name+”.db” +”/” + tableName
III. New table path for the default database.
tablePath = carbon.storeLocation +”/” + tableName
5. The carbon should support hybrid Nameservice path i.e. different operation could be performed in different name service. For example in load command the raw CSV files could be read from the nameservice ns1 i.e. “hdfs://ns1/user/data.csv” and database location could point to viewfs path “viewfs://federation_cluster/dir1/carbonstore”.
Example:
CREATE DATABASE carbon LOCATION “viewfs://federation_cluster/dir1/carbonstore”;
CREATE TABLE carbon.carbon_table (c1 string, c2 int) stored by ‘carbondata’;
LOAD DATA INPATH ‘hdfs://ns1/user/data.csv’ into table carbon.carbon_table;
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mohammadshahidkhan/incubator-carbondata support_db_location
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1418.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1418
----
commit efc5edb8f721bc22edcaa8af16179b5ed734d32b
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-10T05:20:11Z
Support database location:
The database name and database location will be retrieved from the HiveSessionCatalog. The tables will be retrieved by scanning the table folders.
The read/write operation on old databases/ new databases created without specifying the LOCATION attribute should be performed the fixed carbon store location.
commit bdb11c9a57bb5b7adc77a7d73ff242ef438be7cc
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-10T06:00:43Z
Support configurable path for the store modification info file (modifiedTime.mdt).
commit 8bf6ac4b37d40bde6c9223e2b06609e2c771c004
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-10T10:17:00Z
Fixed drop database issue
commit 6a336153297c6edc38010f6d871511a85b916531
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-11T10:00:51Z
Corrected lock file location
commit b60a874ca6ba60f71ec030c96b397d0d20562fbe
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-16T08:58:07Z
Making carbon table path same as hive.
commit d0d15148c0c4cc3f9a7c53c725138bb64def3e61
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-18T15:31:13Z
fixed failing test cases
commit 3a32d6ba03b1a813cf348db21c9e5190ba442c7f
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-19T10:04:04Z
fixed failing test cases
commit 151f25264e3cc1ecd1a78b051b5b9dc408aa3d90
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-19T14:02:53Z
Added database location test and fixed existing failing test cases
commit 92599d21f06085299aa3357d64c6cc5efd075de9
Author: mohammadshahidkhan <mo...@gmail.com>
Date: 2017-10-19T18:11:32Z
Fixed alter rename table + Corrected test cases
Fixed drop table issue
----
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] Support Database Location Configur...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1418
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/881/
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on the issue:
https://github.com/apache/carbondata/pull/1418
retest this please
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1659/
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r150442055
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala ---
@@ -399,16 +396,31 @@ class CarbonFileMetastore extends CarbonMetaStore {
FileFactory.isFileExist(tablePath, fileType)
} catch {
case e: Exception =>
- false
+ // this is identify the stale tables, table physically present but removed from hive
+ try {
--- End diff --
fixed
---
[GitHub] carbondata issue #1418: [WIP] Support db location
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1159/
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r149971910
--- Diff: core/src/main/java/org/apache/carbondata/core/locks/LocalFileLock.java ---
@@ -90,8 +90,9 @@ public LocalFileLock(String lockFileLocation, String lockFile) {
* @param tableIdentifier
* @param lockFile
*/
- public LocalFileLock(CarbonTableIdentifier tableIdentifier, String lockFile) {
- this(tableIdentifier.getDatabaseName() + CarbonCommonConstants.FILE_SEPARATOR + tableIdentifier
+ public LocalFileLock(AbsoluteTableIdentifier tableIdentifier, String lockFile) {
+ this(tableIdentifier.getCarbonTableIdentifier().getDatabaseName()
--- End diff --
Don't use the storelocation to create path, use tablepath
---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] [Integration] Support Database Loc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1418
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1606/
---
[GitHub] carbondata pull request #1418: [CARBONDATA-1573] [Integration] Support Datab...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1418#discussion_r149982091
--- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/CarbonMergerRDD.scala ---
@@ -167,7 +167,7 @@ class CarbonMergerRDD[K, V](
val dataFileMetadataSegMapping: java.util.Map[String, List[DataFileFooter]] =
--- End diff --
remove storepath from CarbonMergerMapping if nobody is using
---