You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by jackylk <gi...@git.apache.org> on 2018/01/02 15:49:57 UTC
[GitHub] carbondata pull request #1749: [CARBONDATA-1968] Add external table support
GitHub user jackylk opened a pull request:
https://github.com/apache/carbondata/pull/1749
[CARBONDATA-1968] Add external table support
This PR adds support for creating external table with existing carbondata files, using Hive syntax.
CREATE EXTERNAL TABLE tableName STORED BY 'carbondata' LOCATION 'path'
- [X] Any interfaces changed?
No
- [X] Any backward compatibility impacted?
No
- [X] Document update required?
Yes
- [X] Testing done
Testcase added
- [X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
NA
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jackylk/incubator-carbondata external_table
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1749.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1749
----
commit 5d4376bdf7bfd99ff72da355cea0103b568f78c0
Author: Jacky Li <ja...@...>
Date: 2018-01-02T15:46:14Z
add external table support
----
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1749
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2771/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1749
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2918/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1749
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2886/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1749
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1627/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1749
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2867/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1749
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2759/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1749
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2660/
---
[GitHub] carbondata pull request #1749: [CARBONDATA-1968] Add external table support
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk closed the pull request at:
https://github.com/apache/carbondata/pull/1749
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1749
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2749/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1749
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2908/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1749
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1528/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1749
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1538/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1749
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1520/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1749
Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1263/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/1749
Merged into carbonstore branch
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/1749
@mohammadshahidkhan Yes, that is correct. Drop table should not delete data. I have modify the testcase to verify it.
I have added a internal table property ("_external"), it will be set to true during table creation in `createCarbonTable`
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1749
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1518/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/1749
I think the first commit of these 5 commits (hash code 177d6d26c58e0d06c2cdb93d4a03a27472884559) need to be drop, and squash other 4 commits.
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1749
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2750/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1749
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2488/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1749
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1554/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1749
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2863/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1749
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2858/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on the issue:
https://github.com/apache/carbondata/pull/1749
@jackylk
I have couple of Queries.
For carbon external table, how to identify the table is external table?
What would be drop table behaviour for external carbon table?
As per hive For an external table, dropping the table only involves changes to metadata in the metastore database.
The data in the table will NOT be deleted from the file system.
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1749
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1525/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1749
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1685/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1749
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1550/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1749
@jackylk I can see other commits from other closed PR , is it accidentally added or you want to merge those commits here?
And also can you answer this question
How the external table feature works for enabling carbon hive metastore option. Schema will not be stored in file if hive metastore is enabled so we cannot create external table from it.
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1749
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2783/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/1749
retest this please
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1749
LGTM
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1749
@jackylk How the external table feature works for enabling carbon hive metastore option. Schema will not be stored in file if hive metastore is enabled so we cannot create external table from it.
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/1749
If the table data is generated without enabling carbon hive metastore option, the table data includes schema file, so this table folder can be used to create another external table.
If the table data is generated with enabling carbon hive metastore option, the table data does not include schema file. When creating external table using such table folder, we can generate the schema file by either deserializing the carbonSchema string in hive metastore or read the header of data file.
This is not implemented in this PR.
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1749
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2827/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1749
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2752/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1749
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1517/
---
[GitHub] carbondata pull request #1749: [CARBONDATA-1968] Add external table support
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1749#discussion_r161400186
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala ---
@@ -571,4 +572,22 @@ class CarbonFileMetastore extends CarbonMetaStore {
case _ => throw new NoSuchTableException(tableIdentifier.database.get, tableIdentifier.table)
}
}
+
+ override def getTableInfo(identifier: AbsoluteTableIdentifier)
--- End diff --
All ready API exists in SchemReader , better use from there. As this API is only for filebased schema
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1749
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2941/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1749
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2878/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1749
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1506/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1749
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1592/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1749
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2738/
---
[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1749
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2877/
---