You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org> on 2016/07/05 19:20:06 UTC

[Impala-CR](cdh5-trunk) Kimpala (Preview) 

Bikramjeet Vig has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/3571

Change subject: Kimpala (Preview)
......................................................................

Kimpala (Preview)

Since this is just a preview and not the final patch, request reviewers to please
focus their comments more on the design aspect. This is to demonstrate that
Impala can be used with Kudu only and not rely on Hive Metastore at all.
It introduces a Metastore interface that has an implementation which talks
to both Hive and Kudu. A simple commandline argument (--kudu_only) can be
specified to switch between "Kudu Only, Non-HMS" and
"Full functionality"(How Implala currently works) mode.

To run Kudu-only mode run the impala cluster as follows:
$IMPALA_HOME/bin/start-impala-cluster.py --kudu_only

For detailed implementation on the design please see the design doc at:
https://docs.google.com/a/cloudera.com/document/d/1mjQm158AmfCluNg4Ik-vU3H_OKDNKOwgVPmxHEbLkXs/edit?usp=sharing

Change-Id: Idfa139e8393ed9f390ea8f3d1e352912f519953a
---
M be/src/catalog/catalog.cc
M bin/start-impala-cluster.py
M fe/src/main/java/com/cloudera/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java
M fe/src/main/java/com/cloudera/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/com/cloudera/impala/catalog/DataSourceTable.java
M fe/src/main/java/com/cloudera/impala/catalog/HBaseTable.java
M fe/src/main/java/com/cloudera/impala/catalog/HdfsTable.java
A fe/src/main/java/com/cloudera/impala/catalog/HiveAndKuduClient.java
M fe/src/main/java/com/cloudera/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/com/cloudera/impala/catalog/IncompleteTable.java
M fe/src/main/java/com/cloudera/impala/catalog/KuduTable.java
A fe/src/main/java/com/cloudera/impala/catalog/MetaStoreClientObject.java
M fe/src/main/java/com/cloudera/impala/catalog/MetaStoreClientPool.java
M fe/src/main/java/com/cloudera/impala/catalog/Table.java
M fe/src/main/java/com/cloudera/impala/catalog/TableLoader.java
M fe/src/main/java/com/cloudera/impala/catalog/View.java
M fe/src/main/java/com/cloudera/impala/service/CatalogOpExecutor.java
M fe/src/main/java/com/cloudera/impala/service/JniCatalog.java
M fe/src/main/java/com/cloudera/impala/service/KuduCatalogOpExecutor.java
M fe/src/main/java/com/cloudera/impala/util/KuduUtil.java
M fe/src/test/java/com/cloudera/impala/catalog/CatalogTest.java
M fe/src/test/java/com/cloudera/impala/testutil/CatalogServiceTestCatalog.java
M fe/src/test/java/com/cloudera/impala/testutil/ImpaladTestCatalog.java
24 files changed, 904 insertions(+), 272 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/71/3571/1
-- 
To view, visit http://gerrit.cloudera.org:8080/3571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Idfa139e8393ed9f390ea8f3d1e352912f519953a
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>

[Impala-CR](cdh5-trunk) Kimpala (Preview)

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Bikramjeet Vig has uploaded a new patch set (#3).

Change subject: Kimpala (Preview)
......................................................................

Kimpala (Preview)

Since this is just a preview and not the final patch, request reviewers to please
focus their comments more on the design aspect. This is to demonstrate that
Impala can be used with Kudu only and not rely on Hive Metastore at all.
It introduces a Metastore interface that has an implementation which talks
to both Hive and Kudu. A simple commandline argument (--kudu_only) can be
specified to switch between "Kudu Only, Non-HMS" and
"Full functionality"(How Implala currently works) mode.

To run Kudu-only mode run the impala cluster as follows:
$IMPALA_HOME/bin/start-impala-cluster.py --kudu_only

Or you can directly run the following command which will only start
Kudu service and run Impala in Kudu only mode:
$IMPALA_HOME/testdata/bin/run-kimpala.sh

To shut down the cluster use:
$IMPALA_HOME/testdata/bin/run-kimpala.sh -kill

For detailed implementation on the design please see the design doc at:
https://docs.google.com/a/cloudera.com/document/d/1mjQm158AmfCluNg4Ik-vU3H_OKDNKOwgVPmxHEbLkXs/edit?usp=sharing

Change-Id: Idfa139e8393ed9f390ea8f3d1e352912f519953a
---
M be/src/catalog/catalog.cc
M be/src/service/frontend.cc
M bin/start-impala-cluster.py
M fe/src/main/java/com/cloudera/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java
M fe/src/main/java/com/cloudera/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/com/cloudera/impala/catalog/DataSourceTable.java
M fe/src/main/java/com/cloudera/impala/catalog/HBaseTable.java
M fe/src/main/java/com/cloudera/impala/catalog/HdfsTable.java
A fe/src/main/java/com/cloudera/impala/catalog/HiveAndKuduClient.java
M fe/src/main/java/com/cloudera/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/com/cloudera/impala/catalog/IncompleteTable.java
M fe/src/main/java/com/cloudera/impala/catalog/KuduTable.java
A fe/src/main/java/com/cloudera/impala/catalog/MetaStoreClientObject.java
M fe/src/main/java/com/cloudera/impala/catalog/MetaStoreClientPool.java
M fe/src/main/java/com/cloudera/impala/catalog/Table.java
M fe/src/main/java/com/cloudera/impala/catalog/TableLoader.java
M fe/src/main/java/com/cloudera/impala/catalog/View.java
M fe/src/main/java/com/cloudera/impala/service/CatalogOpExecutor.java
M fe/src/main/java/com/cloudera/impala/service/Frontend.java
M fe/src/main/java/com/cloudera/impala/service/JniCatalog.java
M fe/src/main/java/com/cloudera/impala/service/JniFrontend.java
M fe/src/main/java/com/cloudera/impala/service/KuduCatalogOpExecutor.java
M fe/src/main/java/com/cloudera/impala/util/KuduUtil.java
M fe/src/test/java/com/cloudera/impala/catalog/CatalogTest.java
M fe/src/test/java/com/cloudera/impala/testutil/CatalogServiceTestCatalog.java
M fe/src/test/java/com/cloudera/impala/testutil/ImpaladTestCatalog.java
A testdata/bin/run-kimpala.sh
28 files changed, 962 insertions(+), 291 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/71/3571/3
-- 
To view, visit http://gerrit.cloudera.org:8080/3571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idfa139e8393ed9f390ea8f3d1e352912f519953a
Gerrit-PatchSet: 3
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>

[Impala-CR](cdh5-trunk) Kimpala (Preview)

Posted by "Dimitris Tsirogiannis (Code Review)" <ge...@cloudera.org>.
Dimitris Tsirogiannis has posted comments on this change.

Change subject: Kimpala (Preview)
......................................................................


Patch Set 4:

Your editor seems to be auto-formatting the code, causing many lines to be incorrectly marked as modified. That makes reviewing an already big patch even harder. Can you plz revert these changes?

-- 
To view, visit http://gerrit.cloudera.org:8080/3571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Idfa139e8393ed9f390ea8f3d1e352912f519953a
Gerrit-PatchSet: 4
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-HasComments: No

[Impala-CR](cdh5-trunk) Kimpala (Preview)

Posted by "Jim Apple (Code Review)" <ge...@cloudera.org>.
Jim Apple has abandoned this change.

Change subject: Kimpala (Preview)
......................................................................


Abandoned

-- 
To view, visit http://gerrit.cloudera.org:8080/3571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: abandon
Gerrit-Change-Id: Idfa139e8393ed9f390ea8f3d1e352912f519953a
Gerrit-PatchSet: 4
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@cloudera.com>

[Impala-CR](cdh5-trunk) Kimpala (Preview)

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Bikramjeet Vig has uploaded a new patch set (#4).

Change subject: Kimpala (Preview)
......................................................................

Kimpala (Preview)

Since this is just a preview and not the final patch, request reviewers to please
focus their comments more on the design aspect. This is to demonstrate that
Impala can be used with Kudu only and not rely on Hive Metastore at all.

It introduces a Metastore interface that has an implementation which talks
to both Hive and Kudu. It also adds functionality for persistent
table level column statistics in "kudu only" mode.

A simple commandline argument (--kudu_only) can be
specified to switch between "Kudu Only, Non-HMS" and
"Full functionality"(How Implala currently works) mode.
To run Kudu-only mode run the impala cluster as follows:
$IMPALA_HOME/bin/start-impala-cluster.py --kudu_only

Or you can directly run the following command which will only start
Kudu service and run Impala in Kudu only mode:
$IMPALA_HOME/testdata/bin/run-kimpala.sh

To shut down the cluster use:
$IMPALA_HOME/testdata/bin/run-kimpala.sh -kill

For detailed implementation on the design please see the design doc at:
https://docs.google.com/a/cloudera.com/document/d/1mjQm158AmfCluNg4Ik-vU3H_OKDNKOwgVPmxHEbLkXs/edit?usp=sharing

Change-Id: Idfa139e8393ed9f390ea8f3d1e352912f519953a
---
M be/src/catalog/catalog.cc
M be/src/service/frontend.cc
M bin/start-impala-cluster.py
M fe/src/main/java/com/cloudera/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java
M fe/src/main/java/com/cloudera/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/com/cloudera/impala/catalog/DataSourceTable.java
M fe/src/main/java/com/cloudera/impala/catalog/HBaseTable.java
M fe/src/main/java/com/cloudera/impala/catalog/HdfsTable.java
A fe/src/main/java/com/cloudera/impala/catalog/HiveAndKuduClient.java
M fe/src/main/java/com/cloudera/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/com/cloudera/impala/catalog/IncompleteTable.java
A fe/src/main/java/com/cloudera/impala/catalog/KuduColumnStatsTable.java
M fe/src/main/java/com/cloudera/impala/catalog/KuduTable.java
A fe/src/main/java/com/cloudera/impala/catalog/MetaStoreClientObject.java
M fe/src/main/java/com/cloudera/impala/catalog/MetaStoreClientPool.java
M fe/src/main/java/com/cloudera/impala/catalog/Table.java
M fe/src/main/java/com/cloudera/impala/catalog/TableLoader.java
M fe/src/main/java/com/cloudera/impala/catalog/View.java
M fe/src/main/java/com/cloudera/impala/service/CatalogOpExecutor.java
M fe/src/main/java/com/cloudera/impala/service/Frontend.java
M fe/src/main/java/com/cloudera/impala/service/JniCatalog.java
M fe/src/main/java/com/cloudera/impala/service/JniFrontend.java
M fe/src/main/java/com/cloudera/impala/service/KuduCatalogOpExecutor.java
M fe/src/main/java/com/cloudera/impala/util/KuduUtil.java
M fe/src/main/java/com/cloudera/impala/util/MetaStoreUtil.java
M fe/src/test/java/com/cloudera/impala/catalog/CatalogTest.java
M fe/src/test/java/com/cloudera/impala/testutil/CatalogServiceTestCatalog.java
M fe/src/test/java/com/cloudera/impala/testutil/ImpaladTestCatalog.java
A testdata/bin/run-kimpala.sh
30 files changed, 1,376 insertions(+), 297 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/71/3571/4
-- 
To view, visit http://gerrit.cloudera.org:8080/3571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idfa139e8393ed9f390ea8f3d1e352912f519953a
Gerrit-PatchSet: 4
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>

[Impala-CR](cdh5-trunk) Kimpala (Preview)

Posted by "Jim Apple (Code Review)" <ge...@cloudera.org>.
Jim Apple has posted comments on this change.

Change subject: Kimpala (Preview)
......................................................................


Patch Set 4:

Please update to using the new gerrit project, "Impala-ASF".
Instructions are here:

https://cwiki.apache.org/confluence/display/IMPALA/How+to+switch+to+Apache-hosted+git

Pushes to this project will be disabled on October 1.

-- 
To view, visit http://gerrit.cloudera.org:8080/3571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Idfa139e8393ed9f390ea8f3d1e352912f519953a
Gerrit-PatchSet: 4
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@cloudera.com>
Gerrit-HasComments: No

[Impala-CR](cdh5-trunk) Kimpala (Preview)

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Bikramjeet Vig has uploaded a new patch set (#2).

Change subject: Kimpala (Preview)
......................................................................

Kimpala (Preview)

Since this is just a preview and not the final patch, request reviewers to please
focus their comments more on the design aspect. This is to demonstrate that
Impala can be used with Kudu only and not rely on Hive Metastore at all.
It introduces a Metastore interface that has an implementation which talks
to both Hive and Kudu. A simple commandline argument (--kudu_only) can be
specified to switch between "Kudu Only, Non-HMS" and
"Full functionality"(How Implala currently works) mode.

To run Kudu-only mode run the impala cluster as follows:
$IMPALA_HOME/bin/start-impala-cluster.py --kudu_only

Or you can directly run the following command which will only start
Kudu service and run Impala in Kudu only mode:
$IMPALA_HOME/testdata/bin/run-kimpala.sh

To shut down the cluster use:
$IMPALA_HOME/testdata/bin/run-kimpala.sh -kill

For detailed implementation on the design please see the design doc at:
https://docs.google.com/a/cloudera.com/document/d/1mjQm158AmfCluNg4Ik-vU3H_OKDNKOwgVPmxHEbLkXs/edit?usp=sharing

Change-Id: Idfa139e8393ed9f390ea8f3d1e352912f519953a
---
M be/src/catalog/catalog.cc
M be/src/service/frontend.cc
M bin/start-impala-cluster.py
M fe/src/main/java/com/cloudera/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java
M fe/src/main/java/com/cloudera/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/com/cloudera/impala/catalog/DataSourceTable.java
M fe/src/main/java/com/cloudera/impala/catalog/HBaseTable.java
M fe/src/main/java/com/cloudera/impala/catalog/HdfsTable.java
A fe/src/main/java/com/cloudera/impala/catalog/HiveAndKuduClient.java
M fe/src/main/java/com/cloudera/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/com/cloudera/impala/catalog/IncompleteTable.java
M fe/src/main/java/com/cloudera/impala/catalog/KuduTable.java
A fe/src/main/java/com/cloudera/impala/catalog/MetaStoreClientObject.java
M fe/src/main/java/com/cloudera/impala/catalog/MetaStoreClientPool.java
M fe/src/main/java/com/cloudera/impala/catalog/Table.java
M fe/src/main/java/com/cloudera/impala/catalog/TableLoader.java
M fe/src/main/java/com/cloudera/impala/catalog/View.java
M fe/src/main/java/com/cloudera/impala/service/CatalogOpExecutor.java
M fe/src/main/java/com/cloudera/impala/service/JniCatalog.java
M fe/src/main/java/com/cloudera/impala/service/JniFrontend.java
M fe/src/main/java/com/cloudera/impala/service/KuduCatalogOpExecutor.java
M fe/src/main/java/com/cloudera/impala/util/KuduUtil.java
M fe/src/test/java/com/cloudera/impala/catalog/CatalogTest.java
M fe/src/test/java/com/cloudera/impala/testutil/CatalogServiceTestCatalog.java
M fe/src/test/java/com/cloudera/impala/testutil/ImpaladTestCatalog.java
A testdata/bin/run-kimpala.sh
27 files changed, 945 insertions(+), 279 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/71/3571/2
-- 
To view, visit http://gerrit.cloudera.org:8080/3571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idfa139e8393ed9f390ea8f3d1e352912f519953a
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>