You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hawq.apache.org by shivzone <gi...@git.apache.org> on 2016/04/05 00:42:51 UTC
[GitHub] incubator-hawq pull request: HAWQ-615. Handle incomptible tables w...
GitHub user shivzone opened a pull request:
https://github.com/apache/incubator-hawq/pull/551
HAWQ-615. Handle incomptible tables with getMetadata PXF API
Incompatible tables will be ignored when getMetadata is invoked with a pattern that matches > 1 table
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/incubator-hawq HAWQ-615
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hawq/pull/551.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #551
----
commit 676396fa0c40bb600aadda6f1dc7e8763a0bdde8
Author: Shivram Mani <sh...@gmail.com>
Date: 2016-04-04T22:41:08Z
HAWQ-615. Handle incomptible tables with getMetadata PXF API
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-615. Handle incomptible tables w...
Posted by shivzone <gi...@git.apache.org>.
Github user shivzone commented on the pull request:
https://github.com/apache/incubator-hawq/pull/551#issuecomment-210174733
@hornn the reason we went with returning an empty response was we wanted to be consistent with the way we treat the metadata api based on the given pattern.
If the pattern has wildcards in it, we will never throw an exception in any case based on table incompatibilities. If the pattern corresponds to a specific table, we will throw the appropriate exception.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-615. Handle incomptible tables w...
Posted by hornn <gi...@git.apache.org>.
Github user hornn commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/551#discussion_r58473430
--- Diff: pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveMetadataFetcher.java ---
@@ -62,11 +74,24 @@ public HiveMetadataFetcher(InputData md) {
List<Metadata> metadataList = new ArrayList<Metadata>();
+ if(tblsDesc.size() > 1) {
--- End diff --
What if there are 2 tables but both are not supported? in that case the function will return 0 tables. Is that an acceptable scenario? if so, perhaps add a test to cover it?
Did you consider changing the condition to have at least one 'good' table - check in the end of the loop if there metadataList is not empty. If it is, throw an exception.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-615. Handle incomptible tables w...
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/551#discussion_r58462813
--- Diff: pxf/pxf-hive/src/test/java/org/apache/hawq/pxf/plugins/hive/HiveMetadataFetcherTest.java ---
@@ -154,6 +155,115 @@ public void getTableMetadata() throws Exception {
assertEquals("int4", field.getType());
}
+ @Test
+ public void getTableMetadataWithMultipleTables() throws Exception {
+ prepareConstruction();
+
+ fetcher = new HiveMetadataFetcher(inputData);
+
+ String tablepattern = "*";
+ String dbpattern = "*";
+ String dbname = "default";
+ String tablenamebase = "regulartable";
+ String pattern = dbpattern + "." + tablepattern;
+
+ List<String> dbNames = new ArrayList<String>(Arrays.asList(dbname));
+ List<String> tableNames = new ArrayList<String>();
+
+ // Prepare for tables
+ List<FieldSchema> fields = new ArrayList<FieldSchema>();
+ fields.add(new FieldSchema("field1", "string", null));
+ fields.add(new FieldSchema("field2", "int", null));
+ StorageDescriptor sd = new StorageDescriptor();
+ sd.setCols(fields);
+
+ // Mock hive tables returned from hive client
+ for(int index=1;index<=2;index++) {
--- End diff --
Add comment why starting from 1 not from 0?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-615. Handle incomptible tables w...
Posted by hornn <gi...@git.apache.org>.
Github user hornn commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/551#discussion_r58473077
--- Diff: pxf/pxf-hive/src/test/java/org/apache/hawq/pxf/plugins/hive/HiveMetadataFetcherTest.java ---
@@ -154,6 +155,115 @@ public void getTableMetadata() throws Exception {
assertEquals("int4", field.getType());
}
+ @Test
+ public void getTableMetadataWithMultipleTables() throws Exception {
+ prepareConstruction();
+
+ fetcher = new HiveMetadataFetcher(inputData);
+
+ String tablepattern = "*";
+ String dbpattern = "*";
+ String dbname = "default";
+ String tablenamebase = "regulartable";
+ String pattern = dbpattern + "." + tablepattern;
+
+ List<String> dbNames = new ArrayList<String>(Arrays.asList(dbname));
+ List<String> tableNames = new ArrayList<String>();
+
+ // Prepare for tables
+ List<FieldSchema> fields = new ArrayList<FieldSchema>();
+ fields.add(new FieldSchema("field1", "string", null));
+ fields.add(new FieldSchema("field2", "int", null));
+ StorageDescriptor sd = new StorageDescriptor();
+ sd.setCols(fields);
+
+ // Mock hive tables returned from hive client
+ for(int index=1;index<=2;index++) {
+ String tableName = tablenamebase + index;
+ tableNames.add(tableName);;
--- End diff --
two ;
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-615. Handle incomptible tables w...
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on the pull request:
https://github.com/apache/incubator-hawq/pull/551#issuecomment-205532919
LGTM other than small cosmetic comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-615. Handle incomptible tables w...
Posted by shivzone <gi...@git.apache.org>.
Github user shivzone commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/551#discussion_r58466610
--- Diff: pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveMetadataFetcher.java ---
@@ -50,9 +50,21 @@ public HiveMetadataFetcher(InputData md) {
client = HiveUtilities.initHiveClient();
}
+ /**
+ * Fetches metadata of hive tables corresponding to the given pattern
+ * For patterns matching more than one table, the tables are skipped.
--- End diff --
will update
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-615. Handle incomptible tables w...
Posted by shivzone <gi...@git.apache.org>.
Github user shivzone closed the pull request at:
https://github.com/apache/incubator-hawq/pull/551
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-615. Handle incomptible tables w...
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/551#discussion_r58462587
--- Diff: pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveMetadataFetcher.java ---
@@ -50,9 +50,21 @@ public HiveMetadataFetcher(InputData md) {
client = HiveUtilities.initHiveClient();
}
+ /**
+ * Fetches metadata of hive tables corresponding to the given pattern
+ * For patterns matching more than one table, the tables are skipped.
--- End diff --
Maybe For patterns matching more than one table, the unsupported tables are skipped.?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---