[jira] [Comment Edited] (HIVE-20720) Add partition column option to JDBC handler

["Jesus Camacho Rodriguez (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-20720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649718#comment-16649718 ] 

Jesus Camacho Rodriguez edited comment on HIVE-20720 at 10/15/18 4:53 AM:
--------------------------------------------------------------------------

[~daijy], thanks! Minor comment (but important), can you use unique names for the derby databases and tables in the new q files? Before I have observed clashes with other q files which caused flaky failures in ptest runs (see HIVE-20706 for instance and the patch I submitted). I will review the rest of the patch shortly.


was (Author: jcamachorodriguez):
[~daijy], thanks! Minor comment (but important), can you use unique names for the derby databases and tables in the new q files? Before I have observed clashes with other q files which caused flaky failures in ptest runs (see HIVE-20706 for instance and the patch I submitted).

> Add partition column option to JDBC handler
> -------------------------------------------
>
>                 Key: HIVE-20720
>                 URL: https://issues.apache.org/jira/browse/HIVE-20720
>             Project: Hive
>          Issue Type: New Feature
>          Components: StorageHandler
>            Reporter: Daniel Dai
>            Assignee: Daniel Dai
>            Priority: Major
>         Attachments: HIVE-20720.1.patch, HIVE-20720.2.patch, HIVE-20720.3.patch, HIVE-20720.4.patch, HIVE-20720.5.patch
>
>
> Currently JdbcStorageHandler does not split input in Tez. The reason is numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" in Tez. And "mapreduce.job.maps" is not a valid param if authorizer(eg. SQLStdAuth) is in use. User ends up always use 1 split.
> We need to rely on this new feature if we want to support multi-splits. Here is my proposal:
> 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound in tblproperties if user want to split jdbc data source. In case lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min query to get this in planner. We can currently limit partitionColumn to only numeric/date/timestamp column for simplicity
> 2. If partitionColumn/numPartitions are not specified, don't split input
> 3. Splits are equal intervals without respect to data distribution
> 4. There is also a "hive.sql.query.split" flag vetos the split (can be set manually or automatically by calcite)
> 5. If partitionColumn is not defined, but numPartitions is defined, use original limit/offset logic (however, don't rely on numSplit).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)