[jira] [Resolved] (KUDU-1820) Improve KuduContext with Range Partitioning

["Grant Henke (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KUDU-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Grant Henke resolved KUDU-1820.
-------------------------------
       Resolution: Duplicate
    Fix Version/s: n/a

> Improve KuduContext with Range Partitioning
> -------------------------------------------
>
>                 Key: KUDU-1820
>                 URL: https://issues.apache.org/jira/browse/KUDU-1820
>             Project: Kudu
>          Issue Type: Improvement
>          Components: spark
>    Affects Versions: 1.0.1
>            Reporter: Ryan Bosshart
>            Assignee: Cam Quoc Mach
>            Priority: Minor
>             Fix For: n/a
>
>
> With hash partitions, I use the KuduContext like so: 
> kuduContext.createTable(modifiedTable, predictions.schema, Seq("movieid","userid"),
>   new CreateTableOptions().addHashPartitions(ImmutableList.of("movieid"), 3).setNumReplicas(1))
> There isn't a clean way to use KuduContext with range partitions however. I have it working below, but CreateTableOptions and KuduContext both take a schema, but of different formats (one a list of ColumnSchema, the other a StructType).
> val fixSchema: Schema = {
>   val columns = ImmutableList.of(
>     new ColumnSchemaBuilder("clordid", Type.STRING).key(true).build(),
>     new ColumnSchemaBuilder("transacttime", Type.INT64).key(true).build(),
>       ....
>     new ColumnSchemaBuilder("lastupdated", Type.INT64).key(false).build())
>   new Schema(columns)
> }
> val schema =
>   StructType(
>     StructField("clordid", StringType, false) ::
>     StructField("transacttime", LongType, false) ::
>        ....
>     StructField("lastupdated", LongType, true) :: Nil)
> val kuduContext = new KuduContext(kuduMaster)
> val options = new CreateTableOptions()
>   .setRangePartitionColumns(ImmutableList.of("transacttime"))
>   .addHashPartitions(ImmutableList.of("clordid"), 3)
>   .setNumReplicas(1)
> val today = new DateTime().withTimeAtStartOfDay()
> val dayInMillis = TimeUnit.MILLISECONDS.convert(1, TimeUnit.DAYS)
> for (i <- 0 until numberOfDays ){
>   val lowerBound = fixSchema.newPartialRow()
>   val lbMillis = today.plusDays(i).getMillis
>   lowerBound.addLong("transacttime", lbMillis)
>   val upperBound = fixSchema.newPartialRow()
>   upperBound.addLong("transacttime", (lbMillis+dayInMillis-1))
>   options.addRangePartition(lowerBound, upperBound)
> }
> kuduContext.createTable(tableName, schema, Seq("clordid","transacttime"),options)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)