You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by patcharee <Pa...@uni.no> on 2015/03/16 14:59:08 UTC
insert hive partitioned table
Hi,
I tried to insert into a hive partitioned table
val ZONE: Int = Integer.valueOf(args(2))
val MONTH: Int = Integer.valueOf(args(3))
val YEAR: Int = Integer.valueOf(args(4))
val weightedUVToDF = weightedUVToRecord.toDF()
weightedUVToDF.registerTempTable("speeddata")
hiveContext.sql("INSERT OVERWRITE table speed partition (year=" + YEAR +
",month=" + MONTH + ",zone=" + ZONE + ")
select key, speed, direction from speeddata")
First I registered a temporary table "speeddata". The value of the
partitioned column (year, month, zone) is from user input. If I would
like to get the value of the partitioned column from the temporary
table, how can I do that?
BR,
Patcharee
Re: insert hive partitioned table
Posted by Cheng Lian <li...@gmail.com>.
I see. Since all Spark SQL queries must be issued from the driver side,
you'll have to first collect all interested values to the driver side,
and then use them to compose one or more insert statements.
Cheng
On 3/16/15 10:33 PM, patcharee wrote:
> I would like to insert the table, and the value of the partition
> column to be inserted must be from temporary registered table/dataframe.
>
> Patcharee
>
>
> On 16. mars 2015 15:26, Cheng Lian wrote:
>>
>> Not quite sure whether I understand your question properly. But if
>> you just want to read the partition columns, it’s pretty easy. Take
>> the “year” column as an example, you may do this in HiveQL:
>>
>> |hiveContext.sql("SELECT year FROM speed")
>> |
>>
>> or in DataFrame DSL:
>>
>> |hiveContext.table("speed").select("year")
>> |
>>
>> Cheng
>>
>> On 3/16/15 9:59 PM, patcharee wrote:
>>
>>> Hi,
>>>
>>> I tried to insert into a hive partitioned table
>>>
>>> val ZONE: Int = Integer.valueOf(args(2))
>>> val MONTH: Int = Integer.valueOf(args(3))
>>> val YEAR: Int = Integer.valueOf(args(4))
>>>
>>> val weightedUVToDF = weightedUVToRecord.toDF()
>>> weightedUVToDF.registerTempTable("speeddata")
>>> hiveContext.sql("INSERT OVERWRITE table speed partition (year=" +
>>> YEAR + ",month=" + MONTH + ",zone=" + ZONE + ")
>>> select key, speed, direction from speeddata")
>>>
>>> First I registered a temporary table "speeddata". The value of the
>>> partitioned column (year, month, zone) is from user input. If I
>>> would like to get the value of the partitioned column from the
>>> temporary table, how can I do that?
>>>
>>> BR,
>>> Patcharee
>>
>
Re: insert hive partitioned table
Posted by patcharee <Pa...@uni.no>.
I would like to insert the table, and the value of the partition column
to be inserted must be from temporary registered table/dataframe.
Patcharee
On 16. mars 2015 15:26, Cheng Lian wrote:
>
> Not quite sure whether I understand your question properly. But if you
> just want to read the partition columns, it’s pretty easy. Take the
> “year” column as an example, you may do this in HiveQL:
>
> |hiveContext.sql("SELECT year FROM speed")
> |
>
> or in DataFrame DSL:
>
> |hiveContext.table("speed").select("year")
> |
>
> Cheng
>
> On 3/16/15 9:59 PM, patcharee wrote:
>
>> Hi,
>>
>> I tried to insert into a hive partitioned table
>>
>> val ZONE: Int = Integer.valueOf(args(2))
>> val MONTH: Int = Integer.valueOf(args(3))
>> val YEAR: Int = Integer.valueOf(args(4))
>>
>> val weightedUVToDF = weightedUVToRecord.toDF()
>> weightedUVToDF.registerTempTable("speeddata")
>> hiveContext.sql("INSERT OVERWRITE table speed partition (year=" +
>> YEAR + ",month=" + MONTH + ",zone=" + ZONE + ")
>> select key, speed, direction from speeddata")
>>
>> First I registered a temporary table "speeddata". The value of the
>> partitioned column (year, month, zone) is from user input. If I would
>> like to get the value of the partitioned column from the temporary
>> table, how can I do that?
>>
>> BR,
>> Patcharee
>
Re: insert hive partitioned table
Posted by Cheng Lian <li...@gmail.com>.
Not quite sure whether I understand your question properly. But if you
just want to read the partition columns, it’s pretty easy. Take the
“year” column as an example, you may do this in HiveQL:
|hiveContext.sql("SELECT year FROM speed")
|
or in DataFrame DSL:
|hiveContext.table("speed").select("year")
|
Cheng
On 3/16/15 9:59 PM, patcharee wrote:
> Hi,
>
> I tried to insert into a hive partitioned table
>
> val ZONE: Int = Integer.valueOf(args(2))
> val MONTH: Int = Integer.valueOf(args(3))
> val YEAR: Int = Integer.valueOf(args(4))
>
> val weightedUVToDF = weightedUVToRecord.toDF()
> weightedUVToDF.registerTempTable("speeddata")
> hiveContext.sql("INSERT OVERWRITE table speed partition (year=" + YEAR
> + ",month=" + MONTH + ",zone=" + ZONE + ")
> select key, speed, direction from speeddata")
>
> First I registered a temporary table "speeddata". The value of the
> partitioned column (year, month, zone) is from user input. If I would
> like to get the value of the partitioned column from the temporary
> table, how can I do that?
>
> BR,
> Patcharee