You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Bijay Kumar Pathak <bk...@mtu.edu> on 2016/04/22 00:20:13 UTC

Spark SQL insert overwrite table not showing all the partition.

Hi,

I have a job which writes to the Hive table with dynamic partition. Inside
the job,  I am writing into the table two-time but I am only seeing the
partition with last write although I can see in the Spark UI it is
processing data fro both the partition.

Below is the query I am using to write to the table.

hive_c.sql("""INSERT OVERWRITE TABLE base_table PARTITION (date='{1}', date_2)
              SELECT * from temp_table
                """.format(date_val)
                 )



Thanks,
Bijay

Re: Spark SQL insert overwrite table not showing all the partition.

Posted by Bijay Kumar Pathak <bk...@mtu.edu>.

Hi Zhan,

I tried with IF NOT EXISTS clause and still I cannot see the first
partition only the partition with last insert overwrite is present in
the table.

Thanks,
Bijay

On Thu, Apr 21, 2016 at 11:18 PM, Zhan Zhang <zz...@hortonworks.com> wrote:

> INSERT OVERWRITE will overwrite any existing data in the table or partition
>
>    - unless IF NOT EXISTS is provided for a partition (as of Hive 0.9.0
>    <https://issues.apache.org/jira/browse/HIVE-2612>).
>
>
>
> Thanks.
>
> Zhan Zhang
>
> On Apr 21, 2016, at 3:20 PM, Bijay Kumar Pathak <bk...@mtu.edu> wrote:
>
> Hi,
>
> I have a job which writes to the Hive table with dynamic partition. Inside
> the job,  I am writing into the table two-time but I am only seeing the
> partition with last write although I can see in the Spark UI it is
> processing data fro both the partition.
>
> Below is the query I am using to write to the table.
>
> hive_c.sql("""INSERT OVERWRITE TABLE base_table PARTITION (date='{1}', date_2)
>               SELECT * from temp_table
>                 """.format(date_val)
>                  )
>
>
>
> Thanks,
> Bijay
>
>
>

Re: Spark SQL insert overwrite table not showing all the partition.

Posted by Zhan Zhang <zz...@hortonworks.com>.

INSERT OVERWRITE will overwrite any existing data in the table or partition

  *   unless IF NOT EXISTS is provided for a partition (as of Hive 0.9.0<https://issues.apache.org/jira/browse/HIVE-2612>).


Thanks.

Zhan Zhang

On Apr 21, 2016, at 3:20 PM, Bijay Kumar Pathak <bk...@mtu.edu>> wrote:

Hi,

I have a job which writes to the Hive table with dynamic partition. Inside the job,  I am writing into the table two-time but I am only seeing the partition with last write although I can see in the Spark UI it is processing data fro both the partition.

Below is the query I am using to write to the table.

hive_c.sql("""INSERT OVERWRITE TABLE base_table PARTITION (date='{1}', date_2)
              SELECT * from temp_table
                """.format(date_val)
                 )


Thanks,
Bijay