You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hudi.apache.org by Umesh Kacha <um...@gmail.com> on 2019/04/16 18:01:51 UTC

Can we bulk insert without using physical partition column/folder in hdfs?

Hi I am trying to bulk insert using the following code but I dont have
partitioned directory so I just skipped that configuration. It throws some
databricks Avro schema exception? Is string allowed as time key column?
Please guide.

df1.write
      .format("com.uber.hoodie")
      .option(DataSourceWriteOptions.STORAGE_TYPE_OPT_KEY,
HoodieTableType.COPY_ON_WRITE.name
<http://hoodietabletype.copy_on_write.name/>())
      .option(DataSourceWriteOptions.OPERATION_OPT_KEY,
DataSourceWriteOptions.BULK_INSERT_OPERATION_OPT_VAL) // insert
      .option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY, "customer_id")
      .option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY,
"review_date")
      .option(HoodieWriteConfig.TABLE_NAME, "hoodie_test_table")
      .mode(SaveMode.Overwrite)
      .save("/tmp/hoodie/test_hoodie")

Re: Can we bulk insert without using physical partition column/folder in hdfs?

Posted by Vinoth Chandar <vi...@apache.org>.

Hi Umesh,

Could you share the exact error? Also have you tried using
com.uber.hoodie.NonpartitionedKeyGenerator.class as the key generator
class?

>> Is string allowed as time key column?
Sorry not fully following. Please clarify.

Thanks
Vinoth

On Tue, Apr 16, 2019 at 11:02 AM Umesh Kacha <um...@gmail.com> wrote:

> Hi I am trying to bulk insert using the following code but I dont have
> partitioned directory so I just skipped that configuration. It throws some
> databricks Avro schema exception? Is string allowed as time key column?
> Please guide.
>
> df1.write
>       .format("com.uber.hoodie")
>       .option(DataSourceWriteOptions.STORAGE_TYPE_OPT_KEY,
> HoodieTableType.COPY_ON_WRITE.name
> <http://hoodietabletype.copy_on_write.name/>())
>       .option(DataSourceWriteOptions.OPERATION_OPT_KEY,
> DataSourceWriteOptions.BULK_INSERT_OPERATION_OPT_VAL) // insert
>       .option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY,
> "customer_id")
>       .option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY,
> "review_date")
>       .option(HoodieWriteConfig.TABLE_NAME, "hoodie_test_table")
>       .mode(SaveMode.Overwrite)
>       .save("/tmp/hoodie/test_hoodie")
>