You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hudi.apache.org by on 2019/03/07 13:00:59 UTC
How i can pass Hoodie configurations to spark job
Hi All
I have found hoodie related configurations in http://hudi.apache.org/configurations.html. Please tell how we can pass these configurations to the spark job. Also please tell for file size related configs in which way i need to give the value for MB/GB/Bytes.
Thanks & Regards
Rahul P
Re: How i can pass Hoodie configurations to spark job
Posted by .
On 2019/03/08 19:25:25, Vinoth Chandar <vi...@apache.org> wrote:
> Hi Rahul,
>
> I have improved the docs around configs here
> https://github.com/apache/incubator-hudi/pull/10
>
> Please take a look and share thoughts..
> @others, please review as well
>
> Thanks
> Vinoth
>
> On Fri, Mar 8, 2019 at 10:09 AM Vinoth Chandar <vi...@apache.org> wrote:
>
> > Hi Rahul,
> >
> > You mentioned you are doing inserts alone correct? Then , there won't be
> > any log files and thus no compaction..
> > is this what you are seeing
> >
> > Thanks
> > Vinoth
> >
> > On Fri, Mar 8, 2019 at 2:15 AM rahuledavalath@gmail.com <
> > rahuledavalath@gmail.com> wrote:
> >
> >>
> >>
> >> On 2019/03/08 10:12:18, rahuledavalath@gmail.com <
> >> rahuledavalath@gmail.com> wrote:
> >> >
> >> >
> >> > On 2019/03/08 10:05:16, rahuledavalath@gmail.com <
> >> rahuledavalath@gmail.com> wrote:
> >> > >
> >> > >
> >> > > On 2019/03/08 09:29:20, "vbalaji@apache.org" <vb...@apache.org>
> >> wrote:
> >> > > >
> >> > > > This warn messages might be misleading in this context. Looks like
> >> it it coming from Kafka Source. At very high-level DeltaStreamer uses a
> >> single properties object to configure all things and this error could be
> >> coming from a component that do not understand other configs. Can you let
> >> the delta-streamer run to completion. Are you seeing errors/exception
> >> stopping the job ?
> >> > > > On Friday, March 8, 2019, 12:05:50 AM PST, <
> >> rahuledavalath@gmail.com> wrote:
> >> > > >
> >> > > >
> >> > > >
> >> > > > On 2019/03/07 18:35:54, Vinoth Chandar <vi...@apache.org> wrote:
> >> > > > > Hi Rahul,
> >> > > > >
> >> > > > > Can you please subscribe to the mailing list? Otherwise, each
> >> reply
> >> > > > > requires a moderator to approve before it can show up :)
> >> > > > >
> >> > > > > Thanks
> >> > > > > Vinoth
> >> > > > >
> >> > > > > On Thu, Mar 7, 2019 at 9:02 AM Balaji Varadarajan
> >> > > > > <v....@ymail.com.invalid> wrote:
> >> > > > >
> >> > > > > > It depends on which mechanism you use :
> >> > > > > > 1. For Spark DataSource route, you can use the "options" API
> >> > > > > > of DataFrameWriter to pass in these configs. Here is an example
> >> from
> >> > > > > > http://hudi.apache.org/incremental_processing.html
> >> > > > > > inputDF.write()
> >> > > > > > .format("com.uber.hoodie")
> >> > > > > > .options(clientOpts) // any of the Hudi client opts can
> >> be passed
> >> > > > > > in as well
> >> > > > > > .option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY(),
> >> > > > > > "_row_key")
> >> > > > > >
> >> .option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY(),
> >> > > > > > "partition")
> >> > > > > >
> >> .option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY(),
> >> > > > > > "timestamp")
> >> > > > > > .option(HoodieWriteConfig.TABLE_NAME, tableName)
> >> > > > > > .mode(SaveMode.Append)
> >> > > > > > .save(basePath);
> >> > > > > > 2. For an approach involving using HoodieWriteClient directly,
> >> you can
> >> > > > > > simply construct HoodieWriteConfig object with the configs in
> >> the link you
> >> > > > > > mentioned.
> >> > > > > > 3. When using HoodieDeltaStreamer tool to ingest, you can set
> >> the configs
> >> > > > > > in properties file and pass the file as the cmdline argument
> >> "--props"
> >> > > > > >
> >> > > > > > All the file-size configs must be in bytes denomination
> >> > > > > > Balaji.V
> >> > > > > >
> >> > > > > >
> >> > > > > > On Thursday, March 7, 2019, 7:00:55 AM PST,
> >> rahuledavalath@gmail.com <
> >> > > > > > rahuledavalath@gmail.com> wrote:
> >> > > > > >
> >> > > > > > Hi All
> >> > > > > > I have found hoodie related configurations in
> >> > > > > > http://hudi.apache.org/configurations.html. Please tell how
> >> we can pass
> >> > > > > > these configurations to the spark job. Also please tell for
> >> file size
> >> > > > > > related configs in which way i need to give the value for
> >> MB/GB/Bytes.
> >> > > > > >
> >> > > > > > Thanks & Regards
> >> > > > > > Rahul P
> >> > > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > I tried testing the properties with --props options. I added some
> >> of the properties in the kafka.source.properties file and passed with
> >> --props option. But the configuration are not valid. I think it is not for
> >> the deletastreamer. Can you please give proper properties for the
> >> deltastreamer. My intention is inline compaction.
> >> > > >
> >> > > > WARN VerifiableProperties: Property OPERATION_OPT_KEY is not valid
> >> > > > INFO VerifiableProperties: Property auto.offset.reset is overridden
> >> to smallest
> >> > > > WARN VerifiableProperties: Property compactionSmallFileSize is not
> >> valid
> >> > > > WARN VerifiableProperties: Property hoodie.compact.inline is not
> >> valid
> >> > > >
> >> > > >
> >> > > >
> >> > > > Thanks & Regards
> >> > > > Rahul P
> >> > > >
> >> > >
> >> > >
> >> > > It is not compacting, and nothing related to compaction i am
> >> observing in spark job logs. Can you please tell in which java class the
> >> "hoodie.compact.inline" option's value is taking.
> >> > >
> >> > >
> >> > >
> >> > >
> >> > FYI
> >> > I am using below command and checking whether it is compaction while
> >> injecting data throgh deltastreamer
> >> >
> >> > spark-submit --class
> >> com.uber.hoodie.utilities.deltastreamer.HoodieDeltaStreamer
> >> hoodie-utilities-0.4.5.jar --storage-type MERGE_ON_READ --source-class
> >> --source-ordering-field evemt_time --target-base-path /MOR10
> >> --target-table MOR10 --props /hudi/kafka-source.properties
> >> --schemaprovider-class
> >> com.uber.hoodie.utilities.schema.FilebasedSchemaProvider
> >> >
> >> ++Added missed source class argument
> >>
> >> I am using below command and checking whether it is compacting while
> >> injecting data throgh deltastreamer
> >>
> >> spark-submit --class
> >> com.uber.hoodie.utilities.deltastreamer.HoodieDeltaStreamer
> >> hoodie-utilities-0.4.5.jar --storage-type MERGE_ON_READ --source-class
> >> com.uber.hoodie.utilities.sources.JsonKafkaSource --source-ordering-field
> >> evemt_time --target-base-path /MOR10 --target-table MOR10 --props
> >> /hudi/kafka-source.properties --schemaprovider-class
> >> com.uber.hoodie.utilities.schema.FilebasedSchemaProvider
> >>
> >
> Dear Vinoth
Thanks for the improving configuration related docs. It's more clear now
Thanks & Regards
Rahul
Re: How i can pass Hoodie configurations to spark job
Posted by Vinoth Chandar <vi...@apache.org>.
Hi Rahul,
I have improved the docs around configs here
https://github.com/apache/incubator-hudi/pull/10
Please take a look and share thoughts..
@others, please review as well
Thanks
Vinoth
On Fri, Mar 8, 2019 at 10:09 AM Vinoth Chandar <vi...@apache.org> wrote:
> Hi Rahul,
>
> You mentioned you are doing inserts alone correct? Then , there won't be
> any log files and thus no compaction..
> is this what you are seeing
>
> Thanks
> Vinoth
>
> On Fri, Mar 8, 2019 at 2:15 AM rahuledavalath@gmail.com <
> rahuledavalath@gmail.com> wrote:
>
>>
>>
>> On 2019/03/08 10:12:18, rahuledavalath@gmail.com <
>> rahuledavalath@gmail.com> wrote:
>> >
>> >
>> > On 2019/03/08 10:05:16, rahuledavalath@gmail.com <
>> rahuledavalath@gmail.com> wrote:
>> > >
>> > >
>> > > On 2019/03/08 09:29:20, "vbalaji@apache.org" <vb...@apache.org>
>> wrote:
>> > > >
>> > > > This warn messages might be misleading in this context. Looks like
>> it it coming from Kafka Source. At very high-level DeltaStreamer uses a
>> single properties object to configure all things and this error could be
>> coming from a component that do not understand other configs. Can you let
>> the delta-streamer run to completion. Are you seeing errors/exception
>> stopping the job ?
>> > > > On Friday, March 8, 2019, 12:05:50 AM PST, <
>> rahuledavalath@gmail.com> wrote:
>> > > >
>> > > >
>> > > >
>> > > > On 2019/03/07 18:35:54, Vinoth Chandar <vi...@apache.org> wrote:
>> > > > > Hi Rahul,
>> > > > >
>> > > > > Can you please subscribe to the mailing list? Otherwise, each
>> reply
>> > > > > requires a moderator to approve before it can show up :)
>> > > > >
>> > > > > Thanks
>> > > > > Vinoth
>> > > > >
>> > > > > On Thu, Mar 7, 2019 at 9:02 AM Balaji Varadarajan
>> > > > > <v....@ymail.com.invalid> wrote:
>> > > > >
>> > > > > > It depends on which mechanism you use :
>> > > > > > 1. For Spark DataSource route, you can use the "options" API
>> > > > > > of DataFrameWriter to pass in these configs. Here is an example
>> from
>> > > > > > http://hudi.apache.org/incremental_processing.html
>> > > > > > inputDF.write()
>> > > > > > .format("com.uber.hoodie")
>> > > > > > .options(clientOpts) // any of the Hudi client opts can
>> be passed
>> > > > > > in as well
>> > > > > > .option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY(),
>> > > > > > "_row_key")
>> > > > > >
>> .option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY(),
>> > > > > > "partition")
>> > > > > >
>> .option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY(),
>> > > > > > "timestamp")
>> > > > > > .option(HoodieWriteConfig.TABLE_NAME, tableName)
>> > > > > > .mode(SaveMode.Append)
>> > > > > > .save(basePath);
>> > > > > > 2. For an approach involving using HoodieWriteClient directly,
>> you can
>> > > > > > simply construct HoodieWriteConfig object with the configs in
>> the link you
>> > > > > > mentioned.
>> > > > > > 3. When using HoodieDeltaStreamer tool to ingest, you can set
>> the configs
>> > > > > > in properties file and pass the file as the cmdline argument
>> "--props"
>> > > > > >
>> > > > > > All the file-size configs must be in bytes denomination
>> > > > > > Balaji.V
>> > > > > >
>> > > > > >
>> > > > > > On Thursday, March 7, 2019, 7:00:55 AM PST,
>> rahuledavalath@gmail.com <
>> > > > > > rahuledavalath@gmail.com> wrote:
>> > > > > >
>> > > > > > Hi All
>> > > > > > I have found hoodie related configurations in
>> > > > > > http://hudi.apache.org/configurations.html. Please tell how
>> we can pass
>> > > > > > these configurations to the spark job. Also please tell for
>> file size
>> > > > > > related configs in which way i need to give the value for
>> MB/GB/Bytes.
>> > > > > >
>> > > > > > Thanks & Regards
>> > > > > > Rahul P
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > I tried testing the properties with --props options. I added some
>> of the properties in the kafka.source.properties file and passed with
>> --props option. But the configuration are not valid. I think it is not for
>> the deletastreamer. Can you please give proper properties for the
>> deltastreamer. My intention is inline compaction.
>> > > >
>> > > > WARN VerifiableProperties: Property OPERATION_OPT_KEY is not valid
>> > > > INFO VerifiableProperties: Property auto.offset.reset is overridden
>> to smallest
>> > > > WARN VerifiableProperties: Property compactionSmallFileSize is not
>> valid
>> > > > WARN VerifiableProperties: Property hoodie.compact.inline is not
>> valid
>> > > >
>> > > >
>> > > >
>> > > > Thanks & Regards
>> > > > Rahul P
>> > > >
>> > >
>> > >
>> > > It is not compacting, and nothing related to compaction i am
>> observing in spark job logs. Can you please tell in which java class the
>> "hoodie.compact.inline" option's value is taking.
>> > >
>> > >
>> > >
>> > >
>> > FYI
>> > I am using below command and checking whether it is compaction while
>> injecting data throgh deltastreamer
>> >
>> > spark-submit --class
>> com.uber.hoodie.utilities.deltastreamer.HoodieDeltaStreamer
>> hoodie-utilities-0.4.5.jar --storage-type MERGE_ON_READ --source-class
>> --source-ordering-field evemt_time --target-base-path /MOR10
>> --target-table MOR10 --props /hudi/kafka-source.properties
>> --schemaprovider-class
>> com.uber.hoodie.utilities.schema.FilebasedSchemaProvider
>> >
>> ++Added missed source class argument
>>
>> I am using below command and checking whether it is compacting while
>> injecting data throgh deltastreamer
>>
>> spark-submit --class
>> com.uber.hoodie.utilities.deltastreamer.HoodieDeltaStreamer
>> hoodie-utilities-0.4.5.jar --storage-type MERGE_ON_READ --source-class
>> com.uber.hoodie.utilities.sources.JsonKafkaSource --source-ordering-field
>> evemt_time --target-base-path /MOR10 --target-table MOR10 --props
>> /hudi/kafka-source.properties --schemaprovider-class
>> com.uber.hoodie.utilities.schema.FilebasedSchemaProvider
>>
>
Re: How i can pass Hoodie configurations to spark job
Posted by Vinoth Chandar <vi...@apache.org>.
Hi Rahul,
You mentioned you are doing inserts alone correct? Then , there won't be
any log files and thus no compaction..
is this what you are seeing
Thanks
Vinoth
On Fri, Mar 8, 2019 at 2:15 AM rahuledavalath@gmail.com <
rahuledavalath@gmail.com> wrote:
>
>
> On 2019/03/08 10:12:18, rahuledavalath@gmail.com <ra...@gmail.com>
> wrote:
> >
> >
> > On 2019/03/08 10:05:16, rahuledavalath@gmail.com <
> rahuledavalath@gmail.com> wrote:
> > >
> > >
> > > On 2019/03/08 09:29:20, "vbalaji@apache.org" <vb...@apache.org>
> wrote:
> > > >
> > > > This warn messages might be misleading in this context. Looks like
> it it coming from Kafka Source. At very high-level DeltaStreamer uses a
> single properties object to configure all things and this error could be
> coming from a component that do not understand other configs. Can you let
> the delta-streamer run to completion. Are you seeing errors/exception
> stopping the job ?
> > > > On Friday, March 8, 2019, 12:05:50 AM PST, <
> rahuledavalath@gmail.com> wrote:
> > > >
> > > >
> > > >
> > > > On 2019/03/07 18:35:54, Vinoth Chandar <vi...@apache.org> wrote:
> > > > > Hi Rahul,
> > > > >
> > > > > Can you please subscribe to the mailing list? Otherwise, each reply
> > > > > requires a moderator to approve before it can show up :)
> > > > >
> > > > > Thanks
> > > > > Vinoth
> > > > >
> > > > > On Thu, Mar 7, 2019 at 9:02 AM Balaji Varadarajan
> > > > > <v....@ymail.com.invalid> wrote:
> > > > >
> > > > > > It depends on which mechanism you use :
> > > > > > 1. For Spark DataSource route, you can use the "options" API
> > > > > > of DataFrameWriter to pass in these configs. Here is an example
> from
> > > > > > http://hudi.apache.org/incremental_processing.html
> > > > > > inputDF.write()
> > > > > > .format("com.uber.hoodie")
> > > > > > .options(clientOpts) // any of the Hudi client opts can
> be passed
> > > > > > in as well
> > > > > > .option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY(),
> > > > > > "_row_key")
> > > > > >
> .option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY(),
> > > > > > "partition")
> > > > > > .option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY(),
> > > > > > "timestamp")
> > > > > > .option(HoodieWriteConfig.TABLE_NAME, tableName)
> > > > > > .mode(SaveMode.Append)
> > > > > > .save(basePath);
> > > > > > 2. For an approach involving using HoodieWriteClient directly,
> you can
> > > > > > simply construct HoodieWriteConfig object with the configs in
> the link you
> > > > > > mentioned.
> > > > > > 3. When using HoodieDeltaStreamer tool to ingest, you can set
> the configs
> > > > > > in properties file and pass the file as the cmdline argument
> "--props"
> > > > > >
> > > > > > All the file-size configs must be in bytes denomination
> > > > > > Balaji.V
> > > > > >
> > > > > >
> > > > > > On Thursday, March 7, 2019, 7:00:55 AM PST,
> rahuledavalath@gmail.com <
> > > > > > rahuledavalath@gmail.com> wrote:
> > > > > >
> > > > > > Hi All
> > > > > > I have found hoodie related configurations in
> > > > > > http://hudi.apache.org/configurations.html. Please tell how we
> can pass
> > > > > > these configurations to the spark job. Also please tell for
> file size
> > > > > > related configs in which way i need to give the value for
> MB/GB/Bytes.
> > > > > >
> > > > > > Thanks & Regards
> > > > > > Rahul P
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > I tried testing the properties with --props options. I added some of
> the properties in the kafka.source.properties file and passed with --props
> option. But the configuration are not valid. I think it is not for the
> deletastreamer. Can you please give proper properties for the
> deltastreamer. My intention is inline compaction.
> > > >
> > > > WARN VerifiableProperties: Property OPERATION_OPT_KEY is not valid
> > > > INFO VerifiableProperties: Property auto.offset.reset is overridden
> to smallest
> > > > WARN VerifiableProperties: Property compactionSmallFileSize is not
> valid
> > > > WARN VerifiableProperties: Property hoodie.compact.inline is not
> valid
> > > >
> > > >
> > > >
> > > > Thanks & Regards
> > > > Rahul P
> > > >
> > >
> > >
> > > It is not compacting, and nothing related to compaction i am
> observing in spark job logs. Can you please tell in which java class the
> "hoodie.compact.inline" option's value is taking.
> > >
> > >
> > >
> > >
> > FYI
> > I am using below command and checking whether it is compaction while
> injecting data throgh deltastreamer
> >
> > spark-submit --class
> com.uber.hoodie.utilities.deltastreamer.HoodieDeltaStreamer
> hoodie-utilities-0.4.5.jar --storage-type MERGE_ON_READ --source-class
> --source-ordering-field evemt_time --target-base-path /MOR10
> --target-table MOR10 --props /hudi/kafka-source.properties
> --schemaprovider-class
> com.uber.hoodie.utilities.schema.FilebasedSchemaProvider
> >
> ++Added missed source class argument
>
> I am using below command and checking whether it is compacting while
> injecting data throgh deltastreamer
>
> spark-submit --class
> com.uber.hoodie.utilities.deltastreamer.HoodieDeltaStreamer
> hoodie-utilities-0.4.5.jar --storage-type MERGE_ON_READ --source-class
> com.uber.hoodie.utilities.sources.JsonKafkaSource --source-ordering-field
> evemt_time --target-base-path /MOR10 --target-table MOR10 --props
> /hudi/kafka-source.properties --schemaprovider-class
> com.uber.hoodie.utilities.schema.FilebasedSchemaProvider
>
Re: How i can pass Hoodie configurations to spark job
Posted by .
On 2019/03/08 10:12:18, rahuledavalath@gmail.com <ra...@gmail.com> wrote:
>
>
> On 2019/03/08 10:05:16, rahuledavalath@gmail.com <ra...@gmail.com> wrote:
> >
> >
> > On 2019/03/08 09:29:20, "vbalaji@apache.org" <vb...@apache.org> wrote:
> > >
> > > This warn messages might be misleading in this context. Looks like it it coming from Kafka Source. At very high-level DeltaStreamer uses a single properties object to configure all things and this error could be coming from a component that do not understand other configs. Can you let the delta-streamer run to completion. Are you seeing errors/exception stopping the job ?
> > > On Friday, March 8, 2019, 12:05:50 AM PST, <ra...@gmail.com> wrote:
> > >
> > >
> > >
> > > On 2019/03/07 18:35:54, Vinoth Chandar <vi...@apache.org> wrote:
> > > > Hi Rahul,
> > > >
> > > > Can you please subscribe to the mailing list? Otherwise, each reply
> > > > requires a moderator to approve before it can show up :)
> > > >
> > > > Thanks
> > > > Vinoth
> > > >
> > > > On Thu, Mar 7, 2019 at 9:02 AM Balaji Varadarajan
> > > > <v....@ymail.com.invalid> wrote:
> > > >
> > > > > It depends on which mechanism you use :
> > > > > 1. For Spark DataSource route, you can use the "options" API
> > > > > of DataFrameWriter to pass in these configs. Here is an example from
> > > > > http://hudi.apache.org/incremental_processing.html
> > > > > inputDF.write()
> > > > > .format("com.uber.hoodie")
> > > > > .options(clientOpts) // any of the Hudi client opts can be passed
> > > > > in as well
> > > > > .option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY(),
> > > > > "_row_key")
> > > > > .option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY(),
> > > > > "partition")
> > > > > .option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY(),
> > > > > "timestamp")
> > > > > .option(HoodieWriteConfig.TABLE_NAME, tableName)
> > > > > .mode(SaveMode.Append)
> > > > > .save(basePath);
> > > > > 2. For an approach involving using HoodieWriteClient directly, you can
> > > > > simply construct HoodieWriteConfig object with the configs in the link you
> > > > > mentioned.
> > > > > 3. When using HoodieDeltaStreamer tool to ingest, you can set the configs
> > > > > in properties file and pass the file as the cmdline argument "--props"
> > > > >
> > > > > All the file-size configs must be in bytes denomination
> > > > > Balaji.V
> > > > >
> > > > >
> > > > > On Thursday, March 7, 2019, 7:00:55 AM PST, rahuledavalath@gmail.com <
> > > > > rahuledavalath@gmail.com> wrote:
> > > > >
> > > > > Hi All
> > > > > I have found hoodie related configurations in
> > > > > http://hudi.apache.org/configurations.html. Please tell how we can pass
> > > > > these configurations to the spark job. Also please tell for file size
> > > > > related configs in which way i need to give the value for MB/GB/Bytes.
> > > > >
> > > > > Thanks & Regards
> > > > > Rahul P
> > > > >
> > > >
> > >
> > >
> > >
> > > I tried testing the properties with --props options. I added some of the properties in the kafka.source.properties file and passed with --props option. But the configuration are not valid. I think it is not for the deletastreamer. Can you please give proper properties for the deltastreamer. My intention is inline compaction.
> > >
> > > WARN VerifiableProperties: Property OPERATION_OPT_KEY is not valid
> > > INFO VerifiableProperties: Property auto.offset.reset is overridden to smallest
> > > WARN VerifiableProperties: Property compactionSmallFileSize is not valid
> > > WARN VerifiableProperties: Property hoodie.compact.inline is not valid
> > >
> > >
> > >
> > > Thanks & Regards
> > > Rahul P
> > >
> >
> >
> > It is not compacting, and nothing related to compaction i am observing in spark job logs. Can you please tell in which java class the "hoodie.compact.inline" option's value is taking.
> >
> >
> >
> >
> FYI
> I am using below command and checking whether it is compaction while injecting data throgh deltastreamer
>
> spark-submit --class com.uber.hoodie.utilities.deltastreamer.HoodieDeltaStreamer hoodie-utilities-0.4.5.jar --storage-type MERGE_ON_READ --source-class --source-ordering-field evemt_time --target-base-path /MOR10 --target-table MOR10 --props /hudi/kafka-source.properties --schemaprovider-class com.uber.hoodie.utilities.schema.FilebasedSchemaProvider
>
++Added missed source class argument
I am using below command and checking whether it is compacting while injecting data throgh deltastreamer
spark-submit --class com.uber.hoodie.utilities.deltastreamer.HoodieDeltaStreamer hoodie-utilities-0.4.5.jar --storage-type MERGE_ON_READ --source-class com.uber.hoodie.utilities.sources.JsonKafkaSource --source-ordering-field evemt_time --target-base-path /MOR10 --target-table MOR10 --props /hudi/kafka-source.properties --schemaprovider-class com.uber.hoodie.utilities.schema.FilebasedSchemaProvider
Re: How i can pass Hoodie configurations to spark job
Posted by .
On 2019/03/08 10:05:16, rahuledavalath@gmail.com <ra...@gmail.com> wrote:
>
>
> On 2019/03/08 09:29:20, "vbalaji@apache.org" <vb...@apache.org> wrote:
> >
> > This warn messages might be misleading in this context. Looks like it it coming from Kafka Source. At very high-level DeltaStreamer uses a single properties object to configure all things and this error could be coming from a component that do not understand other configs. Can you let the delta-streamer run to completion. Are you seeing errors/exception stopping the job ?
> > On Friday, March 8, 2019, 12:05:50 AM PST, <ra...@gmail.com> wrote:
> >
> >
> >
> > On 2019/03/07 18:35:54, Vinoth Chandar <vi...@apache.org> wrote:
> > > Hi Rahul,
> > >
> > > Can you please subscribe to the mailing list? Otherwise, each reply
> > > requires a moderator to approve before it can show up :)
> > >
> > > Thanks
> > > Vinoth
> > >
> > > On Thu, Mar 7, 2019 at 9:02 AM Balaji Varadarajan
> > > <v....@ymail.com.invalid> wrote:
> > >
> > > > It depends on which mechanism you use :
> > > > 1. For Spark DataSource route, you can use the "options" API
> > > > of DataFrameWriter to pass in these configs. Here is an example from
> > > > http://hudi.apache.org/incremental_processing.html
> > > > inputDF.write()
> > > > .format("com.uber.hoodie")
> > > > .options(clientOpts) // any of the Hudi client opts can be passed
> > > > in as well
> > > > .option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY(),
> > > > "_row_key")
> > > > .option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY(),
> > > > "partition")
> > > > .option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY(),
> > > > "timestamp")
> > > > .option(HoodieWriteConfig.TABLE_NAME, tableName)
> > > > .mode(SaveMode.Append)
> > > > .save(basePath);
> > > > 2. For an approach involving using HoodieWriteClient directly, you can
> > > > simply construct HoodieWriteConfig object with the configs in the link you
> > > > mentioned.
> > > > 3. When using HoodieDeltaStreamer tool to ingest, you can set the configs
> > > > in properties file and pass the file as the cmdline argument "--props"
> > > >
> > > > All the file-size configs must be in bytes denomination
> > > > Balaji.V
> > > >
> > > >
> > > > On Thursday, March 7, 2019, 7:00:55 AM PST, rahuledavalath@gmail.com <
> > > > rahuledavalath@gmail.com> wrote:
> > > >
> > > > Hi All
> > > > I have found hoodie related configurations in
> > > > http://hudi.apache.org/configurations.html. Please tell how we can pass
> > > > these configurations to the spark job. Also please tell for file size
> > > > related configs in which way i need to give the value for MB/GB/Bytes.
> > > >
> > > > Thanks & Regards
> > > > Rahul P
> > > >
> > >
> >
> >
> >
> > I tried testing the properties with --props options. I added some of the properties in the kafka.source.properties file and passed with --props option. But the configuration are not valid. I think it is not for the deletastreamer. Can you please give proper properties for the deltastreamer. My intention is inline compaction.
> >
> > WARN VerifiableProperties: Property OPERATION_OPT_KEY is not valid
> > INFO VerifiableProperties: Property auto.offset.reset is overridden to smallest
> > WARN VerifiableProperties: Property compactionSmallFileSize is not valid
> > WARN VerifiableProperties: Property hoodie.compact.inline is not valid
> >
> >
> >
> > Thanks & Regards
> > Rahul P
> >
>
>
> It is not compacting, and nothing related to compaction i am observing in spark job logs. Can you please tell in which java class the "hoodie.compact.inline" option's value is taking.
>
>
>
>
FYI
I am using below command and checking whether it is compaction while injecting data throgh deltastreamer
spark-submit --class com.uber.hoodie.utilities.deltastreamer.HoodieDeltaStreamer hoodie-utilities-0.4.5.jar --storage-type MERGE_ON_READ --source-class --source-ordering-field evemt_time --target-base-path /MOR10 --target-table MOR10 --props /hudi/kafka-source.properties --schemaprovider-class com.uber.hoodie.utilities.schema.FilebasedSchemaProvider
Re: How i can pass Hoodie configurations to spark job
Posted by .
On 2019/03/08 09:29:20, "vbalaji@apache.org" <vb...@apache.org> wrote:
>
> This warn messages might be misleading in this context. Looks like it it coming from Kafka Source. At very high-level DeltaStreamer uses a single properties object to configure all things and this error could be coming from a component that do not understand other configs. Can you let the delta-streamer run to completion. Are you seeing errors/exception stopping the job ?
> On Friday, March 8, 2019, 12:05:50 AM PST, <ra...@gmail.com> wrote:
>
>
>
> On 2019/03/07 18:35:54, Vinoth Chandar <vi...@apache.org> wrote:
> > Hi Rahul,
> >
> > Can you please subscribe to the mailing list? Otherwise, each reply
> > requires a moderator to approve before it can show up :)
> >
> > Thanks
> > Vinoth
> >
> > On Thu, Mar 7, 2019 at 9:02 AM Balaji Varadarajan
> > <v....@ymail.com.invalid> wrote:
> >
> > > It depends on which mechanism you use :
> > > 1. For Spark DataSource route, you can use the "options" API
> > > of DataFrameWriter to pass in these configs. Here is an example from
> > > http://hudi.apache.org/incremental_processing.html
> > > inputDF.write()
> > > .format("com.uber.hoodie")
> > > .options(clientOpts) // any of the Hudi client opts can be passed
> > > in as well
> > > .option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY(),
> > > "_row_key")
> > > .option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY(),
> > > "partition")
> > > .option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY(),
> > > "timestamp")
> > > .option(HoodieWriteConfig.TABLE_NAME, tableName)
> > > .mode(SaveMode.Append)
> > > .save(basePath);
> > > 2. For an approach involving using HoodieWriteClient directly, you can
> > > simply construct HoodieWriteConfig object with the configs in the link you
> > > mentioned.
> > > 3. When using HoodieDeltaStreamer tool to ingest, you can set the configs
> > > in properties file and pass the file as the cmdline argument "--props"
> > >
> > > All the file-size configs must be in bytes denomination
> > > Balaji.V
> > >
> > >
> > > On Thursday, March 7, 2019, 7:00:55 AM PST, rahuledavalath@gmail.com <
> > > rahuledavalath@gmail.com> wrote:
> > >
> > > Hi All
> > > I have found hoodie related configurations in
> > > http://hudi.apache.org/configurations.html. Please tell how we can pass
> > > these configurations to the spark job. Also please tell for file size
> > > related configs in which way i need to give the value for MB/GB/Bytes.
> > >
> > > Thanks & Regards
> > > Rahul P
> > >
> >
>
>
>
> I tried testing the properties with --props options. I added some of the properties in the kafka.source.properties file and passed with --props option. But the configuration are not valid. I think it is not for the deletastreamer. Can you please give proper properties for the deltastreamer. My intention is inline compaction.
>
> WARN VerifiableProperties: Property OPERATION_OPT_KEY is not valid
> INFO VerifiableProperties: Property auto.offset.reset is overridden to smallest
> WARN VerifiableProperties: Property compactionSmallFileSize is not valid
> WARN VerifiableProperties: Property hoodie.compact.inline is not valid
>
>
>
> Thanks & Regards
> Rahul P
>
It is not compacting, and nothing related to compaction i am observing in spark job logs. Can you please tell in which java class the "hoodie.compact.inline" option's value is taking.
Re: How i can pass Hoodie configurations to spark job
Posted by "vbalaji@apache.org" <vb...@apache.org>.
This warn messages might be misleading in this context. Looks like it it coming from Kafka Source. At very high-level DeltaStreamer uses a single properties object to configure all things and this error could be coming from a component that do not understand other configs. Can you let the delta-streamer run to completion. Are you seeing errors/exception stopping the job ?
On Friday, March 8, 2019, 12:05:50 AM PST, <ra...@gmail.com> wrote:
On 2019/03/07 18:35:54, Vinoth Chandar <vi...@apache.org> wrote:
> Hi Rahul,
>
> Can you please subscribe to the mailing list? Otherwise, each reply
> requires a moderator to approve before it can show up :)
>
> Thanks
> Vinoth
>
> On Thu, Mar 7, 2019 at 9:02 AM Balaji Varadarajan
> <v....@ymail.com.invalid> wrote:
>
> > It depends on which mechanism you use :
> > 1. For Spark DataSource route, you can use the "options" API
> > of DataFrameWriter to pass in these configs. Here is an example from
> > http://hudi.apache.org/incremental_processing.html
> > inputDF.write()
> > .format("com.uber.hoodie")
> > .options(clientOpts) // any of the Hudi client opts can be passed
> > in as well
> > .option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY(),
> > "_row_key")
> > .option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY(),
> > "partition")
> > .option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY(),
> > "timestamp")
> > .option(HoodieWriteConfig.TABLE_NAME, tableName)
> > .mode(SaveMode.Append)
> > .save(basePath);
> > 2. For an approach involving using HoodieWriteClient directly, you can
> > simply construct HoodieWriteConfig object with the configs in the link you
> > mentioned.
> > 3. When using HoodieDeltaStreamer tool to ingest, you can set the configs
> > in properties file and pass the file as the cmdline argument "--props"
> >
> > All the file-size configs must be in bytes denomination
> > Balaji.V
> >
> >
> > On Thursday, March 7, 2019, 7:00:55 AM PST, rahuledavalath@gmail.com <
> > rahuledavalath@gmail.com> wrote:
> >
> > Hi All
> > I have found hoodie related configurations in
> > http://hudi.apache.org/configurations.html. Please tell how we can pass
> > these configurations to the spark job. Also please tell for file size
> > related configs in which way i need to give the value for MB/GB/Bytes.
> >
> > Thanks & Regards
> > Rahul P
> >
>
I tried testing the properties with --props options. I added some of the properties in the kafka.source.properties file and passed with --props option. But the configuration are not valid. I think it is not for the deletastreamer. Can you please give proper properties for the deltastreamer. My intention is inline compaction.
WARN VerifiableProperties: Property OPERATION_OPT_KEY is not valid
INFO VerifiableProperties: Property auto.offset.reset is overridden to smallest
WARN VerifiableProperties: Property compactionSmallFileSize is not valid
WARN VerifiableProperties: Property hoodie.compact.inline is not valid
Thanks & Regards
Rahul P
Re: How i can pass Hoodie configurations to spark job
Posted by .
On 2019/03/07 18:35:54, Vinoth Chandar <vi...@apache.org> wrote:
> Hi Rahul,
>
> Can you please subscribe to the mailing list? Otherwise, each reply
> requires a moderator to approve before it can show up :)
>
> Thanks
> Vinoth
>
> On Thu, Mar 7, 2019 at 9:02 AM Balaji Varadarajan
> <v....@ymail.com.invalid> wrote:
>
> > It depends on which mechanism you use :
> > 1. For Spark DataSource route, you can use the "options" API
> > of DataFrameWriter to pass in these configs. Here is an example from
> > http://hudi.apache.org/incremental_processing.html
> > inputDF.write()
> > .format("com.uber.hoodie")
> > .options(clientOpts) // any of the Hudi client opts can be passed
> > in as well
> > .option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY(),
> > "_row_key")
> > .option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY(),
> > "partition")
> > .option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY(),
> > "timestamp")
> > .option(HoodieWriteConfig.TABLE_NAME, tableName)
> > .mode(SaveMode.Append)
> > .save(basePath);
> > 2. For an approach involving using HoodieWriteClient directly, you can
> > simply construct HoodieWriteConfig object with the configs in the link you
> > mentioned.
> > 3. When using HoodieDeltaStreamer tool to ingest, you can set the configs
> > in properties file and pass the file as the cmdline argument "--props"
> >
> > All the file-size configs must be in bytes denomination
> > Balaji.V
> >
> >
> > On Thursday, March 7, 2019, 7:00:55 AM PST, rahuledavalath@gmail.com <
> > rahuledavalath@gmail.com> wrote:
> >
> > Hi All
> > I have found hoodie related configurations in
> > http://hudi.apache.org/configurations.html. Please tell how we can pass
> > these configurations to the spark job. Also please tell for file size
> > related configs in which way i need to give the value for MB/GB/Bytes.
> >
> > Thanks & Regards
> > Rahul P
> >
>
I tried testing the properties with --props options. I added some of the properties in the kafka.source.properties file and passed with --props option. But the configuration are not valid. I think it is not for the deletastreamer. Can you please give proper properties for the deltastreamer. My intention is inline compaction.
WARN VerifiableProperties: Property OPERATION_OPT_KEY is not valid
INFO VerifiableProperties: Property auto.offset.reset is overridden to smallest
WARN VerifiableProperties: Property compactionSmallFileSize is not valid
WARN VerifiableProperties: Property hoodie.compact.inline is not valid
Thanks & Regards
Rahul P
Re: How i can pass Hoodie configurations to spark job
Posted by Vinoth Chandar <vi...@apache.org>.
Hi Rahul,
Can you please subscribe to the mailing list? Otherwise, each reply
requires a moderator to approve before it can show up :)
Thanks
Vinoth
On Thu, Mar 7, 2019 at 9:02 AM Balaji Varadarajan
<v....@ymail.com.invalid> wrote:
> It depends on which mechanism you use :
> 1. For Spark DataSource route, you can use the "options" API
> of DataFrameWriter to pass in these configs. Here is an example from
> http://hudi.apache.org/incremental_processing.html
> inputDF.write()
> .format("com.uber.hoodie")
> .options(clientOpts) // any of the Hudi client opts can be passed
> in as well
> .option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY(),
> "_row_key")
> .option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY(),
> "partition")
> .option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY(),
> "timestamp")
> .option(HoodieWriteConfig.TABLE_NAME, tableName)
> .mode(SaveMode.Append)
> .save(basePath);
> 2. For an approach involving using HoodieWriteClient directly, you can
> simply construct HoodieWriteConfig object with the configs in the link you
> mentioned.
> 3. When using HoodieDeltaStreamer tool to ingest, you can set the configs
> in properties file and pass the file as the cmdline argument "--props"
>
> All the file-size configs must be in bytes denomination
> Balaji.V
>
>
> On Thursday, March 7, 2019, 7:00:55 AM PST, rahuledavalath@gmail.com <
> rahuledavalath@gmail.com> wrote:
>
> Hi All
> I have found hoodie related configurations in
> http://hudi.apache.org/configurations.html. Please tell how we can pass
> these configurations to the spark job. Also please tell for file size
> related configs in which way i need to give the value for MB/GB/Bytes.
>
> Thanks & Regards
> Rahul P
>
Re: How i can pass Hoodie configurations to spark job
Posted by Balaji Varadarajan <v....@ymail.com.INVALID>.
It depends on which mechanism you use :
1. For Spark DataSource route, you can use the "options" API of DataFrameWriter to pass in these configs. Here is an example from http://hudi.apache.org/incremental_processing.html
inputDF.write()
.format("com.uber.hoodie")
.options(clientOpts) // any of the Hudi client opts can be passed in as well
.option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY(), "_row_key")
.option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY(), "partition")
.option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY(), "timestamp")
.option(HoodieWriteConfig.TABLE_NAME, tableName)
.mode(SaveMode.Append)
.save(basePath);
2. For an approach involving using HoodieWriteClient directly, you can simply construct HoodieWriteConfig object with the configs in the link you mentioned.
3. When using HoodieDeltaStreamer tool to ingest, you can set the configs in properties file and pass the file as the cmdline argument "--props"
All the file-size configs must be in bytes denomination
Balaji.V
On Thursday, March 7, 2019, 7:00:55 AM PST, rahuledavalath@gmail.com <ra...@gmail.com> wrote:
Hi All
I have found hoodie related configurations in http://hudi.apache.org/configurations.html. Please tell how we can pass these configurations to the spark job. Also please tell for file size related configs in which way i need to give the value for MB/GB/Bytes.
Thanks & Regards
Rahul P