You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hudi.apache.org by Jack Wang <ja...@vungle.com.INVALID> on 2019/04/24 11:07:20 UTC
How to use HoodieDeltaStreamer for upsert on JsonDFSSource
Hi forks,
Doesn't anyone know how to use HoodieDeltaStreamer for upsert on
JsonDFSSource? Highly appreciated if you could provide a demo on that.
thanks and regards,
Jack
--
[image: vshapesaqua11553186012.gif] <https://vungle.com/> *Jianbin Wang*
Sr. Engineer II, Data
+86 18633600964
[image: in1552694272.png] <https://www.linkedin.com/company/vungle> [image:
fb1552694203.png] <https://facebook.com/vungle> [image:
tw1552694330.png] <https://twitter.com/vungle> [image:
ig1552694392.png] <https://www.instagram.com/vungle>
Units 3801, 3804, 38F, C Block, Beijing Yintai Center, Beijing, China
Re: How to use HoodieDeltaStreamer for upsert on JsonDFSSource
Posted by Jack Wang <ja...@vungle.com.INVALID>.
Thanks Balaji, this helps a lot!
On Thu, Apr 25, 2019 at 9:41 AM Balaji Varadarajan
<v....@ymail.com.invalid> wrote:
> Hi Jack,
> I have a hastily written notes for using demo with Json DFS source
> https://gist.github.com/bvaradar/6eba5c0fab7a0ab630206d64367d6cde
>
> Let me know if this works and provide suggestions/comments. We can see if
> this is can be incorporated to demo steps in future.
> Balaji.V
> On Wednesday, April 24, 2019, 6:24:31 PM PDT, Jack Wang
> <ja...@vungle.com.INVALID> wrote:
>
> Got you, thanks much for the detailed info.
>
> On Thu, Apr 25, 2019 at 9:17 AM Vinoth Chandar <vi...@apache.org> wrote:
>
> > ah right. Swapping it out with --source-class
> > com.uber.hoodie.utilities.sources.JsonDFSSource and specifying
> > the following config in the property file should work
> >
> > hoodie.deltastreamer.source.dfs.root=file:///path/to/json/root
> >
> > I see, it already uses the FileBasedSchemaProvider.
> >
> >
> > On Wed, Apr 24, 2019 at 6:11 PM Jack Wang <ja...@vungle.com.invalid>
> > wrote:
> >
> > > The docker demo by your link below uses JsonKafkaSource not the
> > > JsonDFSSource.
> > >
> > > On Thu, Apr 25, 2019 at 8:27 AM Vinoth Chandar <vi...@apache.org>
> > wrote:
> > >
> > > > The demo here https://hudi.apache.org/docker_demo.html actually
> > invokes
> > > > this path.. Is that helpful?
> > > >
> > > > Balaji, please correct me if I am wrong.
> > > >
> > > > Thanks
> > > > Vinoth
> > > >
> > > > On Wed, Apr 24, 2019 at 4:07 AM Jack Wang
> <jack.wang@vungle.com.invalid
> > >
> > > > wrote:
> > > >
> > > > > Hi forks,
> > > > >
> > > > > Doesn't anyone know how to use HoodieDeltaStreamer for upsert on
> > > > > JsonDFSSource? Highly appreciated if you could provide a demo on
> > that.
> > > > >
> > > > > thanks and regards,
> > > > > Jack
> > > > >
> > > > > --
> > > > > [image: vshapesaqua11553186012.gif] <https://vungle.com/>
> *Jianbin
> > > > Wang*
> > > > > Sr. Engineer II, Data
> > > > > +86 18633600964
> > > > >
> > > > > [image: in1552694272.png] <https://www.linkedin.com/company/vungle
> >
> > > > > [image:
> > > > > fb1552694203.png] <https://facebook.com/vungle> [image:
> > > > > tw1552694330.png] <https://twitter.com/vungle> [image:
> > > > > ig1552694392.png] <https://www.instagram.com/vungle>
> > > > > Units 3801, 3804, 38F, C Block, Beijing Yintai Center, Beijing,
> China
> > > > >
> > > >
> > >
> > >
> > > --
> > > [image: vshapesaqua11553186012.gif] <https://vungle.com/> *Jianbin
> > Wang*
> > > Sr. Engineer II, Data
> > > +86 18633600964
> > >
> > > [image: in1552694272.png] <https://www.linkedin.com/company/vungle>
> > > [image:
> > > fb1552694203.png] <https://facebook.com/vungle> [image:
> > > tw1552694330.png] <https://twitter.com/vungle> [image:
> > > ig1552694392.png] <https://www.instagram.com/vungle>
> > > Units 3801, 3804, 38F, C Block, Beijing Yintai Center, Beijing, China
> > >
> >
>
>
> --
> [image: vshapesaqua11553186012.gif] <https://vungle.com/> *Jianbin Wang*
> Sr. Engineer II, Data
> +86 18633600964
>
> [image: in1552694272.png] <https://www.linkedin.com/company/vungle>
> [image:
> fb1552694203.png] <https://facebook.com/vungle> [image:
> tw1552694330.png] <https://twitter.com/vungle> [image:
> ig1552694392.png] <https://www.instagram.com/vungle>
> Units 3801, 3804, 38F, C Block, Beijing Yintai Center, Beijing, China
>
--
[image: vshapesaqua11553186012.gif] <https://vungle.com/> *Jianbin Wang*
Sr. Engineer II, Data
+86 18633600964
[image: in1552694272.png] <https://www.linkedin.com/company/vungle> [image:
fb1552694203.png] <https://facebook.com/vungle> [image:
tw1552694330.png] <https://twitter.com/vungle> [image:
ig1552694392.png] <https://www.instagram.com/vungle>
Units 3801, 3804, 38F, C Block, Beijing Yintai Center, Beijing, China
Re: How to use HoodieDeltaStreamer for upsert on JsonDFSSource
Posted by Balaji Varadarajan <v....@ymail.com.INVALID>.
Hi Jack,
I have a hastily written notes for using demo with Json DFS source
https://gist.github.com/bvaradar/6eba5c0fab7a0ab630206d64367d6cde
Let me know if this works and provide suggestions/comments. We can see if this is can be incorporated to demo steps in future.
Balaji.V
On Wednesday, April 24, 2019, 6:24:31 PM PDT, Jack Wang <ja...@vungle.com.INVALID> wrote:
Got you, thanks much for the detailed info.
On Thu, Apr 25, 2019 at 9:17 AM Vinoth Chandar <vi...@apache.org> wrote:
> ah right. Swapping it out with --source-class
> com.uber.hoodie.utilities.sources.JsonDFSSource and specifying
> the following config in the property file should work
>
> hoodie.deltastreamer.source.dfs.root=file:///path/to/json/root
>
> I see, it already uses the FileBasedSchemaProvider.
>
>
> On Wed, Apr 24, 2019 at 6:11 PM Jack Wang <ja...@vungle.com.invalid>
> wrote:
>
> > The docker demo by your link below uses JsonKafkaSource not the
> > JsonDFSSource.
> >
> > On Thu, Apr 25, 2019 at 8:27 AM Vinoth Chandar <vi...@apache.org>
> wrote:
> >
> > > The demo here https://hudi.apache.org/docker_demo.html actually
> invokes
> > > this path.. Is that helpful?
> > >
> > > Balaji, please correct me if I am wrong.
> > >
> > > Thanks
> > > Vinoth
> > >
> > > On Wed, Apr 24, 2019 at 4:07 AM Jack Wang <jack.wang@vungle.com.invalid
> >
> > > wrote:
> > >
> > > > Hi forks,
> > > >
> > > > Doesn't anyone know how to use HoodieDeltaStreamer for upsert on
> > > > JsonDFSSource? Highly appreciated if you could provide a demo on
> that.
> > > >
> > > > thanks and regards,
> > > > Jack
> > > >
> > > > --
> > > > [image: vshapesaqua11553186012.gif] <https://vungle.com/> *Jianbin
> > > Wang*
> > > > Sr. Engineer II, Data
> > > > +86 18633600964
> > > >
> > > > [image: in1552694272.png] <https://www.linkedin.com/company/vungle>
> > > > [image:
> > > > fb1552694203.png] <https://facebook.com/vungle> [image:
> > > > tw1552694330.png] <https://twitter.com/vungle> [image:
> > > > ig1552694392.png] <https://www.instagram.com/vungle>
> > > > Units 3801, 3804, 38F, C Block, Beijing Yintai Center, Beijing, China
> > > >
> > >
> >
> >
> > --
> > [image: vshapesaqua11553186012.gif] <https://vungle.com/> *Jianbin
> Wang*
> > Sr. Engineer II, Data
> > +86 18633600964
> >
> > [image: in1552694272.png] <https://www.linkedin.com/company/vungle>
> > [image:
> > fb1552694203.png] <https://facebook.com/vungle> [image:
> > tw1552694330.png] <https://twitter.com/vungle> [image:
> > ig1552694392.png] <https://www.instagram.com/vungle>
> > Units 3801, 3804, 38F, C Block, Beijing Yintai Center, Beijing, China
> >
>
--
[image: vshapesaqua11553186012.gif] <https://vungle.com/> *Jianbin Wang*
Sr. Engineer II, Data
+86 18633600964
[image: in1552694272.png] <https://www.linkedin.com/company/vungle> [image:
fb1552694203.png] <https://facebook.com/vungle> [image:
tw1552694330.png] <https://twitter.com/vungle> [image:
ig1552694392.png] <https://www.instagram.com/vungle>
Units 3801, 3804, 38F, C Block, Beijing Yintai Center, Beijing, China
Re: How to use HoodieDeltaStreamer for upsert on JsonDFSSource
Posted by Jack Wang <ja...@vungle.com.INVALID>.
Got you, thanks much for the detailed info.
On Thu, Apr 25, 2019 at 9:17 AM Vinoth Chandar <vi...@apache.org> wrote:
> ah right. Swapping it out with --source-class
> com.uber.hoodie.utilities.sources.JsonDFSSource and specifying
> the following config in the property file should work
>
> hoodie.deltastreamer.source.dfs.root=file:///path/to/json/root
>
> I see, it already uses the FileBasedSchemaProvider.
>
>
> On Wed, Apr 24, 2019 at 6:11 PM Jack Wang <ja...@vungle.com.invalid>
> wrote:
>
> > The docker demo by your link below uses JsonKafkaSource not the
> > JsonDFSSource.
> >
> > On Thu, Apr 25, 2019 at 8:27 AM Vinoth Chandar <vi...@apache.org>
> wrote:
> >
> > > The demo here https://hudi.apache.org/docker_demo.html actually
> invokes
> > > this path.. Is that helpful?
> > >
> > > Balaji, please correct me if I am wrong.
> > >
> > > Thanks
> > > Vinoth
> > >
> > > On Wed, Apr 24, 2019 at 4:07 AM Jack Wang <jack.wang@vungle.com.invalid
> >
> > > wrote:
> > >
> > > > Hi forks,
> > > >
> > > > Doesn't anyone know how to use HoodieDeltaStreamer for upsert on
> > > > JsonDFSSource? Highly appreciated if you could provide a demo on
> that.
> > > >
> > > > thanks and regards,
> > > > Jack
> > > >
> > > > --
> > > > [image: vshapesaqua11553186012.gif] <https://vungle.com/> *Jianbin
> > > Wang*
> > > > Sr. Engineer II, Data
> > > > +86 18633600964
> > > >
> > > > [image: in1552694272.png] <https://www.linkedin.com/company/vungle>
> > > > [image:
> > > > fb1552694203.png] <https://facebook.com/vungle> [image:
> > > > tw1552694330.png] <https://twitter.com/vungle> [image:
> > > > ig1552694392.png] <https://www.instagram.com/vungle>
> > > > Units 3801, 3804, 38F, C Block, Beijing Yintai Center, Beijing, China
> > > >
> > >
> >
> >
> > --
> > [image: vshapesaqua11553186012.gif] <https://vungle.com/> *Jianbin
> Wang*
> > Sr. Engineer II, Data
> > +86 18633600964
> >
> > [image: in1552694272.png] <https://www.linkedin.com/company/vungle>
> > [image:
> > fb1552694203.png] <https://facebook.com/vungle> [image:
> > tw1552694330.png] <https://twitter.com/vungle> [image:
> > ig1552694392.png] <https://www.instagram.com/vungle>
> > Units 3801, 3804, 38F, C Block, Beijing Yintai Center, Beijing, China
> >
>
--
[image: vshapesaqua11553186012.gif] <https://vungle.com/> *Jianbin Wang*
Sr. Engineer II, Data
+86 18633600964
[image: in1552694272.png] <https://www.linkedin.com/company/vungle> [image:
fb1552694203.png] <https://facebook.com/vungle> [image:
tw1552694330.png] <https://twitter.com/vungle> [image:
ig1552694392.png] <https://www.instagram.com/vungle>
Units 3801, 3804, 38F, C Block, Beijing Yintai Center, Beijing, China
Re: How to use HoodieDeltaStreamer for upsert on JsonDFSSource
Posted by Vinoth Chandar <vi...@apache.org>.
ah right. Swapping it out with --source-class
com.uber.hoodie.utilities.sources.JsonDFSSource and specifying
the following config in the property file should work
hoodie.deltastreamer.source.dfs.root=file:///path/to/json/root
I see, it already uses the FileBasedSchemaProvider.
On Wed, Apr 24, 2019 at 6:11 PM Jack Wang <ja...@vungle.com.invalid>
wrote:
> The docker demo by your link below uses JsonKafkaSource not the
> JsonDFSSource.
>
> On Thu, Apr 25, 2019 at 8:27 AM Vinoth Chandar <vi...@apache.org> wrote:
>
> > The demo here https://hudi.apache.org/docker_demo.html actually invokes
> > this path.. Is that helpful?
> >
> > Balaji, please correct me if I am wrong.
> >
> > Thanks
> > Vinoth
> >
> > On Wed, Apr 24, 2019 at 4:07 AM Jack Wang <ja...@vungle.com.invalid>
> > wrote:
> >
> > > Hi forks,
> > >
> > > Doesn't anyone know how to use HoodieDeltaStreamer for upsert on
> > > JsonDFSSource? Highly appreciated if you could provide a demo on that.
> > >
> > > thanks and regards,
> > > Jack
> > >
> > > --
> > > [image: vshapesaqua11553186012.gif] <https://vungle.com/> *Jianbin
> > Wang*
> > > Sr. Engineer II, Data
> > > +86 18633600964
> > >
> > > [image: in1552694272.png] <https://www.linkedin.com/company/vungle>
> > > [image:
> > > fb1552694203.png] <https://facebook.com/vungle> [image:
> > > tw1552694330.png] <https://twitter.com/vungle> [image:
> > > ig1552694392.png] <https://www.instagram.com/vungle>
> > > Units 3801, 3804, 38F, C Block, Beijing Yintai Center, Beijing, China
> > >
> >
>
>
> --
> [image: vshapesaqua11553186012.gif] <https://vungle.com/> *Jianbin Wang*
> Sr. Engineer II, Data
> +86 18633600964
>
> [image: in1552694272.png] <https://www.linkedin.com/company/vungle>
> [image:
> fb1552694203.png] <https://facebook.com/vungle> [image:
> tw1552694330.png] <https://twitter.com/vungle> [image:
> ig1552694392.png] <https://www.instagram.com/vungle>
> Units 3801, 3804, 38F, C Block, Beijing Yintai Center, Beijing, China
>
Re: How to use HoodieDeltaStreamer for upsert on JsonDFSSource
Posted by Jack Wang <ja...@vungle.com.INVALID>.
The docker demo by your link below uses JsonKafkaSource not the
JsonDFSSource.
On Thu, Apr 25, 2019 at 8:27 AM Vinoth Chandar <vi...@apache.org> wrote:
> The demo here https://hudi.apache.org/docker_demo.html actually invokes
> this path.. Is that helpful?
>
> Balaji, please correct me if I am wrong.
>
> Thanks
> Vinoth
>
> On Wed, Apr 24, 2019 at 4:07 AM Jack Wang <ja...@vungle.com.invalid>
> wrote:
>
> > Hi forks,
> >
> > Doesn't anyone know how to use HoodieDeltaStreamer for upsert on
> > JsonDFSSource? Highly appreciated if you could provide a demo on that.
> >
> > thanks and regards,
> > Jack
> >
> > --
> > [image: vshapesaqua11553186012.gif] <https://vungle.com/> *Jianbin
> Wang*
> > Sr. Engineer II, Data
> > +86 18633600964
> >
> > [image: in1552694272.png] <https://www.linkedin.com/company/vungle>
> > [image:
> > fb1552694203.png] <https://facebook.com/vungle> [image:
> > tw1552694330.png] <https://twitter.com/vungle> [image:
> > ig1552694392.png] <https://www.instagram.com/vungle>
> > Units 3801, 3804, 38F, C Block, Beijing Yintai Center, Beijing, China
> >
>
--
[image: vshapesaqua11553186012.gif] <https://vungle.com/> *Jianbin Wang*
Sr. Engineer II, Data
+86 18633600964
[image: in1552694272.png] <https://www.linkedin.com/company/vungle> [image:
fb1552694203.png] <https://facebook.com/vungle> [image:
tw1552694330.png] <https://twitter.com/vungle> [image:
ig1552694392.png] <https://www.instagram.com/vungle>
Units 3801, 3804, 38F, C Block, Beijing Yintai Center, Beijing, China
Re: How to use HoodieDeltaStreamer for upsert on JsonDFSSource
Posted by Vinoth Chandar <vi...@apache.org>.
The demo here https://hudi.apache.org/docker_demo.html actually invokes
this path.. Is that helpful?
Balaji, please correct me if I am wrong.
Thanks
Vinoth
On Wed, Apr 24, 2019 at 4:07 AM Jack Wang <ja...@vungle.com.invalid>
wrote:
> Hi forks,
>
> Doesn't anyone know how to use HoodieDeltaStreamer for upsert on
> JsonDFSSource? Highly appreciated if you could provide a demo on that.
>
> thanks and regards,
> Jack
>
> --
> [image: vshapesaqua11553186012.gif] <https://vungle.com/> *Jianbin Wang*
> Sr. Engineer II, Data
> +86 18633600964
>
> [image: in1552694272.png] <https://www.linkedin.com/company/vungle>
> [image:
> fb1552694203.png] <https://facebook.com/vungle> [image:
> tw1552694330.png] <https://twitter.com/vungle> [image:
> ig1552694392.png] <https://www.instagram.com/vungle>
> Units 3801, 3804, 38F, C Block, Beijing Yintai Center, Beijing, China
>