You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hudi.apache.org by "vbalaji@apache.org" <vb...@apache.org> on 2019/03/15 16:30:55 UTC
Re: DISCUSS HIP 1 Csv Source Support for Hudi DeltaStreamer
Hi Rahul,
Thanks for creating the HIP. I have reformatted your HIP just to standardize it.
https://docs.google.com/document/d/1bj-xpkRomVtbzvLb_4BRngDIGkkMR5yzxXRRzkA7QVo/edit?usp=sharing
Few points you need to think and elaborate in the HIP section:
1. You can mention how the schema of the csv file is going to be handled. We should probably use the current schemaProvider and use it decode csv data. 2. Using a config, make the source generic to support any configurable delimiters (e:g - tab instead of comma).3. Also, you can write about how to handle presence/absence of heading. I am assuming for kafka, this should not be a concern but not sure if there is a standard way of storing CSV files in data-lake. Do they include heading or not ?
Let us know if you have any thoughts/questions and we will be happy to help.
Also, If not already done, can you create a JIRA account and we can assign the ticket.
Balaji.V
On Friday, March 15, 2019, 7:34:18 AM PDT, <ra...@gmail.com> wrote:
On 2019/03/15 06:09:04, Umesh Kacha <um...@gmail.com> wrote:
> Hi Rahul I am happy to volunteer for this task in case you don't have
> bandwidth for the same. Please advice.
>
> Regards,
> Umesh.
>
> On Fri, Mar 15, 2019, 6:59 AM vbalaji@apache.org <vb...@apache.org> wrote:
>
> >
> > Hi Rahul,
> > We do not have any ready made csv support in deltastreamer yet. But it
> > should be simple to extend the DeltaStreamer by implementing a CSV Source.
> > Would you be interested in writing a HIP -
> > https://cwiki.apache.org/confluence/display/HUDI/Hudi+Improvement+Plan+Details+and+Process for
> > CSV support and implementing it ?
> > We will be very happy to assist you on this.
> > Thanks,Balaji.V
> >
> > On Thursday, March 14, 2019, 2:48:49 AM PDT, <ra...@gmail.com>
> > wrote:
> >
> > Dear Team
> >
> > I tested DeltaStreamer with JsonKafka,JsonDFS ..etc sources. If possible
> > please suggest how i can consume CSV data from kaka/HDFS and insert it into
> > hudi.
> >
> >
> > Thanks & Regards
> > Rahul
> >
>
Dear Balaji
I am initiating a HIP for Csv Source Support for Hudi DeltaStreamer.
Please find the HIP document in the below link.
https://docs.google.com/document/d/1bj-xpkRomVtbzvLb_4BRngDIGkkMR5yzxXRRzkA7QVo/edit?usp=sharing
I am new to this kind of open source project discussions. If there is any mistake in my HIP requested way please correct me.
@Umesh thanks you are always welcome.
Thanks & Regards
Rahul P
Re: DISCUSS HIP 1 Csv Source Support for Hudi DeltaStreamer
Posted by "vbalaji@apache.org" <vb...@apache.org>.
Hi Rahul,
I have added some comments in the HIP with ideas around implementation. Take a look and proceed with working on the implementation. Let us know in this thread if you need any help. Looking forward to your PR.
Balaji.V
On Monday, March 18, 2019, 3:44:21 AM PDT, <ra...@gmail.com> wrote:
On 2019/03/15 16:30:55, "vbalaji@apache.org" <vb...@apache.org> wrote:
>
> Hi Rahul,
> Thanks for creating the HIP. I have reformatted your HIP just to standardize it.
> https://docs.google.com/document/d/1bj-xpkRomVtbzvLb_4BRngDIGkkMR5yzxXRRzkA7QVo/edit?usp=sharing
>
> Few points you need to think and elaborate in the HIP section:
> 1. You can mention how the schema of the csv file is going to be handled. We should probably use the current schemaProvider and use it decode csv data. 2. Using a config, make the source generic to support any configurable delimiters (e:g - tab instead of comma).3. Also, you can write about how to handle presence/absence of heading. I am assuming for kafka, this should not be a concern but not sure if there is a standard way of storing CSV files in data-lake. Do they include heading or not ?
>
> Let us know if you have any thoughts/questions and we will be happy to help.
> Also, If not already done, can you create a JIRA account and we can assign the ticket.
> Balaji.V
> On Friday, March 15, 2019, 7:34:18 AM PDT, <ra...@gmail.com> wrote:
>
>
>
> On 2019/03/15 06:09:04, Umesh Kacha <um...@gmail.com> wrote:
> > Hi Rahul I am happy to volunteer for this task in case you don't have
> > bandwidth for the same. Please advice.
> >
> > Regards,
> > Umesh.
> >
> > On Fri, Mar 15, 2019, 6:59 AM vbalaji@apache.org <vb...@apache.org> wrote:
> >
> > >
> > > Hi Rahul,
> > > We do not have any ready made csv support in deltastreamer yet. But it
> > > should be simple to extend the DeltaStreamer by implementing a CSV Source.
> > > Would you be interested in writing a HIP -
> > > https://cwiki.apache.org/confluence/display/HUDI/Hudi+Improvement+Plan+Details+and+Process for
> > > CSV support and implementing it ?
> > > We will be very happy to assist you on this.
> > > Thanks,Balaji.V
> > >
> > > On Thursday, March 14, 2019, 2:48:49 AM PDT, <ra...@gmail.com>
> > > wrote:
> > >
> > > Dear Team
> > >
> > > I tested DeltaStreamer with JsonKafka,JsonDFS ..etc sources. If possible
> > > please suggest how i can consume CSV data from kaka/HDFS and insert it into
> > > hudi.
> > >
> > >
> > > Thanks & Regards
> > > Rahul
> > >
> >
> Dear Balaji
>
> I am initiating a HIP for Csv Source Support for Hudi DeltaStreamer.
> Please find the HIP document in the below link.
> https://docs.google.com/document/d/1bj-xpkRomVtbzvLb_4BRngDIGkkMR5yzxXRRzkA7QVo/edit?usp=sharing
>
> I am new to this kind of open source project discussions. If there is any mistake in my HIP requested way please correct me.
>
> @Umesh thanks you are always welcome.
>
> Thanks & Regards
> Rahul P
>
>
>
Dear Balaji
I have edited the HIP as per your suggestion. Please advise if any further modification is required.
Jira Id: rahuledavalath
Thanks & Regards
Rahul
Re: DISCUSS HIP 1 Csv Source Support for Hudi DeltaStreamer
Posted by .
On 2019/03/15 16:30:55, "vbalaji@apache.org" <vb...@apache.org> wrote:
>
> Hi Rahul,
> Thanks for creating the HIP. I have reformatted your HIP just to standardize it.
> https://docs.google.com/document/d/1bj-xpkRomVtbzvLb_4BRngDIGkkMR5yzxXRRzkA7QVo/edit?usp=sharing
>
> Few points you need to think and elaborate in the HIP section:
> 1. You can mention how the schema of the csv file is going to be handled. We should probably use the current schemaProvider and use it decode csv data. 2. Using a config, make the source generic to support any configurable delimiters (e:g - tab instead of comma).3. Also, you can write about how to handle presence/absence of heading. I am assuming for kafka, this should not be a concern but not sure if there is a standard way of storing CSV files in data-lake. Do they include heading or not ?
>
> Let us know if you have any thoughts/questions and we will be happy to help.
> Also, If not already done, can you create a JIRA account and we can assign the ticket.
> Balaji.V
> On Friday, March 15, 2019, 7:34:18 AM PDT, <ra...@gmail.com> wrote:
>
>
>
> On 2019/03/15 06:09:04, Umesh Kacha <um...@gmail.com> wrote:
> > Hi Rahul I am happy to volunteer for this task in case you don't have
> > bandwidth for the same. Please advice.
> >
> > Regards,
> > Umesh.
> >
> > On Fri, Mar 15, 2019, 6:59 AM vbalaji@apache.org <vb...@apache.org> wrote:
> >
> > >
> > > Hi Rahul,
> > > We do not have any ready made csv support in deltastreamer yet. But it
> > > should be simple to extend the DeltaStreamer by implementing a CSV Source.
> > > Would you be interested in writing a HIP -
> > > https://cwiki.apache.org/confluence/display/HUDI/Hudi+Improvement+Plan+Details+and+Process for
> > > CSV support and implementing it ?
> > > We will be very happy to assist you on this.
> > > Thanks,Balaji.V
> > >
> > > On Thursday, March 14, 2019, 2:48:49 AM PDT, <ra...@gmail.com>
> > > wrote:
> > >
> > > Dear Team
> > >
> > > I tested DeltaStreamer with JsonKafka,JsonDFS ..etc sources. If possible
> > > please suggest how i can consume CSV data from kaka/HDFS and insert it into
> > > hudi.
> > >
> > >
> > > Thanks & Regards
> > > Rahul
> > >
> >
> Dear Balaji
>
> I am initiating a HIP for Csv Source Support for Hudi DeltaStreamer.
> Please find the HIP document in the below link.
> https://docs.google.com/document/d/1bj-xpkRomVtbzvLb_4BRngDIGkkMR5yzxXRRzkA7QVo/edit?usp=sharing
>
> I am new to this kind of open source project discussions. If there is any mistake in my HIP requested way please correct me.
>
> @Umesh thanks you are always welcome.
>
> Thanks & Regards
> Rahul P
>
>
>
Dear Balaji
I have edited the HIP as per your suggestion. Please advise if any further modification is required.
Jira Id: rahuledavalath
Thanks & Regards
Rahul