You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hudi.apache.org by Kabeer Ahmed <ka...@linuxmail.org> on 2019/03/15 05:45:54 UTC
Timestamp
Hi,
https://github.com/apache/incubator-hudi/issues/547 (https://link.getmailspring.com/link/8516D513-FFCA-4F2F-B8BB-FE1387AE20AF@getmailspring.com/0?redirect=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-hudi%2Fissues%2F547&recipient=ZGV2QGh1ZGkuYXBhY2hlLm9yZw%3D%3D) has resulted in the jira https://issues.apache.org/jira/browse/HUDI-12 (https://link.getmailspring.com/link/8516D513-FFCA-4F2F-B8BB-FE1387AE20AF@getmailspring.com/1?redirect=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHUDI-12&recipient=ZGV2QGh1ZGkuYXBhY2hlLm9yZw%3D%3D).
The requirement is to be able to interpret timestamp from CSV and store it in the parquet table. Does anyone have a working example on these lines?
Going by the Hudi example from the GitHub:
Timestamp is being encoded in avro as double: https://github.com/apache/incubator-hudi/blob/master/hoodie-client/src/test/java/com/uber/hoodie/common/HoodieTestDataGenerator.java#L69 (https://link.getmailspring.com/link/8516D513-FFCA-4F2F-B8BB-FE1387AE20AF@getmailspring.com/2?redirect=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-hudi%2Fblob%2Fmaster%2Fhoodie-client%2Fsrc%2Ftest%2Fjava%2Fcom%2Fuber%2Fhoodie%2Fcommon%2FHoodieTestDataGenerator.java%23L69&recipient=ZGV2QGh1ZGkuYXBhY2hlLm9yZw%3D%3D)
The end result is that parquet field for timestamp is not of timestamp (INT96).
My best guess is that this would have been a requirement at Uber (tracking trips in minutes and seconds) and how is it being handled.
If anyone else has handled this and has an example that can be shared, it will be much appreciated.
Kabeer Ahmed, http://www.linkedin.com/in/kabeerahmed
Re: Timestamp
Posted by Kabeer Ahmed <ka...@linuxmail.org>.
Hi Vinoth
Thank you for your time. None of these 2 issues are blocking as of now. But priority should be given to S3 connections.
Further to our discussion, I shall focus on the strategy below:
Disable dynamic allocation and run the job and check if clean job is parallelized or running on a single executor.
Try to find the exact option on the lines of: "df.option("hoodie.clean.automatic", "false")" to disable cleaning job and see if that solves the issue.
I shall relay back my findings tomorrow.
Thanks for all the support,
Kabeer.
On Mar 26 2019, at 2:06 am, Vinoth Chandar <vi...@apache.org> wrote:
> Will give this a shot as well. Between this and the S3 thing, what's
> blocking progress? both? ;) ?
>
> On Sat, Mar 23, 2019 at 7:10 PM Kabeer Ahmed <ka...@linuxmail.org> wrote:
> > Hi Vinoth,
> > Thank you for looking into this. I am planning to try out this over this
> > weekend if it is possible. Just downloaded the 0.4.6 version of Hudi.
> > I think we can start with a very simple schema as below (copied from the
> > Hudi's own example)?
> >
> > val EXAMPLE_SCHEMA = "{\"type\": \"record\"," + "\"name\": \"hudirec\"," +
> > "\"fields\": [ " +
> > "{\"name\": \"timestamp\", \"type\": \"double\"}," +
> > "{\"name\": \"_row_key\", \"type\": \"string\"}," +
> > "{\"name\": \"trade_date\", \"type\": \"string\"}," +
> > "{\"name\": \"bats\", \"type\": \"int\"}]}";
> >
> > And sample data could be (cricket bats):
> > kabeer,2018-11-15T07:35:54.387Z,7
> > vinoth,2018-11-16T09:35:54.387Z,9
> > I did try passing several combinations in the timestamp field above to
> > indicate the logicalType to timestamp but no success. I was using Hive 1.1
> > compile time flag but I was not worried about reading data through Hive. I
> > could see in the generated parquet that the timestamp field was NOT INT96
> > timestamp format that the parquet expects.
> >
> > Keep me posted as to how you get along with this and I shall keep you
> > posted if I find any joy sooner than yourself.
> > Thanks
> > Kabeer.
> >
> > On Mar 23 2019, at 12:05 am, Vinoth Chandar <vi...@apache.org> wrote:
> > > Hi Kabeer,
> > >
> > > I spent time looking at the issue and its other linked issues as well.
> > > High level, seems like we need to change the data type mappings for these
> > > date/timestamp types..
> > > It does seem doable, given Avro also supports date/timestamp types..
> > >
> > > Do you have some sample schema/data generation that we can start with?
> > > Thanks
> > > Vinoth
> > >
> > > On Fri, Mar 15, 2019 at 11:19 AM Vinoth Chandar <vi...@apache.org>
> > wrote:
> > > > Hi Kabeer,
> > > > Thanks for bringing this up. I don't think we have actually hit this
> > > > before :)
> > > >
> > > > Let me spend sometime understanding the issue and get back to you
> > > > Thanks
> > > > Vinoth
> > > >
> > > > On Thu, Mar 14, 2019 at 10:46 PM Kabeer Ahmed <ka...@linuxmail.org>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > > https://github.com/apache/incubator-hudi/issues/547 (
> > > > >
> > https://link.getmailspring.com/link/8516D513-FFCA-4F2F-B8BB-FE1387AE20AF@getmailspring.com/0?redirect=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-hudi%2Fissues%2F547&recipient=ZGV2QGh1ZGkuYXBhY2hlLm9yZw%3D%3D
> > )
> > > > > has resulted in the jira
> > > >
> > >
> >
> > https://issues.apache.org/jira/browse/HUDI-12 (
> > > > >
> > https://link.getmailspring.com/link/8516D513-FFCA-4F2F-B8BB-FE1387AE20AF@getmailspring.com/1?redirect=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHUDI-12&recipient=ZGV2QGh1ZGkuYXBhY2hlLm9yZw%3D%3D
> > > > > ).
> > > > > The requirement is to be able to interpret timestamp from CSV and
> > > >
> > >
> >
> > store
> > > > > it in the parquet table. Does anyone have a working example on these
> > > >
> > >
> >
> > lines?
> > > > > Going by the Hudi example from the GitHub:
> > > > > Timestamp is being encoded in avro as double:
> > > > >
> > https://github.com/apache/incubator-hudi/blob/master/hoodie-client/src/test/java/com/uber/hoodie/common/HoodieTestDataGenerator.java#L69
> > > > > (
> > > > >
> > https://link.getmailspring.com/link/8516D513-FFCA-4F2F-B8BB-FE1387AE20AF@getmailspring.com/2?redirect=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-hudi%2Fblob%2Fmaster%2Fhoodie-client%2Fsrc%2Ftest%2Fjava%2Fcom%2Fuber%2Fhoodie%2Fcommon%2FHoodieTestDataGenerator.java%23L69&recipient=ZGV2QGh1ZGkuYXBhY2hlLm9yZw%3D%3D
> > > > > )
> > > > >
> > > > > The end result is that parquet field for timestamp is not of
> > timestamp
> > > > > (INT96).
> > > > >
> > > > > My best guess is that this would have been a requirement at Uber
> > > > > (tracking trips in minutes and seconds) and how is it being handled.
> > > > >
> > > > > If anyone else has handled this and has an example that can be
> > shared, it
> > > > > will be much appreciated.
> > > > > Kabeer Ahmed, http://www.linkedin.com/in/kabeerahmed
> > > >
> > >
> >
>
>
Re: Timestamp
Posted by Vinoth Chandar <vi...@apache.org>.
Will give this a shot as well. Between this and the S3 thing, what's
blocking progress? both? ;) ?
On Sat, Mar 23, 2019 at 7:10 PM Kabeer Ahmed <ka...@linuxmail.org> wrote:
> Hi Vinoth,
>
> Thank you for looking into this. I am planning to try out this over this
> weekend if it is possible. Just downloaded the 0.4.6 version of Hudi.
> I think we can start with a very simple schema as below (copied from the
> Hudi's own example)?
>
> val EXAMPLE_SCHEMA = "{\"type\": \"record\"," + "\"name\": \"hudirec\"," +
> "\"fields\": [ " +
> "{\"name\": \"timestamp\", \"type\": \"double\"}," +
> "{\"name\": \"_row_key\", \"type\": \"string\"}," +
> "{\"name\": \"trade_date\", \"type\": \"string\"}," +
> "{\"name\": \"bats\", \"type\": \"int\"}]}";
>
> And sample data could be (cricket bats):
> kabeer,2018-11-15T07:35:54.387Z,7
> vinoth,2018-11-16T09:35:54.387Z,9
> I did try passing several combinations in the timestamp field above to
> indicate the logicalType to timestamp but no success. I was using Hive 1.1
> compile time flag but I was not worried about reading data through Hive. I
> could see in the generated parquet that the timestamp field was NOT INT96
> timestamp format that the parquet expects.
>
> Keep me posted as to how you get along with this and I shall keep you
> posted if I find any joy sooner than yourself.
> Thanks
> Kabeer.
>
> On Mar 23 2019, at 12:05 am, Vinoth Chandar <vi...@apache.org> wrote:
> > Hi Kabeer,
> >
> > I spent time looking at the issue and its other linked issues as well.
> > High level, seems like we need to change the data type mappings for these
> > date/timestamp types..
> > It does seem doable, given Avro also supports date/timestamp types..
> >
> > Do you have some sample schema/data generation that we can start with?
> > Thanks
> > Vinoth
> >
> > On Fri, Mar 15, 2019 at 11:19 AM Vinoth Chandar <vi...@apache.org>
> wrote:
> > > Hi Kabeer,
> > > Thanks for bringing this up. I don't think we have actually hit this
> > > before :)
> > >
> > > Let me spend sometime understanding the issue and get back to you
> > > Thanks
> > > Vinoth
> > >
> > > On Thu, Mar 14, 2019 at 10:46 PM Kabeer Ahmed <ka...@linuxmail.org>
> > > wrote:
> > >
> > > > Hi,
> > > > https://github.com/apache/incubator-hudi/issues/547 (
> > > >
> https://link.getmailspring.com/link/8516D513-FFCA-4F2F-B8BB-FE1387AE20AF@getmailspring.com/0?redirect=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-hudi%2Fissues%2F547&recipient=ZGV2QGh1ZGkuYXBhY2hlLm9yZw%3D%3D
> )
> > > > has resulted in the jira
> https://issues.apache.org/jira/browse/HUDI-12 (
> > > >
> https://link.getmailspring.com/link/8516D513-FFCA-4F2F-B8BB-FE1387AE20AF@getmailspring.com/1?redirect=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHUDI-12&recipient=ZGV2QGh1ZGkuYXBhY2hlLm9yZw%3D%3D
> > > > ).
> > > > The requirement is to be able to interpret timestamp from CSV and
> store
> > > > it in the parquet table. Does anyone have a working example on these
> lines?
> > > > Going by the Hudi example from the GitHub:
> > > > Timestamp is being encoded in avro as double:
> > > >
> https://github.com/apache/incubator-hudi/blob/master/hoodie-client/src/test/java/com/uber/hoodie/common/HoodieTestDataGenerator.java#L69
> > > > (
> > > >
> https://link.getmailspring.com/link/8516D513-FFCA-4F2F-B8BB-FE1387AE20AF@getmailspring.com/2?redirect=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-hudi%2Fblob%2Fmaster%2Fhoodie-client%2Fsrc%2Ftest%2Fjava%2Fcom%2Fuber%2Fhoodie%2Fcommon%2FHoodieTestDataGenerator.java%23L69&recipient=ZGV2QGh1ZGkuYXBhY2hlLm9yZw%3D%3D
> > > > )
> > > >
> > > > The end result is that parquet field for timestamp is not of
> timestamp
> > > > (INT96).
> > > >
> > > > My best guess is that this would have been a requirement at Uber
> > > > (tracking trips in minutes and seconds) and how is it being handled.
> > > >
> > > > If anyone else has handled this and has an example that can be
> shared, it
> > > > will be much appreciated.
> > > > Kabeer Ahmed, http://www.linkedin.com/in/kabeerahmed
> > >
> >
> >
>
>
Re: Timestamp
Posted by Kabeer Ahmed <ka...@linuxmail.org>.
Hi Vinoth,
Thank you for looking into this. I am planning to try out this over this weekend if it is possible. Just downloaded the 0.4.6 version of Hudi.
I think we can start with a very simple schema as below (copied from the Hudi's own example)?
val EXAMPLE_SCHEMA = "{\"type\": \"record\"," + "\"name\": \"hudirec\"," + "\"fields\": [ " +
"{\"name\": \"timestamp\", \"type\": \"double\"}," +
"{\"name\": \"_row_key\", \"type\": \"string\"}," +
"{\"name\": \"trade_date\", \"type\": \"string\"}," +
"{\"name\": \"bats\", \"type\": \"int\"}]}";
And sample data could be (cricket bats):
kabeer,2018-11-15T07:35:54.387Z,7
vinoth,2018-11-16T09:35:54.387Z,9
I did try passing several combinations in the timestamp field above to indicate the logicalType to timestamp but no success. I was using Hive 1.1 compile time flag but I was not worried about reading data through Hive. I could see in the generated parquet that the timestamp field was NOT INT96 timestamp format that the parquet expects.
Keep me posted as to how you get along with this and I shall keep you posted if I find any joy sooner than yourself.
Thanks
Kabeer.
On Mar 23 2019, at 12:05 am, Vinoth Chandar <vi...@apache.org> wrote:
> Hi Kabeer,
>
> I spent time looking at the issue and its other linked issues as well.
> High level, seems like we need to change the data type mappings for these
> date/timestamp types..
> It does seem doable, given Avro also supports date/timestamp types..
>
> Do you have some sample schema/data generation that we can start with?
> Thanks
> Vinoth
>
> On Fri, Mar 15, 2019 at 11:19 AM Vinoth Chandar <vi...@apache.org> wrote:
> > Hi Kabeer,
> > Thanks for bringing this up. I don't think we have actually hit this
> > before :)
> >
> > Let me spend sometime understanding the issue and get back to you
> > Thanks
> > Vinoth
> >
> > On Thu, Mar 14, 2019 at 10:46 PM Kabeer Ahmed <ka...@linuxmail.org>
> > wrote:
> >
> > > Hi,
> > > https://github.com/apache/incubator-hudi/issues/547 (
> > > https://link.getmailspring.com/link/8516D513-FFCA-4F2F-B8BB-FE1387AE20AF@getmailspring.com/0?redirect=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-hudi%2Fissues%2F547&recipient=ZGV2QGh1ZGkuYXBhY2hlLm9yZw%3D%3D)
> > > has resulted in the jira https://issues.apache.org/jira/browse/HUDI-12 (
> > > https://link.getmailspring.com/link/8516D513-FFCA-4F2F-B8BB-FE1387AE20AF@getmailspring.com/1?redirect=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHUDI-12&recipient=ZGV2QGh1ZGkuYXBhY2hlLm9yZw%3D%3D
> > > ).
> > > The requirement is to be able to interpret timestamp from CSV and store
> > > it in the parquet table. Does anyone have a working example on these lines?
> > > Going by the Hudi example from the GitHub:
> > > Timestamp is being encoded in avro as double:
> > > https://github.com/apache/incubator-hudi/blob/master/hoodie-client/src/test/java/com/uber/hoodie/common/HoodieTestDataGenerator.java#L69
> > > (
> > > https://link.getmailspring.com/link/8516D513-FFCA-4F2F-B8BB-FE1387AE20AF@getmailspring.com/2?redirect=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-hudi%2Fblob%2Fmaster%2Fhoodie-client%2Fsrc%2Ftest%2Fjava%2Fcom%2Fuber%2Fhoodie%2Fcommon%2FHoodieTestDataGenerator.java%23L69&recipient=ZGV2QGh1ZGkuYXBhY2hlLm9yZw%3D%3D
> > > )
> > >
> > > The end result is that parquet field for timestamp is not of timestamp
> > > (INT96).
> > >
> > > My best guess is that this would have been a requirement at Uber
> > > (tracking trips in minutes and seconds) and how is it being handled.
> > >
> > > If anyone else has handled this and has an example that can be shared, it
> > > will be much appreciated.
> > > Kabeer Ahmed, http://www.linkedin.com/in/kabeerahmed
> >
>
>
Re: Timestamp
Posted by Vinoth Chandar <vi...@apache.org>.
Hi Kabeer,
I spent time looking at the issue and its other linked issues as well.
High level, seems like we need to change the data type mappings for these
date/timestamp types..
It does seem doable, given Avro also supports date/timestamp types..
Do you have some sample schema/data generation that we can start with?
Thanks
Vinoth
On Fri, Mar 15, 2019 at 11:19 AM Vinoth Chandar <vi...@apache.org> wrote:
> Hi Kabeer,
>
> Thanks for bringing this up. I don't think we have actually hit this
> before :)
>
> Let me spend sometime understanding the issue and get back to you
>
> Thanks
> Vinoth
>
> On Thu, Mar 14, 2019 at 10:46 PM Kabeer Ahmed <ka...@linuxmail.org>
> wrote:
>
>> Hi,
>>
>> https://github.com/apache/incubator-hudi/issues/547 (
>> https://link.getmailspring.com/link/8516D513-FFCA-4F2F-B8BB-FE1387AE20AF@getmailspring.com/0?redirect=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-hudi%2Fissues%2F547&recipient=ZGV2QGh1ZGkuYXBhY2hlLm9yZw%3D%3D)
>> has resulted in the jira https://issues.apache.org/jira/browse/HUDI-12 (
>> https://link.getmailspring.com/link/8516D513-FFCA-4F2F-B8BB-FE1387AE20AF@getmailspring.com/1?redirect=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHUDI-12&recipient=ZGV2QGh1ZGkuYXBhY2hlLm9yZw%3D%3D
>> ).
>> The requirement is to be able to interpret timestamp from CSV and store
>> it in the parquet table. Does anyone have a working example on these lines?
>> Going by the Hudi example from the GitHub:
>> Timestamp is being encoded in avro as double:
>> https://github.com/apache/incubator-hudi/blob/master/hoodie-client/src/test/java/com/uber/hoodie/common/HoodieTestDataGenerator.java#L69
>> (
>> https://link.getmailspring.com/link/8516D513-FFCA-4F2F-B8BB-FE1387AE20AF@getmailspring.com/2?redirect=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-hudi%2Fblob%2Fmaster%2Fhoodie-client%2Fsrc%2Ftest%2Fjava%2Fcom%2Fuber%2Fhoodie%2Fcommon%2FHoodieTestDataGenerator.java%23L69&recipient=ZGV2QGh1ZGkuYXBhY2hlLm9yZw%3D%3D
>> )
>>
>> The end result is that parquet field for timestamp is not of timestamp
>> (INT96).
>>
>> My best guess is that this would have been a requirement at Uber
>> (tracking trips in minutes and seconds) and how is it being handled.
>>
>> If anyone else has handled this and has an example that can be shared, it
>> will be much appreciated.
>> Kabeer Ahmed, http://www.linkedin.com/in/kabeerahmed
>>
>>
Re: Timestamp
Posted by Vinoth Chandar <vi...@apache.org>.
Hi Kabeer,
Thanks for bringing this up. I don't think we have actually hit this before
:)
Let me spend sometime understanding the issue and get back to you
Thanks
Vinoth
On Thu, Mar 14, 2019 at 10:46 PM Kabeer Ahmed <ka...@linuxmail.org> wrote:
> Hi,
>
> https://github.com/apache/incubator-hudi/issues/547 (
> https://link.getmailspring.com/link/8516D513-FFCA-4F2F-B8BB-FE1387AE20AF@getmailspring.com/0?redirect=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-hudi%2Fissues%2F547&recipient=ZGV2QGh1ZGkuYXBhY2hlLm9yZw%3D%3D)
> has resulted in the jira https://issues.apache.org/jira/browse/HUDI-12 (
> https://link.getmailspring.com/link/8516D513-FFCA-4F2F-B8BB-FE1387AE20AF@getmailspring.com/1?redirect=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHUDI-12&recipient=ZGV2QGh1ZGkuYXBhY2hlLm9yZw%3D%3D
> ).
> The requirement is to be able to interpret timestamp from CSV and store it
> in the parquet table. Does anyone have a working example on these lines?
> Going by the Hudi example from the GitHub:
> Timestamp is being encoded in avro as double:
> https://github.com/apache/incubator-hudi/blob/master/hoodie-client/src/test/java/com/uber/hoodie/common/HoodieTestDataGenerator.java#L69
> (
> https://link.getmailspring.com/link/8516D513-FFCA-4F2F-B8BB-FE1387AE20AF@getmailspring.com/2?redirect=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-hudi%2Fblob%2Fmaster%2Fhoodie-client%2Fsrc%2Ftest%2Fjava%2Fcom%2Fuber%2Fhoodie%2Fcommon%2FHoodieTestDataGenerator.java%23L69&recipient=ZGV2QGh1ZGkuYXBhY2hlLm9yZw%3D%3D
> )
>
> The end result is that parquet field for timestamp is not of timestamp
> (INT96).
>
> My best guess is that this would have been a requirement at Uber (tracking
> trips in minutes and seconds) and how is it being handled.
>
> If anyone else has handled this and has an example that can be shared, it
> will be much appreciated.
> Kabeer Ahmed, http://www.linkedin.com/in/kabeerahmed
>
>