You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "kyle minmaxcorp.com" <ky...@minmaxcorp.com> on 2022/04/13 18:22:20 UTC

Duration spec change

When it comes to a duration type, both years and months are not standard increments. We would have to know the start datetime and end datetime to calculate the actual values.

Current Text:

Duration

The duration logical type represents an amount of time defined by a number of months, days and milliseconds. This is not equivalent to a number of milliseconds, because, depending on the moment in time from which the duration is measured, the number of days in the month and number of milliseconds in a day may differ. Other standard periods such as years, quarters, hours and minutes can be expressed through these basic periods.



A duration logical type annotates Avro fixed type of size 12, which stores three little-endian unsigned integers that represent durations at different granularities of time. The first stores a number in months, the second stores a number in days, and the third stores a number in milliseconds.



Updated text:

Duration

The duration logical type represents an amount of time defined by the number of weeks, days, and milliseconds. Other standard periods such as hours, minutes, and seconds can be expressed through these basic periods.



A duration logical type annotates Avro fixed type of size 12, which stores three little-endian unsigned integers that represent durations at different granularities of time. The first stores a number in weeks, the second stores a number in days, and the third stores a number in milliseconds.


Additional discussions:
https://issues.apache.org/jira/browse/AVRO-739?focusedCommentId=14083646&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14083646

Re: Duration spec change

Posted by Oscar Westra van Holthe - Kind <os...@westravanholthe.nl>.
Hi all,

One of the difficulties is that time intervals and periods have different
use cases,  but they share the same type in Avro.

Often, a time interval is used within a day only. Thus,  it can be
expressed using milliseconds.
Larger periods usually span entire days, and don't need milliseconds. But
they do need months and days.

Although translating the Avro type to a specific language may be tricky
(Avro durations support more values than Java intervals and periods, for
example), I think it's best to keep it as is.

Kind regards,
Oscar

-- 
Oscar Westra van Holthe - Kind <os...@westravanholthe.nl>

Op wo 13 apr. 2022 23:11 schreef kyle minmaxcorp.com <ky...@minmaxcorp.com>:

> If I'm being honest I would only do days, hours, milliseconds. Months
> doesn't really have a defined standard (unless I know the datetime that I
> am adding to or subtracting to). I can't add or subtract 30 days and know
> that it is a month. The actual is same for a year. So, based on some of the
> implementations I'm seeing. The first value isn't month, but should be
> referred to as 30 day intervals.
>
> > On April 13, 2022 12:05 PM Micah Kornfield <em...@gmail.com>
> wrote:
> >
> >
> > Hi Kyle,
> > > We would have to know the start datetime and end datetime to calculate
> the actual values.
> > I believe this is intended. There are two types of durations, those that
> relate to calendars (you need to know when they start) and those that are
> independent of any notion of calendar and measure an absolute time delta.
> Both are useful. It's unfortunate that there isn't common terminology,
> Avro's duration is referred to as interval in other systems.
> >
> > > The duration logical type represents an amount of time defined by the
> number of weeks, days, and milliseconds.
> > Changing to weeks seems like a fundamentally breaking change? What value
> do weeks have over days? For context in SQL systems there are generally two
> different durations (Year-Month) and (Days/Seconds). Some systems combine
> the two (e.g. Postgres and BigQuery).
> >
> > Cheers,
> > Micah
> >
> >
> > On Wed, Apr 13, 2022 at 11:22 AM kyle minmaxcorp.com (
> http://minmaxcorp.com) <ky...@minmaxcorp.com> wrote:
> > > When it comes to a duration type, both years and months are not
> standard increments. We would have to know the start datetime and end
> datetime to calculate the actual values.
> > >
> > > Current Text:
> > >
> > > Duration
> > >
> > > The duration logical type represents an amount of time defined by a
> number of months, days and milliseconds. This is not equivalent to a number
> of milliseconds, because, depending on the moment in time from which the
> duration is measured, the number of days in the month and number of
> milliseconds in a day may differ. Other standard periods such as years,
> quarters, hours and minutes can be expressed through these basic periods.
> > >
> > >
> > >
> > > A duration logical type annotates Avro fixed type of size 12, which
> stores three little-endian unsigned integers that represent durations at
> different granularities of time. The first stores a number in months, the
> second stores a number in days, and the third stores a number in
> milliseconds.
> > >
> > >
> > >
> > > Updated text:
> > >
> > > Duration
> > >
> > > The duration logical type represents an amount of time defined by the
> number of weeks, days, and milliseconds. Other standard periods such as
> hours, minutes, and seconds can be expressed through these basic periods.
> > >
> > >
> > >
> > > A duration logical type annotates Avro fixed type of size 12, which
> stores three little-endian unsigned integers that represent durations at
> different granularities of time. The first stores a number in weeks, the
> second stores a number in days, and the third stores a number in
> milliseconds.
> > >
> > >
> > > Additional discussions:
> > >
> https://issues.apache.org/jira/browse/AVRO-739?focusedCommentId=14083646&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14083646
>

Re: Duration spec change

Posted by "kyle minmaxcorp.com" <ky...@minmaxcorp.com>.
Should there be a definition of intention of the interval of month in the spec then?  My concern is it is left up to interpretation, which can cause more of these discussions.



> On April 13, 2022 2:39 PM Martin Grigorov <mg...@apache.org> wrote:
> 
> 
> 
> 
> 
> On Thu, Apr 14, 2022 at 12:11 AM kyle minmaxcorp.com (http://minmaxcorp.com) <ky...@minmaxcorp.com> wrote:
> > If I'm being honest I would only do days, hours, milliseconds. Months doesn't really have a defined standard (unless I know the datetime that I am adding to or subtracting to). I can't add or subtract 30 days and know that it is a month. The actual is same for a year. So, based on some of the implementations I'm seeing. The first value isn't month, but should be referred to as 30 day intervals.
> 
> The javadoc of java.time.temporal.ChronoUnit#MONTHS explains the idea:
> ----
> Unit that represents the concept of a month. For the ISO calendar system, the length of the month varies by month-of-year. The estimated duration of a month is one twelfth of 365.2425 Days.
> When used with other calendar systems it must correspond to an integral number of days.
> ----
> So it is neither 28, 29, 30 nor 31 but Duration.ofSeconds(31556952L / 12)
> 
> > 
> > > On April 13, 2022 12:05 PM Micah Kornfield <em...@gmail.com> wrote:
> > > 
> > > 
> > > Hi Kyle,
> > > > We would have to know the start datetime and end datetime to calculate the actual values.
> > > I believe this is intended. There are two types of durations, those that relate to calendars (you need to know when they start) and those that are independent of any notion of calendar and measure an absolute time delta. Both are useful. It's unfortunate that there isn't common terminology, Avro's duration is referred to as interval in other systems.
> > > 
> > > > The duration logical type represents an amount of time defined by the number of weeks, days, and milliseconds.
> > > Changing to weeks seems like a fundamentally breaking change? What value do weeks have over days? For context in SQL systems there are generally two different durations (Year-Month) and (Days/Seconds). Some systems combine the two (e.g. Postgres and BigQuery).
> > > 
> > > Cheers,
> > > Micah
> > > 
> > > 
> > > On Wed, Apr 13, 2022 at 11:22 AM kyle minmaxcorp.com (http://minmaxcorp.com) (http://minmaxcorp.com) <ky...@minmaxcorp.com> wrote:
> > > > When it comes to a duration type, both years and months are not standard increments. We would have to know the start datetime and end datetime to calculate the actual values.
> > > > 
> > > > Current Text:
> > > > 
> > > > Duration
> > > > 
> > > > The duration logical type represents an amount of time defined by a number of months, days and milliseconds. This is not equivalent to a number of milliseconds, because, depending on the moment in time from which the duration is measured, the number of days in the month and number of milliseconds in a day may differ. Other standard periods such as years, quarters, hours and minutes can be expressed through these basic periods.
> > > > 
> > > > 
> > > > 
> > > > A duration logical type annotates Avro fixed type of size 12, which stores three little-endian unsigned integers that represent durations at different granularities of time. The first stores a number in months, the second stores a number in days, and the third stores a number in milliseconds.
> > > > 
> > > > 
> > > > 
> > > > Updated text:
> > > > 
> > > > Duration
> > > > 
> > > > The duration logical type represents an amount of time defined by the number of weeks, days, and milliseconds. Other standard periods such as hours, minutes, and seconds can be expressed through these basic periods.
> > > > 
> > > > 
> > > > 
> > > > A duration logical type annotates Avro fixed type of size 12, which stores three little-endian unsigned integers that represent durations at different granularities of time. The first stores a number in weeks, the second stores a number in days, and the third stores a number in milliseconds.
> > > > 
> > > > 
> > > > Additional discussions:
> > > > https://issues.apache.org/jira/browse/AVRO-739?focusedCommentId=14083646&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14083646

Re: Duration spec change

Posted by Martin Grigorov <mg...@apache.org>.
On Thu, Apr 14, 2022 at 12:11 AM kyle minmaxcorp.com <ky...@minmaxcorp.com>
wrote:

> If I'm being honest I would only do days, hours, milliseconds. Months
> doesn't really have a defined standard (unless I know the datetime that I
> am adding to or subtracting to). I can't add or subtract 30 days and know
> that it is a month. The actual is same for a year. So, based on some of the
> implementations I'm seeing. The first value isn't month, but should be
> referred to as 30 day intervals.
>

The javadoc of java.time.temporal.ChronoUnit#MONTHS explains the idea:
----
Unit that represents the concept of a month. For the ISO calendar system,
the length of the month varies by month-of-year. The estimated duration of
a month is one twelfth of 365.2425 Days.
When used with other calendar systems it must correspond to an integral
number of days.
----
So it is neither 28, 29, 30 nor 31 but Duration.ofSeconds(31556952L / 12)


> > On April 13, 2022 12:05 PM Micah Kornfield <em...@gmail.com>
> wrote:
> >
> >
> > Hi Kyle,
> > > We would have to know the start datetime and end datetime to calculate
> the actual values.
> > I believe this is intended. There are two types of durations, those that
> relate to calendars (you need to know when they start) and those that are
> independent of any notion of calendar and measure an absolute time delta.
> Both are useful. It's unfortunate that there isn't common terminology,
> Avro's duration is referred to as interval in other systems.
> >
> > > The duration logical type represents an amount of time defined by the
> number of weeks, days, and milliseconds.
> > Changing to weeks seems like a fundamentally breaking change? What value
> do weeks have over days? For context in SQL systems there are generally two
> different durations (Year-Month) and (Days/Seconds). Some systems combine
> the two (e.g. Postgres and BigQuery).
> >
> > Cheers,
> > Micah
> >
> >
> > On Wed, Apr 13, 2022 at 11:22 AM kyle minmaxcorp.com (
> http://minmaxcorp.com) <ky...@minmaxcorp.com> wrote:
> > > When it comes to a duration type, both years and months are not
> standard increments. We would have to know the start datetime and end
> datetime to calculate the actual values.
> > >
> > > Current Text:
> > >
> > > Duration
> > >
> > > The duration logical type represents an amount of time defined by a
> number of months, days and milliseconds. This is not equivalent to a number
> of milliseconds, because, depending on the moment in time from which the
> duration is measured, the number of days in the month and number of
> milliseconds in a day may differ. Other standard periods such as years,
> quarters, hours and minutes can be expressed through these basic periods.
> > >
> > >
> > >
> > > A duration logical type annotates Avro fixed type of size 12, which
> stores three little-endian unsigned integers that represent durations at
> different granularities of time. The first stores a number in months, the
> second stores a number in days, and the third stores a number in
> milliseconds.
> > >
> > >
> > >
> > > Updated text:
> > >
> > > Duration
> > >
> > > The duration logical type represents an amount of time defined by the
> number of weeks, days, and milliseconds. Other standard periods such as
> hours, minutes, and seconds can be expressed through these basic periods.
> > >
> > >
> > >
> > > A duration logical type annotates Avro fixed type of size 12, which
> stores three little-endian unsigned integers that represent durations at
> different granularities of time. The first stores a number in weeks, the
> second stores a number in days, and the third stores a number in
> milliseconds.
> > >
> > >
> > > Additional discussions:
> > >
> https://issues.apache.org/jira/browse/AVRO-739?focusedCommentId=14083646&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14083646
>

Re: Duration spec change

Posted by "kyle minmaxcorp.com" <ky...@minmaxcorp.com>.
If I'm being honest I would only do days, hours, milliseconds. Months doesn't really have a defined standard (unless I know the datetime that I am adding to or subtracting to). I can't add or subtract 30 days and know that it is a month. The actual is same for a year. So, based on some of the implementations I'm seeing. The first value isn't month, but should be referred to as 30 day intervals.

> On April 13, 2022 12:05 PM Micah Kornfield <em...@gmail.com> wrote:
> 
> 
> Hi Kyle,
> > We would have to know the start datetime and end datetime to calculate the actual values.
> I believe this is intended. There are two types of durations, those that relate to calendars (you need to know when they start) and those that are independent of any notion of calendar and measure an absolute time delta. Both are useful. It's unfortunate that there isn't common terminology, Avro's duration is referred to as interval in other systems.
> 
> > The duration logical type represents an amount of time defined by the number of weeks, days, and milliseconds.
> Changing to weeks seems like a fundamentally breaking change? What value do weeks have over days? For context in SQL systems there are generally two different durations (Year-Month) and (Days/Seconds). Some systems combine the two (e.g. Postgres and BigQuery).
> 
> Cheers,
> Micah
> 
> 
> On Wed, Apr 13, 2022 at 11:22 AM kyle minmaxcorp.com (http://minmaxcorp.com) <ky...@minmaxcorp.com> wrote:
> > When it comes to a duration type, both years and months are not standard increments. We would have to know the start datetime and end datetime to calculate the actual values.
> > 
> > Current Text:
> > 
> > Duration
> > 
> > The duration logical type represents an amount of time defined by a number of months, days and milliseconds. This is not equivalent to a number of milliseconds, because, depending on the moment in time from which the duration is measured, the number of days in the month and number of milliseconds in a day may differ. Other standard periods such as years, quarters, hours and minutes can be expressed through these basic periods.
> > 
> > 
> > 
> > A duration logical type annotates Avro fixed type of size 12, which stores three little-endian unsigned integers that represent durations at different granularities of time. The first stores a number in months, the second stores a number in days, and the third stores a number in milliseconds.
> > 
> > 
> > 
> > Updated text:
> > 
> > Duration
> > 
> > The duration logical type represents an amount of time defined by the number of weeks, days, and milliseconds. Other standard periods such as hours, minutes, and seconds can be expressed through these basic periods.
> > 
> > 
> > 
> > A duration logical type annotates Avro fixed type of size 12, which stores three little-endian unsigned integers that represent durations at different granularities of time. The first stores a number in weeks, the second stores a number in days, and the third stores a number in milliseconds.
> > 
> > 
> > Additional discussions:
> > https://issues.apache.org/jira/browse/AVRO-739?focusedCommentId=14083646&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14083646

Re: Duration spec change

Posted by Micah Kornfield <em...@gmail.com>.
Hi Kyle,

> We would have to know the start datetime and end datetime to calculate the
> actual values.

I believe this is intended.  There are two types of durations, those that
relate to calendars (you need to know when they start) and those that are
independent of any notion of calendar and measure an absolute time delta.
Both are useful.  It's unfortunate that there isn't common terminology,
Avro's duration is referred to as interval in other systems.

The duration logical type represents an amount of time defined by the
> number of weeks, days, and milliseconds.

Changing to weeks seems like a fundamentally breaking change?  What value
do weeks have over days?   For context in SQL systems there are generally
two different durations (Year-Month) and (Days/Seconds).  Some systems
combine the two (e.g. Postgres and BigQuery).

Cheers,
Micah

On Wed, Apr 13, 2022 at 11:22 AM kyle minmaxcorp.com <ky...@minmaxcorp.com>
wrote:

> When it comes to a duration type, both years and months are not standard
> increments. We would have to know the start datetime and end datetime to
> calculate the actual values.
>
> Current Text:
>
> Duration
>
> The duration logical type represents an amount of time defined by a number
> of months, days and milliseconds. This is not equivalent to a number of
> milliseconds, because, depending on the moment in time from which the
> duration is measured, the number of days in the month and number of
> milliseconds in a day may differ. Other standard periods such as years,
> quarters, hours and minutes can be expressed through these basic periods.
>
>
>
> A duration logical type annotates Avro fixed type of size 12, which stores
> three little-endian unsigned integers that represent durations at different
> granularities of time. The first stores a number in months, the second
> stores a number in days, and the third stores a number in milliseconds.
>
>
>
> Updated text:
>
> Duration
>
> The duration logical type represents an amount of time defined by the
> number of weeks, days, and milliseconds. Other standard periods such as
> hours, minutes, and seconds can be expressed through these basic periods.
>
>
>
> A duration logical type annotates Avro fixed type of size 12, which stores
> three little-endian unsigned integers that represent durations at different
> granularities of time. The first stores a number in weeks, the second
> stores a number in days, and the third stores a number in milliseconds.
>
>
> Additional discussions:
>
> https://issues.apache.org/jira/browse/AVRO-739?focusedCommentId=14083646&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14083646
>