You are viewing a plain text version of this content. The canonical link for it is here.
Posted to devnull@infra.apache.org by "snuyanzin (GitHub)" <gi...@apache.org> on 2018/08/26 11:00:20 UTC

[GitHub] [flink] snuyanzin opened pull request #6614: [FLINK-9432] Support extract timeunit: DECADE, EPOCH, ISODOW, ISOYEAR…

## What is the purpose of the change
The PR provides implementation of `extract` function for timeunits: `decade`, `epoch`, `isodow`, `isoyear`, 'millisecond`,`microsecond`

## Brief change log

- support timeunits `extract`
- documentation updates (mention all possible timeunits for `extract`)
- tests

## Verifying this change
  - *Added tests validates the result of `extract` call*
  - *Added tests validates that `extract` of `isodow`, `isodoy`, `epoch`, `decade` for time is incorrect*

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (yes)
  - If yes, how is the feature documented? (docs)
    functions.md updated


[ Full content available at: https://github.com/apache/flink/pull/6614 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] pnowojski commented on pull request #6614: [FLINK-9432] [table] Support extract timeunit: DECADE, EPOCH, ISODOW, ISOYEAR…

Posted by "pnowojski (GitHub)" <gi...@apache.org>.
Why `default` throws `AssertionError`? Shouldn't it be `ValidationException` or `UnsupportedOperationException`? If it should never happen then it should be `IllegalStateException`.

[ Full content available at: https://github.com/apache/flink/pull/6614 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] pnowojski commented on pull request #6614: [FLINK-9432] [table] Support extract timeunit: DECADE, EPOCH, ISODOW, ISOYEAR…

Posted by "pnowojski (GitHub)" <gi...@apache.org>.
Side question (maybe stupid one). Why do we need `ExtractCallGen` class at all?

[ Full content available at: https://github.com/apache/flink/pull/6614 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] pnowojski commented on pull request #6614: [FLINK-9432] [table] Support extract timeunit: DECADE, EPOCH, ISODOW, ISOYEAR…

Posted by "pnowojski (GitHub)" <gi...@apache.org>.
Could you change it just to:
```
public static final long SECONDS_PER_DAY = 24 * 60 * 60;
```
Same readability, less duplication. Could you also change it for other constants in this file like `MILLIS_PER_HOUR` and `MILLIS_PER_DAY`?

[ Full content available at: https://github.com/apache/flink/pull/6614 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] pnowojski commented on pull request #6614: [FLINK-9432] [table] Support extract timeunit: DECADE, EPOCH, ISODOW, ISOYEAR…

Posted by "pnowojski (GitHub)" <gi...@apache.org>.
1. Please extract this code into some method
2. add some comments what is going on here
3. Please add more tests for `ISOYEAR` covering those edge cases (when `ISOYEAR` differs from `YEAR` in either of the directions).

[ Full content available at: https://github.com/apache/flink/pull/6614 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] pnowojski commented on pull request #6614: [FLINK-9432] [table] Support extract timeunit: DECADE, EPOCH, ISODOW, ISOYEAR…

Posted by "pnowojski (GitHub)" <gi...@apache.org>.
Why did you change the old `WEEK` code to `getIso8601WeekNumber`? If they return the same thing, why didn't you implement `getIso8601WeekNumber` as `(julian - fmofw) / 7 + 1;`? 

[ Full content available at: https://github.com/apache/flink/pull/6614 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] pnowojski commented on pull request #6614: [FLINK-9432] [table] Support extract timeunit: DECADE, EPOCH, ISODOW, ISOYEAR…

Posted by "pnowojski (GitHub)" <gi...@apache.org>.
ditto

[ Full content available at: https://github.com/apache/flink/pull/6614 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] pnowojski commented on pull request #6614: [FLINK-9432] [table] Support extract timeunit: DECADE, EPOCH, ISODOW, ISOYEAR…

Posted by "pnowojski (GitHub)" <gi...@apache.org>.
Why does the PostgreSQL (which seems to be the only major DB that supports EPOCHs?) returns different results here? The results are 10 seconds of, which seems disturbing.

http://sqlfiddle.com/#!17/9eecb/21451

It might have something to do with leap seconds...

[ Full content available at: https://github.com/apache/flink/pull/6614 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] pnowojski commented on pull request #6614: [FLINK-9432] [table] Support extract timeunit: DECADE, EPOCH, ISODOW, ISOYEAR…

Posted by "pnowojski (GitHub)" <gi...@apache.org>.
What is this code block doing?

[ Full content available at: https://github.com/apache/flink/pull/6614 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] pnowojski commented on pull request #6614: [FLINK-9432] [table] Support extract timeunit: DECADE, EPOCH, ISODOW, ISOYEAR…

Posted by "pnowojski (GitHub)" <gi...@apache.org>.
Why didn't you add support and tests to table API as well? Why do we support only subset of ranges in table api?

[ Full content available at: https://github.com/apache/flink/pull/6614 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] pnowojski commented on pull request #6614: [FLINK-9432] [table] Support extract timeunit: DECADE, EPOCH, ISODOW, ISOYEAR…

Posted by "pnowojski (GitHub)" <gi...@apache.org>.
1. please deduplicate the code in those `EXTRACT(XYZ FROM TIME t)` tests. Either use parametrized tests from junit4 ( https://stackoverflow.com/questions/4399881/parameterized-unit-tests-in-scala-with-junit4 ), or create a for loop over units.
2. Add expected exception message pattern like `unit .* can not be applied to time variable`?

[ Full content available at: https://github.com/apache/flink/pull/6614 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] pnowojski commented on pull request #6614: [FLINK-9432] [table] Support extract timeunit: DECADE, EPOCH, ISODOW, ISOYEAR…

Posted by "pnowojski (GitHub)" <gi...@apache.org>.
? what's the problem here? If it returns incorrect result, we should change it into throwing an exception or not providing `EXTRACT(MILLISECOND` feature at all.

[ Full content available at: https://github.com/apache/flink/pull/6614 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] pnowojski commented on pull request #6614: [FLINK-9432] [table] Support extract timeunit: DECADE, EPOCH, ISODOW, ISOYEAR…

Posted by "pnowojski (GitHub)" <gi...@apache.org>.
I'm a little bit worried about the `julian` thing in this method name... I have a bad feeling that it supposed to be Gregorian calendar here...

[ Full content available at: https://github.com/apache/flink/pull/6614 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] twalthr commented on pull request #6614: [FLINK-9432] [table] Support extract timeunit: DECADE, EPOCH, ISODOW, ISOYEAR…

Posted by "twalthr (GitHub)" <gi...@apache.org>.
This class should not be changed. It is copied from Calcite until a certain issue is fixed. If we apply changes here, then just to update it to the latest Calcite code base.

[ Full content available at: https://github.com/apache/flink/pull/6614 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org