You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "snuyanzin (via GitHub)" <gi...@apache.org> on 2026/03/28 16:26:44 UTC
[PR] [FLINK-39348][table] Add `Interval` to reuse code for `FRESHESS` and `START MODE` [flink]
snuyanzin opened a new pull request, #27845:
URL: https://github.com/apache/flink/pull/27845
## What is the purpose of the change
Currently there is `IntervalFreshness` which has some interval functionality
in FLIP-557 we need similar functionality for `START_MODE`, however `IntervalFreshness` is not reusable in this way.
This PR introduces `Interval` adapts `IntervalFreshness` code to use it and prepares code for `START_MODE` implementation
## Brief change log
Interval
adaptation of IntervalFreshness
tests
## Verifying this change
Existing tests
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): ( no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no)
- The serializers: ( no)
- The runtime per-record code paths (performance sensitive): ( no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: ( no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? ( no)
- If yes, how is the feature documented? (not applicable)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39348][table] Add `Interval` to reuse code for `FRESHNESS` and `START MODE` [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin merged PR #27845:
URL: https://github.com/apache/flink/pull/27845
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39348][table] Add `Interval` to reuse code for `FRESHNESS` and `START MODE` [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #27845:
URL: https://github.com/apache/flink/pull/27845#discussion_r3011496146
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/utils/MaterializedTableUtils.java:
##########
@@ -68,29 +71,110 @@ public class MaterializedTableUtils {
public static IntervalFreshness getMaterializedTableFreshness(
SqlIntervalLiteral sqlIntervalLiteral) {
- if (sqlIntervalLiteral.signum() < 0) {
- throw new ValidationException(
- "Materialized table freshness doesn't support negative value.");
+ return new IntervalFreshness(getFreshnessInterval(sqlIntervalLiteral));
+ }
+
+ private static Interval getFreshnessInterval(SqlIntervalLiteral sqlIntervalLiteral) {
+ final IntervalValue intervalValue = sqlIntervalLiteral.getValueAs(IntervalValue.class);
+ final SqlTypeName typeName = intervalValue.getIntervalQualifier().typeName();
+
+ if (isDateTimeInterval(typeName)) {
+ final Interval freshnessInterval =
+ getDayTimeInterval(
+ intervalValue,
+ typeName,
+ sqlIntervalLiteral.getValueAs(BigDecimal.class),
+ "freshness");
+ final int interval = freshnessInterval.getInterval();
+ // Freshness interval might be only positive
+ if (interval <= 0) {
+ throw new ValidationException(
+ String.format(
+ "The freshness interval currently only supports positive integer type values. But was: %d",
+ interval));
+ }
+ return freshnessInterval;
}
- if (sqlIntervalLiteral.getTypeName().getFamily() != SqlTypeFamily.INTERVAL_DAY_TIME) {
- throw new ValidationException(
- "Materialized table freshness only support SECOND, MINUTE, HOUR, DAY as the time unit.");
+
+ throw new ValidationException(
+ "Materialized table freshness only supports SECOND, MINUTE, HOUR, DAY, WEEK as the time unit.");
+ }
+
+ private static Interval intervalFrom(
Review Comment:
yes it is correct, this branch is not used for freshness, it will be used for start_mode
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39348][table] Add `Interval` to reuse code for `FRESHNESS` and `START MODE` [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #27845:
URL: https://github.com/apache/flink/pull/27845#discussion_r3008532979
##########
flink-table/flink-sql-gateway/src/test/java/org/apache/flink/table/gateway/service/materializedtable/MaterializedTableManagerTest.java:
##########
@@ -136,66 +136,66 @@ static Stream<TestSpec> testData() {
// The interval of freshness match the partition specified by the 'date-formatter'.
TestSpec.create()
.schedulerTime("2024-01-01 00:00:00")
- .freshness(IntervalFreshness.ofDay("1"))
+ .freshness(IntervalFreshness.ofDay(1))
Review Comment:
hm, there is no `START_MODE` tests yet since it is just a prep PR to extract `Interval`, so not sure what tests for `START_MODE` you are referring to
>String parameters may cause runtime parsing errors; recommend unifying to integer parameters.
I didn't get that, can you clarify what you mean here?
From PR you can see that it is introduced int arg methods
string args are deprecated, however we can not simply remove them since it is public api
>Repeated test logic can be extracted into factory methods or generated loops to reduce maintenance.
are you referring to some specific code lines?
here it was only change input arg, for refactoring (if needed) it would make sense to have a separate PR since it will require more changes in tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39348][table] Add `Interval` to reuse code for `FRESHNESS` and `START MODE` [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #27845:
URL: https://github.com/apache/flink/pull/27845#discussion_r3010574706
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/Interval.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.catalog;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.table.api.ValidationException;
+
+import java.time.Duration;
+import java.time.Period;
+import java.util.Objects;
+
+/** A class representing day-time and year month intervals. */
+@PublicEvolving
+public class Interval {
+ private final Duration duration;
+ private final Period period;
+ private final TimeUnit timeUnit;
+
+ private Interval(TimeUnit timeUnit, Duration duration, Period period) {
+ this.timeUnit = timeUnit;
+ this.duration = duration;
+ this.period = period;
+ }
+
+ public static Interval of(Duration duration, TimeUnit timeUnit) {
+ return new Interval(timeUnit, duration, null);
+ }
+
+ public static Interval of(Period period, TimeUnit timeUnit) {
+ return new Interval(timeUnit, null, period);
+ }
+
+ public static Interval of(int duration, TimeUnit timeUnit) {
+ if (timeUnit.isDayTime) {
+ return of(toDuration(duration, timeUnit), timeUnit);
+ } else {
+ return of(toPeriod(duration, timeUnit), timeUnit);
+ }
+ }
+
+ public Duration getDuration() {
+ return duration;
+ }
+
+ public Period getPeriod() {
+ return period;
+ }
+
+ public TimeUnit getTimeUnit() {
+ return timeUnit;
+ }
+
+ public int getInterval() {
+ switch (timeUnit) {
+ case SECOND:
+ return (int) duration.toSeconds();
+ case MINUTE:
+ return (int) duration.toMinutes();
+ case HOUR:
+ return (int) duration.toHours();
+ case DAY:
+ return (int) duration.toDays();
+ case WEEK:
+ return (int) (duration.toDays() / 7);
+ case MONTH:
+ return period.getMonths();
+ case QUARTER:
+ return period.getMonths() / 3;
+ case YEAR:
+ return period.getYears();
+ default:
+ // Could happen if new TimeUnit is introduced and not supported here
+ throw new ValidationException(
+ String.format("TimeUnit %s is not supported", timeUnit));
+ }
+ }
+
+ private static Duration toDuration(int interval, TimeUnit timeUnit) {
+ switch (timeUnit) {
+ case SECOND:
+ return Duration.ofSeconds(interval);
+ case MINUTE:
+ return Duration.ofMinutes(interval);
+ case HOUR:
+ return Duration.ofHours(interval);
+ case DAY:
+ return Duration.ofDays(interval);
+ case WEEK:
+ return Duration.ofDays(interval * 7L);
+ default:
+ throw new IllegalArgumentException("Unsupported time unit: " + timeUnit);
+ }
+ }
+
+ private static Period toPeriod(int interval, TimeUnit timeUnit) {
+ switch (timeUnit) {
+ case MONTH:
+ return Period.ofMonths(interval);
+ case QUARTER:
+ return Period.ofMonths(interval * 3);
+ case YEAR:
+ return Period.ofYears(interval);
+ default:
+ throw new IllegalArgumentException("Unsupported time unit: " + timeUnit);
+ }
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ Interval interval = (Interval) o;
+ return Objects.equals(duration, interval.duration)
+ && Objects.equals(period, interval.period)
+ && timeUnit == interval.timeUnit;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(duration, period, timeUnit);
+ }
+
+ @Override
+ public String toString() {
+ return "INTERVAL '" + getInterval() + "' " + timeUnit;
+ }
+
+ // --------------------------------------------------------------------------------------------
+ // TimeUnit enums
+ // --------------------------------------------------------------------------------------------
+
+ /** An enumeration of time unit representing the unit of interval freshness. */
Review Comment:
also not sure about mentioning in javadoc all the values once more...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39348][table] Add `Interval` to reuse code for `FRESHNESS` and `START MODE` [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #27845:
URL: https://github.com/apache/flink/pull/27845#discussion_r3011495231
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/utils/MaterializedTableUtils.java:
##########
@@ -68,29 +71,110 @@ public class MaterializedTableUtils {
public static IntervalFreshness getMaterializedTableFreshness(
SqlIntervalLiteral sqlIntervalLiteral) {
- if (sqlIntervalLiteral.signum() < 0) {
- throw new ValidationException(
- "Materialized table freshness doesn't support negative value.");
+ return new IntervalFreshness(getFreshnessInterval(sqlIntervalLiteral));
+ }
+
+ private static Interval getFreshnessInterval(SqlIntervalLiteral sqlIntervalLiteral) {
+ final IntervalValue intervalValue = sqlIntervalLiteral.getValueAs(IntervalValue.class);
+ final SqlTypeName typeName = intervalValue.getIntervalQualifier().typeName();
+
+ if (isDateTimeInterval(typeName)) {
+ final Interval freshnessInterval =
+ getDayTimeInterval(
+ intervalValue,
+ typeName,
+ sqlIntervalLiteral.getValueAs(BigDecimal.class),
+ "freshness");
+ final int interval = freshnessInterval.getInterval();
+ // Freshness interval might be only positive
+ if (interval <= 0) {
+ throw new ValidationException(
+ String.format(
+ "The freshness interval currently only supports positive integer type values. But was: %d",
+ interval));
+ }
+ return freshnessInterval;
}
- if (sqlIntervalLiteral.getTypeName().getFamily() != SqlTypeFamily.INTERVAL_DAY_TIME) {
- throw new ValidationException(
- "Materialized table freshness only support SECOND, MINUTE, HOUR, DAY as the time unit.");
+
+ throw new ValidationException(
+ "Materialized table freshness only supports SECOND, MINUTE, HOUR, DAY, WEEK as the time unit.");
+ }
+
+ private static Interval intervalFrom(
+ SqlIntervalLiteral sqlIntervalLiteral, String intervalDescription) {
+
+ final IntervalValue intervalValue = sqlIntervalLiteral.getValueAs(IntervalValue.class);
+ final SqlTypeName typeName = intervalValue.getIntervalQualifier().typeName();
+ if (intervalValue.getIntervalQualifier().isYearMonth()) {
+ return getYearMonthInterval(
+ intervalValue,
+ typeName,
+ sqlIntervalLiteral.getValueAs(BigDecimal.class),
+ intervalDescription);
+ }
+
+ if (isDateTimeInterval(typeName)) {
+ return getDayTimeInterval(
+ intervalValue,
+ typeName,
+ sqlIntervalLiteral.getValueAs(BigDecimal.class),
+ intervalDescription);
}
- IntervalValue intervalValue = sqlIntervalLiteral.getValueAs(IntervalValue.class);
- String interval = intervalValue.getIntervalLiteral();
- switch (intervalValue.getIntervalQualifier().typeName()) {
+ throw new ValidationException(
+ String.format(
+ "Materialized table %s only supports SECOND, MINUTE, HOUR, DAY, WEEK, MONTH, QUARTER, YEAR as the time unit.",
+ intervalDescription));
Review Comment:
yes it is correct, this branch is not used for freshness, it will be used for `start_mode`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39348][table] Add `Interval` to reuse code for `FRESHNESS` and `START MODE` [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #27845:
URL: https://github.com/apache/flink/pull/27845#discussion_r3010605382
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/Interval.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.catalog;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.table.api.ValidationException;
+
+import java.time.Duration;
+import java.time.Period;
+import java.util.Objects;
+
+/** A class representing day-time and year month intervals. */
+@PublicEvolving
+public class Interval {
+ private final Duration duration;
+ private final Period period;
+ private final TimeUnit timeUnit;
+
+ private Interval(TimeUnit timeUnit, Duration duration, Period period) {
+ this.timeUnit = timeUnit;
+ this.duration = duration;
+ this.period = period;
+ }
+
+ public static Interval of(Duration duration, TimeUnit timeUnit) {
+ return new Interval(timeUnit, duration, null);
+ }
+
+ public static Interval of(Period period, TimeUnit timeUnit) {
+ return new Interval(timeUnit, null, period);
+ }
+
+ public static Interval of(int duration, TimeUnit timeUnit) {
+ if (timeUnit.isDayTime) {
+ return of(toDuration(duration, timeUnit), timeUnit);
+ } else {
+ return of(toPeriod(duration, timeUnit), timeUnit);
+ }
+ }
+
+ public Duration getDuration() {
+ return duration;
+ }
+
+ public Period getPeriod() {
+ return period;
+ }
+
+ public TimeUnit getTimeUnit() {
+ return timeUnit;
+ }
+
+ public int getInterval() {
+ switch (timeUnit) {
+ case SECOND:
+ return (int) duration.toSeconds();
+ case MINUTE:
+ return (int) duration.toMinutes();
+ case HOUR:
+ return (int) duration.toHours();
+ case DAY:
+ return (int) duration.toDays();
+ case WEEK:
+ return (int) (duration.toDays() / 7);
+ case MONTH:
+ return period.getMonths();
+ case QUARTER:
+ return period.getMonths() / 3;
+ case YEAR:
+ return period.getYears();
+ default:
+ // Could happen if new TimeUnit is introduced and not supported here
+ throw new ValidationException(
+ String.format("TimeUnit %s is not supported", timeUnit));
+ }
+ }
+
+ private static Duration toDuration(int interval, TimeUnit timeUnit) {
+ switch (timeUnit) {
+ case SECOND:
+ return Duration.ofSeconds(interval);
+ case MINUTE:
+ return Duration.ofMinutes(interval);
+ case HOUR:
+ return Duration.ofHours(interval);
+ case DAY:
+ return Duration.ofDays(interval);
+ case WEEK:
+ return Duration.ofDays(interval * 7L);
+ default:
+ throw new IllegalArgumentException("Unsupported time unit: " + timeUnit);
+ }
+ }
+
+ private static Period toPeriod(int interval, TimeUnit timeUnit) {
+ switch (timeUnit) {
+ case MONTH:
+ return Period.ofMonths(interval);
+ case QUARTER:
+ return Period.ofMonths(interval * 3);
+ case YEAR:
+ return Period.ofYears(interval);
+ default:
+ throw new IllegalArgumentException("Unsupported time unit: " + timeUnit);
+ }
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ Interval interval = (Interval) o;
+ return Objects.equals(duration, interval.duration)
+ && Objects.equals(period, interval.period)
+ && timeUnit == interval.timeUnit;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(duration, period, timeUnit);
+ }
+
+ @Override
+ public String toString() {
+ return "INTERVAL '" + getInterval() + "' " + timeUnit;
+ }
+
+ // --------------------------------------------------------------------------------------------
+ // TimeUnit enums
+ // --------------------------------------------------------------------------------------------
+
+ /** An enumeration of time unit representing the unit of interval freshness. */
Review Comment:
Sure thing, we can keep it shorter 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39348][table] Add `Interval` to reuse code for `FRESHNESS` and `START MODE` [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #27845:
URL: https://github.com/apache/flink/pull/27845#discussion_r3011503885
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/utils/MaterializedTableUtils.java:
##########
@@ -68,29 +71,110 @@ public class MaterializedTableUtils {
public static IntervalFreshness getMaterializedTableFreshness(
SqlIntervalLiteral sqlIntervalLiteral) {
- if (sqlIntervalLiteral.signum() < 0) {
- throw new ValidationException(
- "Materialized table freshness doesn't support negative value.");
+ return new IntervalFreshness(getFreshnessInterval(sqlIntervalLiteral));
+ }
+
+ private static Interval getFreshnessInterval(SqlIntervalLiteral sqlIntervalLiteral) {
+ final IntervalValue intervalValue = sqlIntervalLiteral.getValueAs(IntervalValue.class);
+ final SqlTypeName typeName = intervalValue.getIntervalQualifier().typeName();
+
+ if (isDateTimeInterval(typeName)) {
Review Comment:
it is enough
if you look at implementation
```java
private static Interval getDayTimeInterval(
final IntervalValue intervalValue,
final SqlTypeName typeName,
final BigDecimal interval,
final String intervalDescription) {
final int intervalInt = (int) (interval.longValue() / 1000);
switch (typeName) {
case INTERVAL_DAY:
final int amountOfDays = intervalInt / 60 / 60 / 24;
if (intervalValue.getIntervalQualifier().timeUnitRange.startUnit
== org.apache.calcite.avatica.util.TimeUnit.WEEK) {
return Interval.of(amountOfDays / 7, TimeUnit.WEEK);
}
return Interval.of(amountOfDays, TimeUnit.DAY);
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39348][table] Add `Interval` to reuse code for `FRESHNESS` and `START MODE` [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #27845:
URL: https://github.com/apache/flink/pull/27845#discussion_r3010571407
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/Interval.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.catalog;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.table.api.ValidationException;
+
+import java.time.Duration;
+import java.time.Period;
+import java.util.Objects;
+
+/** A class representing day-time and year month intervals. */
+@PublicEvolving
+public class Interval {
+ private final Duration duration;
+ private final Period period;
+ private final TimeUnit timeUnit;
+
+ private Interval(TimeUnit timeUnit, Duration duration, Period period) {
+ this.timeUnit = timeUnit;
+ this.duration = duration;
+ this.period = period;
+ }
+
+ public static Interval of(Duration duration, TimeUnit timeUnit) {
+ return new Interval(timeUnit, duration, null);
+ }
+
+ public static Interval of(Period period, TimeUnit timeUnit) {
+ return new Interval(timeUnit, null, period);
+ }
+
+ public static Interval of(int duration, TimeUnit timeUnit) {
+ if (timeUnit.isDayTime) {
+ return of(toDuration(duration, timeUnit), timeUnit);
+ } else {
+ return of(toPeriod(duration, timeUnit), timeUnit);
+ }
+ }
+
+ public Duration getDuration() {
+ return duration;
+ }
+
+ public Period getPeriod() {
+ return period;
+ }
+
+ public TimeUnit getTimeUnit() {
+ return timeUnit;
+ }
+
+ public int getInterval() {
+ switch (timeUnit) {
+ case SECOND:
+ return (int) duration.toSeconds();
+ case MINUTE:
+ return (int) duration.toMinutes();
+ case HOUR:
+ return (int) duration.toHours();
+ case DAY:
+ return (int) duration.toDays();
+ case WEEK:
+ return (int) (duration.toDays() / 7);
+ case MONTH:
+ return period.getMonths();
+ case QUARTER:
+ return period.getMonths() / 3;
+ case YEAR:
+ return period.getYears();
+ default:
+ // Could happen if new TimeUnit is introduced and not supported here
+ throw new ValidationException(
+ String.format("TimeUnit %s is not supported", timeUnit));
+ }
+ }
+
+ private static Duration toDuration(int interval, TimeUnit timeUnit) {
+ switch (timeUnit) {
+ case SECOND:
+ return Duration.ofSeconds(interval);
+ case MINUTE:
+ return Duration.ofMinutes(interval);
+ case HOUR:
+ return Duration.ofHours(interval);
+ case DAY:
+ return Duration.ofDays(interval);
+ case WEEK:
+ return Duration.ofDays(interval * 7L);
+ default:
+ throw new IllegalArgumentException("Unsupported time unit: " + timeUnit);
+ }
+ }
+
+ private static Period toPeriod(int interval, TimeUnit timeUnit) {
+ switch (timeUnit) {
+ case MONTH:
+ return Period.ofMonths(interval);
+ case QUARTER:
+ return Period.ofMonths(interval * 3);
+ case YEAR:
+ return Period.ofYears(interval);
+ default:
+ throw new IllegalArgumentException("Unsupported time unit: " + timeUnit);
+ }
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ Interval interval = (Interval) o;
+ return Objects.equals(duration, interval.duration)
+ && Objects.equals(period, interval.period)
+ && timeUnit == interval.timeUnit;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(duration, period, timeUnit);
+ }
+
+ @Override
+ public String toString() {
+ return "INTERVAL '" + getInterval() + "' " + timeUnit;
+ }
+
+ // --------------------------------------------------------------------------------------------
+ // TimeUnit enums
+ // --------------------------------------------------------------------------------------------
+
+ /** An enumeration of time unit representing the unit of interval freshness. */
Review Comment:
agree about freshness however not about mentioning all these things.
Mentioning `Period` and `Duration` is too implementation specific
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39348][table] Add `Interval` to reuse code for `FRESHNESS` and `START MODE` [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #27845:
URL: https://github.com/apache/flink/pull/27845#discussion_r3008536584
##########
flink-table/flink-table-api-java/src/test/java/org/apache/flink/table/api/internal/ShowCreateUtilTest.java:
##########
@@ -313,7 +313,7 @@ private static Collection<Arguments> argsForShowCreateMaterializedTable() {
null,
List.of(),
null,
- IntervalFreshness.ofMinute("1"),
+ IntervalFreshness.ofMinute(1),
Review Comment:
What do you mean by this?
can you give an example?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39348][table] Add `Interval` to reuse code for `FRESHNESS` and `START MODE` [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on PR #27845:
URL: https://github.com/apache/flink/pull/27845#issuecomment-4149663532
@flinkbot run azure re-run
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39348][table] Add `Interval` to reuse code for `FRESHNESS` and `START MODE` [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #27845:
URL: https://github.com/apache/flink/pull/27845#discussion_r3011466004
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/utils/MaterializedTableUtils.java:
##########
@@ -68,29 +71,110 @@ public class MaterializedTableUtils {
public static IntervalFreshness getMaterializedTableFreshness(
SqlIntervalLiteral sqlIntervalLiteral) {
- if (sqlIntervalLiteral.signum() < 0) {
- throw new ValidationException(
- "Materialized table freshness doesn't support negative value.");
+ return new IntervalFreshness(getFreshnessInterval(sqlIntervalLiteral));
+ }
+
+ private static Interval getFreshnessInterval(SqlIntervalLiteral sqlIntervalLiteral) {
+ final IntervalValue intervalValue = sqlIntervalLiteral.getValueAs(IntervalValue.class);
+ final SqlTypeName typeName = intervalValue.getIntervalQualifier().typeName();
+
+ if (isDateTimeInterval(typeName)) {
+ final Interval freshnessInterval =
+ getDayTimeInterval(
+ intervalValue,
+ typeName,
+ sqlIntervalLiteral.getValueAs(BigDecimal.class),
+ "freshness");
+ final int interval = freshnessInterval.getInterval();
+ // Freshness interval might be only positive
+ if (interval <= 0) {
+ throw new ValidationException(
+ String.format(
+ "The freshness interval currently only supports positive integer type values. But was: %d",
+ interval));
+ }
+ return freshnessInterval;
}
- if (sqlIntervalLiteral.getTypeName().getFamily() != SqlTypeFamily.INTERVAL_DAY_TIME) {
- throw new ValidationException(
- "Materialized table freshness only support SECOND, MINUTE, HOUR, DAY as the time unit.");
+
+ throw new ValidationException(
+ "Materialized table freshness only supports SECOND, MINUTE, HOUR, DAY, WEEK as the time unit.");
+ }
+
+ private static Interval intervalFrom(
+ SqlIntervalLiteral sqlIntervalLiteral, String intervalDescription) {
+
+ final IntervalValue intervalValue = sqlIntervalLiteral.getValueAs(IntervalValue.class);
+ final SqlTypeName typeName = intervalValue.getIntervalQualifier().typeName();
+ if (intervalValue.getIntervalQualifier().isYearMonth()) {
+ return getYearMonthInterval(
+ intervalValue,
+ typeName,
+ sqlIntervalLiteral.getValueAs(BigDecimal.class),
+ intervalDescription);
+ }
+
+ if (isDateTimeInterval(typeName)) {
Review Comment:
this part is generic and will be executed for `START_MODE` where we have both day-time and year-month
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39348][table] Add `Interval` to reuse code for `FRESHNESS` and `START MODE` [flink]
Posted by "featzhang (via GitHub)" <gi...@apache.org>.
featzhang commented on code in PR #27845:
URL: https://github.com/apache/flink/pull/27845#discussion_r3008354306
##########
flink-table/flink-sql-gateway/src/test/java/org/apache/flink/table/gateway/service/materializedtable/MaterializedTableManagerTest.java:
##########
@@ -136,66 +136,66 @@ static Stream<TestSpec> testData() {
// The interval of freshness match the partition specified by the 'date-formatter'.
TestSpec.create()
.schedulerTime("2024-01-01 00:00:00")
- .freshness(IntervalFreshness.ofDay("1"))
+ .freshness(IntervalFreshness.ofDay(1))
Review Comment:
Recommend adding boundary combination tests for Interval and START MODE.
##########
flink-table/flink-sql-gateway/src/test/java/org/apache/flink/table/gateway/service/materializedtable/MaterializedTableManagerTest.java:
##########
@@ -136,66 +136,66 @@ static Stream<TestSpec> testData() {
// The interval of freshness match the partition specified by the 'date-formatter'.
TestSpec.create()
.schedulerTime("2024-01-01 00:00:00")
- .freshness(IntervalFreshness.ofDay("1"))
+ .freshness(IntervalFreshness.ofDay(1))
Review Comment:
String parameters may cause runtime parsing errors; recommend unifying to integer parameters.
##########
flink-table/flink-sql-gateway/src/test/java/org/apache/flink/table/gateway/service/materializedtable/MaterializedTableManagerTest.java:
##########
@@ -136,66 +136,66 @@ static Stream<TestSpec> testData() {
// The interval of freshness match the partition specified by the 'date-formatter'.
TestSpec.create()
.schedulerTime("2024-01-01 00:00:00")
- .freshness(IntervalFreshness.ofDay("1"))
+ .freshness(IntervalFreshness.ofDay(1))
Review Comment:
Repeated test logic can be extracted into factory methods or generated loops to reduce maintenance.
##########
flink-table/flink-table-api-java/src/test/java/org/apache/flink/table/api/internal/ShowCreateUtilTest.java:
##########
@@ -313,7 +313,7 @@ private static Collection<Arguments> argsForShowCreateMaterializedTable() {
null,
List.of(),
null,
- IntervalFreshness.ofMinute("1"),
+ IntervalFreshness.ofMinute(1),
Review Comment:
Annotate Interval unit (day/hour/minute) in test comments for clarity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39348][table] Add `Interval` to reuse code for `FRESHNESS` and `START MODE` [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #27845:
URL: https://github.com/apache/flink/pull/27845#discussion_r3008642376
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/Interval.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.catalog;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.table.api.ValidationException;
+
+import java.time.Duration;
+import java.time.Period;
+import java.util.Objects;
+
+/** A class representing day-time and year month intervals. */
Review Comment:
```suggestion
/**
* Represents a time interval consisting of a numeric value and a {@link TimeUnit}.
*
* <p>Depending on the time unit,
* the interval is backed by either a {@link Duration} (for day-time units like {@code SECOND},
* {@code MINUTE}, {@code HOUR}, {@code DAY}, {@code WEEK}) or a {@link Period} (for year-month
* units like {@code MONTH}, {@code QUARTER}, {@code YEAR}).
*
* <p>Intervals can be created using the {@link #of} factory methods:
*
* <pre>{@code
* Interval.of(5, TimeUnit.MINUTE) // 5 minutes (day-time)
* Interval.of(3, TimeUnit.MONTH) // 3 months (year-month)
* }</pre>
*/
```
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/Interval.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.catalog;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.table.api.ValidationException;
+
+import java.time.Duration;
+import java.time.Period;
+import java.util.Objects;
+
+/** A class representing day-time and year month intervals. */
+@PublicEvolving
+public class Interval {
+ private final Duration duration;
+ private final Period period;
+ private final TimeUnit timeUnit;
+
+ private Interval(TimeUnit timeUnit, Duration duration, Period period) {
+ this.timeUnit = timeUnit;
+ this.duration = duration;
+ this.period = period;
+ }
+
+ public static Interval of(Duration duration, TimeUnit timeUnit) {
+ return new Interval(timeUnit, duration, null);
+ }
+
+ public static Interval of(Period period, TimeUnit timeUnit) {
+ return new Interval(timeUnit, null, period);
+ }
+
+ public static Interval of(int duration, TimeUnit timeUnit) {
+ if (timeUnit.isDayTime) {
+ return of(toDuration(duration, timeUnit), timeUnit);
+ } else {
+ return of(toPeriod(duration, timeUnit), timeUnit);
+ }
+ }
+
+ public Duration getDuration() {
+ return duration;
+ }
+
+ public Period getPeriod() {
+ return period;
+ }
+
+ public TimeUnit getTimeUnit() {
+ return timeUnit;
+ }
+
+ public int getInterval() {
+ switch (timeUnit) {
+ case SECOND:
+ return (int) duration.toSeconds();
+ case MINUTE:
+ return (int) duration.toMinutes();
+ case HOUR:
+ return (int) duration.toHours();
+ case DAY:
+ return (int) duration.toDays();
+ case WEEK:
+ return (int) (duration.toDays() / 7);
+ case MONTH:
+ return period.getMonths();
+ case QUARTER:
+ return period.getMonths() / 3;
+ case YEAR:
+ return period.getYears();
+ default:
+ // Could happen if new TimeUnit is introduced and not supported here
+ throw new ValidationException(
+ String.format("TimeUnit %s is not supported", timeUnit));
+ }
+ }
+
+ private static Duration toDuration(int interval, TimeUnit timeUnit) {
+ switch (timeUnit) {
+ case SECOND:
+ return Duration.ofSeconds(interval);
+ case MINUTE:
+ return Duration.ofMinutes(interval);
+ case HOUR:
+ return Duration.ofHours(interval);
+ case DAY:
+ return Duration.ofDays(interval);
+ case WEEK:
+ return Duration.ofDays(interval * 7L);
+ default:
+ throw new IllegalArgumentException("Unsupported time unit: " + timeUnit);
+ }
+ }
+
+ private static Period toPeriod(int interval, TimeUnit timeUnit) {
+ switch (timeUnit) {
+ case MONTH:
+ return Period.ofMonths(interval);
+ case QUARTER:
+ return Period.ofMonths(interval * 3);
+ case YEAR:
+ return Period.ofYears(interval);
+ default:
+ throw new IllegalArgumentException("Unsupported time unit: " + timeUnit);
+ }
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ Interval interval = (Interval) o;
+ return Objects.equals(duration, interval.duration)
+ && Objects.equals(period, interval.period)
+ && timeUnit == interval.timeUnit;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(duration, period, timeUnit);
+ }
+
+ @Override
+ public String toString() {
+ return "INTERVAL '" + getInterval() + "' " + timeUnit;
+ }
+
+ // --------------------------------------------------------------------------------------------
+ // TimeUnit enums
+ // --------------------------------------------------------------------------------------------
+
+ /** An enumeration of time unit representing the unit of interval freshness. */
Review Comment:
Better if we leave freshness out of the docs
```suggestion
/**
* Supported time units for intervals.
*
* <p>Day-time units ({@code SECOND}, {@code MINUTE}, {@code HOUR}, {@code DAY}, {@code WEEK})
* produce {@link Duration}-backed intervals. Year-month units ({@code MONTH}, {@code QUARTER},
* {@code YEAR}) produce {@link Period}-backed intervals.
*/
```
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/Interval.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.catalog;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.table.api.ValidationException;
+
+import java.time.Duration;
+import java.time.Period;
+import java.util.Objects;
+
+/** A class representing day-time and year month intervals. */
+@PublicEvolving
+public class Interval {
+ private final Duration duration;
+ private final Period period;
+ private final TimeUnit timeUnit;
+
+ private Interval(TimeUnit timeUnit, Duration duration, Period period) {
+ this.timeUnit = timeUnit;
+ this.duration = duration;
+ this.period = period;
+ }
+
+ public static Interval of(Duration duration, TimeUnit timeUnit) {
+ return new Interval(timeUnit, duration, null);
+ }
+
+ public static Interval of(Period period, TimeUnit timeUnit) {
+ return new Interval(timeUnit, null, period);
+ }
+
+ public static Interval of(int duration, TimeUnit timeUnit) {
+ if (timeUnit.isDayTime) {
+ return of(toDuration(duration, timeUnit), timeUnit);
+ } else {
+ return of(toPeriod(duration, timeUnit), timeUnit);
+ }
+ }
+
+ public Duration getDuration() {
+ return duration;
+ }
+
+ public Period getPeriod() {
+ return period;
+ }
+
+ public TimeUnit getTimeUnit() {
+ return timeUnit;
+ }
+
+ public int getInterval() {
+ switch (timeUnit) {
+ case SECOND:
+ return (int) duration.toSeconds();
+ case MINUTE:
+ return (int) duration.toMinutes();
+ case HOUR:
+ return (int) duration.toHours();
+ case DAY:
+ return (int) duration.toDays();
+ case WEEK:
+ return (int) (duration.toDays() / 7);
Review Comment:
Should we make a const out of 7? e.g., `DAYS_IN_WEEK`
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/IntervalFreshness.java:
##########
@@ -42,20 +43,29 @@ public class IntervalFreshness {
private static final long MINUTE_CRON_UPPER_BOUND = 60;
private static final long HOUR_CRON_UPPER_BOUND = 24;
- private final int interval;
- private final TimeUnit timeUnit;
+ private final Interval interval;
- private IntervalFreshness(int interval, TimeUnit timeUnit) {
+ public IntervalFreshness(Interval interval) {
this.interval = interval;
- this.timeUnit = timeUnit;
}
public static IntervalFreshness of(String interval, TimeUnit timeUnit) {
- final int validateIntervalInput = validateIntervalInput(interval);
- return new IntervalFreshness(validateIntervalInput, timeUnit);
+ final int validateIntervalInput = validateIntervalStringInput(interval);
+ return new IntervalFreshness(Interval.of(validateIntervalInput, timeUnit));
}
- private static int validateIntervalInput(final String interval) {
+ public static IntervalFreshness of(int intervalDuration, TimeUnit timeUnit) {
+ validateIntervalIntInput(intervalDuration);
+ return new IntervalFreshness(Interval.of(intervalDuration, timeUnit));
+ }
+
+ public static IntervalFreshness of(Duration duration, TimeUnit timeUnit) {
+ // just check it is positive
+ validateIntervalIntInput((int) duration.toSeconds());
Review Comment:
How about we rename the method to what is actually doing?
```suggestion
isIntervalPositive((int) duration.toSeconds());
```
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/utils/MaterializedTableUtils.java:
##########
@@ -68,29 +71,110 @@ public class MaterializedTableUtils {
public static IntervalFreshness getMaterializedTableFreshness(
SqlIntervalLiteral sqlIntervalLiteral) {
- if (sqlIntervalLiteral.signum() < 0) {
- throw new ValidationException(
- "Materialized table freshness doesn't support negative value.");
+ return new IntervalFreshness(getFreshnessInterval(sqlIntervalLiteral));
+ }
+
+ private static Interval getFreshnessInterval(SqlIntervalLiteral sqlIntervalLiteral) {
+ final IntervalValue intervalValue = sqlIntervalLiteral.getValueAs(IntervalValue.class);
+ final SqlTypeName typeName = intervalValue.getIntervalQualifier().typeName();
+
+ if (isDateTimeInterval(typeName)) {
+ final Interval freshnessInterval =
+ getDayTimeInterval(
+ intervalValue,
+ typeName,
+ sqlIntervalLiteral.getValueAs(BigDecimal.class),
+ "freshness");
+ final int interval = freshnessInterval.getInterval();
+ // Freshness interval might be only positive
+ if (interval <= 0) {
+ throw new ValidationException(
+ String.format(
+ "The freshness interval currently only supports positive integer type values. But was: %d",
+ interval));
+ }
Review Comment:
The method in IntervalFreshness can be moved here and be used here. WDYT?
```java
public static void validateIntervalIntInput(final int interval) {
if (interval <= 0) {
throw new ValidationException(
String.format(
"The freshness interval currently only supports positive integer type values. But was: %d",
interval));
}
}
```
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/utils/MaterializedTableUtils.java:
##########
@@ -68,29 +71,110 @@ public class MaterializedTableUtils {
public static IntervalFreshness getMaterializedTableFreshness(
SqlIntervalLiteral sqlIntervalLiteral) {
- if (sqlIntervalLiteral.signum() < 0) {
- throw new ValidationException(
- "Materialized table freshness doesn't support negative value.");
+ return new IntervalFreshness(getFreshnessInterval(sqlIntervalLiteral));
+ }
+
+ private static Interval getFreshnessInterval(SqlIntervalLiteral sqlIntervalLiteral) {
+ final IntervalValue intervalValue = sqlIntervalLiteral.getValueAs(IntervalValue.class);
+ final SqlTypeName typeName = intervalValue.getIntervalQualifier().typeName();
+
+ if (isDateTimeInterval(typeName)) {
+ final Interval freshnessInterval =
+ getDayTimeInterval(
+ intervalValue,
+ typeName,
+ sqlIntervalLiteral.getValueAs(BigDecimal.class),
+ "freshness");
+ final int interval = freshnessInterval.getInterval();
+ // Freshness interval might be only positive
+ if (interval <= 0) {
+ throw new ValidationException(
+ String.format(
+ "The freshness interval currently only supports positive integer type values. But was: %d",
+ interval));
+ }
+ return freshnessInterval;
}
- if (sqlIntervalLiteral.getTypeName().getFamily() != SqlTypeFamily.INTERVAL_DAY_TIME) {
- throw new ValidationException(
- "Materialized table freshness only support SECOND, MINUTE, HOUR, DAY as the time unit.");
+
+ throw new ValidationException(
+ "Materialized table freshness only supports SECOND, MINUTE, HOUR, DAY, WEEK as the time unit.");
+ }
+
+ private static Interval intervalFrom(
Review Comment:
This method is not used anywhere
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/IntervalFreshness.java:
##########
@@ -42,20 +43,29 @@ public class IntervalFreshness {
private static final long MINUTE_CRON_UPPER_BOUND = 60;
private static final long HOUR_CRON_UPPER_BOUND = 24;
- private final int interval;
- private final TimeUnit timeUnit;
+ private final Interval interval;
- private IntervalFreshness(int interval, TimeUnit timeUnit) {
+ public IntervalFreshness(Interval interval) {
this.interval = interval;
- this.timeUnit = timeUnit;
}
Review Comment:
To keep the object creation consistent with static factories I would suggest to make this private and introduce
```java
public static IntervalFreshness of(Interval interval) {
return new IntervalFreshness(interval);
}
```
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/utils/MaterializedTableUtils.java:
##########
@@ -68,29 +71,110 @@ public class MaterializedTableUtils {
public static IntervalFreshness getMaterializedTableFreshness(
SqlIntervalLiteral sqlIntervalLiteral) {
- if (sqlIntervalLiteral.signum() < 0) {
- throw new ValidationException(
- "Materialized table freshness doesn't support negative value.");
+ return new IntervalFreshness(getFreshnessInterval(sqlIntervalLiteral));
+ }
+
+ private static Interval getFreshnessInterval(SqlIntervalLiteral sqlIntervalLiteral) {
+ final IntervalValue intervalValue = sqlIntervalLiteral.getValueAs(IntervalValue.class);
+ final SqlTypeName typeName = intervalValue.getIntervalQualifier().typeName();
+
+ if (isDateTimeInterval(typeName)) {
+ final Interval freshnessInterval =
+ getDayTimeInterval(
+ intervalValue,
+ typeName,
+ sqlIntervalLiteral.getValueAs(BigDecimal.class),
+ "freshness");
+ final int interval = freshnessInterval.getInterval();
+ // Freshness interval might be only positive
+ if (interval <= 0) {
+ throw new ValidationException(
+ String.format(
+ "The freshness interval currently only supports positive integer type values. But was: %d",
+ interval));
+ }
+ return freshnessInterval;
}
- if (sqlIntervalLiteral.getTypeName().getFamily() != SqlTypeFamily.INTERVAL_DAY_TIME) {
- throw new ValidationException(
- "Materialized table freshness only support SECOND, MINUTE, HOUR, DAY as the time unit.");
+
+ throw new ValidationException(
+ "Materialized table freshness only supports SECOND, MINUTE, HOUR, DAY, WEEK as the time unit.");
+ }
+
+ private static Interval intervalFrom(
+ SqlIntervalLiteral sqlIntervalLiteral, String intervalDescription) {
+
+ final IntervalValue intervalValue = sqlIntervalLiteral.getValueAs(IntervalValue.class);
+ final SqlTypeName typeName = intervalValue.getIntervalQualifier().typeName();
+ if (intervalValue.getIntervalQualifier().isYearMonth()) {
+ return getYearMonthInterval(
+ intervalValue,
+ typeName,
+ sqlIntervalLiteral.getValueAs(BigDecimal.class),
+ intervalDescription);
+ }
+
+ if (isDateTimeInterval(typeName)) {
Review Comment:
You do the validation of the type two times here. So the `getDayTimeInterval` default branch will be never executed. We should remove it from here.
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/utils/MaterializedTableUtils.java:
##########
@@ -68,29 +71,110 @@ public class MaterializedTableUtils {
public static IntervalFreshness getMaterializedTableFreshness(
SqlIntervalLiteral sqlIntervalLiteral) {
- if (sqlIntervalLiteral.signum() < 0) {
- throw new ValidationException(
- "Materialized table freshness doesn't support negative value.");
+ return new IntervalFreshness(getFreshnessInterval(sqlIntervalLiteral));
+ }
+
+ private static Interval getFreshnessInterval(SqlIntervalLiteral sqlIntervalLiteral) {
+ final IntervalValue intervalValue = sqlIntervalLiteral.getValueAs(IntervalValue.class);
+ final SqlTypeName typeName = intervalValue.getIntervalQualifier().typeName();
+
+ if (isDateTimeInterval(typeName)) {
Review Comment:
This only checks for DAY, HOUR, MINUTE, SECOND. But the validation message in the else mentions also WEEK.
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/Interval.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.catalog;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.table.api.ValidationException;
+
+import java.time.Duration;
+import java.time.Period;
+import java.util.Objects;
+
+/** A class representing day-time and year month intervals. */
+@PublicEvolving
+public class Interval {
+ private final Duration duration;
+ private final Period period;
+ private final TimeUnit timeUnit;
+
+ private Interval(TimeUnit timeUnit, Duration duration, Period period) {
+ this.timeUnit = timeUnit;
+ this.duration = duration;
+ this.period = period;
+ }
+
+ public static Interval of(Duration duration, TimeUnit timeUnit) {
+ return new Interval(timeUnit, duration, null);
+ }
+
+ public static Interval of(Period period, TimeUnit timeUnit) {
+ return new Interval(timeUnit, null, period);
+ }
+
+ public static Interval of(int duration, TimeUnit timeUnit) {
+ if (timeUnit.isDayTime) {
+ return of(toDuration(duration, timeUnit), timeUnit);
+ } else {
+ return of(toPeriod(duration, timeUnit), timeUnit);
+ }
+ }
+
+ public Duration getDuration() {
+ return duration;
+ }
+
+ public Period getPeriod() {
+ return period;
+ }
+
+ public TimeUnit getTimeUnit() {
+ return timeUnit;
+ }
+
+ public int getInterval() {
+ switch (timeUnit) {
+ case SECOND:
+ return (int) duration.toSeconds();
+ case MINUTE:
+ return (int) duration.toMinutes();
+ case HOUR:
+ return (int) duration.toHours();
+ case DAY:
+ return (int) duration.toDays();
+ case WEEK:
+ return (int) (duration.toDays() / 7);
+ case MONTH:
+ return period.getMonths();
+ case QUARTER:
+ return period.getMonths() / 3;
Review Comment:
Extract to const `MONTHS_IN_QUARTER`
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/utils/MaterializedTableUtils.java:
##########
@@ -68,29 +71,110 @@ public class MaterializedTableUtils {
public static IntervalFreshness getMaterializedTableFreshness(
SqlIntervalLiteral sqlIntervalLiteral) {
- if (sqlIntervalLiteral.signum() < 0) {
- throw new ValidationException(
- "Materialized table freshness doesn't support negative value.");
+ return new IntervalFreshness(getFreshnessInterval(sqlIntervalLiteral));
+ }
+
+ private static Interval getFreshnessInterval(SqlIntervalLiteral sqlIntervalLiteral) {
+ final IntervalValue intervalValue = sqlIntervalLiteral.getValueAs(IntervalValue.class);
+ final SqlTypeName typeName = intervalValue.getIntervalQualifier().typeName();
+
+ if (isDateTimeInterval(typeName)) {
+ final Interval freshnessInterval =
+ getDayTimeInterval(
+ intervalValue,
+ typeName,
+ sqlIntervalLiteral.getValueAs(BigDecimal.class),
+ "freshness");
+ final int interval = freshnessInterval.getInterval();
+ // Freshness interval might be only positive
+ if (interval <= 0) {
+ throw new ValidationException(
+ String.format(
+ "The freshness interval currently only supports positive integer type values. But was: %d",
+ interval));
+ }
+ return freshnessInterval;
}
- if (sqlIntervalLiteral.getTypeName().getFamily() != SqlTypeFamily.INTERVAL_DAY_TIME) {
- throw new ValidationException(
- "Materialized table freshness only support SECOND, MINUTE, HOUR, DAY as the time unit.");
+
+ throw new ValidationException(
+ "Materialized table freshness only supports SECOND, MINUTE, HOUR, DAY, WEEK as the time unit.");
+ }
+
+ private static Interval intervalFrom(
+ SqlIntervalLiteral sqlIntervalLiteral, String intervalDescription) {
+
+ final IntervalValue intervalValue = sqlIntervalLiteral.getValueAs(IntervalValue.class);
+ final SqlTypeName typeName = intervalValue.getIntervalQualifier().typeName();
+ if (intervalValue.getIntervalQualifier().isYearMonth()) {
+ return getYearMonthInterval(
+ intervalValue,
+ typeName,
+ sqlIntervalLiteral.getValueAs(BigDecimal.class),
+ intervalDescription);
+ }
+
+ if (isDateTimeInterval(typeName)) {
+ return getDayTimeInterval(
+ intervalValue,
+ typeName,
+ sqlIntervalLiteral.getValueAs(BigDecimal.class),
+ intervalDescription);
}
- IntervalValue intervalValue = sqlIntervalLiteral.getValueAs(IntervalValue.class);
- String interval = intervalValue.getIntervalLiteral();
- switch (intervalValue.getIntervalQualifier().typeName()) {
+ throw new ValidationException(
+ String.format(
+ "Materialized table %s only supports SECOND, MINUTE, HOUR, DAY, WEEK, MONTH, QUARTER, YEAR as the time unit.",
+ intervalDescription));
Review Comment:
This message correct in this place?
```suggestion
String.format(
"Materialized table %s only supports SECOND, MINUTE, HOUR, DAY as the time unit.",
intervalDescription));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39348][table] Add `Interval` to reuse code for `FRESHESS` and `START MODE` [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #27845:
URL: https://github.com/apache/flink/pull/27845#issuecomment-4148384140
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "566bed20ddbae2ecf0bf2685cf2beeb923e87fb3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "566bed20ddbae2ecf0bf2685cf2beeb923e87fb3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 566bed20ddbae2ecf0bf2685cf2beeb923e87fb3 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39348][table] Add `Interval` to reuse code for `FRESHNESS` and `START MODE` [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #27845:
URL: https://github.com/apache/flink/pull/27845#discussion_r3008536584
##########
flink-table/flink-table-api-java/src/test/java/org/apache/flink/table/api/internal/ShowCreateUtilTest.java:
##########
@@ -313,7 +313,7 @@ private static Collection<Arguments> argsForShowCreateMaterializedTable() {
null,
List.of(),
null,
- IntervalFreshness.ofMinute("1"),
+ IntervalFreshness.ofMinute(1),
Review Comment:
What do you mean by this?
can you give an example?
Also I'm not a fan of lots of comments especially in case of non public code or tests, because people might forget to update them updating/refactoring the actual code... (just my opinion)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39348][table] Add `Interval` to reuse code for `FRESHNESS` and `START MODE` [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #27845:
URL: https://github.com/apache/flink/pull/27845#discussion_r3011754240
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/utils/MaterializedTableUtils.java:
##########
@@ -68,29 +71,110 @@ public class MaterializedTableUtils {
public static IntervalFreshness getMaterializedTableFreshness(
SqlIntervalLiteral sqlIntervalLiteral) {
- if (sqlIntervalLiteral.signum() < 0) {
- throw new ValidationException(
- "Materialized table freshness doesn't support negative value.");
+ return new IntervalFreshness(getFreshnessInterval(sqlIntervalLiteral));
+ }
+
+ private static Interval getFreshnessInterval(SqlIntervalLiteral sqlIntervalLiteral) {
+ final IntervalValue intervalValue = sqlIntervalLiteral.getValueAs(IntervalValue.class);
+ final SqlTypeName typeName = intervalValue.getIntervalQualifier().typeName();
+
+ if (isDateTimeInterval(typeName)) {
+ final Interval freshnessInterval =
+ getDayTimeInterval(
+ intervalValue,
+ typeName,
+ sqlIntervalLiteral.getValueAs(BigDecimal.class),
+ "freshness");
+ final int interval = freshnessInterval.getInterval();
+ // Freshness interval might be only positive
+ if (interval <= 0) {
+ throw new ValidationException(
+ String.format(
+ "The freshness interval currently only supports positive integer type values. But was: %d",
+ interval));
+ }
Review Comment:
we can not reuse it in intervalfreshness if we move it
we can have a copy of that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39348][table] Add `Interval` to reuse code for `FRESHNESS` and `START MODE` [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #27845:
URL: https://github.com/apache/flink/pull/27845#discussion_r3011756139
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/IntervalFreshness.java:
##########
@@ -42,20 +43,29 @@ public class IntervalFreshness {
private static final long MINUTE_CRON_UPPER_BOUND = 60;
private static final long HOUR_CRON_UPPER_BOUND = 24;
- private final int interval;
- private final TimeUnit timeUnit;
+ private final Interval interval;
- private IntervalFreshness(int interval, TimeUnit timeUnit) {
+ public IntervalFreshness(Interval interval) {
this.interval = interval;
- this.timeUnit = timeUnit;
}
public static IntervalFreshness of(String interval, TimeUnit timeUnit) {
- final int validateIntervalInput = validateIntervalInput(interval);
- return new IntervalFreshness(validateIntervalInput, timeUnit);
+ final int validateIntervalInput = validateIntervalStringInput(interval);
+ return new IntervalFreshness(Interval.of(validateIntervalInput, timeUnit));
}
- private static int validateIntervalInput(final String interval) {
+ public static IntervalFreshness of(int intervalDuration, TimeUnit timeUnit) {
+ validateIntervalIntInput(intervalDuration);
+ return new IntervalFreshness(Interval.of(intervalDuration, timeUnit));
+ }
+
+ public static IntervalFreshness of(Duration duration, TimeUnit timeUnit) {
+ // just check it is positive
+ validateIntervalIntInput((int) duration.toSeconds());
Review Comment:
renamed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39348][table] Add `Interval` to reuse code for `FRESHNESS` and `START MODE` [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on PR #27845:
URL: https://github.com/apache/flink/pull/27845#issuecomment-4161609567
Thanks for the feedback, merging
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org