You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "harshmotw-db (via GitHub)" <gi...@apache.org> on 2024/07/24 20:01:50 UTC
[PR] [SPARK-45891][SQL][PySpark][VARIANT] Add support for interval types in the Variant Spec [spark]
harshmotw-db opened a new pull request, #47473:
URL: https://github.com/apache/spark/pull/47473
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
This PR adds support for the [YearMonthIntervalType](https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/types/YearMonthIntervalType.html) and [DayTimeIntervalType](https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/types/DayTimeIntervalType.html) as new primitive types in the Variant spec. As part of this task, the PR adds support for casting between intervals and variants and support for interval types in all the relevant variant expressions. This PR also adds support for these types on the PySpark side.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
The variant spec should be compatible with all SQL Standard data types.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Yes, it allows users to cast interval types to variants and vice versa.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
Unit tests in VariantExpressionSuite.scala and test_types.py
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
Yes, I used perplexity.ai to get guidance on converting some Scala code to Java code and Java code to Python code.
Generated-by: perplexity.ai
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PYTHON][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "harshmotw-db (via GitHub)" <gi...@apache.org>.
harshmotw-db commented on PR #47473:
URL: https://github.com/apache/spark/pull/47473#issuecomment-2294330037
Hi @LuciferYang I have made your requested changes in [this PR](https://github.com/apache/spark/pull/47792).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PySpark][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "chenhao-db (via GitHub)" <gi...@apache.org>.
chenhao-db commented on code in PR #47473:
URL: https://github.com/apache/spark/pull/47473#discussion_r1690378462
##########
common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java:
##########
@@ -377,11 +402,51 @@ public static long getLong(byte[] value, int pos) {
case TIMESTAMP:
case TIMESTAMP_NTZ:
return readLong(value, pos + 1, 8);
+ case YEAR_MONTH_INTERVAL:
+ return readLong(value, pos + 2, 4);
+ case DAY_TIME_INTERVAL:
+ return readLong(value, pos + 2, 8);
default:
throw new IllegalStateException(exceptionMessage);
}
}
+ public static class IntervalFields {
+ public IntervalFields(byte startField, byte endField) {
+ this.startField = startField;
+ this.endField = endField;
+ }
+
+ public byte startField;
Review Comment:
Can be `final`.
##########
common/variant/src/main/java/org/apache/spark/types/variant/Variant.java:
##########
@@ -316,6 +334,22 @@ static void toJsonImpl(byte[] value, byte[] metadata, int pos, StringBuilder sb,
case BINARY:
appendQuoted(sb, Base64.getEncoder().encodeToString(VariantUtil.getBinary(value, pos)));
break;
+ case YEAR_MONTH_INTERVAL:
+ IntervalFields ymFields = VariantUtil.getYearMonthIntervalFields(value, pos);
+ int ymValue = (int) VariantUtil.getLong(value, pos);
+ appendQuoted(sb, YearMonthIntervalUtils
+ .toYearMonthIntervalANSIString(ymValue, ymFields.startField, ymFields.endField));
+ break;
+ case DAY_TIME_INTERVAL:
+ IntervalFields dtFields = VariantUtil.getDayTimeIntervalFields(value, pos);
+ long dtValue = VariantUtil.getLong(value, pos);
+ try {
Review Comment:
Why is this block in a `try` statement, but the year-month code is not?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PYTHON][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "harshmotw-db (via GitHub)" <gi...@apache.org>.
harshmotw-db commented on code in PR #47473:
URL: https://github.com/apache/spark/pull/47473#discussion_r1690728245
##########
common/utils/src/main/scala/org/apache/spark/util/DayTimeIntervalUtils.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util;
+
+import org.apache.spark.SparkException;
+
+import java.math.BigDecimal;
+import java.util.ArrayList;
+
+// Replicating code from SparkIntervalUtils so code in the 'common' space can work with
Review Comment:
I agree. However, I believe that should be a different PR as it would be a big change. For the purposes of this PR, these functions only support ANSIStyle while the functions on SQL also expect Hive style sometimes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PYTHON][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "harshmotw-db (via GitHub)" <gi...@apache.org>.
harshmotw-db commented on code in PR #47473:
URL: https://github.com/apache/spark/pull/47473#discussion_r1690728245
##########
common/utils/src/main/scala/org/apache/spark/util/DayTimeIntervalUtils.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util;
+
+import org.apache.spark.SparkException;
+
+import java.math.BigDecimal;
+import java.util.ArrayList;
+
+// Replicating code from SparkIntervalUtils so code in the 'common' space can work with
Review Comment:
I agree. However, I believe that should be a different PR as it would be a big change in itself. For the purposes of this PR, these functions only support ANSIStyle while the functions on SQL also expect Hive style sometimes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PYTHON][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #47473:
URL: https://github.com/apache/spark/pull/47473#discussion_r1690727975
##########
common/variant/src/main/java/org/apache/spark/types/variant/Variant.java:
##########
@@ -316,6 +334,22 @@ static void toJsonImpl(byte[] value, byte[] metadata, int pos, StringBuilder sb,
case BINARY:
appendQuoted(sb, Base64.getEncoder().encodeToString(VariantUtil.getBinary(value, pos)));
break;
+ case YEAR_MONTH_INTERVAL:
+ IntervalFields ymFields = VariantUtil.getYearMonthIntervalFields(value, pos);
+ int ymValue = (int) VariantUtil.getLong(value, pos);
Review Comment:
shouldn't we call `getInt`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PYTHON][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #47473:
URL: https://github.com/apache/spark/pull/47473#discussion_r1694436695
##########
common/utils/src/main/scala/org/apache/spark/util/DayTimeIntervalUtils.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util;
+
+import org.apache.spark.SparkException;
+
+import java.math.BigDecimal;
+import java.util.ArrayList;
+
+// Replicating code from SparkIntervalUtils so code in the 'common' space can work with
+// year-month intervals.
+public class DayTimeIntervalUtils {
+ private static byte DAY = 0;
+ private static byte HOUR = 1;
+ private static byte MINUTE = 2;
+ private static byte SECOND = 3;
+ private static long HOURS_PER_DAY = 24;
+ private static long MINUTES_PER_HOUR = 60;
+ private static long SECONDS_PER_MINUTE = 60;
+ private static long MILLIS_PER_SECOND = 1000;
+ private static long MICROS_PER_MILLIS = 1000;
+ private static long MICROS_PER_SECOND = MICROS_PER_MILLIS * MILLIS_PER_SECOND;
+ private static long MICROS_PER_MINUTE = SECONDS_PER_MINUTE * MICROS_PER_SECOND;
+ private static long MICROS_PER_HOUR = MINUTES_PER_HOUR * MICROS_PER_MINUTE;
+ private static long MICROS_PER_DAY = HOURS_PER_DAY * MICROS_PER_HOUR;
+ private static long MAX_DAY = Long.MAX_VALUE / MICROS_PER_DAY;
+ private static long MAX_HOUR = Long.MAX_VALUE / MICROS_PER_HOUR;
+ private static long MAX_MINUTE = Long.MAX_VALUE / MICROS_PER_MINUTE;
+ private static long MAX_SECOND = Long.MAX_VALUE / MICROS_PER_SECOND;
+
+ public static String fieldToString(byte field) throws SparkException {
+ if (field == DAY) {
+ return "DAY";
+ } else if (field == HOUR) {
+ return "HOUR";
+ } else if (field == MINUTE) {
+ return "MINUTE";
+ } else if (field == SECOND) {
+ return "SECOND";
+ } else {
+ throw new SparkException("Invalid field in day-time interval: " + field +
+ ". Supported fields are: DAY, HOUR, MINUTE, SECOND");
+ }
+ }
+
+ // Used to convert microseconds representing a day-time interval with given start and end fields
+ // to its ANSI SQL string representation. Throws a SparkException if startField or endField are
+ // out of bounds.
+ public static String toDayTimeIntervalANSIString(long micros, byte startField, byte endField)
+ throws SparkException {
+ String sign = "";
+ long rest = micros;
+ try {
+ String from = fieldToString(startField).toUpperCase();
+ String to = fieldToString(endField).toUpperCase();
+ String prefix = "INTERVAL '";
+ String postfix = startField == endField ? "' " + from : "' " + from + " TO " + to;
+ if (micros < 0) {
+ if (micros == Long.MIN_VALUE) {
+ // Especial handling of minimum `Long` value because negate op overflows `Long`.
+ // seconds = 106751991 * (24 * 60 * 60) + 4 * 60 * 60 + 54 = 9223372036854
+ // microseconds = -9223372036854000000L-775808 == Long.MinValue
+ String baseStr = "-106751991 04:00:54.775808000";
+ String firstStr = "-" + (startField == DAY ? Long.toString(MAX_DAY) :
+ (startField == HOUR ? Long.toString(MAX_HOUR) :
+ (startField == MINUTE ? Long.toString(MAX_MINUTE) :
+ Long.toString(MAX_SECOND) + ".775808")));
+ if (startField == endField) {
+ return prefix + firstStr + postfix;
+ } else {
+ int substrStart = startField == DAY ? 10 : (startField == HOUR ? 13 : 16);
+ int substrEnd = endField == HOUR ? 13 : (endField == MINUTE ? 16 : 26);
+ return prefix + firstStr + baseStr.substring(substrStart, substrEnd) + postfix;
+ }
+ } else {
+ sign = "-";
+ rest = -rest;
+ }
+ }
+ StringBuilder formatBuilder = new StringBuilder(sign);
+ ArrayList<Long> formatArgs = new ArrayList<>();
+ if (startField == DAY) {
+ formatBuilder.append(rest / MICROS_PER_DAY);
+ rest %= MICROS_PER_DAY;
+ } else if (startField == HOUR) {
+ formatBuilder.append("%02d");
+ formatArgs.add(rest / MICROS_PER_HOUR);
+ rest %= MICROS_PER_HOUR;
+ } else if (startField == MINUTE) {
+ formatBuilder.append("%02d");
+ formatArgs.add(rest / MICROS_PER_MINUTE);
+ rest %= MICROS_PER_MINUTE;
+ } else if (startField == SECOND) {
+ String leadZero = rest < 10 * MICROS_PER_SECOND ? "0" : "";
+ formatBuilder.append(leadZero + BigDecimal.valueOf(rest, 6)
+ .stripTrailingZeros().toPlainString());
+ }
+
+ if (startField < HOUR && HOUR <= endField) {
+ formatBuilder.append(" %02d");
+ formatArgs.add(rest / MICROS_PER_HOUR);
+ rest %= MICROS_PER_HOUR;
+ }
+ if (startField < MINUTE && MINUTE <= endField) {
+ formatBuilder.append(":%02d");
+ formatArgs.add(rest / MICROS_PER_MINUTE);
+ rest %= MICROS_PER_MINUTE;
+ }
+ if (startField < SECOND && SECOND <= endField) {
+ String leadZero = rest < 10 * MICROS_PER_SECOND ? "0" : "";
+ formatBuilder.append(":" + leadZero + BigDecimal.valueOf(rest, 6)
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PySpark][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "harshmotw-db (via GitHub)" <gi...@apache.org>.
harshmotw-db commented on code in PR #47473:
URL: https://github.com/apache/spark/pull/47473#discussion_r1690393525
##########
common/variant/src/main/java/org/apache/spark/types/variant/Variant.java:
##########
@@ -316,6 +334,22 @@ static void toJsonImpl(byte[] value, byte[] metadata, int pos, StringBuilder sb,
case BINARY:
appendQuoted(sb, Base64.getEncoder().encodeToString(VariantUtil.getBinary(value, pos)));
break;
+ case YEAR_MONTH_INTERVAL:
+ IntervalFields ymFields = VariantUtil.getYearMonthIntervalFields(value, pos);
+ int ymValue = (int) VariantUtil.getLong(value, pos);
+ appendQuoted(sb, YearMonthIntervalUtils
+ .toYearMonthIntervalANSIString(ymValue, ymFields.startField, ymFields.endField));
+ break;
+ case DAY_TIME_INTERVAL:
+ IntervalFields dtFields = VariantUtil.getDayTimeIntervalFields(value, pos);
+ long dtValue = VariantUtil.getLong(value, pos);
+ try {
Review Comment:
It can throw an exception when the field ranges are out of bounds. That being said, `getDayTimeIntervalFields` ensures that they are always within bounds the try block is unnecessary. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PYTHON][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #47473:
URL: https://github.com/apache/spark/pull/47473#issuecomment-2253782672
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PYTHON][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "harshmotw-db (via GitHub)" <gi...@apache.org>.
harshmotw-db commented on PR #47473:
URL: https://github.com/apache/spark/pull/47473#issuecomment-2256405172
@LuciferYang Thanks for the comments! I'll address them in a future follow up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PySpark][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on PR #47473:
URL: https://github.com/apache/spark/pull/47473#issuecomment-2249092323
nit: `[PySpark]` in the title should be `[PYTHON]`. You can see the full list of PR categories from https://spark-prs.appspot.com/ :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PYTHON][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "harshmotw-db (via GitHub)" <gi...@apache.org>.
harshmotw-db commented on code in PR #47473:
URL: https://github.com/apache/spark/pull/47473#discussion_r1691939585
##########
common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java:
##########
@@ -377,11 +405,52 @@ public static long getLong(byte[] value, int pos) {
case TIMESTAMP:
case TIMESTAMP_NTZ:
return readLong(value, pos + 1, 8);
+ case YEAR_MONTH_INTERVAL:
+ return readLong(value, pos + 2, 4);
+ case DAY_TIME_INTERVAL:
+ return readLong(value, pos + 2, 8);
default:
throw new IllegalStateException(exceptionMessage);
}
}
+ // Class used to pass around start and end fields of year-month and day-time interval values.
+ public static class IntervalFields {
+ public IntervalFields(byte startField, byte endField) {
+ this.startField = startField;
+ this.endField = endField;
+ }
+
+ public final byte startField;
+ public final byte endField;
+ }
+
+ // Get the start and end fields of a variant value representing a year-month interval value. The
+ // returned array contains the start field at the zeroth index and the end field at the first
+ // index.
+ public static IntervalFields getYearMonthIntervalFields(byte[] value, int pos) {
+ long fieldInfo = readLong(value, pos + 1, 1);
+ IntervalFields intervalFields = new IntervalFields((byte) (fieldInfo & 0x1),
+ (byte) ((fieldInfo >> 1) & 0x1));
+ if (intervalFields.endField < intervalFields.startField) {
Review Comment:
Yes, that would have been a good idea. `checkIndex` is performed by `readLong` anyway though. Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PYTHON][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #47473: [SPARK-45891][SQL][PYTHON][VARIANT] Add support for interval types in the Variant Spec
URL: https://github.com/apache/spark/pull/47473
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PySpark][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "harshmotw-db (via GitHub)" <gi...@apache.org>.
harshmotw-db commented on code in PR #47473:
URL: https://github.com/apache/spark/pull/47473#discussion_r1690384029
##########
common/variant/src/main/java/org/apache/spark/types/variant/Variant.java:
##########
@@ -316,6 +334,22 @@ static void toJsonImpl(byte[] value, byte[] metadata, int pos, StringBuilder sb,
case BINARY:
appendQuoted(sb, Base64.getEncoder().encodeToString(VariantUtil.getBinary(value, pos)));
break;
+ case YEAR_MONTH_INTERVAL:
+ IntervalFields ymFields = VariantUtil.getYearMonthIntervalFields(value, pos);
+ int ymValue = (int) VariantUtil.getLong(value, pos);
+ appendQuoted(sb, YearMonthIntervalUtils
+ .toYearMonthIntervalANSIString(ymValue, ymFields.startField, ymFields.endField));
+ break;
+ case DAY_TIME_INTERVAL:
+ IntervalFields dtFields = VariantUtil.getDayTimeIntervalFields(value, pos);
+ long dtValue = VariantUtil.getLong(value, pos);
+ try {
Review Comment:
`toDayTimeIntervalANSIString` can throw an exception while `toYearMonthIntervalANSIString` cannot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PYTHON][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #47473:
URL: https://github.com/apache/spark/pull/47473#discussion_r1694437591
##########
common/utils/src/main/scala/org/apache/spark/util/DayTimeIntervalUtils.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util;
+
+import org.apache.spark.SparkException;
+
+import java.math.BigDecimal;
Review Comment:
The import order rule in the newly added Java file should be consistent with that of the Scala files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PYTHON][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #47473:
URL: https://github.com/apache/spark/pull/47473#discussion_r1690726878
##########
common/utils/src/main/scala/org/apache/spark/util/DayTimeIntervalUtils.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util;
+
+import org.apache.spark.SparkException;
+
+import java.math.BigDecimal;
+import java.util.ArrayList;
+
+// Replicating code from SparkIntervalUtils so code in the 'common' space can work with
Review Comment:
why replicating? I think other modules depend on `common/utils` and it's fine we move interval utils to this module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PYTHON][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "harshmotw-db (via GitHub)" <gi...@apache.org>.
harshmotw-db commented on code in PR #47473:
URL: https://github.com/apache/spark/pull/47473#discussion_r1691852875
##########
common/variant/src/main/java/org/apache/spark/types/variant/Variant.java:
##########
@@ -113,6 +126,11 @@ public String getString() {
return VariantUtil.getString(value, pos);
}
+ // Get the type info bits from a variant value.
+ public int getTypeInfo() {
Review Comment:
It is used when throwing an `UNKNOWN_PRIMITIVE_TYPE_IN_VARIANT` exception from scopes which don't directly have access to the type info.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PySpark][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "harshmotw-db (via GitHub)" <gi...@apache.org>.
harshmotw-db commented on code in PR #47473:
URL: https://github.com/apache/spark/pull/47473#discussion_r1690573509
##########
common/variant/README.md:
##########
@@ -335,27 +335,29 @@ The Decimal type contains a scale, but no precision. The implied precision of a
| Object | `2` | A collection of (string-key, variant-value) pairs |
| Array | `3` | An ordered sequence of variant values |
-| Primitive Type | Type ID | Equivalent Parquet Type | Binary format |
-|-----------------------------|---------|---------------------------|-----------------------------------------------------------------------------------------------------------|
-| null | `0` | any | none |
-| boolean (True) | `1` | BOOLEAN | none |
-| boolean (False) | `2` | BOOLEAN | none |
-| int8 | `3` | INT(8, signed) | 1 byte |
-| int16 | `4` | INT(16, signed) | 2 byte little-endian |
-| int32 | `5` | INT(32, signed) | 4 byte little-endian |
-| int64 | `6` | INT(64, signed) | 8 byte little-endian |
-| double | `7` | DOUBLE | IEEE little-endian |
-| decimal4 | `8` | DECIMAL(precision, scale) | 1 byte scale in range [0, 38], followed by little-endian unscaled value (see decimal table) |
-| decimal8 | `9` | DECIMAL(precision, scale) | 1 byte scale in range [0, 38], followed by little-endian unscaled value (see decimal table) |
-| decimal16 | `10` | DECIMAL(precision, scale) | 1 byte scale in range [0, 38], followed by little-endian unscaled value (see decimal table) |
-| date | `11` | DATE | 4 byte little-endian |
-| timestamp | `12` | TIMESTAMP(true, MICROS) | 8-byte little-endian |
-| timestamp without time zone | `13` | TIMESTAMP(false, MICROS) | 8-byte little-endian |
-| float | `14` | FLOAT | IEEE little-endian |
-| binary | `15` | BINARY | 4 byte little-endian size, followed by bytes |
-| string | `16` | STRING | 4 byte little-endian size, followed by UTF-8 encoded bytes |
-| binary from metadata | `17` | BINARY | Little-endian index into the metadata dictionary. Number of bytes is equal to the metadata `offset_size`. |
-| string from metadata | `18` | STRING | Little-endian index into the metadata dictionary. Number of bytes is equal to the metadata `offset_size`. |
+| Primitive Type | Type ID | Equivalent Parquet Type | Binary format |
+|-----------------------------|---------|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------------|
+| null | `0` | any | none |
+| boolean (True) | `1` | BOOLEAN | none |
+| boolean (False) | `2` | BOOLEAN | none |
+| int8 | `3` | INT(8, signed) | 1 byte |
+| int16 | `4` | INT(16, signed) | 2 byte little-endian |
+| int32 | `5` | INT(32, signed) | 4 byte little-endian |
+| int64 | `6` | INT(64, signed) | 8 byte little-endian |
+| double | `7` | DOUBLE | IEEE little-endian |
+| decimal4 | `8` | DECIMAL(precision, scale) | 1 byte scale in range [0, 38], followed by little-endian unscaled value (see decimal table) |
+| decimal8 | `9` | DECIMAL(precision, scale) | 1 byte scale in range [0, 38], followed by little-endian unscaled value (see decimal table) |
+| decimal16 | `10` | DECIMAL(precision, scale) | 1 byte scale in range [0, 38], followed by little-endian unscaled value (see decimal table) |
+| date | `11` | DATE | 4 byte little-endian |
+| timestamp | `12` | TIMESTAMP(true, MICROS) | 8-byte little-endian |
+| timestamp without time zone | `13` | TIMESTAMP(false, MICROS) | 8-byte little-endian |
+| float | `14` | FLOAT | IEEE little-endian |
+| binary | `15` | BINARY | 4 byte little-endian size, followed by bytes |
+| string | `16` | STRING | 4 byte little-endian size, followed by UTF-8 encoded bytes |
+| binary from metadata | `17` | BINARY | Little-endian index into the metadata dictionary. Number of bytes is equal to the metadata `offset_size`. |
+| string from metadata | `18` | STRING | Little-endian index into the metadata dictionary. Number of bytes is equal to the metadata `offset_size`. |
+| year-month interval | `19` | YearMonthIntervalType(start_field, end_field) | 1 byte denoting start field (1 bit) and end field (1 bit) starting at LSB followed by 4-byte little-endian value. |
Review Comment:
I had mistakenly put in the equivalent spark types here earlier. I have removed the parquet types for now as I am investigating the parquet types.
The details about the start and end field are in a paragraph after this table in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PySpark][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "harshmotw-db (via GitHub)" <gi...@apache.org>.
harshmotw-db commented on code in PR #47473:
URL: https://github.com/apache/spark/pull/47473#discussion_r1690393525
##########
common/variant/src/main/java/org/apache/spark/types/variant/Variant.java:
##########
@@ -316,6 +334,22 @@ static void toJsonImpl(byte[] value, byte[] metadata, int pos, StringBuilder sb,
case BINARY:
appendQuoted(sb, Base64.getEncoder().encodeToString(VariantUtil.getBinary(value, pos)));
break;
+ case YEAR_MONTH_INTERVAL:
+ IntervalFields ymFields = VariantUtil.getYearMonthIntervalFields(value, pos);
+ int ymValue = (int) VariantUtil.getLong(value, pos);
+ appendQuoted(sb, YearMonthIntervalUtils
+ .toYearMonthIntervalANSIString(ymValue, ymFields.startField, ymFields.endField));
+ break;
+ case DAY_TIME_INTERVAL:
+ IntervalFields dtFields = VariantUtil.getDayTimeIntervalFields(value, pos);
+ long dtValue = VariantUtil.getLong(value, pos);
+ try {
Review Comment:
It can throw an exception when the field ranges are out of bounds. That being said, `getDayTimeIntervalFields` ensures that they are always within bounds so the try block is unnecessary. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PySpark][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "chenhao-db (via GitHub)" <gi...@apache.org>.
chenhao-db commented on code in PR #47473:
URL: https://github.com/apache/spark/pull/47473#discussion_r1690387596
##########
common/variant/src/main/java/org/apache/spark/types/variant/Variant.java:
##########
@@ -316,6 +334,22 @@ static void toJsonImpl(byte[] value, byte[] metadata, int pos, StringBuilder sb,
case BINARY:
appendQuoted(sb, Base64.getEncoder().encodeToString(VariantUtil.getBinary(value, pos)));
break;
+ case YEAR_MONTH_INTERVAL:
+ IntervalFields ymFields = VariantUtil.getYearMonthIntervalFields(value, pos);
+ int ymValue = (int) VariantUtil.getLong(value, pos);
+ appendQuoted(sb, YearMonthIntervalUtils
+ .toYearMonthIntervalANSIString(ymValue, ymFields.startField, ymFields.endField));
+ break;
+ case DAY_TIME_INTERVAL:
+ IntervalFields dtFields = VariantUtil.getDayTimeIntervalFields(value, pos);
+ long dtValue = VariantUtil.getLong(value, pos);
+ try {
Review Comment:
Why `toDayTimeIntervalANSIString` can throw an exception? I don't think it can. If it really can, maybe need some explanation and comments on the function itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PySpark][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "gene-db (via GitHub)" <gi...@apache.org>.
gene-db commented on code in PR #47473:
URL: https://github.com/apache/spark/pull/47473#discussion_r1690470525
##########
common/utils/src/main/resources/error/error-conditions.json:
##########
@@ -4186,6 +4186,12 @@
],
"sqlState" : "42846"
},
+ "UNKNOWN_PRIMITIVE_TYPE_IN_VARIANT" : {
+ "message" : [
+ "Unknown primitive type with id <id> was found in a variant value. The type might be supported in a newer version."
Review Comment:
I think we can remove the second part.
```suggestion
"Unknown primitive type with id <id> was found in a variant value."
```
##########
common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:
##########
@@ -214,6 +214,22 @@ public void appendTimestampNtz(long microsSinceEpoch) {
writePos += 8;
}
+ public void appendYearMonthInterval(long value, byte startField, byte endField) {
+ checkCapacity(1 + 5);
+ writeBuffer[writePos++] = primitiveHeader(YEAR_MONTH_INTERVAL);
+ writeBuffer[writePos++] = (byte) (startField | (endField << 1));
Review Comment:
Should we mask `startField` and `endField` to make sure we are only using the desired bits?
##########
common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java:
##########
@@ -377,11 +402,51 @@ public static long getLong(byte[] value, int pos) {
case TIMESTAMP:
case TIMESTAMP_NTZ:
return readLong(value, pos + 1, 8);
+ case YEAR_MONTH_INTERVAL:
Review Comment:
Oh, I see how you updated `getLong`. Could you update the function comments to include details about the new behavior?
##########
common/variant/README.md:
##########
@@ -335,27 +335,29 @@ The Decimal type contains a scale, but no precision. The implied precision of a
| Object | `2` | A collection of (string-key, variant-value) pairs |
| Array | `3` | An ordered sequence of variant values |
-| Primitive Type | Type ID | Equivalent Parquet Type | Binary format |
-|-----------------------------|---------|---------------------------|-----------------------------------------------------------------------------------------------------------|
-| null | `0` | any | none |
-| boolean (True) | `1` | BOOLEAN | none |
-| boolean (False) | `2` | BOOLEAN | none |
-| int8 | `3` | INT(8, signed) | 1 byte |
-| int16 | `4` | INT(16, signed) | 2 byte little-endian |
-| int32 | `5` | INT(32, signed) | 4 byte little-endian |
-| int64 | `6` | INT(64, signed) | 8 byte little-endian |
-| double | `7` | DOUBLE | IEEE little-endian |
-| decimal4 | `8` | DECIMAL(precision, scale) | 1 byte scale in range [0, 38], followed by little-endian unscaled value (see decimal table) |
-| decimal8 | `9` | DECIMAL(precision, scale) | 1 byte scale in range [0, 38], followed by little-endian unscaled value (see decimal table) |
-| decimal16 | `10` | DECIMAL(precision, scale) | 1 byte scale in range [0, 38], followed by little-endian unscaled value (see decimal table) |
-| date | `11` | DATE | 4 byte little-endian |
-| timestamp | `12` | TIMESTAMP(true, MICROS) | 8-byte little-endian |
-| timestamp without time zone | `13` | TIMESTAMP(false, MICROS) | 8-byte little-endian |
-| float | `14` | FLOAT | IEEE little-endian |
-| binary | `15` | BINARY | 4 byte little-endian size, followed by bytes |
-| string | `16` | STRING | 4 byte little-endian size, followed by UTF-8 encoded bytes |
-| binary from metadata | `17` | BINARY | Little-endian index into the metadata dictionary. Number of bytes is equal to the metadata `offset_size`. |
-| string from metadata | `18` | STRING | Little-endian index into the metadata dictionary. Number of bytes is equal to the metadata `offset_size`. |
+| Primitive Type | Type ID | Equivalent Parquet Type | Binary format |
+|-----------------------------|---------|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------------|
+| null | `0` | any | none |
+| boolean (True) | `1` | BOOLEAN | none |
+| boolean (False) | `2` | BOOLEAN | none |
+| int8 | `3` | INT(8, signed) | 1 byte |
+| int16 | `4` | INT(16, signed) | 2 byte little-endian |
+| int32 | `5` | INT(32, signed) | 4 byte little-endian |
+| int64 | `6` | INT(64, signed) | 8 byte little-endian |
+| double | `7` | DOUBLE | IEEE little-endian |
+| decimal4 | `8` | DECIMAL(precision, scale) | 1 byte scale in range [0, 38], followed by little-endian unscaled value (see decimal table) |
+| decimal8 | `9` | DECIMAL(precision, scale) | 1 byte scale in range [0, 38], followed by little-endian unscaled value (see decimal table) |
+| decimal16 | `10` | DECIMAL(precision, scale) | 1 byte scale in range [0, 38], followed by little-endian unscaled value (see decimal table) |
+| date | `11` | DATE | 4 byte little-endian |
+| timestamp | `12` | TIMESTAMP(true, MICROS) | 8-byte little-endian |
+| timestamp without time zone | `13` | TIMESTAMP(false, MICROS) | 8-byte little-endian |
+| float | `14` | FLOAT | IEEE little-endian |
+| binary | `15` | BINARY | 4 byte little-endian size, followed by bytes |
+| string | `16` | STRING | 4 byte little-endian size, followed by UTF-8 encoded bytes |
+| binary from metadata | `17` | BINARY | Little-endian index into the metadata dictionary. Number of bytes is equal to the metadata `offset_size`. |
+| string from metadata | `18` | STRING | Little-endian index into the metadata dictionary. Number of bytes is equal to the metadata `offset_size`. |
+| year-month interval | `19` | YearMonthIntervalType(start_field, end_field) | 1 byte denoting start field (1 bit) and end field (1 bit) starting at LSB followed by 4-byte little-endian value. |
Review Comment:
What would the parquet types be for these intervals?
Also, we need to describe what the start/end fields are and how exactly they are encoded in the byte.
##########
common/variant/src/main/java/org/apache/spark/types/variant/Variant.java:
##########
@@ -316,6 +334,18 @@ static void toJsonImpl(byte[] value, byte[] metadata, int pos, StringBuilder sb,
case BINARY:
appendQuoted(sb, Base64.getEncoder().encodeToString(VariantUtil.getBinary(value, pos)));
break;
+ case YEAR_MONTH_INTERVAL:
+ IntervalFields ymFields = VariantUtil.getYearMonthIntervalFields(value, pos);
+ int ymValue = (int) VariantUtil.getLong(value, pos);
Review Comment:
I'm not sure how this works. I thought the first byte at `pos` would include the start and end info. Then, I thought the remaining would be a 4 byte int? My confusion is:
- Shouldn't the `pos` read be `pos + 1`?
- Doesn't `VariantUtil.getLong(value, pos);` read variable length variant-encoded int? I thought we just need to read a 4-byte int, since the 4 bytes won't have any variant type info?
(similar question for `DAY_TIME_INTERVAL` below)
##########
common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java:
##########
@@ -377,11 +402,51 @@ public static long getLong(byte[] value, int pos) {
case TIMESTAMP:
case TIMESTAMP_NTZ:
return readLong(value, pos + 1, 8);
+ case YEAR_MONTH_INTERVAL:
+ return readLong(value, pos + 2, 4);
+ case DAY_TIME_INTERVAL:
+ return readLong(value, pos + 2, 8);
default:
throw new IllegalStateException(exceptionMessage);
}
}
+ public static class IntervalFields {
Review Comment:
Can you comment on this class, and what it represents?
##########
common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:
##########
@@ -214,6 +214,22 @@ public void appendTimestampNtz(long microsSinceEpoch) {
writePos += 8;
}
+ public void appendYearMonthInterval(long value, byte startField, byte endField) {
+ checkCapacity(1 + 5);
+ writeBuffer[writePos++] = primitiveHeader(YEAR_MONTH_INTERVAL);
+ writeBuffer[writePos++] = (byte) (startField | (endField << 1));
+ writeLong(writeBuffer, writePos, value, 4);
+ writePos += 4;
+ }
+
+ public void appendDayTimeInterval(long value, byte startField, byte endField) {
+ checkCapacity(1 + 9);
+ writeBuffer[writePos++] = primitiveHeader(DAY_TIME_INTERVAL);
+ writeBuffer[writePos++] = (byte) (startField | (endField << 2));
Review Comment:
Should we mask `startField` and `endField` to make sure we are only using the desired bits?
Or, we could do checks on the inputs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PYTHON][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "harshmotw-db (via GitHub)" <gi...@apache.org>.
harshmotw-db commented on code in PR #47473:
URL: https://github.com/apache/spark/pull/47473#discussion_r1691942388
##########
common/variant/src/main/java/org/apache/spark/types/variant/Variant.java:
##########
@@ -88,6 +91,16 @@ public long getLong() {
return VariantUtil.getLong(value, pos);
}
+ // Get the start and end fields of a year-month interval from the variant.
+ public IntervalFields getYearMonthIntervalFields() {
Review Comment:
`schemaOf`, `cast from variant` and `to_json`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PYTHON][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #47473:
URL: https://github.com/apache/spark/pull/47473#discussion_r1694436591
##########
common/utils/src/main/scala/org/apache/spark/util/DayTimeIntervalUtils.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util;
+
+import org.apache.spark.SparkException;
+
+import java.math.BigDecimal;
+import java.util.ArrayList;
+
+// Replicating code from SparkIntervalUtils so code in the 'common' space can work with
+// year-month intervals.
+public class DayTimeIntervalUtils {
+ private static byte DAY = 0;
+ private static byte HOUR = 1;
+ private static byte MINUTE = 2;
+ private static byte SECOND = 3;
+ private static long HOURS_PER_DAY = 24;
+ private static long MINUTES_PER_HOUR = 60;
+ private static long SECONDS_PER_MINUTE = 60;
+ private static long MILLIS_PER_SECOND = 1000;
+ private static long MICROS_PER_MILLIS = 1000;
+ private static long MICROS_PER_SECOND = MICROS_PER_MILLIS * MILLIS_PER_SECOND;
+ private static long MICROS_PER_MINUTE = SECONDS_PER_MINUTE * MICROS_PER_SECOND;
+ private static long MICROS_PER_HOUR = MINUTES_PER_HOUR * MICROS_PER_MINUTE;
+ private static long MICROS_PER_DAY = HOURS_PER_DAY * MICROS_PER_HOUR;
+ private static long MAX_DAY = Long.MAX_VALUE / MICROS_PER_DAY;
+ private static long MAX_HOUR = Long.MAX_VALUE / MICROS_PER_HOUR;
+ private static long MAX_MINUTE = Long.MAX_VALUE / MICROS_PER_MINUTE;
+ private static long MAX_SECOND = Long.MAX_VALUE / MICROS_PER_SECOND;
+
+ public static String fieldToString(byte field) throws SparkException {
+ if (field == DAY) {
+ return "DAY";
+ } else if (field == HOUR) {
+ return "HOUR";
+ } else if (field == MINUTE) {
+ return "MINUTE";
+ } else if (field == SECOND) {
+ return "SECOND";
+ } else {
+ throw new SparkException("Invalid field in day-time interval: " + field +
+ ". Supported fields are: DAY, HOUR, MINUTE, SECOND");
+ }
+ }
+
+ // Used to convert microseconds representing a day-time interval with given start and end fields
+ // to its ANSI SQL string representation. Throws a SparkException if startField or endField are
+ // out of bounds.
+ public static String toDayTimeIntervalANSIString(long micros, byte startField, byte endField)
+ throws SparkException {
+ String sign = "";
+ long rest = micros;
+ try {
+ String from = fieldToString(startField).toUpperCase();
+ String to = fieldToString(endField).toUpperCase();
+ String prefix = "INTERVAL '";
+ String postfix = startField == endField ? "' " + from : "' " + from + " TO " + to;
+ if (micros < 0) {
+ if (micros == Long.MIN_VALUE) {
+ // Especial handling of minimum `Long` value because negate op overflows `Long`.
+ // seconds = 106751991 * (24 * 60 * 60) + 4 * 60 * 60 + 54 = 9223372036854
+ // microseconds = -9223372036854000000L-775808 == Long.MinValue
+ String baseStr = "-106751991 04:00:54.775808000";
+ String firstStr = "-" + (startField == DAY ? Long.toString(MAX_DAY) :
+ (startField == HOUR ? Long.toString(MAX_HOUR) :
+ (startField == MINUTE ? Long.toString(MAX_MINUTE) :
+ Long.toString(MAX_SECOND) + ".775808")));
+ if (startField == endField) {
+ return prefix + firstStr + postfix;
+ } else {
+ int substrStart = startField == DAY ? 10 : (startField == HOUR ? 13 : 16);
+ int substrEnd = endField == HOUR ? 13 : (endField == MINUTE ? 16 : 26);
+ return prefix + firstStr + baseStr.substring(substrStart, substrEnd) + postfix;
+ }
+ } else {
+ sign = "-";
+ rest = -rest;
+ }
+ }
+ StringBuilder formatBuilder = new StringBuilder(sign);
+ ArrayList<Long> formatArgs = new ArrayList<>();
+ if (startField == DAY) {
+ formatBuilder.append(rest / MICROS_PER_DAY);
+ rest %= MICROS_PER_DAY;
+ } else if (startField == HOUR) {
+ formatBuilder.append("%02d");
+ formatArgs.add(rest / MICROS_PER_HOUR);
+ rest %= MICROS_PER_HOUR;
+ } else if (startField == MINUTE) {
+ formatBuilder.append("%02d");
+ formatArgs.add(rest / MICROS_PER_MINUTE);
+ rest %= MICROS_PER_MINUTE;
+ } else if (startField == SECOND) {
+ String leadZero = rest < 10 * MICROS_PER_SECOND ? "0" : "";
+ formatBuilder.append(leadZero + BigDecimal.valueOf(rest, 6)
Review Comment:
should use chained calls, like
```java
formatBuilder.append(leadZero).append(BigDecimal.valueOf(rest, 6).stripTrailingZeros().toPlainString());
```
otherwise `leadZero + BigDecimal.valueOf(rest, 6).stripTrailingZeros().toPlainString()` will result in another string concatenation."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PYTHON][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #47473:
URL: https://github.com/apache/spark/pull/47473#discussion_r1694436983
##########
common/utils/src/main/scala/org/apache/spark/util/DayTimeIntervalUtils.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util;
+
+import org.apache.spark.SparkException;
+
+import java.math.BigDecimal;
+import java.util.ArrayList;
+
+// Replicating code from SparkIntervalUtils so code in the 'common' space can work with
+// year-month intervals.
+public class DayTimeIntervalUtils {
+ private static byte DAY = 0;
+ private static byte HOUR = 1;
+ private static byte MINUTE = 2;
+ private static byte SECOND = 3;
+ private static long HOURS_PER_DAY = 24;
+ private static long MINUTES_PER_HOUR = 60;
+ private static long SECONDS_PER_MINUTE = 60;
+ private static long MILLIS_PER_SECOND = 1000;
+ private static long MICROS_PER_MILLIS = 1000;
+ private static long MICROS_PER_SECOND = MICROS_PER_MILLIS * MILLIS_PER_SECOND;
+ private static long MICROS_PER_MINUTE = SECONDS_PER_MINUTE * MICROS_PER_SECOND;
+ private static long MICROS_PER_HOUR = MINUTES_PER_HOUR * MICROS_PER_MINUTE;
+ private static long MICROS_PER_DAY = HOURS_PER_DAY * MICROS_PER_HOUR;
+ private static long MAX_DAY = Long.MAX_VALUE / MICROS_PER_DAY;
+ private static long MAX_HOUR = Long.MAX_VALUE / MICROS_PER_HOUR;
+ private static long MAX_MINUTE = Long.MAX_VALUE / MICROS_PER_MINUTE;
+ private static long MAX_SECOND = Long.MAX_VALUE / MICROS_PER_SECOND;
+
+ public static String fieldToString(byte field) throws SparkException {
+ if (field == DAY) {
+ return "DAY";
+ } else if (field == HOUR) {
+ return "HOUR";
+ } else if (field == MINUTE) {
+ return "MINUTE";
+ } else if (field == SECOND) {
+ return "SECOND";
+ } else {
+ throw new SparkException("Invalid field in day-time interval: " + field +
+ ". Supported fields are: DAY, HOUR, MINUTE, SECOND");
+ }
+ }
+
+ // Used to convert microseconds representing a day-time interval with given start and end fields
+ // to its ANSI SQL string representation. Throws a SparkException if startField or endField are
+ // out of bounds.
+ public static String toDayTimeIntervalANSIString(long micros, byte startField, byte endField)
+ throws SparkException {
+ String sign = "";
+ long rest = micros;
+ try {
+ String from = fieldToString(startField).toUpperCase();
+ String to = fieldToString(endField).toUpperCase();
+ String prefix = "INTERVAL '";
+ String postfix = startField == endField ? "' " + from : "' " + from + " TO " + to;
+ if (micros < 0) {
+ if (micros == Long.MIN_VALUE) {
+ // Especial handling of minimum `Long` value because negate op overflows `Long`.
+ // seconds = 106751991 * (24 * 60 * 60) + 4 * 60 * 60 + 54 = 9223372036854
+ // microseconds = -9223372036854000000L-775808 == Long.MinValue
+ String baseStr = "-106751991 04:00:54.775808000";
+ String firstStr = "-" + (startField == DAY ? Long.toString(MAX_DAY) :
+ (startField == HOUR ? Long.toString(MAX_HOUR) :
+ (startField == MINUTE ? Long.toString(MAX_MINUTE) :
+ Long.toString(MAX_SECOND) + ".775808")));
+ if (startField == endField) {
+ return prefix + firstStr + postfix;
+ } else {
+ int substrStart = startField == DAY ? 10 : (startField == HOUR ? 13 : 16);
+ int substrEnd = endField == HOUR ? 13 : (endField == MINUTE ? 16 : 26);
+ return prefix + firstStr + baseStr.substring(substrStart, substrEnd) + postfix;
+ }
+ } else {
+ sign = "-";
+ rest = -rest;
+ }
+ }
+ StringBuilder formatBuilder = new StringBuilder(sign);
+ ArrayList<Long> formatArgs = new ArrayList<>();
Review Comment:
```
List<Long> formatArgs = new ArrayList<>();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PYTHON][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "harshmotw-db (via GitHub)" <gi...@apache.org>.
harshmotw-db commented on code in PR #47473:
URL: https://github.com/apache/spark/pull/47473#discussion_r1690730843
##########
common/variant/src/main/java/org/apache/spark/types/variant/Variant.java:
##########
@@ -316,6 +334,22 @@ static void toJsonImpl(byte[] value, byte[] metadata, int pos, StringBuilder sb,
case BINARY:
appendQuoted(sb, Base64.getEncoder().encodeToString(VariantUtil.getBinary(value, pos)));
break;
+ case YEAR_MONTH_INTERVAL:
+ IntervalFields ymFields = VariantUtil.getYearMonthIntervalFields(value, pos);
+ int ymValue = (int) VariantUtil.getLong(value, pos);
Review Comment:
`getLong` in `VariantUtils.java` looks at the first byte in the variant and decides which type of int should be extracted. Therefore, functions like `getInt` or `getShort` don't exist. I have added cases in `getLong` for `YEAR_MONTH_INTERVAL` and `DAY_TIME_INTERVAL`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PySpark][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "harshmotw-db (via GitHub)" <gi...@apache.org>.
harshmotw-db commented on code in PR #47473:
URL: https://github.com/apache/spark/pull/47473#discussion_r1690587491
##########
common/variant/README.md:
##########
@@ -335,27 +335,29 @@ The Decimal type contains a scale, but no precision. The implied precision of a
| Object | `2` | A collection of (string-key, variant-value) pairs |
| Array | `3` | An ordered sequence of variant values |
-| Primitive Type | Type ID | Equivalent Parquet Type | Binary format |
-|-----------------------------|---------|---------------------------|-----------------------------------------------------------------------------------------------------------|
-| null | `0` | any | none |
-| boolean (True) | `1` | BOOLEAN | none |
-| boolean (False) | `2` | BOOLEAN | none |
-| int8 | `3` | INT(8, signed) | 1 byte |
-| int16 | `4` | INT(16, signed) | 2 byte little-endian |
-| int32 | `5` | INT(32, signed) | 4 byte little-endian |
-| int64 | `6` | INT(64, signed) | 8 byte little-endian |
-| double | `7` | DOUBLE | IEEE little-endian |
-| decimal4 | `8` | DECIMAL(precision, scale) | 1 byte scale in range [0, 38], followed by little-endian unscaled value (see decimal table) |
-| decimal8 | `9` | DECIMAL(precision, scale) | 1 byte scale in range [0, 38], followed by little-endian unscaled value (see decimal table) |
-| decimal16 | `10` | DECIMAL(precision, scale) | 1 byte scale in range [0, 38], followed by little-endian unscaled value (see decimal table) |
-| date | `11` | DATE | 4 byte little-endian |
-| timestamp | `12` | TIMESTAMP(true, MICROS) | 8-byte little-endian |
-| timestamp without time zone | `13` | TIMESTAMP(false, MICROS) | 8-byte little-endian |
-| float | `14` | FLOAT | IEEE little-endian |
-| binary | `15` | BINARY | 4 byte little-endian size, followed by bytes |
-| string | `16` | STRING | 4 byte little-endian size, followed by UTF-8 encoded bytes |
-| binary from metadata | `17` | BINARY | Little-endian index into the metadata dictionary. Number of bytes is equal to the metadata `offset_size`. |
-| string from metadata | `18` | STRING | Little-endian index into the metadata dictionary. Number of bytes is equal to the metadata `offset_size`. |
+| Primitive Type | Type ID | Equivalent Parquet Type | Binary format |
+|-----------------------------|---------|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------------|
+| null | `0` | any | none |
+| boolean (True) | `1` | BOOLEAN | none |
+| boolean (False) | `2` | BOOLEAN | none |
+| int8 | `3` | INT(8, signed) | 1 byte |
+| int16 | `4` | INT(16, signed) | 2 byte little-endian |
+| int32 | `5` | INT(32, signed) | 4 byte little-endian |
+| int64 | `6` | INT(64, signed) | 8 byte little-endian |
+| double | `7` | DOUBLE | IEEE little-endian |
+| decimal4 | `8` | DECIMAL(precision, scale) | 1 byte scale in range [0, 38], followed by little-endian unscaled value (see decimal table) |
+| decimal8 | `9` | DECIMAL(precision, scale) | 1 byte scale in range [0, 38], followed by little-endian unscaled value (see decimal table) |
+| decimal16 | `10` | DECIMAL(precision, scale) | 1 byte scale in range [0, 38], followed by little-endian unscaled value (see decimal table) |
+| date | `11` | DATE | 4 byte little-endian |
+| timestamp | `12` | TIMESTAMP(true, MICROS) | 8-byte little-endian |
+| timestamp without time zone | `13` | TIMESTAMP(false, MICROS) | 8-byte little-endian |
+| float | `14` | FLOAT | IEEE little-endian |
+| binary | `15` | BINARY | 4 byte little-endian size, followed by bytes |
+| string | `16` | STRING | 4 byte little-endian size, followed by UTF-8 encoded bytes |
+| binary from metadata | `17` | BINARY | Little-endian index into the metadata dictionary. Number of bytes is equal to the metadata `offset_size`. |
+| string from metadata | `18` | STRING | Little-endian index into the metadata dictionary. Number of bytes is equal to the metadata `offset_size`. |
+| year-month interval | `19` | YearMonthIntervalType(start_field, end_field) | 1 byte denoting start field (1 bit) and end field (1 bit) starting at LSB followed by 4-byte little-endian value. |
Review Comment:
I ran the following Python script on a parquet table containing these interval types and found that these intervals are intrinsically stored as int/long and the type info is stored in the metadata. I'll update the table to reflect this.
```
>>> import pyarrow.parquet as pq
>>> table = pq.read_table('/home/harsh.motwani/tables/part-00000-tid-8067172485220669242-1687c1be-9e28-455a-817a-449a862b4a05-0-1-c000.snappy.parquet')
>>> table.schema
ymi0: int32 not null
ymi1: int32 not null
ymi2: int32
dti0: int64
dti1: int64
-- schema metadata --
org.apache.spark.version: '4.0.0'
org.apache.spark.sql.parquet.row.metadata: '{"type":"struct","fields":[{"' + 375
>>> table.schema.metadata
OrderedDict([(b'org.apache.spark.version', b'4.0.0'), (b'org.apache.spark.sql.parquet.row.metadata', b'{"type":"struct","fields":[{"name":"ymi0","type":"interval year to month","nullable":false,"metadata":{}},{"name":"ymi1","type":"interval year","nullable":false,"metadata":{}},{"name":"ymi2","type":"interval month","nullable":true,"metadata":{}},{"name":"dti0","type":"interval day to second","nullable":true,"metadata":{}},{"name":"dti1","type":"interval hour to minute","nullable":true,"metadata":{}}]}')])
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PYTHON][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "gene-db (via GitHub)" <gi...@apache.org>.
gene-db commented on code in PR #47473:
URL: https://github.com/apache/spark/pull/47473#discussion_r1691812112
##########
common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java:
##########
@@ -120,6 +120,12 @@ public class VariantUtil {
// Long string value. The content is (4-byte little-endian unsigned integer representing the
// string size) + (size bytes of string content).
public static final int LONG_STR = 16;
+ // year-month interval value. The content is one byte representing the start and end field values
+ // (1 bit each starting at least significant bits) and a 4-byte little-endian signed integer
+ public static final int YEAR_MONTH_INTERVAL = 19;
+ // day-time interval value. The content is one byte representing the start and end field values
+ // (2 bits each starting at least significant bits) and an 8-byte little-endian signed integer
Review Comment:
Similar to the README, we should document what 0, 1, 2, 3 represent for the start and end fields.
##########
common/variant/src/main/java/org/apache/spark/types/variant/Variant.java:
##########
@@ -88,6 +91,16 @@ public long getLong() {
return VariantUtil.getLong(value, pos);
}
+ // Get the start and end fields of a year-month interval from the variant.
+ public IntervalFields getYearMonthIntervalFields() {
Review Comment:
What uses these apis?
##########
common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java:
##########
@@ -377,11 +405,52 @@ public static long getLong(byte[] value, int pos) {
case TIMESTAMP:
case TIMESTAMP_NTZ:
return readLong(value, pos + 1, 8);
+ case YEAR_MONTH_INTERVAL:
+ return readLong(value, pos + 2, 4);
+ case DAY_TIME_INTERVAL:
+ return readLong(value, pos + 2, 8);
default:
throw new IllegalStateException(exceptionMessage);
}
}
+ // Class used to pass around start and end fields of year-month and day-time interval values.
+ public static class IntervalFields {
+ public IntervalFields(byte startField, byte endField) {
+ this.startField = startField;
+ this.endField = endField;
+ }
+
+ public final byte startField;
+ public final byte endField;
+ }
+
+ // Get the start and end fields of a variant value representing a year-month interval value. The
+ // returned array contains the start field at the zeroth index and the end field at the first
+ // index.
+ public static IntervalFields getYearMonthIntervalFields(byte[] value, int pos) {
+ long fieldInfo = readLong(value, pos + 1, 1);
+ IntervalFields intervalFields = new IntervalFields((byte) (fieldInfo & 0x1),
+ (byte) ((fieldInfo >> 1) & 0x1));
+ if (intervalFields.endField < intervalFields.startField) {
Review Comment:
Do we need to check the type to be year-month interval, and the length? Some other functions call `checkIndex`.
##########
common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java:
##########
@@ -355,9 +380,12 @@ public static boolean getBoolean(byte[] value, int pos) {
// Get a long value from variant value `value[pos...]`.
// It is only legal to call it if `getType` returns one of `Type.LONG/DATE/TIMESTAMP/
- // TIMESTAMP_NTZ`. If the type is `DATE`, the return value is guaranteed to fit into an int and
- // represents the number of days from the Unix epoch. If the type is `TIMESTAMP/TIMESTAMP_NTZ`,
- // the return value represents the number of microseconds from the Unix epoch.
+ // TIMESTAMP_NTZ/YEAR_MONTH_INTERVAL/DAY_TIME_INTERVAL`. If the type is `DATE`, the return value
+ // is guaranteed to fit into an int and represents the number of days from the Unix epoch.
+ // If the type is `TIMESTAMP/TIMESTAMP_NTZ`, the return value represents the number of
+ // microseconds from the Unix epoch. If the type is `YEAR_MONTH_INTERVAL`, the return value
Review Comment:
We should also mention that the year-month one is guaranteed to fit into an int.
##########
common/variant/src/main/java/org/apache/spark/types/variant/Variant.java:
##########
@@ -113,6 +126,11 @@ public String getString() {
return VariantUtil.getString(value, pos);
}
+ // Get the type info bits from a variant value.
+ public int getTypeInfo() {
Review Comment:
I can't easily tell, but what uses this api?
##########
common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java:
##########
@@ -120,6 +120,12 @@ public class VariantUtil {
// Long string value. The content is (4-byte little-endian unsigned integer representing the
// string size) + (size bytes of string content).
public static final int LONG_STR = 16;
+ // year-month interval value. The content is one byte representing the start and end field values
+ // (1 bit each starting at least significant bits) and a 4-byte little-endian signed integer
Review Comment:
Similar to the README, we should document what 0 and 1 represent for the start and end fields.
##########
common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java:
##########
@@ -233,6 +245,13 @@ public enum Type {
TIMESTAMP_NTZ,
FLOAT,
BINARY,
+ YEAR_MONTH_INTERVAL,
+ DAY_TIME_INTERVAL,
+ }
+
+ public static int getTypeInfo(byte[] value, int pos) {
Review Comment:
Similar to an earlier question, what uses this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PYTHON][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #47473:
URL: https://github.com/apache/spark/pull/47473#discussion_r1694435200
##########
common/utils/src/main/scala/org/apache/spark/util/DayTimeIntervalUtils.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util;
+
+import org.apache.spark.SparkException;
+
+import java.math.BigDecimal;
+import java.util.ArrayList;
+
+// Replicating code from SparkIntervalUtils so code in the 'common' space can work with
+// year-month intervals.
+public class DayTimeIntervalUtils {
+ private static byte DAY = 0;
+ private static byte HOUR = 1;
+ private static byte MINUTE = 2;
+ private static byte SECOND = 3;
+ private static long HOURS_PER_DAY = 24;
+ private static long MINUTES_PER_HOUR = 60;
+ private static long SECONDS_PER_MINUTE = 60;
+ private static long MILLIS_PER_SECOND = 1000;
+ private static long MICROS_PER_MILLIS = 1000;
+ private static long MICROS_PER_SECOND = MICROS_PER_MILLIS * MILLIS_PER_SECOND;
+ private static long MICROS_PER_MINUTE = SECONDS_PER_MINUTE * MICROS_PER_SECOND;
+ private static long MICROS_PER_HOUR = MINUTES_PER_HOUR * MICROS_PER_MINUTE;
+ private static long MICROS_PER_DAY = HOURS_PER_DAY * MICROS_PER_HOUR;
+ private static long MAX_DAY = Long.MAX_VALUE / MICROS_PER_DAY;
+ private static long MAX_HOUR = Long.MAX_VALUE / MICROS_PER_HOUR;
+ private static long MAX_MINUTE = Long.MAX_VALUE / MICROS_PER_MINUTE;
+ private static long MAX_SECOND = Long.MAX_VALUE / MICROS_PER_SECOND;
+
+ public static String fieldToString(byte field) throws SparkException {
+ if (field == DAY) {
+ return "DAY";
+ } else if (field == HOUR) {
+ return "HOUR";
+ } else if (field == MINUTE) {
+ return "MINUTE";
+ } else if (field == SECOND) {
+ return "SECOND";
+ } else {
+ throw new SparkException("Invalid field in day-time interval: " + field +
+ ". Supported fields are: DAY, HOUR, MINUTE, SECOND");
+ }
+ }
+
+ // Used to convert microseconds representing a day-time interval with given start and end fields
+ // to its ANSI SQL string representation. Throws a SparkException if startField or endField are
+ // out of bounds.
+ public static String toDayTimeIntervalANSIString(long micros, byte startField, byte endField)
+ throws SparkException {
+ String sign = "";
+ long rest = micros;
+ try {
+ String from = fieldToString(startField).toUpperCase();
+ String to = fieldToString(endField).toUpperCase();
+ String prefix = "INTERVAL '";
+ String postfix = startField == endField ? "' " + from : "' " + from + " TO " + to;
+ if (micros < 0) {
+ if (micros == Long.MIN_VALUE) {
+ // Especial handling of minimum `Long` value because negate op overflows `Long`.
+ // seconds = 106751991 * (24 * 60 * 60) + 4 * 60 * 60 + 54 = 9223372036854
+ // microseconds = -9223372036854000000L-775808 == Long.MinValue
+ String baseStr = "-106751991 04:00:54.775808000";
+ String firstStr = "-" + (startField == DAY ? Long.toString(MAX_DAY) :
+ (startField == HOUR ? Long.toString(MAX_HOUR) :
+ (startField == MINUTE ? Long.toString(MAX_MINUTE) :
+ Long.toString(MAX_SECOND) + ".775808")));
+ if (startField == endField) {
+ return prefix + firstStr + postfix;
+ } else {
+ int substrStart = startField == DAY ? 10 : (startField == HOUR ? 13 : 16);
+ int substrEnd = endField == HOUR ? 13 : (endField == MINUTE ? 16 : 26);
+ return prefix + firstStr + baseStr.substring(substrStart, substrEnd) + postfix;
+ }
+ } else {
+ sign = "-";
+ rest = -rest;
+ }
+ }
+ StringBuilder formatBuilder = new StringBuilder(sign);
+ ArrayList<Long> formatArgs = new ArrayList<>();
+ if (startField == DAY) {
+ formatBuilder.append(rest / MICROS_PER_DAY);
+ rest %= MICROS_PER_DAY;
+ } else if (startField == HOUR) {
+ formatBuilder.append("%02d");
+ formatArgs.add(rest / MICROS_PER_HOUR);
+ rest %= MICROS_PER_HOUR;
+ } else if (startField == MINUTE) {
+ formatBuilder.append("%02d");
+ formatArgs.add(rest / MICROS_PER_MINUTE);
+ rest %= MICROS_PER_MINUTE;
+ } else if (startField == SECOND) {
+ String leadZero = rest < 10 * MICROS_PER_SECOND ? "0" : "";
+ formatBuilder.append(leadZero + BigDecimal.valueOf(rest, 6)
+ .stripTrailingZeros().toPlainString());
+ }
+
+ if (startField < HOUR && HOUR <= endField) {
+ formatBuilder.append(" %02d");
+ formatArgs.add(rest / MICROS_PER_HOUR);
+ rest %= MICROS_PER_HOUR;
+ }
+ if (startField < MINUTE && MINUTE <= endField) {
+ formatBuilder.append(":%02d");
+ formatArgs.add(rest / MICROS_PER_MINUTE);
+ rest %= MICROS_PER_MINUTE;
+ }
+ if (startField < SECOND && SECOND <= endField) {
+ String leadZero = rest < 10 * MICROS_PER_SECOND ? "0" : "";
+ formatBuilder.append(":" + leadZero + BigDecimal.valueOf(rest, 6)
+ .stripTrailingZeros().toPlainString());
+ }
+ return prefix + String.format(formatBuilder.toString(), formatArgs.toArray()) + postfix;
+ } catch (SparkException e) {
Review Comment:
The `try-catch` here seems a bit redundant. Why do we need to catch the `SparkException` only to rethrow it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45891][SQL][PYTHON][VARIANT] Add support for interval types in the Variant Spec [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #47473:
URL: https://github.com/apache/spark/pull/47473#discussion_r1694438032
##########
common/utils/src/main/scala/org/apache/spark/util/DayTimeIntervalUtils.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util;
+
+import org.apache.spark.SparkException;
+
+import java.math.BigDecimal;
+import java.util.ArrayList;
+
+// Replicating code from SparkIntervalUtils so code in the 'common' space can work with
+// year-month intervals.
+public class DayTimeIntervalUtils {
+ private static byte DAY = 0;
+ private static byte HOUR = 1;
+ private static byte MINUTE = 2;
+ private static byte SECOND = 3;
+ private static long HOURS_PER_DAY = 24;
+ private static long MINUTES_PER_HOUR = 60;
+ private static long SECONDS_PER_MINUTE = 60;
+ private static long MILLIS_PER_SECOND = 1000;
+ private static long MICROS_PER_MILLIS = 1000;
+ private static long MICROS_PER_SECOND = MICROS_PER_MILLIS * MILLIS_PER_SECOND;
+ private static long MICROS_PER_MINUTE = SECONDS_PER_MINUTE * MICROS_PER_SECOND;
+ private static long MICROS_PER_HOUR = MINUTES_PER_HOUR * MICROS_PER_MINUTE;
+ private static long MICROS_PER_DAY = HOURS_PER_DAY * MICROS_PER_HOUR;
+ private static long MAX_DAY = Long.MAX_VALUE / MICROS_PER_DAY;
+ private static long MAX_HOUR = Long.MAX_VALUE / MICROS_PER_HOUR;
+ private static long MAX_MINUTE = Long.MAX_VALUE / MICROS_PER_MINUTE;
+ private static long MAX_SECOND = Long.MAX_VALUE / MICROS_PER_SECOND;
+
+ public static String fieldToString(byte field) throws SparkException {
+ if (field == DAY) {
+ return "DAY";
+ } else if (field == HOUR) {
+ return "HOUR";
+ } else if (field == MINUTE) {
+ return "MINUTE";
+ } else if (field == SECOND) {
+ return "SECOND";
+ } else {
+ throw new SparkException("Invalid field in day-time interval: " + field +
+ ". Supported fields are: DAY, HOUR, MINUTE, SECOND");
+ }
+ }
+
+ // Used to convert microseconds representing a day-time interval with given start and end fields
+ // to its ANSI SQL string representation. Throws a SparkException if startField or endField are
+ // out of bounds.
+ public static String toDayTimeIntervalANSIString(long micros, byte startField, byte endField)
+ throws SparkException {
+ String sign = "";
+ long rest = micros;
+ try {
+ String from = fieldToString(startField).toUpperCase();
+ String to = fieldToString(endField).toUpperCase();
+ String prefix = "INTERVAL '";
+ String postfix = startField == endField ? "' " + from : "' " + from + " TO " + to;
+ if (micros < 0) {
+ if (micros == Long.MIN_VALUE) {
+ // Especial handling of minimum `Long` value because negate op overflows `Long`.
+ // seconds = 106751991 * (24 * 60 * 60) + 4 * 60 * 60 + 54 = 9223372036854
+ // microseconds = -9223372036854000000L-775808 == Long.MinValue
+ String baseStr = "-106751991 04:00:54.775808000";
+ String firstStr = "-" + (startField == DAY ? Long.toString(MAX_DAY) :
+ (startField == HOUR ? Long.toString(MAX_HOUR) :
+ (startField == MINUTE ? Long.toString(MAX_MINUTE) :
+ Long.toString(MAX_SECOND) + ".775808")));
Review Comment:
`MAX_SECOND + ".775808"`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org