You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/03/07 19:38:49 UTC

[GitHub] [spark] MaxGekk opened a new pull request #35756: [WIP][SPARK-38437][SQL] Dynamic serialization of Java datetime objects to micros/days

MaxGekk opened a new pull request #35756:
URL: https://github.com/apache/spark/pull/35756


   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
     8. If you want to add or modify an error type or message, please read the guideline first in
        'core/src/main/resources/error/README.md'.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MaxGekk edited a comment on pull request #35756: [SPARK-38437][SQL] Lenient serialization of datetime from datasource

Posted by GitBox <gi...@apache.org>.
MaxGekk edited a comment on pull request #35756:
URL: https://github.com/apache/spark/pull/35756#issuecomment-1062675605


   > ... the per record `instanceof` should have nearly 0 overhead ...
   
   The datetime ops are pretty expensive, especially when need to look up to history data to calculate time zone offsets. Such op could take milliseconds (or hundreds microseconds) comparing to `instanceof` which takes a few nanoseconds.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MaxGekk commented on pull request #35756: [WIP][SPARK-38437][SQL] Dynamic serialization of Java datetime objects to micros/days

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #35756:
URL: https://github.com/apache/spark/pull/35756#issuecomment-1061448283


   > Previously we have compile/analysis-time checks which generate code specifically tailored to either SQL or Java-native types.
   
   @xkrogen This PR doesn't weak any compile/analysis-time checks. The goal is to improve user experience with Spark SQL, and make it more flexible to user's input. Currently, Spark support 2 external Java types for Catalyst's timestamp type: `java.sql.Timestamp` and `java.time.Instant`, and users/datasource connector can use both but Spark accepts only one based its config `spark.sql.datetime.java8API.enabled`. Let's imagine the situation when a datasource connector is going to be re-used with new Spark version when the config has been already added and enabled. The datasource doesn't aware of new config, and still pushes old `java.sql.Timestamp` to Spark but Spark rejects them even it can properly handle it.
   
   > After this PR, we would **relax** that compile-time check and instead perform per-row runtime checks on the object type.
   
   No, it doesn't relax any compile-time checks. We declare that Spark support both Java types for timestamps depending on Spark SQL config. After the PR, Spark will accept both independently from the config.
   
   cc @cloud-fan 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MaxGekk commented on pull request #35756: [WIP][SPARK-38437][SQL] Dynamic serialization of Java datetime objects to micros/days

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #35756:
URL: https://github.com/apache/spark/pull/35756#issuecomment-1061624794


   > Can we narrow the scope down to only data source scan?
   
   Sure. I am trying this now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xkrogen commented on pull request #35756: [WIP][SPARK-38437][SQL] Dynamic serialization of Java datetime objects to micros/days

Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #35756:
URL: https://github.com/apache/spark/pull/35756#issuecomment-1061174940


   This PR seems to me to be moving in the wrong direction. Previously we have compile/analysis-time checks which generate code specifically tailored to either SQL or Java-native types. After this PR, we would relax that compile-time check and instead perform per-row runtime checks on the object type. I would expect this to be detrimental to performance, and is generally contradictory to the approach of performing more analysis at query compile-time to avoid having to do checks at runtime. LMK if I'm missing anything.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xkrogen commented on pull request #35756: [SPARK-38437][SQL] Lenient serialization of datetime from datasource

Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #35756:
URL: https://github.com/apache/spark/pull/35756#issuecomment-1062097418


   Perhaps I should have been more careful with my wording. I agree that no **checks** are weakened. However, we previously generated code that was specialized to a single type, either Java native or SQL type. Now, we perform a type-check at runtime, meaning we have to do an `instanceof` call for _every single record_. My understanding is that `instanceof` is pretty fast on modern JVMs, but still, it is additional overhead on a per-record basis, so there is a performance degradation.
   
   I think the PR looks much more reasonable now that it is scoped to only the Datasource API, and I do understand the problem you're addressing, though it still imposes extra overhead for well-behaved datasources that obey the `java8API.enabled` config. I am wondering if this behavior should be configurable? That would enable performance-sensitive users to generate code which strictly obeys the `java8API.enabled` config, and users who need to accommodate legacy datasources (which don't obey the config) can accept the small performance hit. We would just need to change `true` in this line to a configurable value:
   ```scala
   val toRow = RowEncoder(StructType.fromAttributes(output), lenient = true).createSerializer()
   ```
   
   Not a strong concern, I think it is likely that the performance difference is small, but something to consider.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #35756: [WIP][SPARK-38437][SQL] Dynamic serialization of Java datetime objects to micros/days

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #35756:
URL: https://github.com/apache/spark/pull/35756#issuecomment-1061533653


   Can we narrow the scope down to only data source scan? For dataset, the type is known at compile-time and we can still generate precise code to process either `java.sql.Timestamp` or `java.time.Instant`. It's only a problem in data source scan that we don't know the datetime jave type at the compile time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #35756: [SPARK-38437][SQL] Lenient serialization of datetime from datasource

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #35756:
URL: https://github.com/apache/spark/pull/35756#discussion_r822349667



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala
##########
@@ -66,23 +66,27 @@ import org.apache.spark.sql.types._
  * }}}
  */
 object RowEncoder {
-  def apply(schema: StructType): ExpressionEncoder[Row] = {
+  def apply(schema: StructType, lenient: Boolean): ExpressionEncoder[Row] = {
     val cls = classOf[Row]
     val inputObject = BoundReference(0, ObjectType(cls), nullable = true)
-    val serializer = serializerFor(inputObject, schema)
+    val serializer = serializerFor(inputObject, schema, lenient)
     val deserializer = deserializerFor(GetColumnByOrdinal(0, serializer.dataType), schema)
     new ExpressionEncoder[Row](
       serializer,
       deserializer,
       ClassTag(cls))
   }
+  def apply(schema: StructType): ExpressionEncoder[Row] = {

Review comment:
       nit: put a blank line between methods.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #35756: [SPARK-38437][SQL] Lenient serialization of datetime from datasource

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #35756:
URL: https://github.com/apache/spark/pull/35756#issuecomment-1062589104


   as long as the data source always returns one type of datetime class, the branch prediction will work pretty well on JVM and the per record `instanceof` should have nearly 0 overhead.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MaxGekk closed pull request #35756: [SPARK-38437][SQL] Lenient serialization of datetime from datasource

Posted by GitBox <gi...@apache.org>.
MaxGekk closed pull request #35756:
URL: https://github.com/apache/spark/pull/35756


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xkrogen commented on pull request #35756: [SPARK-38437][SQL] Lenient serialization of datetime from datasource

Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #35756:
URL: https://github.com/apache/spark/pull/35756#issuecomment-1063160793


   Fair points from both of you on branch prediction and relative cost of `instanceof` vs. datetime ops. Thanks for taking the time to address my concerns.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MaxGekk commented on pull request #35756: [SPARK-38437][SQL] Lenient serialization of datetime from datasource

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #35756:
URL: https://github.com/apache/spark/pull/35756#issuecomment-1062675605


   > ... the per record `instanceof` should have nearly 0 overhead ...
   
   The datetime ops are pretty expensive, especially when need to look up to history data to calculate time zone offsets. Such op could take milliseconds comparing to `instanceof` which takes a few nanoseconds.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MaxGekk commented on pull request #35756: [SPARK-38437][SQL] Lenient serialization of datetime from datasource

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #35756:
URL: https://github.com/apache/spark/pull/35756#issuecomment-1062676965


   Merging to master. Thank you, @xkrogen and @cloud-fan for review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org