You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/12/20 15:19:43 UTC
[GitHub] [spark] EnricoMi opened a new pull request #26969: Adds a stricter
version of `as[T]`
EnricoMi opened a new pull request #26969: Adds a stricter version of `as[T]`
URL: https://github.com/apache/spark/pull/26969
### What changes were proposed in this pull request?
Some aspects of `as[T]` are not intuitive and expected behaviour is not provided elsewhere:
* Extra columns that are not part of the type `T` are not dropped.
* Order of columns is not aligned with schema of `T`.
* Columns are not cast to the types of `T`'s fields. They have to be cast explicitly.
**This PR adds a stricter version of `as[T]` to `Dataset`.**
### Why are the changes needed?
The behaviour of `as[T]` is not intuitive when you read code like `df.as[T].write.csv("data.csv")`. The result depends on the actual schema of `df`, where `def as[T](): Dataset[T]` should be agnostic to the schema of `df`.
A method that enforces schema of `T` on a given Dataset would be very convenient and allows to articulate and guarantee above assumptions about your data with the native Spark Dataset API. This method plays a more explicit and enforcing role than `as[T]` with respect to columns, column order and column type.
### Does this PR introduce any user-facing change?
Yes, it adds a new method to `Dataset`. It does not touch the existing `as[T]`.
Possible naming of a stricter version of `as[T]`:
* `as[T](strict = true)`
* `toDS[T]` (as in `toDF`)
* `selectAs[T]` (as this is merely selecting the columns of schema `T`)
This PR chooses the `toDS[T]` naming.
### How was this patch tested?
Existing tests for `as[T]` have been extended to assert the actual schema and emphasize the differences between `as[T]` and `toDS[T]`. Tests for `toDS[T]` are based on the `as[T]` tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on a change in pull request #26969:
[SPARK-30319][SQL] Add a stricter version of `as[T]`
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on a change in pull request #26969: [SPARK-30319][SQL] Add a stricter version of `as[T]`
URL: https://github.com/apache/spark/pull/26969#discussion_r367849005
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -495,6 +495,25 @@ class Dataset[T] private[sql](
select(newCols : _*)
}
+ /**
+ * Returns a new Dataset where each record has been mapped on to the specified type.
+ * This only supports `U` being a class. Fields for the class will be mapped to columns of the
+ * same name (case sensitivity is determined by `spark.sql.caseSensitive`).
+ *
+ * If the schema of the Dataset does not match the desired `U` type, you can use `select`
+ * along with `alias` or `as` to rearrange or rename as required.
+ *
+ * This method eagerly projects away any columns that are not present in the specified class.
+ * It further guarantees the order of columns as well as data types to match `U`.
+ *
+ * @group basic
+ * @since 3.0.0
+ */
+ def toDS[U : Encoder]: Dataset[U] = {
+ val columns = implicitly[Encoder[U]].schema.fields.map(f => col(f.name).cast(f.dataType))
Review comment:
Do you agree that the lazy-ness of `as[T]` is an inconsistency of the API where all other transformations' schema instantly reflect the result's schema? With `as[T]` you need a `map(identity)` to really become a `Dataset[T]`.
Which properties of `toDS[T]` would you require?
1. case classes, tuples and inner fields supported
2. arbitrary implementations of `Encoder` supported
3. data (not just schema) of result `Dataset[T]` are identical to `as[T].map(identity)`, which is given when they follow schema `encoder.schema`
4. uses projection where possible (efficient), falls back to `map(identity)` when needed (expensive)
The idea is to turn the dataset into the same shape as it would look like when the encoder would produce it (`map(identity)`). In case of `ExpressionEncoder`, the encoder can tell you which columns it serialises / deserialises. If the schema of those columns does not change (e.g. unchanged inner fields), than a simple projection suffices: select those fields in the right order. The data then comply to `encoder.schema`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on a change in pull request #26969:
[SPARK-30319][SQL] Add a stricter version of `as[T]`
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on a change in pull request #26969: [SPARK-30319][SQL] Add a stricter version of `as[T]`
URL: https://github.com/apache/spark/pull/26969#discussion_r366256637
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -495,6 +495,25 @@ class Dataset[T] private[sql](
select(newCols : _*)
}
+ /**
+ * Returns a new Dataset where each record has been mapped on to the specified type.
+ * This only supports `U` being a class. Fields for the class will be mapped to columns of the
+ * same name (case sensitivity is determined by `spark.sql.caseSensitive`).
+ *
+ * If the schema of the Dataset does not match the desired `U` type, you can use `select`
+ * along with `alias` or `as` to rearrange or rename as required.
+ *
+ * This method eagerly projects away any columns that are not present in the specified class.
+ * It further guarantees the order of columns as well as data types to match `U`.
+ *
+ * @group basic
+ * @since 3.0.0
+ */
+ def toDS[U : Encoder]: Dataset[U] = {
+ val columns = implicitly[Encoder[U]].schema.fields.map(f => col(f.name).cast(f.dataType))
Review comment:
As the documentation of that method says, this only supports U being a class. Tuples or structs are not supported. Those would require knowledge that only the encoder does have.
For classes a simple projection is sufficient, which is very cheap. An encoder round trip should be avoided in this use case. The stricter as method could be extended later to other encoders and kinds of Us, but I think classes are a wide use case that justify adding it already.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on a change in pull request #26969:
[SPARK-30319][SQL] Add a stricter version of `as[T]`
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on a change in pull request #26969: [SPARK-30319][SQL] Add a stricter version of `as[T]`
URL: https://github.com/apache/spark/pull/26969#discussion_r377677022
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -495,6 +495,25 @@ class Dataset[T] private[sql](
select(newCols : _*)
}
+ /**
+ * Returns a new Dataset where each record has been mapped on to the specified type.
+ * This only supports `U` being a class. Fields for the class will be mapped to columns of the
+ * same name (case sensitivity is determined by `spark.sql.caseSensitive`).
+ *
+ * If the schema of the Dataset does not match the desired `U` type, you can use `select`
+ * along with `alias` or `as` to rearrange or rename as required.
+ *
+ * This method eagerly projects away any columns that are not present in the specified class.
+ * It further guarantees the order of columns as well as data types to match `U`.
+ *
+ * @group basic
+ * @since 3.0.0
+ */
+ def toDS[U : Encoder]: Dataset[U] = {
+ val columns = implicitly[Encoder[U]].schema.fields.map(f => col(f.name).cast(f.dataType))
Review comment:
@cloud-fan what do you think about this implementation of `toDS[T]` and its scope?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on a change in pull request #26969:
[SPARK-30319][SQL] Add a stricter version of `as[T]`
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on a change in pull request #26969: [SPARK-30319][SQL] Add a stricter version of `as[T]`
URL: https://github.com/apache/spark/pull/26969#discussion_r369655551
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -495,6 +495,25 @@ class Dataset[T] private[sql](
select(newCols : _*)
}
+ /**
+ * Returns a new Dataset where each record has been mapped on to the specified type.
+ * This only supports `U` being a class. Fields for the class will be mapped to columns of the
+ * same name (case sensitivity is determined by `spark.sql.caseSensitive`).
+ *
+ * If the schema of the Dataset does not match the desired `U` type, you can use `select`
+ * along with `alias` or `as` to rearrange or rename as required.
+ *
+ * This method eagerly projects away any columns that are not present in the specified class.
+ * It further guarantees the order of columns as well as data types to match `U`.
+ *
+ * @group basic
+ * @since 3.0.0
+ */
+ def toDS[U : Encoder]: Dataset[U] = {
+ val columns = implicitly[Encoder[U]].schema.fields.map(f => col(f.name).cast(f.dataType))
Review comment:
I have added checks to `toDS[T]` that throw a meaningful `AnalysisException` when columns do not line up with encoder's schema. The method is now general enough to support any encoder with a schema that can be created merely by projection, which is true for standard encoder of case classes (without classes that have inner case classes with extra fields or different field order), seq and maps of case classes (without extra fields or different field order) as well as any Java classes (`Encoders.javaSerialisation`). It throws `as[T]`'s Exception if columns do not line up with encoder's schema and `as[T]` would fail either.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26969:
[SPARK-30319][SQL] Add a stricter version of `as[T]`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #26969: [SPARK-30319][SQL] Add a stricter version of `as[T]`
URL: https://github.com/apache/spark/pull/26969#discussion_r367908939
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -495,6 +495,25 @@ class Dataset[T] private[sql](
select(newCols : _*)
}
+ /**
+ * Returns a new Dataset where each record has been mapped on to the specified type.
+ * This only supports `U` being a class. Fields for the class will be mapped to columns of the
+ * same name (case sensitivity is determined by `spark.sql.caseSensitive`).
+ *
+ * If the schema of the Dataset does not match the desired `U` type, you can use `select`
+ * along with `alias` or `as` to rearrange or rename as required.
+ *
+ * This method eagerly projects away any columns that are not present in the specified class.
+ * It further guarantees the order of columns as well as data types to match `U`.
+ *
+ * @group basic
+ * @since 3.0.0
+ */
+ def toDS[U : Encoder]: Dataset[U] = {
+ val columns = implicitly[Encoder[U]].schema.fields.map(f => col(f.name).cast(f.dataType))
Review comment:
I'm OK to make the scope small and only support common cases like case class. But we must make sure the implementation is defensive and we give a clear better error message for unsupported cases.
Now this just throws attribute not found exception if the encoder is `Encoders.javaSerialization`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26969:
[SPARK-30319][SQL] Add a stricter version of `as[T]`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #26969: [SPARK-30319][SQL] Add a stricter version of `as[T]`
URL: https://github.com/apache/spark/pull/26969#discussion_r364217710
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -495,6 +495,25 @@ class Dataset[T] private[sql](
select(newCols : _*)
}
+ /**
+ * Returns a new Dataset where each record has been mapped on to the specified type.
+ * This only supports `U` being a class. Fields for the class will be mapped to columns of the
+ * same name (case sensitivity is determined by `spark.sql.caseSensitive`).
+ *
+ * If the schema of the Dataset does not match the desired `U` type, you can use `select`
+ * along with `alias` or `as` to rearrange or rename as required.
+ *
+ * This method eagerly projects away any columns that are not present in the specified class.
+ * It further guarantees the order of columns as well as data types to match `U`.
+ *
+ * @group basic
+ * @since 3.0.0
+ */
+ def toDS[U : Encoder]: Dataset[U] = {
+ val columns = implicitly[Encoder[U]].schema.fields.map(f => col(f.name).cast(f.dataType))
Review comment:
and the encoder can do anything, e.g. `Encoders.javaSerialization`, whose schema is just a binary column.
I think the only way is to execute the encoder and do a round trip, which is effectively `df.as[T].map(identity)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26969:
[SPARK-30319][SQL] Add a stricter version of `as[T]`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26969: [SPARK-30319][SQL] Add a stricter version of `as[T]`
URL: https://github.com/apache/spark/pull/26969#issuecomment-567974187
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26969:
[SPARK-30319][SQL] Add a stricter version of `as[T]`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #26969: [SPARK-30319][SQL] Add a stricter version of `as[T]`
URL: https://github.com/apache/spark/pull/26969#discussion_r364216809
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -495,6 +495,25 @@ class Dataset[T] private[sql](
select(newCols : _*)
}
+ /**
+ * Returns a new Dataset where each record has been mapped on to the specified type.
+ * This only supports `U` being a class. Fields for the class will be mapped to columns of the
+ * same name (case sensitivity is determined by `spark.sql.caseSensitive`).
+ *
+ * If the schema of the Dataset does not match the desired `U` type, you can use `select`
+ * along with `alias` or `as` to rearrange or rename as required.
+ *
+ * This method eagerly projects away any columns that are not present in the specified class.
+ * It further guarantees the order of columns as well as data types to match `U`.
+ *
+ * @group basic
+ * @since 3.0.0
+ */
+ def toDS[U : Encoder]: Dataset[U] = {
+ val columns = implicitly[Encoder[U]].schema.fields.map(f => col(f.name).cast(f.dataType))
Review comment:
what about inner fields? e.g.
```
case class Inner(a: Int)
case class Outer(i: Inner)
```
and the schema is `<col: struct<a: int, b: int>>`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26969: [SPARK-30319][SQL]
Add a stricter version of `as[T]`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26969: [SPARK-30319][SQL] Add a stricter version of `as[T]`
URL: https://github.com/apache/spark/pull/26969#issuecomment-567980169
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on a change in pull request #26969:
[SPARK-30319][SQL] Add a stricter version of `as[T]`
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on a change in pull request #26969: [SPARK-30319][SQL] Add a stricter version of `as[T]`
URL: https://github.com/apache/spark/pull/26969#discussion_r383277727
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -495,6 +495,25 @@ class Dataset[T] private[sql](
select(newCols : _*)
}
+ /**
+ * Returns a new Dataset where each record has been mapped on to the specified type.
+ * This only supports `U` being a class. Fields for the class will be mapped to columns of the
+ * same name (case sensitivity is determined by `spark.sql.caseSensitive`).
+ *
+ * If the schema of the Dataset does not match the desired `U` type, you can use `select`
+ * along with `alias` or `as` to rearrange or rename as required.
+ *
+ * This method eagerly projects away any columns that are not present in the specified class.
+ * It further guarantees the order of columns as well as data types to match `U`.
+ *
+ * @group basic
+ * @since 3.0.0
+ */
+ def toDS[U : Encoder]: Dataset[U] = {
+ val columns = implicitly[Encoder[U]].schema.fields.map(f => col(f.name).cast(f.dataType))
Review comment:
@cloud-fan any thoughts on this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26969:
[SPARK-30319][SQL] Add a stricter version of `as[T]`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #26969: [SPARK-30319][SQL] Add a stricter version of `as[T]`
URL: https://github.com/apache/spark/pull/26969#discussion_r366331841
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -495,6 +495,25 @@ class Dataset[T] private[sql](
select(newCols : _*)
}
+ /**
+ * Returns a new Dataset where each record has been mapped on to the specified type.
+ * This only supports `U` being a class. Fields for the class will be mapped to columns of the
+ * same name (case sensitivity is determined by `spark.sql.caseSensitive`).
+ *
+ * If the schema of the Dataset does not match the desired `U` type, you can use `select`
+ * along with `alias` or `as` to rearrange or rename as required.
+ *
+ * This method eagerly projects away any columns that are not present in the specified class.
+ * It further guarantees the order of columns as well as data types to match `U`.
+ *
+ * @group basic
+ * @since 3.0.0
+ */
+ def toDS[U : Encoder]: Dataset[U] = {
+ val columns = implicitly[Encoder[U]].schema.fields.map(f => col(f.name).cast(f.dataType))
Review comment:
This doesn't work even for simple classes when the encoder is `Encoders.javaSerialization`. I agree that this is a valid use case, and people can create a Util function to do it. But I don't agree this is general enough to be put in a method named `toDS`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on a change in pull request #26969:
[SPARK-30319][SQL] Add a stricter version of `as[T]`
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on a change in pull request #26969: [SPARK-30319][SQL] Add a stricter version of `as[T]`
URL: https://github.com/apache/spark/pull/26969#discussion_r369655551
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -495,6 +495,25 @@ class Dataset[T] private[sql](
select(newCols : _*)
}
+ /**
+ * Returns a new Dataset where each record has been mapped on to the specified type.
+ * This only supports `U` being a class. Fields for the class will be mapped to columns of the
+ * same name (case sensitivity is determined by `spark.sql.caseSensitive`).
+ *
+ * If the schema of the Dataset does not match the desired `U` type, you can use `select`
+ * along with `alias` or `as` to rearrange or rename as required.
+ *
+ * This method eagerly projects away any columns that are not present in the specified class.
+ * It further guarantees the order of columns as well as data types to match `U`.
+ *
+ * @group basic
+ * @since 3.0.0
+ */
+ def toDS[U : Encoder]: Dataset[U] = {
+ val columns = implicitly[Encoder[U]].schema.fields.map(f => col(f.name).cast(f.dataType))
Review comment:
I have added checks to `toDS{T]` that throw a meaningful `AnalysisException` when columns do not line up with encoder's schema. The method is now general enough to support any encoder with a schema that can be created merely by projection, which is true for standard encoder of case classes (without classes that have inner case classes with extra fields or different field order), seq and maps of case classes (without extra fields or different field order) as well as any Java classes (`Encoders.javaSerialisation`). It throws `as[T]`'s Exception if columns do not line up with encoder's schema and `as[T]` would fail either.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] planga82 commented on issue #26969: [SPARK-30319][SQL] Add
a stricter version of `as[T]`
Posted by GitBox <gi...@apache.org>.
planga82 commented on issue #26969: [SPARK-30319][SQL] Add a stricter version of `as[T]`
URL: https://github.com/apache/spark/pull/26969#issuecomment-571785768
Hi @EnricoMi,
I think it is an interesting change. We have an utility in our proyect to solve this situation and avoid problems.
About the name I prefer the first option, but all are good. Thanks for the work!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on a change in pull request #26969:
[SPARK-30319][SQL] Add a stricter version of `as[T]`
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on a change in pull request #26969: [SPARK-30319][SQL] Add a stricter version of `as[T]`
URL: https://github.com/apache/spark/pull/26969#discussion_r367849005
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -495,6 +495,25 @@ class Dataset[T] private[sql](
select(newCols : _*)
}
+ /**
+ * Returns a new Dataset where each record has been mapped on to the specified type.
+ * This only supports `U` being a class. Fields for the class will be mapped to columns of the
+ * same name (case sensitivity is determined by `spark.sql.caseSensitive`).
+ *
+ * If the schema of the Dataset does not match the desired `U` type, you can use `select`
+ * along with `alias` or `as` to rearrange or rename as required.
+ *
+ * This method eagerly projects away any columns that are not present in the specified class.
+ * It further guarantees the order of columns as well as data types to match `U`.
+ *
+ * @group basic
+ * @since 3.0.0
+ */
+ def toDS[U : Encoder]: Dataset[U] = {
+ val columns = implicitly[Encoder[U]].schema.fields.map(f => col(f.name).cast(f.dataType))
Review comment:
Do you agree that the lazy-ness of `as[T]` is an inconsistency of the API where all other transformations' schema instantly reflect the result's schema? With `as[T]` you need a `map(identity)` to really become a `Dataset[T]`.
Which properties of `toDS[T]` would you require?
# case classes, tuples and inner fields supported
# arbitrary implementations of `Encoder` supported
# data (not just schema) of result `Dataset[T]` are identical to `as[T].map(identity)`, which is given when they follow schema `encoder.schema`
# uses projection where possible (efficient), falls back to `map(identity)` when needed (expensive)
The idea is to turn the dataset into the same shape as it would look like when the encoder would produce it (`map(identity)`). In case of `ExpressionEncoder`, the encoder can tell you which columns it serialises / deserialises. If the schema of those columns does not change (e.g. unchanged inner fields), than a simple projection suffices: select those fields in the right order. The data then comply to `encoder.schema`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26969: [SPARK-30319][SQL]
Add a stricter version of `as[T]`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26969: [SPARK-30319][SQL] Add a stricter version of `as[T]`
URL: https://github.com/apache/spark/pull/26969#issuecomment-567974187
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org