You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "hvanhovell (via GitHub)" <gi...@apache.org> on 2023/08/06 19:19:54 UTC
[GitHub] [spark] hvanhovell opened a new pull request, #42366: [SPARK-44686][CONNECT][SQL] Add the ability to create a RowEncoder in Encoders.scala.
hvanhovell opened a new pull request, #42366:
URL: https://github.com/apache/spark/pull/42366
### What changes were proposed in this pull request?
### Why are the changes needed?
It is currently not possible to create a `RowEncoder` using public API. The internal APIs for this will change in Spark 3.5, this means that library maintainers have to update their code if they use a RowEncoder. To avoid happening again, we add this method to the public API.
### Does this PR introduce _any_ user-facing change?
Yes. It adds the `row` method to `Encoders`.
### How was this patch tested?
Added tests to connect and sql.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on pull request #42366: [SPARK-44686][CONNECT][SQL] Add the ability to create a RowEncoder in Encoders.scala.
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on PR #42366:
URL: https://github.com/apache/spark/pull/42366#issuecomment-1667827745
Merging
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42366: [SPARK-44686][CONNECT][SQL] Add the ability to create a RowEncoder in Encoders.scala.
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42366:
URL: https://github.com/apache/spark/pull/42366#discussion_r1285276555
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/Encoders.scala:
##########
@@ -178,6 +178,12 @@ object Encoders {
*/
def bean[T](beanClass: Class[T]): Encoder[T] = ExpressionEncoder.javaBean(beanClass)
+ /**
+ * Creates a [[Row]] encoder for schema `schema`.
+ * Since: 3.5.0
Review Comment:
```suggestion
* Creates a [[Row]] encoder for schema `schema`.
*
* @since 3.5.0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42366: [SPARK-44686][CONNECT][SQL] Add the ability to create a RowEncoder in Encoders.scala.
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42366:
URL: https://github.com/apache/spark/pull/42366#discussion_r1285276527
##########
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Encoders.scala:
##########
@@ -168,6 +169,11 @@ object Encoders {
*/
def bean[T](beanClass: Class[T]): Encoder[T] = JavaTypeInference.encoderFor(beanClass)
+ /**
+ * Creates a [[Row]] encoder for schema `schema`. Since: 3.5.0
Review Comment:
```suggestion
* Creates a [[Row]] encoder for schema `schema`.
*
* @since 3.5.0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on pull request #42366: [SPARK-44686][CONNECT][SQL] Add the ability to create a RowEncoder in Encoders.scala.
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on PR #42366:
URL: https://github.com/apache/spark/pull/42366#issuecomment-1666948630
For the reviewers, I intent to backport this API as far back as I can. Please shout if you have concerns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42366: [SPARK-44686][CONNECT][SQL] Add the ability to create a RowEncoder in Encoders.scala.
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42366:
URL: https://github.com/apache/spark/pull/42366#discussion_r1285276555
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/Encoders.scala:
##########
@@ -178,6 +178,12 @@ object Encoders {
*/
def bean[T](beanClass: Class[T]): Encoder[T] = ExpressionEncoder.javaBean(beanClass)
+ /**
+ * Creates a [[Row]] encoder for schema `schema`.
+ * Since: 3.5.0
Review Comment:
```suggestion
* Creates a [[Row]] encoder for schema `schema`.
*
* @since 3.5.0
```
##########
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Encoders.scala:
##########
@@ -168,6 +169,11 @@ object Encoders {
*/
def bean[T](beanClass: Class[T]): Encoder[T] = JavaTypeInference.encoderFor(beanClass)
+ /**
+ * Creates a [[Row]] encoder for schema `schema`. Since: 3.5.0
Review Comment:
```suggestion
* Creates a [[Row]] encoder for schema `schema`.
*
* @since 3.5.0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell closed pull request #42366: [SPARK-44686][CONNECT][SQL] Add the ability to create a RowEncoder in Encoders.scala.
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell closed pull request #42366: [SPARK-44686][CONNECT][SQL] Add the ability to create a RowEncoder in Encoders.scala.
URL: https://github.com/apache/spark/pull/42366
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org