You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by francoisprunier <gi...@git.apache.org> on 2016/03/17 16:42:31 UTC
[GitHub] spark pull request: [SPARK-11319][SQL] Making StructField's nullab...
GitHub user francoisprunier opened a pull request:
https://github.com/apache/spark/pull/11785
[SPARK-11319][SQL] Making StructField's nullable field documentation clearer
## What changes were proposed in this pull request?
Be more descriptive in what the nullable field of StructField is intended for. At the moment, the description does not explain that it's only a hint to the optimizer, and many people are mistakenly thinking that this is a "NOT NULL" type constraint. Making it clear in the doc should help people understand its purpose.
## How was this patch tested?
Not tested as it's only documentation
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/francoisprunier/spark StructFieldNullableDoc
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11785.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11785
----
commit 58514e73352a47579cf56e921172924245356c61
Author: Francois Prunier <fr...@hurence.com>
Date: 2016-03-17T15:29:08Z
making StructField nullable field documentation clearer
commit 2f986682b9c4bf9490b3c29d194e191067d21be6
Author: Francois Prunier <fr...@hurence.com>
Date: 2016-03-17T15:41:33Z
typo
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11319][SQL] Making StructField's nullab...
Posted by kevincox <gi...@git.apache.org>.
Github user kevincox commented on the pull request:
https://github.com/apache/spark/pull/11785#issuecomment-197957375
:+1:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11319][SQL] Making StructField's nullab...
Posted by kevincox <gi...@git.apache.org>.
Github user kevincox commented on the pull request:
https://github.com/apache/spark/pull/11785#issuecomment-210144539
I would love for this to be a constraint.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #11785: [SPARK-11319][SQL] Making StructField's nullable ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/11785
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11319][SQL] Making StructField's nullab...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11785#issuecomment-197941327
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11319][SQL] Making StructField's nullab...
Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11785#discussion_r62391024
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructField.scala ---
@@ -24,7 +24,9 @@ import org.json4s.JsonDSL._
* A field inside a StructType.
* @param name The name of this field.
* @param dataType The data type of this field.
- * @param nullable Indicates if values of this field can be `null` values.
+ * @param nullable Indicates if values of this field can be `null` values. Note that this is ONLY A HINT to the
--- End diff --
According to @marmbrus's [comment here](https://issues.apache.org/jira/browse/SPARK-11319?focusedCommentId=15014678&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15014678) this is actually much stronger than a hint:
> Fair, this is **a contract with the optimizer that you will not produce null data**. If you are not certain you will not produce null values then set this to true.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11319][SQL] Making StructField's nullab...
Posted by blrnw3 <gi...@git.apache.org>.
Github user blrnw3 commented on the pull request:
https://github.com/apache/spark/pull/11785#issuecomment-210134027
👍Agreed, it's very confusing.
Maybe you could also explain why it doesn't act as a type constraint? I feel that would be useful.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org