You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by avati <gi...@git.apache.org> on 2014/08/01 05:03:05 UTC
[GitHub] spark pull request: [SPARK-1812] sql/catalyst - remove scala.NotNu...
GitHub user avati opened a pull request:
https://github.com/apache/spark/pull/1709
[SPARK-1812] sql/catalyst - remove scala.NotNull related tests
scala.NotNull Trait is removed in 2.11
Signed-off-by: Anand Avati <av...@redhat.com>
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/avati/spark SPARK-1812-notnull
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1709.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1709
----
commit 219b4c077a6462cd7e5d53e879068dc4350248fa
Author: Anand Avati <av...@redhat.com>
Date: 2014-07-31T02:36:32Z
SPARK-1812: sql/catalyst - remove scala.NotNull related tests
scala.NotNull Trait is removed in 2.11
Signed-off-by: Anand Avati <av...@redhat.com>
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] sql/catalyst - Provide explicit t...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1709#issuecomment-50950561
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] sql/catalyst - Provide explicit t...
Posted by avati <gi...@git.apache.org>.
Github user avati commented on the pull request:
https://github.com/apache/spark/pull/1709#issuecomment-50941761
The failures seem to be odd and unrelated to the patch itself.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] sql/catalyst - remove scala.NotNu...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1709#issuecomment-50918726
Ah, thanks for the clarification.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] sql/catalyst - remove scala.NotNu...
Posted by avati <gi...@git.apache.org>.
Github user avati commented on the pull request:
https://github.com/apache/spark/pull/1709#issuecomment-50924789
Yeah, I was thinking:
def notNull: AttributeReference = a.withNullability(false)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] sql/catalyst - remove scala.NotNu...
Posted by avati <gi...@git.apache.org>.
Github user avati commented on the pull request:
https://github.com/apache/spark/pull/1709#issuecomment-50850247
> These tests shouldn't be using scala NotNull, but catalyst's .notNull
> <https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala#L199>
> .
>
Hmm, now that I look more carefully what you say makes sense. Trying to
figure out why notNull (catalyst's) was not found when compiling with Scala
2.11. The missing scala.NotNull in 2.11 was just a close coincidence I
think.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] sql/catalyst - remove scala.NotNu...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1709#issuecomment-50916913
Mind closing this PR if its not an issue then? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] sql/catalyst - remove scala.NotNu...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1709#issuecomment-50849795
These tests shouldn't be using scala NotNull, but catalyst's [.notNull](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala#L199).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] sql/catalyst - remove scala.NotNu...
Posted by avati <gi...@git.apache.org>.
Github user avati commented on the pull request:
https://github.com/apache/spark/pull/1709#issuecomment-50919009
This is the exact failure. The error is actually missing at() method
encountered in Scala 2.11
[ERROR] /Users/avati/work/spark/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionEvaluationSuite.scala:370:
value at is not a member of
org.apache.spark.sql.catalyst.expressions.Attribute
[ERROR] val c4_notNull = 'a.boolean.notNull.at(3)
[ERROR] ^
[ERROR] /Users/avati/work/spark/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionEvaluationSuite.scala:371:
value at is not a member of
org.apache.spark.sql.catalyst.expressions.Attribute
[ERROR] val c5_notNull = 'a.boolean.notNull.at(4)
[ERROR] ^
[ERROR] /Users/avati/work/spark/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionEvaluationSuite.scala:372:
value at is not a member of
org.apache.spark.sql.catalyst.expressions.Attribute
[ERROR] val c6_notNull = 'a.boolean.notNull.at(5)
[ERROR] ^
[ERROR] /Users/avati/work/spark/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionEvaluationSuite.scala:558:
value at is not a member of
org.apache.spark.sql.catalyst.expressions.Attribute
[ERROR] val s_notNull = 'a.string.notNull.at(0)
[ERROR] ^
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] sql/catalyst - remove scala.NotNu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1709#issuecomment-50844662
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] sql/catalyst - remove scala.NotNu...
Posted by avati <gi...@git.apache.org>.
Github user avati commented on the pull request:
https://github.com/apache/spark/pull/1709#issuecomment-50923602
So, it looks like with:
a.withNullability (in
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala#L199
)
type inference is selecting the type Attribute at
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala#L60
instead of AttributeReference at
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala#L137
Do you have a preference on how to make the code less ambiguous?
On Fri, Aug 1, 2014 at 11:41 AM, Michael Armbrust <no...@github.com>
wrote:
> at should be added by this implicit conversion
> <https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala#L205>
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/1709#issuecomment-50920007>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] sql/catalyst - remove scala.NotNu...
Posted by avati <gi...@git.apache.org>.
Github user avati commented on the pull request:
https://github.com/apache/spark/pull/1709#issuecomment-50919584
> It is possibly a change with type inference / implicit conversions?
>
Likely. What is the expected type which is supposed to have 'def at()',
that line of code is assuming?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] sql/catalyst - remove scala.NotNu...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1709#issuecomment-50920007
`at` should be added by [this implicit conversion](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala#L205)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] sql/catalyst - Provide explicit t...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/1709
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1812] sql/catalyst - Provide explicit t...
Posted by avati <gi...@git.apache.org>.
Github user avati commented on the pull request:
https://github.com/apache/spark/pull/1709#issuecomment-50926464
@marmbrus - this latest patch fixes the Scala 2.11 compilation breakage
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] sql/catalyst - Provide explicit t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1709#issuecomment-50950687
QA tests have started for PR 1709. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17730/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] sql/catalyst - remove scala.NotNu...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1709#issuecomment-50924529
I haven't looked closely but if we can make it work by specifying more generic explicit return types to our implicits that seems like a good approach.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] sql/catalyst - Provide explicit t...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1709#issuecomment-50926672
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] sql/catalyst - Provide explicit t...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1709#issuecomment-50941817
I agree.
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] sql/catalyst - remove scala.NotNu...
Posted by avati <gi...@git.apache.org>.
Github user avati commented on the pull request:
https://github.com/apache/spark/pull/1709#issuecomment-50918557
> Mind closing this PR if its not an issue then? Thanks!
>
The description of the cause (scala.NotNull trait removal in 2.11) might be
wrong, but those tests are still failing with Scala 2.11. I'm still
investigating.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] sql/catalyst - Provide explicit t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1709#issuecomment-50952324
QA results for PR 1709:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17730/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] sql/catalyst - Provide explicit t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1709#issuecomment-50934277
QA results for PR 1709:<br>- This patch FAILED unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17693/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] sql/catalyst - Provide explicit t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1709#issuecomment-50927062
QA tests have started for PR 1709. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17693/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] sql/catalyst - remove scala.NotNu...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1709#issuecomment-50919110
It is possibly a change with type inference / implicit conversions?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---