You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by rxin <gi...@git.apache.org> on 2015/10/23 06:29:06 UTC
[GitHub] spark pull request: [SPARK-11273][SQL] Move ArrayData/MapData/Data...
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/9239
[SPARK-11273][SQL] Move ArrayData/MapData/DataTypeParser to catalyst.util package
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rxin/spark types-private
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9239.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9239
----
commit b0557f1b5b7d44455e6ae3c24dde2403f44bee9a
Author: Reynold Xin <rx...@databricks.com>
Date: 2015-10-23T04:27:56Z
[SPARK-11273][SQL] Move ArrayData/MapData to catalyst.util package
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11273][SQL] Move ArrayData/MapData/Data...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/9239#issuecomment-150475822
Where is UnsafeArrayMap?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11273][SQL] Move ArrayData/MapData/Data...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9239#issuecomment-150488675
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44198/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11273][SQL] Move ArrayData/MapData/Data...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9239#issuecomment-150467592
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11273][SQL] Move ArrayData/MapData/Data...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/9239
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11273][SQL] Move ArrayData/MapData/Data...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/9239#issuecomment-150467369
cc @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11273][SQL] Move ArrayData/MapData/Data...
Posted by cequencer <gi...@git.apache.org>.
Github user cequencer commented on the pull request:
https://github.com/apache/spark/pull/9239#issuecomment-152584279
Yes, it is clean and pure, up to this commit: 96cf87f66d47245b19e719cb83947042b21546fa
Here are the options I am using with "make-distribution.sh"
--tgz --with-tachyon -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver -Psparkr -Dhadoop.version=2.6.0 -DskipTests clean package
I am not sure if Jenkins have a build using these combination.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11273][SQL] Move ArrayData/MapData/Data...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/9239#issuecomment-150469831
Logically unsafe data structures(UTF8String, UnsafeArrayMap, etc.) does not belong to `type` package, either. But they are public, should we move them too?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11273][SQL] Move ArrayData/MapData/Data...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9239#issuecomment-150488554
**[Test build #44198 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44198/consoleFull)** for PR 9239 at commit [`dde2dfd`](https://github.com/apache/spark/commit/dde2dfdda27237cdce29b9f164f9d5ca253d6945).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11273][SQL] Move ArrayData/MapData/Data...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/9239#issuecomment-152407939
Are you sure your local master branch is latest and clean? This PR has been merge for a week and everything goes well so far.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11273][SQL] Move ArrayData/MapData/Data...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/9239#issuecomment-152697692
It looks to me that you still have those moved files in `/tmp/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/types`, can you check that? And can you git clone a new copy of spark code and try again?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11273][SQL] Move ArrayData/MapData/Data...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9239#issuecomment-150467898
**[Test build #44198 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44198/consoleFull)** for PR 9239 at commit [`dde2dfd`](https://github.com/apache/spark/commit/dde2dfdda27237cdce29b9f164f9d5ca253d6945).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11273][SQL] Move ArrayData/MapData/Data...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9239#issuecomment-150467603
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11273][SQL] Move ArrayData/MapData/Data...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9239#issuecomment-150488674
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11273][SQL] Move ArrayData/MapData/Data...
Posted by cequencer <gi...@git.apache.org>.
Github user cequencer commented on the pull request:
https://github.com/apache/spark/pull/9239#issuecomment-156150278
Thank you for the helpful tip. You are correct that there were previous artifacts that were NOT "cleaned" that caused this compile error. I had to wipe everything and git clone a new copy of spark code to address my issue. Now I learned that the "clean" option I passed in to "make-distribution.sh" does not really "clean" classes that was moved. Sorry for the trouble. Everything is okay now. Thank you!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11273][SQL] Move ArrayData/MapData/Data...
Posted by cequencer <gi...@git.apache.org>.
Github user cequencer commented on the pull request:
https://github.com/apache/spark/pull/9239#issuecomment-152391757
I am trying to build the latest master branch, but I am having compile errors that seemed to be coming from this commit: cdea0174e32a5f4c28fd59899b2e9774994303d5
$ git blame -L30,31 sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala
cdea0174 (Reynold Xin 2015-10-23 00:00:21 -0700 30) import org.apache.spark.sql.catalyst.util._
67d06880 (Volodymyr Lyubinets 2015-04-10 16:27:56 -0700 31) import org.apache.spark.sql.types._
$ git blame -L25,26 sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
cdea0174 (Reynold Xin 2015-10-23 00:00:21 -0700 25) import org.apache.spark.sql.catalyst.util._
f9969098 (Reynold Xin 2015-01-13 17:16:41 -0800 26) import org.apache.spark.sql.types._
I appreciate any feedback and guidance you can provide on what I maybe doing wrong. Thanks!
Compile errors stdout below:
[error] /tmp/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala:154: reference to ArrayData is ambiguous;
[error] it is imported twice in the same scope by
[error] import org.apache.spark.sql.types._
[error] and import org.apache.spark.sql.catalyst.util._
[error] elementType: DataType) extends CatalystTypeConverter[Any, Seq[Any], ArrayData] {
[error] ^
[error] /tmp/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala:196: reference to MapData is ambiguous;
[error] it is imported twice in the same scope by
[error] import org.apache.spark.sql.types._
[error] and import org.apache.spark.sql.catalyst.util._
[error] extends CatalystTypeConverter[Any, Map[Any, Any], MapData] {
[error] ^
[error] /tmp/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala:175: reference to ArrayData is ambiguous;
[error] it is imported twice in the same scope by
[error] import org.apache.spark.sql.types._
[error] and import org.apache.spark.sql.catalyst.util._
[error] override def toScala(catalystValue: ArrayData): Seq[Any] = {
[error] ^
[error] /tmp/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala:231: reference to MapData is ambiguous;
[error] it is imported twice in the same scope by
[error] import org.apache.spark.sql.types._
[error] and import org.apache.spark.sql.catalyst.util._
[error] override def toScala(catalystValue: MapData): Map[Any, Any] = {
[error] ^
[error] /tmp/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala:158: reference to ArrayData is ambiguous;
[error] it is imported twice in the same scope by
[error] import org.apache.spark.sql.types._
[error] and import org.apache.spark.sql.catalyst.util._
[error] override def toCatalystImpl(scalaValue: Any): ArrayData = {
[error] ^
[error] /tmp/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala:161: reference to GenericArrayData is ambiguous;
[error] it is imported twice in the same scope by
[error] import org.apache.spark.sql.types._
[error] and import org.apache.spark.sql.catalyst.util._
[error] new GenericArrayData(a.map(elementConverter.toCatalyst))
[error] ^
[error] /tmp/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala:163: reference to GenericArrayData is ambiguous;
[error] it is imported twice in the same scope by
[error] import org.apache.spark.sql.types._
[error] and import org.apache.spark.sql.catalyst.util._
[error] new GenericArrayData(s.map(elementConverter.toCatalyst).toArray)
[error] ^
[error] /tmp/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala:171: reference to GenericArrayData is ambiguous;
[error] it is imported twice in the same scope by
[error] import org.apache.spark.sql.types._
[error] and import org.apache.spark.sql.catalyst.util._
[error] new GenericArrayData(convertedIterable.toArray)
[error] ^
[error] /tmp/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala:201: reference to MapData is ambiguous;
[error] it is imported twice in the same scope by
[error] import org.apache.spark.sql.types._
[error] and import org.apache.spark.sql.catalyst.util._
[error] override def toCatalystImpl(scalaValue: Any): MapData = scalaValue match {
[error] ^
[error] /tmp/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala:213: reference to ArrayBasedMapData is ambiguous;
[error] it is imported twice in the same scope by
[error] import org.apache.spark.sql.types._
[error] and import org.apache.spark.sql.catalyst.util._
[error] ArrayBasedMapData(convertedKeys, convertedValues)
[error] ^
[error] /tmp/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala:228: reference to ArrayBasedMapData is ambiguous;
[error] it is imported twice in the same scope by
[error] import org.apache.spark.sql.types._
[error] and import org.apache.spark.sql.catalyst.util._
[error] ArrayBasedMapData(convertedKeys, convertedValues)
[error] ^
[error] /tmp/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala:431: reference to GenericArrayData is ambiguous;
[error] it is imported twice in the same scope by
[error] import org.apache.spark.sql.types._
[error] and import org.apache.spark.sql.catalyst.util._
[error] case seq: Seq[Any] => new GenericArrayData(seq.map(convertToCatalyst).toArray)
[error] ^
[error] /tmp/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala:433: reference to GenericArrayData is ambiguous;
[error] it is imported twice in the same scope by
[error] import org.apache.spark.sql.types._
[error] and import org.apache.spark.sql.catalyst.util._
[error] case arr: Array[Any] => new GenericArrayData(arr.map(convertToCatalyst))
[error] ^
[error] /tmp/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala:445: reference to ArrayBasedMapData is ambiguous;
[error] it is imported twice in the same scope by
[error] import org.apache.spark.sql.types._
[error] and import org.apache.spark.sql.catalyst.util._
[error] ArrayBasedMapData(convertedKeys, convertedValues)
[error] ^
[error] /tmp/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:373: reference to ArrayData is ambiguous;
[error] it is imported twice in the same scope by
[error] import org.apache.spark.sql.types._
[error] and import org.apache.spark.sql.catalyst.util._
[error] buildCast[ArrayData](_, array => {
[error] ^
[error] /tmp/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:389: reference to MapData is ambiguous;
[error] it is imported twice in the same scope by
[error] import org.apache.spark.sql.types._
[error] and import org.apache.spark.sql.catalyst.util._
[error] buildCast[MapData](_, map => {
[error] ^
[error] /tmp/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:814: reference to GenericArrayData is ambiguous;
[error] it is imported twice in the same scope by
[error] import org.apache.spark.sql.types._
[error] and import org.apache.spark.sql.catalyst.util._
[error] val arrayClass = classOf[GenericArrayData].getName
[error] ^
[error] /tmp/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:851: reference to ArrayBasedMapData is ambiguous;
[error] it is imported twice in the same scope by
[error] import org.apache.spark.sql.types._
[error] and import org.apache.spark.sql.catalyst.util._
[error] val mapClass = classOf[ArrayBasedMapData].getName
[error] ^
[warn] /tmp/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala:27: imported `DataTypeParser' is permanently hidden by definition of object DataTypeParser in package types
[warn] import org.apache.spark.sql.catalyst.util.DataTypeParser
[warn] ^
[warn] one warning found
[error] 18 errors found
[error] (catalyst/compile:compile) Compilation failed
[error] Total time: 559 s, completed Oct 29, 2015 7:07:47 PM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11273][SQL] Move ArrayData/MapData/Data...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/9239#issuecomment-150524806
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org