You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by rxin <gi...@git.apache.org> on 2015/11/07 00:40:33 UTC
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/9531
[SPARK-11564][SQL] Dataset Java API audit
A few changes:
1. Removed fold
2. Created specific interfaces for each Dataset function (e.g. MapFunction, ReduceFunction, MapPartitionsFunction)
3. Added more documentation and test cases.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rxin/spark SPARK-11564
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9531.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9531
----
commit c5c694f68e54da6fdeb8ededb8f9976bb3c7bdf4
Author: Reynold Xin <rx...@databricks.com>
Date: 2015-11-06T23:38:47Z
[SPARK-11564][SQL] Dataset Java API audit
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9531#issuecomment-154598126
**[Test build #45270 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45270/consoleFull)** for PR 9531 at commit [`5f7acc8`](https://github.com/apache/spark/commit/5f7acc8021ba7012191613cf267b31b302aaad4c).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `final class ShuffleSortDataFormat extends SortDataFormat<PackedRecordPointer, LongArray> `\n * `final class UnsafeSortDataFormat extends SortDataFormat<RecordPointerAndKeyPrefix, LongArray> `\n * `abstract class Writer extends BaseReadWrite `\n * `trait Writable `\n * `abstract class Reader[T] extends BaseReadWrite `\n * `trait Readable[T] `\n * `case class GetInternalRowField(child: Expression, ordinal: Int, dataType: DataType)`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9531#issuecomment-154619132
**[Test build #2001 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2001/consoleFull)** for PR 9531 at commit [`41cee21`](https://github.com/apache/spark/commit/41cee21b1a7c1b8b9d6875ecd6fcf048ca7274c7).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `final class ShuffleSortDataFormat extends SortDataFormat<PackedRecordPointer, LongArray> `\n * `final class UnsafeSortDataFormat extends SortDataFormat<RecordPointerAndKeyPrefix, LongArray> `\n * `abstract class Writer extends BaseReadWrite `\n * `trait Writable `\n * `abstract class Reader[T] extends BaseReadWrite `\n * `trait Readable[T] `\n * `case class GetInternalRowField(child: Expression, ordinal: Int, dataType: DataType)`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9531#discussion_r44202935
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -166,8 +171,7 @@ class Dataset[T] private[sql](
* Returns a new [[Dataset]] that only contains elements where `func` returns `true`.
* @since 1.6.0
*/
- def filter(func: JFunction[T, java.lang.Boolean]): Dataset[T] =
- filter(t => func.call(t).booleanValue())
+ def filter(func: FilterFunction[T]): Dataset[T] = filter(t => func.call(t).booleanValue())
--- End diff --
Get rid of `booleanValue`? I think this is boxing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9531#discussion_r44202667
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala ---
@@ -1478,18 +1478,54 @@ class DataFrame private[sql](
/**
* Returns the first `n` rows in the [[DataFrame]].
+ *
+ * Running take requires moving data into the application's driver process, and doing so on a
+ * very large dataset can crash the driver process with OutOfMemoryError.
--- End diff --
and doing so with a very large `n`? as phrased now it seems more dangerous than it is.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/9531#discussion_r44209647
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala ---
@@ -1478,18 +1478,54 @@ class DataFrame private[sql](
/**
* Returns the first `n` rows in the [[DataFrame]].
+ *
+ * Running take requires moving data into the application's driver process, and doing so on a
+ * very large dataset can crash the driver process with OutOfMemoryError.
+ *
* @group action
* @since 1.3.0
*/
def take(n: Int): Array[Row] = head(n)
/**
+ * Returns the first `n` rows in the [[DataFrame]] as a list.
+ *
+ * Running take requires moving data into the application's driver process, and doing so with
+ * a very large `n` can crash the driver process with OutOfMemoryError.
+ *
+ * @group action
+ * @since 1.6.0
+ */
+ def takeAsList(n: Int): java.util.List[Row] = java.util.Arrays.asList(take(n) : _*)
+
+ /**
* Returns an array that contains all of [[Row]]s in this [[DataFrame]].
+ *
+ * Running take requires moving data into the application's driver process, and doing so with
+ * a very large `n` can crash the driver process with OutOfMemoryError.
+ *
+ * For Java API, use [[collectAsList]].
+ *
* @group action
* @since 1.3.0
*/
def collect(): Array[Row] = collect(needCallback = true)
+ /**
+ * Returns a Java list that contains all of [[Row]]s in this [[DataFrame]].
+ *
+ * Running collect requires moving all the data into the application's driver process, and
+ * doing so on a very large dataset can crash the driver process with OutOfMemoryError.
+ *
+ * @group action
+ * @since 1.3.0
+ */
+ def collectAsList(): java.util.List[Row] = withCallback("collectAsList", this) { _ =>
+ withNewExecutionId {
+ java.util.Arrays.asList(rdd.collect() : _*)
+ }
+ }
--- End diff --
how about:
```
withCallback("collectAsList", this) { df =>
df.collect(needCallback = false).asJava
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9531#issuecomment-154598162
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9531#discussion_r44215283
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala ---
@@ -1478,18 +1478,54 @@ class DataFrame private[sql](
/**
* Returns the first `n` rows in the [[DataFrame]].
+ *
+ * Running take requires moving data into the application's driver process, and doing so on a
+ * very large dataset can crash the driver process with OutOfMemoryError.
+ *
* @group action
* @since 1.3.0
*/
def take(n: Int): Array[Row] = head(n)
/**
+ * Returns the first `n` rows in the [[DataFrame]] as a list.
+ *
+ * Running take requires moving data into the application's driver process, and doing so with
+ * a very large `n` can crash the driver process with OutOfMemoryError.
+ *
+ * @group action
+ * @since 1.6.0
+ */
+ def takeAsList(n: Int): java.util.List[Row] = java.util.Arrays.asList(take(n) : _*)
+
+ /**
* Returns an array that contains all of [[Row]]s in this [[DataFrame]].
+ *
+ * Running take requires moving data into the application's driver process, and doing so with
+ * a very large `n` can crash the driver process with OutOfMemoryError.
--- End diff --
damn fixed the wrong
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9531#discussion_r44202704
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala ---
@@ -1478,18 +1478,54 @@ class DataFrame private[sql](
/**
* Returns the first `n` rows in the [[DataFrame]].
+ *
+ * Running take requires moving data into the application's driver process, and doing so on a
+ * very large dataset can crash the driver process with OutOfMemoryError.
+ *
* @group action
* @since 1.3.0
*/
def take(n: Int): Array[Row] = head(n)
/**
+ * Returns the first `n` rows in the [[DataFrame]] as a list.
+ *
+ * Running take requires moving data into the application's driver process, and doing so on a
+ * very large dataset can crash the driver process with OutOfMemoryError.
+ *
+ * @group action
+ * @since 1.6.0
+ */
+ def takeAsList(n: Int): java.util.List[Row] = java.util.Arrays.asList(take(n) : _*)
+
+ /**
* Returns an array that contains all of [[Row]]s in this [[DataFrame]].
+ *
+ * Running collect requires moving all the data into the application's driver process, and
+ * doing so on a very large dataset can crash the driver process with OutOfMemoryError.
+ *
+ * For Java API, use [[collectAsList]].
+ *
* @group action
* @since 1.3.0
*/
def collect(): Array[Row] = collect(needCallback = true)
+ /**
+ * Returns a Java list that contains all of [[Row]]s in this [[DataFrame]].
+ *
+ * Running collect requires moving all the data into the application's driver process, and
+ * doing so on a very large dataset can crash the driver process with OutOfMemoryError.
+ *
+ * @group action
+ * @since 1.3.0
+ */
+ def collectAsList(): java.util.List[Row] = withCallback("collectAsList", this) { _ =>
--- End diff --
oh hmm, not in the PR, but we should add `withCallback` to dataset too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9531#issuecomment-154579345
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9531#issuecomment-154596370
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9531#issuecomment-154578494
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9531#issuecomment-154596932
**[Test build #45275 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45275/consoleFull)** for PR 9531 at commit [`41cee21`](https://github.com/apache/spark/commit/41cee21b1a7c1b8b9d6875ecd6fcf048ca7274c7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9531#issuecomment-154619784
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/9531#discussion_r44209587
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala ---
@@ -1478,18 +1478,54 @@ class DataFrame private[sql](
/**
* Returns the first `n` rows in the [[DataFrame]].
+ *
+ * Running take requires moving data into the application's driver process, and doing so on a
+ * very large dataset can crash the driver process with OutOfMemoryError.
+ *
* @group action
* @since 1.3.0
*/
def take(n: Int): Array[Row] = head(n)
/**
+ * Returns the first `n` rows in the [[DataFrame]] as a list.
+ *
+ * Running take requires moving data into the application's driver process, and doing so with
+ * a very large `n` can crash the driver process with OutOfMemoryError.
+ *
+ * @group action
+ * @since 1.6.0
+ */
+ def takeAsList(n: Int): java.util.List[Row] = java.util.Arrays.asList(take(n) : _*)
--- End diff --
In other places we usually `import scala.collection.JavaConverters._` and use `asJava` to do the conversion, should we follow that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9531#discussion_r44202818
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala ---
@@ -1478,18 +1478,54 @@ class DataFrame private[sql](
/**
* Returns the first `n` rows in the [[DataFrame]].
+ *
+ * Running take requires moving data into the application's driver process, and doing so on a
+ * very large dataset can crash the driver process with OutOfMemoryError.
+ *
* @group action
* @since 1.3.0
*/
def take(n: Int): Array[Row] = head(n)
/**
+ * Returns the first `n` rows in the [[DataFrame]] as a list.
+ *
+ * Running take requires moving data into the application's driver process, and doing so on a
+ * very large dataset can crash the driver process with OutOfMemoryError.
+ *
+ * @group action
+ * @since 1.6.0
+ */
+ def takeAsList(n: Int): java.util.List[Row] = java.util.Arrays.asList(take(n) : _*)
+
+ /**
* Returns an array that contains all of [[Row]]s in this [[DataFrame]].
+ *
+ * Running collect requires moving all the data into the application's driver process, and
+ * doing so on a very large dataset can crash the driver process with OutOfMemoryError.
+ *
+ * For Java API, use [[collectAsList]].
--- End diff --
`which will return the correct type without casting`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9531#discussion_r44215279
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala ---
@@ -1478,18 +1478,54 @@ class DataFrame private[sql](
/**
* Returns the first `n` rows in the [[DataFrame]].
+ *
+ * Running take requires moving data into the application's driver process, and doing so on a
+ * very large dataset can crash the driver process with OutOfMemoryError.
+ *
* @group action
* @since 1.3.0
*/
def take(n: Int): Array[Row] = head(n)
/**
+ * Returns the first `n` rows in the [[DataFrame]] as a list.
+ *
+ * Running take requires moving data into the application's driver process, and doing so with
+ * a very large `n` can crash the driver process with OutOfMemoryError.
+ *
+ * @group action
+ * @since 1.6.0
+ */
+ def takeAsList(n: Int): java.util.List[Row] = java.util.Arrays.asList(take(n) : _*)
--- End diff --
take returns an array not a seq though
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9531#issuecomment-154597482
**[Test build #45269 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45269/consoleFull)** for PR 9531 at commit [`c5c694f`](https://github.com/apache/spark/commit/c5c694f68e54da6fdeb8ededb8f9976bb3c7bdf4).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `abstract class Writer extends BaseReadWrite `\n * `trait Writable `\n * `abstract class Reader[T] extends BaseReadWrite `\n * `trait Readable[T] `\n * `case class GetInternalRowField(child: Expression, ordinal: Int, dataType: DataType)`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9531#issuecomment-154619724
**[Test build #45275 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45275/consoleFull)** for PR 9531 at commit [`41cee21`](https://github.com/apache/spark/commit/41cee21b1a7c1b8b9d6875ecd6fcf048ca7274c7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `abstract class Writer extends BaseReadWrite `\n * `trait Writable `\n * `abstract class Reader[T] extends BaseReadWrite `\n * `trait Readable[T] `\n * `case class GetInternalRowField(child: Expression, ordinal: Int, dataType: DataType)`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9531#issuecomment-154579148
**[Test build #45269 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45269/consoleFull)** for PR 9531 at commit [`c5c694f`](https://github.com/apache/spark/commit/c5c694f68e54da6fdeb8ededb8f9976bb3c7bdf4).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/9531#issuecomment-154579075
cc @cloud-fan @marmbrus
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/9531
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/9531#discussion_r44209605
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala ---
@@ -1478,18 +1478,54 @@ class DataFrame private[sql](
/**
* Returns the first `n` rows in the [[DataFrame]].
+ *
+ * Running take requires moving data into the application's driver process, and doing so on a
+ * very large dataset can crash the driver process with OutOfMemoryError.
+ *
* @group action
* @since 1.3.0
*/
def take(n: Int): Array[Row] = head(n)
/**
+ * Returns the first `n` rows in the [[DataFrame]] as a list.
+ *
+ * Running take requires moving data into the application's driver process, and doing so on a
+ * very large dataset can crash the driver process with OutOfMemoryError.
+ *
+ * @group action
+ * @since 1.6.0
+ */
+ def takeAsList(n: Int): java.util.List[Row] = java.util.Arrays.asList(take(n) : _*)
+
+ /**
* Returns an array that contains all of [[Row]]s in this [[DataFrame]].
+ *
+ * Running collect requires moving all the data into the application's driver process, and
+ * doing so on a very large dataset can crash the driver process with OutOfMemoryError.
+ *
+ * For Java API, use [[collectAsList]].
+ *
* @group action
* @since 1.3.0
*/
def collect(): Array[Row] = collect(needCallback = true)
+ /**
+ * Returns a Java list that contains all of [[Row]]s in this [[DataFrame]].
+ *
+ * Running collect requires moving all the data into the application's driver process, and
+ * doing so on a very large dataset can crash the driver process with OutOfMemoryError.
+ *
+ * @group action
+ * @since 1.3.0
+ */
+ def collectAsList(): java.util.List[Row] = withCallback("collectAsList", this) { _ =>
--- End diff --
I'm not sure if we need the `xxxAsList` methods. The `DataFrame` is typed with `Row` and the return type of `collect` or `take` is `Array[Row]` which works well at java side.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9531#issuecomment-154600465
**[Test build #2001 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2001/consoleFull)** for PR 9531 at commit [`41cee21`](https://github.com/apache/spark/commit/41cee21b1a7c1b8b9d6875ecd6fcf048ca7274c7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9531#discussion_r44202846
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala ---
@@ -1478,18 +1478,54 @@ class DataFrame private[sql](
/**
* Returns the first `n` rows in the [[DataFrame]].
+ *
+ * Running take requires moving data into the application's driver process, and doing so on a
+ * very large dataset can crash the driver process with OutOfMemoryError.
+ *
* @group action
* @since 1.3.0
*/
def take(n: Int): Array[Row] = head(n)
/**
+ * Returns the first `n` rows in the [[DataFrame]] as a list.
+ *
+ * Running take requires moving data into the application's driver process, and doing so on a
+ * very large dataset can crash the driver process with OutOfMemoryError.
+ *
+ * @group action
+ * @since 1.6.0
+ */
+ def takeAsList(n: Int): java.util.List[Row] = java.util.Arrays.asList(take(n) : _*)
+
+ /**
* Returns an array that contains all of [[Row]]s in this [[DataFrame]].
+ *
+ * Running collect requires moving all the data into the application's driver process, and
+ * doing so on a very large dataset can crash the driver process with OutOfMemoryError.
+ *
+ * For Java API, use [[collectAsList]].
--- End diff --
oh wait, why? The array variance problems don't exist for dataframe.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9531#issuecomment-154578481
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9531#issuecomment-154597511
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9531#issuecomment-154579300
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9531#issuecomment-154579705
**[Test build #45270 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45270/consoleFull)** for PR 9531 at commit [`5f7acc8`](https://github.com/apache/spark/commit/5f7acc8021ba7012191613cf267b31b302aaad4c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9531#discussion_r44204803
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -166,8 +171,7 @@ class Dataset[T] private[sql](
* Returns a new [[Dataset]] that only contains elements where `func` returns `true`.
* @since 1.6.0
*/
- def filter(func: JFunction[T, java.lang.Boolean]): Dataset[T] =
- filter(t => func.call(t).booleanValue())
+ def filter(func: FilterFunction[T]): Dataset[T] = filter(t => func.call(t).booleanValue())
--- End diff --
good catch
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9531#discussion_r44204792
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala ---
@@ -1478,18 +1478,54 @@ class DataFrame private[sql](
/**
* Returns the first `n` rows in the [[DataFrame]].
+ *
+ * Running take requires moving data into the application's driver process, and doing so on a
+ * very large dataset can crash the driver process with OutOfMemoryError.
+ *
* @group action
* @since 1.3.0
*/
def take(n: Int): Array[Row] = head(n)
/**
+ * Returns the first `n` rows in the [[DataFrame]] as a list.
+ *
+ * Running take requires moving data into the application's driver process, and doing so on a
+ * very large dataset can crash the driver process with OutOfMemoryError.
+ *
+ * @group action
+ * @since 1.6.0
+ */
+ def takeAsList(n: Int): java.util.List[Row] = java.util.Arrays.asList(take(n) : _*)
+
+ /**
* Returns an array that contains all of [[Row]]s in this [[DataFrame]].
+ *
+ * Running collect requires moving all the data into the application's driver process, and
+ * doing so on a very large dataset can crash the driver process with OutOfMemoryError.
+ *
+ * For Java API, use [[collectAsList]].
--- End diff --
No it doesn't' exist, but still good to use a java list.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9531#issuecomment-154596389
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL] Dataset Java API audit
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/9531#issuecomment-154655582
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org