You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jeanlyn <gi...@git.apache.org> on 2015/01/06 09:25:33 UTC
[GitHub] spark pull request: Origin/spark 5068
GitHub user jeanlyn opened a pull request:
https://github.com/apache/spark/pull/3907
Origin/spark 5068
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jeanlyn/spark origin/SPARK-5068
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3907.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3907
----
commit 68a2a0742f4aa7aea1ab84f2b84612aee740b23a
Author: jeanlyn <je...@gmail.com>
Date: 2015-01-04T16:26:14Z
SPARK-5068: fix bug query data when path doesn't exists
commit 0db35b32b196921ae8df8367b3ed6d5927f9d785
Author: jeanlyn <je...@gmail.com>
Date: 2015-01-06T08:08:36Z
catch the execption when the path doesn't exists
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5068][SQL]fix bug query data when path ...
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/3907#discussion_r22635771
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -198,12 +190,19 @@ class HadoopRDD[K, V](
if (inputFormat.isInstanceOf[Configurable]) {
inputFormat.asInstanceOf[Configurable].setConf(jobConf)
}
- val inputSplits = inputFormat.getSplits(jobConf, minPartitions)
- val array = new Array[Partition](inputSplits.size)
- for (i <- 0 until inputSplits.size) {
- array(i) = new HadoopPartition(id, i, inputSplits(i))
+ // SPARK-5068:catch the exception when the path is not exists
+ try {
+ val inputSplits = inputFormat.getSplits(jobConf, minPartitions)
+ val array = new Array[Partition](inputSplits.size)
+ for (i <- 0 until inputSplits.size) {
+ array(i) = new HadoopPartition(id, i, inputSplits(i))
+ }
+ array
+ } catch {
+ case e: InvalidInputException => {
--- End diff --
Agree with @chenghao-intel, we'd better handle this in hive module.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [spark 5068][SQL]fix bug query data when path ...
Posted by jeanlyn <gi...@git.apache.org>.
Github user jeanlyn closed the pull request at:
https://github.com/apache/spark/pull/3907
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5068][SQL]fix bug query data when path ...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/3907#discussion_r22630492
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -198,12 +190,19 @@ class HadoopRDD[K, V](
if (inputFormat.isInstanceOf[Configurable]) {
inputFormat.asInstanceOf[Configurable].setConf(jobConf)
}
- val inputSplits = inputFormat.getSplits(jobConf, minPartitions)
- val array = new Array[Partition](inputSplits.size)
- for (i <- 0 until inputSplits.size) {
- array(i) = new HadoopPartition(id, i, inputSplits(i))
+ // SPARK-5068:catch the exception when the path is not exists
+ try {
+ val inputSplits = inputFormat.getSplits(jobConf, minPartitions)
+ val array = new Array[Partition](inputSplits.size)
+ for (i <- 0 until inputSplits.size) {
+ array(i) = new HadoopPartition(id, i, inputSplits(i))
+ }
+ array
+ } catch {
+ case e: InvalidInputException => {
--- End diff --
I guess throwing exception is the way how `InputFormat` handle the non-existed file. I agree with @srowen , we couldn't just ignore the error here. Probably it's better to leave the current implementation unchanged.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5068][SQL]fix bug query data when path ...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/3907#issuecomment-74755965
Is this superseded by https://github.com/apache/spark/pull/4356 ? if so can this be closed?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5068][SQL]fix bug query data when path ...
Posted by jeanlyn <gi...@git.apache.org>.
Github user jeanlyn commented on a diff in the pull request:
https://github.com/apache/spark/pull/3907#discussion_r22633042
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -198,12 +190,19 @@ class HadoopRDD[K, V](
if (inputFormat.isInstanceOf[Configurable]) {
inputFormat.asInstanceOf[Configurable].setConf(jobConf)
}
- val inputSplits = inputFormat.getSplits(jobConf, minPartitions)
- val array = new Array[Partition](inputSplits.size)
- for (i <- 0 until inputSplits.size) {
- array(i) = new HadoopPartition(id, i, inputSplits(i))
+ // SPARK-5068:catch the exception when the path is not exists
+ try {
+ val inputSplits = inputFormat.getSplits(jobConf, minPartitions)
+ val array = new Array[Partition](inputSplits.size)
+ for (i <- 0 until inputSplits.size) {
+ array(i) = new HadoopPartition(id, i, inputSplits(i))
+ }
+ array
+ } catch {
+ case e: InvalidInputException => {
--- End diff --
Thanks for review.In my understanding,this method is get the data for specific partition by return `Array[Partition]`.I think get when we pass the wrong path and we return an empty array is OK.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5068][SQL]fix bug query data when path ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/3907#issuecomment-69111291
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [spark 5068][SQL]fix bug query data when path ...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3907#discussion_r22515175
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -198,12 +190,19 @@ class HadoopRDD[K, V](
if (inputFormat.isInstanceOf[Configurable]) {
inputFormat.asInstanceOf[Configurable].setConf(jobConf)
}
- val inputSplits = inputFormat.getSplits(jobConf, minPartitions)
- val array = new Array[Partition](inputSplits.size)
- for (i <- 0 until inputSplits.size) {
- array(i) = new HadoopPartition(id, i, inputSplits(i))
+ // SPARK-5068:catch the exception when the path is not exists
+ try {
+ val inputSplits = inputFormat.getSplits(jobConf, minPartitions)
+ val array = new Array[Partition](inputSplits.size)
+ for (i <- 0 until inputSplits.size) {
+ array(i) = new HadoopPartition(id, i, inputSplits(i))
+ }
+ array
+ } catch {
+ case e: InvalidInputException => {
--- End diff --
Is there a more direct and reliable way to detect if the path doesn't exist here than catching a general exception? this risks silently swallowing errors
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5068][SQL]fix bug query data when path ...
Posted by jeanlyn <gi...@git.apache.org>.
Github user jeanlyn commented on a diff in the pull request:
https://github.com/apache/spark/pull/3907#discussion_r22634166
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -198,12 +190,19 @@ class HadoopRDD[K, V](
if (inputFormat.isInstanceOf[Configurable]) {
inputFormat.asInstanceOf[Configurable].setConf(jobConf)
}
- val inputSplits = inputFormat.getSplits(jobConf, minPartitions)
- val array = new Array[Partition](inputSplits.size)
- for (i <- 0 until inputSplits.size) {
- array(i) = new HadoopPartition(id, i, inputSplits(i))
+ // SPARK-5068:catch the exception when the path is not exists
+ try {
+ val inputSplits = inputFormat.getSplits(jobConf, minPartitions)
+ val array = new Array[Partition](inputSplits.size)
+ for (i <- 0 until inputSplits.size) {
+ array(i) = new HadoopPartition(id, i, inputSplits(i))
+ }
+ array
+ } catch {
+ case e: InvalidInputException => {
--- End diff --
I found [Shark](https://github.com/amplab/shark/blob/master/src/main/scala/shark/execution/HadoopTableReader.scala#L152) is check for the path when the RDD was constructed(a little bit like [my last PR](https://github.com/apache/spark/pull/3891)),but i agree with @marmbrus that will slow down when constructing RDD especially a lot of partitions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5068][SQL]fix bug query data when path ...
Posted by jeanlyn <gi...@git.apache.org>.
Github user jeanlyn closed the pull request at:
https://github.com/apache/spark/pull/3907
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5068][SQL]fix bug query data when path ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3907#issuecomment-69120679
[Test build #25181 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25181/consoleFull) for PR 3907 at commit [`d4a5b9a`](https://github.com/apache/spark/commit/d4a5b9ae3fc029619366d017cacf69d0beec3479).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [spark 5068][SQL]fix bug query data when path ...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3907#discussion_r22515156
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -26,15 +26,7 @@ import scala.reflect.ClassTag
import scala.collection.mutable.ListBuffer
import org.apache.hadoop.conf.{Configurable, Configuration}
-import org.apache.hadoop.mapred.FileSplit
-import org.apache.hadoop.mapred.InputFormat
-import org.apache.hadoop.mapred.InputSplit
-import org.apache.hadoop.mapred.JobConf
-import org.apache.hadoop.mapred.RecordReader
-import org.apache.hadoop.mapred.Reporter
-import org.apache.hadoop.mapred.JobID
-import org.apache.hadoop.mapred.TaskAttemptID
-import org.apache.hadoop.mapred.TaskID
+import org.apache.hadoop.mapred._
--- End diff --
I think I would leave this as individual imports for consistency.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5068][SQL]fix bug query data when path ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3907#issuecomment-69111516
[Test build #25181 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25181/consoleFull) for PR 3907 at commit [`d4a5b9a`](https://github.com/apache/spark/commit/d4a5b9ae3fc029619366d017cacf69d0beec3479).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [spark 5068][SQL]fix bug query data when path ...
Posted by jeanlyn <gi...@git.apache.org>.
Github user jeanlyn commented on a diff in the pull request:
https://github.com/apache/spark/pull/3907#discussion_r22518647
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -198,12 +190,19 @@ class HadoopRDD[K, V](
if (inputFormat.isInstanceOf[Configurable]) {
inputFormat.asInstanceOf[Configurable].setConf(jobConf)
}
- val inputSplits = inputFormat.getSplits(jobConf, minPartitions)
- val array = new Array[Partition](inputSplits.size)
- for (i <- 0 until inputSplits.size) {
- array(i) = new HadoopPartition(id, i, inputSplits(i))
+ // SPARK-5068:catch the exception when the path is not exists
+ try {
+ val inputSplits = inputFormat.getSplits(jobConf, minPartitions)
+ val array = new Array[Partition](inputSplits.size)
+ for (i <- 0 until inputSplits.size) {
+ array(i) = new HadoopPartition(id, i, inputSplits(i))
+ }
+ array
+ } catch {
+ case e: InvalidInputException => {
--- End diff --
what about to give a log warning when the path is not found?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5068][SQL]fix bug query data when path ...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/3907#discussion_r22635547
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -198,12 +190,19 @@ class HadoopRDD[K, V](
if (inputFormat.isInstanceOf[Configurable]) {
inputFormat.asInstanceOf[Configurable].setConf(jobConf)
}
- val inputSplits = inputFormat.getSplits(jobConf, minPartitions)
- val array = new Array[Partition](inputSplits.size)
- for (i <- 0 until inputSplits.size) {
- array(i) = new HadoopPartition(id, i, inputSplits(i))
+ // SPARK-5068:catch the exception when the path is not exists
+ try {
+ val inputSplits = inputFormat.getSplits(jobConf, minPartitions)
+ val array = new Array[Partition](inputSplits.size)
+ for (i <- 0 until inputSplits.size) {
+ array(i) = new HadoopPartition(id, i, inputSplits(i))
+ }
+ array
+ } catch {
+ case e: InvalidInputException => {
--- End diff --
Ok, I understand the background now.
You're right, we do need to follow the Hive's behavior, the non-existed partition will returns an empty record iterator, other than throw exception. However, this should be LIMITED within Hive TableReader, not here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [spark 5068][SQL]fix bug query data when path ...
Posted by jeanlyn <gi...@git.apache.org>.
Github user jeanlyn commented on a diff in the pull request:
https://github.com/apache/spark/pull/3907#discussion_r22518690
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -26,15 +26,7 @@ import scala.reflect.ClassTag
import scala.collection.mutable.ListBuffer
import org.apache.hadoop.conf.{Configurable, Configuration}
-import org.apache.hadoop.mapred.FileSplit
-import org.apache.hadoop.mapred.InputFormat
-import org.apache.hadoop.mapred.InputSplit
-import org.apache.hadoop.mapred.JobConf
-import org.apache.hadoop.mapred.RecordReader
-import org.apache.hadoop.mapred.Reporter
-import org.apache.hadoop.mapred.JobID
-import org.apache.hadoop.mapred.TaskAttemptID
-import org.apache.hadoop.mapred.TaskID
+import org.apache.hadoop.mapred._
--- End diff --
thx for your suggestions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [spark 5068][SQL]fix bug query data when path ...
Posted by jeanlyn <gi...@git.apache.org>.
GitHub user jeanlyn reopened a pull request:
https://github.com/apache/spark/pull/3907
[spark 5068][SQL]fix bug query data when path doesn't exists
the issue is descript on [SPARK-5068] (https://issues.apache.org/jira/browse/SPARK-5068) and this PR is fix the same problem as [PR3891] (https://github.com/apache/spark/pull/3891).however,this pull request catch the exception if it is thrown instead of doing it serially when constructing the RDD
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jeanlyn/spark origin/SPARK-5068
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3907.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3907
----
commit 68a2a0742f4aa7aea1ab84f2b84612aee740b23a
Author: jeanlyn <je...@gmail.com>
Date: 2015-01-04T16:26:14Z
SPARK-5068: fix bug query data when path doesn't exists
commit 0db35b32b196921ae8df8367b3ed6d5927f9d785
Author: jeanlyn <je...@gmail.com>
Date: 2015-01-06T08:08:36Z
catch the execption when the path doesn't exists
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5068][SQL]fix bug query data when path ...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/3907#discussion_r22635649
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -198,12 +190,19 @@ class HadoopRDD[K, V](
if (inputFormat.isInstanceOf[Configurable]) {
inputFormat.asInstanceOf[Configurable].setConf(jobConf)
}
- val inputSplits = inputFormat.getSplits(jobConf, minPartitions)
- val array = new Array[Partition](inputSplits.size)
- for (i <- 0 until inputSplits.size) {
- array(i) = new HadoopPartition(id, i, inputSplits(i))
+ // SPARK-5068:catch the exception when the path is not exists
+ try {
+ val inputSplits = inputFormat.getSplits(jobConf, minPartitions)
+ val array = new Array[Partition](inputSplits.size)
+ for (i <- 0 until inputSplits.size) {
+ array(i) = new HadoopPartition(id, i, inputSplits(i))
+ }
+ array
+ } catch {
+ case e: InvalidInputException => {
--- End diff --
And of course, we can add a new parameter for `HadoopRDD` (e.g. `throwExceptionOnError: Boolean = true`), to support the case you described.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Origin/spark 5068
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3907#issuecomment-68838456
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5068][SQL]fix bug query data when path ...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/3907#discussion_r22633589
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -198,12 +190,19 @@ class HadoopRDD[K, V](
if (inputFormat.isInstanceOf[Configurable]) {
inputFormat.asInstanceOf[Configurable].setConf(jobConf)
}
- val inputSplits = inputFormat.getSplits(jobConf, minPartitions)
- val array = new Array[Partition](inputSplits.size)
- for (i <- 0 until inputSplits.size) {
- array(i) = new HadoopPartition(id, i, inputSplits(i))
+ // SPARK-5068:catch the exception when the path is not exists
+ try {
+ val inputSplits = inputFormat.getSplits(jobConf, minPartitions)
+ val array = new Array[Partition](inputSplits.size)
+ for (i <- 0 until inputSplits.size) {
+ array(i) = new HadoopPartition(id, i, inputSplits(i))
+ }
+ array
+ } catch {
+ case e: InvalidInputException => {
--- End diff --
The `HadoopRDD` is general API for reading files, I don't think it's acceptable the API just keep silence if the specified path doesn't exist, throwing an exception probably better.
Take the JDK class `FileReader` for example:
http://docs.oracle.com/javase/7/docs/api/java/io/FileReader.html#FileReader(java.io.File)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5068][SQL]fix bug query data when path ...
Posted by jeanlyn <gi...@git.apache.org>.
Github user jeanlyn commented on the pull request:
https://github.com/apache/spark/pull/3907#issuecomment-75260761
OK.I close this one
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5068][SQL]fix bug query data when path ...
Posted by jeanlyn <gi...@git.apache.org>.
Github user jeanlyn commented on a diff in the pull request:
https://github.com/apache/spark/pull/3907#discussion_r22655119
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -198,12 +190,19 @@ class HadoopRDD[K, V](
if (inputFormat.isInstanceOf[Configurable]) {
inputFormat.asInstanceOf[Configurable].setConf(jobConf)
}
- val inputSplits = inputFormat.getSplits(jobConf, minPartitions)
- val array = new Array[Partition](inputSplits.size)
- for (i <- 0 until inputSplits.size) {
- array(i) = new HadoopPartition(id, i, inputSplits(i))
+ // SPARK-5068:catch the exception when the path is not exists
+ try {
+ val inputSplits = inputFormat.getSplits(jobConf, minPartitions)
+ val array = new Array[Partition](inputSplits.size)
+ for (i <- 0 until inputSplits.size) {
+ array(i) = new HadoopPartition(id, i, inputSplits(i))
+ }
+ array
+ } catch {
+ case e: InvalidInputException => {
--- End diff --
Thanks for review!I have two questions:
1.How could i limited within hive by catch exption instead of using `fs.exists` to find out the wrong patition?
2.Could add a new parameter in `HadoopRDD` affect more code?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [spark 5068][SQL]fix bug query data when path ...
Posted by jeanlyn <gi...@git.apache.org>.
Github user jeanlyn commented on the pull request:
https://github.com/apache/spark/pull/3907#issuecomment-68839176
Hi @marmbrus. Any suggestions?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5068][SQL]fix bug query data when path ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3907#issuecomment-69120687
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25181/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org