You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by Leolh <gi...@git.apache.org> on 2015/01/23 10:27:47 UTC
[GitHub] spark pull request: [SPARK-3650][GraphX] There will be an ArrayInd...
GitHub user Leolh opened a pull request:
https://github.com/apache/spark/pull/4176
[SPARK-3650][GraphX] There will be an ArrayIndexOutOfBoundsException if ...
...the format of the source file is wrong
There will be an ArrayIndexOutOfBoundsException if the format of the source file is wrong
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Leolh/spark patch-1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4176.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4176
----
commit 23767f1239341146df49dc4d4c4956d7a3b48e0f
Author: Leolh <le...@gmail.com>
Date: 2015-01-23T09:27:02Z
[SPARK-3650][GraphX] There will be an ArrayIndexOutOfBoundsException if the format of the source file is wrong
There will be an ArrayIndexOutOfBoundsException if the format of the source file is wrong
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5380][GraphX] Solve an ArrayIndexOutOfB...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/4176#issuecomment-73290045
Ah sounds like I should have done more back-porting. I was conservative and only put these several small fixes in master. Go ahead and I'll recalibrate a little accordingly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5380][GraphX] Solve an ArrayIndexOutOfB...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the pull request:
https://github.com/apache/spark/pull/4176#issuecomment-72786951
I wonder if stopping the process is the best solution.
If there is only one illegal entry in a last line, we need to re-try loading a whole file,
which is time-consuming.
An other idea is that illegal entries are silently re-directed into a file or something.
Finally, # of the re-directed entries is only output in the last phase of bulk-loading.
Moreover, I think that it'd be better to add a new API to append these entries into existing Graph
such as GraphOps.addEdges(val edges: RDD[Edge[ED]]).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5380][GraphX] Solve an ArrayIndexOutOfB...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/4176#issuecomment-73289353
I've also merged this into `branch-1.3` (1.3.0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5380][GraphX] Solve an ArrayIndexOutOfB...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/4176#issuecomment-73290941
@srowen From now until we cut a 1.3.0 preview, I'd merge all bug fixes into at least master and `branch-1.3`. We'll raise the bar for getting fixes into `branch-1.3` as we get closer to the RC voting period.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5380][GraphX] Solve an ArrayIndexOutOfB...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/4176#issuecomment-73206257
Since this was a very simple change, I went ahead and merged it as my first test-drive of the commit bit. Looks like it worked!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3650][GraphX] Solve an ArrayIndexOutOfB...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4176#issuecomment-71168185
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5380][GraphX] Solve an ArrayIndexOutOfB...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/4176#issuecomment-71170456
The argument I can see against it is simply that you want to fail fast on bad input, not sort of continue. If all of the file is like this you'll just log a zillion warnings instead of just stopping. How about changing the warning to throw a standard exception like `IllegalArgumentException` using `require`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5380][GraphX] Solve an ArrayIndexOutOfB...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/4176
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org