You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ionized <io...@gmail.com> on 2014/01/28 11:26:10 UTC
Exception in serialization hangs saving-to-disk
I noticed that running the following code results in the process hanging
forever waiting for the Job to complete.
It seems the exception never propagates to the caller.
Should a bug be filed on this?
- Paul
import java.io.IOException;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.List;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
public class SparkSerializationTest {
public static void main(String[] args) {
JavaSparkContext context = new JavaSparkContext("local[3]", "test");
List<MyObject> list = new ArrayList<>();
list.add(new MyObject());
JavaRDD<MyObject> rdd = context.parallelize(list);
rdd.saveAsObjectFile("/tmp/sparkserializationtest");
}
private static final class MyObject implements Serializable {
private static final long serialVersionUID = 1L;
private void readObject(ObjectInputStream in) throws IOException,
ClassNotFoundException {
}
private void writeObject(ObjectOutputStream out) throws IOException {
throw new RuntimeException();
}
}
}
Re: Exception in serialization hangs saving-to-disk
Posted by Ionized <io...@gmail.com>.
What do you mean by "get a copy"? in writeObject I throw new
RuntimeException(). This hangs the process forever waiting for the spark
Job to complete (the main method never returns)
On Tue, Jan 28, 2014 at 9:55 AM, Andrew Ash <an...@andrewash.com> wrote:
> Are you able to get a copy of the exception you refer to?
>
>
> On Tue, Jan 28, 2014 at 2:26 AM, Ionized <io...@gmail.com> wrote:
>
>> I noticed that running the following code results in the process hanging
>> forever waiting for the Job to complete.
>> It seems the exception never propagates to the caller.
>>
>> Should a bug be filed on this?
>>
>> - Paul
>>
>>
>>
>> import java.io.IOException;
>> import java.io.ObjectInputStream;
>> import java.io.ObjectOutputStream;
>> import java.io.Serializable;
>> import java.util.ArrayList;
>> import java.util.List;
>>
>> import org.apache.spark.api.java.JavaRDD;
>> import org.apache.spark.api.java.JavaSparkContext;
>>
>> public class SparkSerializationTest {
>>
>> public static void main(String[] args) {
>> JavaSparkContext context = new JavaSparkContext("local[3]", "test");
>> List<MyObject> list = new ArrayList<>();
>> list.add(new MyObject());
>> JavaRDD<MyObject> rdd = context.parallelize(list);
>> rdd.saveAsObjectFile("/tmp/sparkserializationtest");
>> }
>>
>> private static final class MyObject implements Serializable {
>>
>> private static final long serialVersionUID = 1L;
>>
>> private void readObject(ObjectInputStream in) throws IOException,
>> ClassNotFoundException {
>> }
>>
>> private void writeObject(ObjectOutputStream out) throws IOException {
>> throw new RuntimeException();
>> }
>>
>> }
>> }
>>
>>
>
Re: Exception in serialization hangs saving-to-disk
Posted by Andrew Ash <an...@andrewash.com>.
Are you able to get a copy of the exception you refer to?
On Tue, Jan 28, 2014 at 2:26 AM, Ionized <io...@gmail.com> wrote:
> I noticed that running the following code results in the process hanging
> forever waiting for the Job to complete.
> It seems the exception never propagates to the caller.
>
> Should a bug be filed on this?
>
> - Paul
>
>
>
> import java.io.IOException;
> import java.io.ObjectInputStream;
> import java.io.ObjectOutputStream;
> import java.io.Serializable;
> import java.util.ArrayList;
> import java.util.List;
>
> import org.apache.spark.api.java.JavaRDD;
> import org.apache.spark.api.java.JavaSparkContext;
>
> public class SparkSerializationTest {
>
> public static void main(String[] args) {
> JavaSparkContext context = new JavaSparkContext("local[3]", "test");
> List<MyObject> list = new ArrayList<>();
> list.add(new MyObject());
> JavaRDD<MyObject> rdd = context.parallelize(list);
> rdd.saveAsObjectFile("/tmp/sparkserializationtest");
> }
>
> private static final class MyObject implements Serializable {
>
> private static final long serialVersionUID = 1L;
>
> private void readObject(ObjectInputStream in) throws IOException,
> ClassNotFoundException {
> }
>
> private void writeObject(ObjectOutputStream out) throws IOException {
> throw new RuntimeException();
> }
>
> }
> }
>
>