You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by ARose <As...@telarix.com> on 2015/04/01 23:57:43 UTC
Spark 1.3.0 DataFrame count() method throwing java.io.EOFException
Note: I am running Spark on Windows 7 in standalone mode.
In my app, I run the following:
DataFrame df = sqlContext.sql("SELECT * FROM tbBER");
System.out.println("Count: " + df.count());
tbBER is registered as a temp table in my SQLContext. When I try to print
the number of rows in the DataFrame, the job fails and I get the following
error message:
java.io.EOFException
at
java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:2747)
at java.io.ObjectInputStream.readFully(ObjectInputStream.java:1033)
at
org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
at org.apache.hadoop.io.UTF8.readChars(UTF8.java:216)
at org.apache.hadoop.io.UTF8.readString(UTF8.java:208)
at org.apache.hadoop.mapred.FileSplit.readFields(FileSplit.java:87)
at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:237)
at org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:66)
at
org.apache.spark.SerializableWritable$$anonfun$readObject$1.apply$mcV$sp(SerializableWritable.scala:43)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1137)
at
org.apache.spark.SerializableWritable.readObject(SerializableWritable.scala:39)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:68)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:185)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
This only happens when I try to call df.count(). The rest runs fine. Is the
count() function not supported in standalone mode? The stack trace makes it
appear to be Hadoop functionality...
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-3-0-DataFrame-count-method-throwing-java-io-EOFException-tp22344.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark 1.3.0 DataFrame count() method throwing java.io.EOFException
Posted by Dean Wampler <de...@gmail.com>.
To clarify one thing, is count() the first "action" (
http://spark.apache.org/docs/latest/programming-guide.html#actions) you're
attempting? As defined in the programming guide, an action forces
evaluation of the pipeline of RDDs. It's only then that reading the data
actually occurs. So, count() might not be the issue, but some upstream step
that attempted to read the file.
As a sanity check, if you just read the text file and don't convert the
strings, then call count(), does that work? If so, it might be something
about your JavaBean BERecord after all. Can you post its definition?
Also calling take(1) to grab the first element should also work, even if
the RDD is empty. (It will return an empty RDD in that case, but not throw
an exception.)
dean
Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
<http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
Typesafe <http://typesafe.com>
@deanwampler <http://twitter.com/deanwampler>
http://polyglotprogramming.com
On Thu, Apr 2, 2015 at 10:16 AM, Ashley Rose <As...@telarix.com>
wrote:
> That’s precisely what I was trying to check. It should have 42577
> records in it, because that’s how many there were in the text file I read
> in.
>
>
>
> // Load a text file and convert each line to a JavaBean.
>
> JavaRDD<String> lines = sc.textFile("file.txt");
>
>
>
> JavaRDD<BERecord> tbBER = lines.map(s -> convertToBER(s));
>
>
>
> // Apply a schema to an RDD of JavaBeans and register it as a
> table.
>
> schemaBERecords = sqlContext.createDataFrame(tbBER,
> BERecord.class);
>
> schemaBERecords.registerTempTable("tbBER");
>
>
>
> The BERecord class is a standard Java Bean that implements Serializable,
> so that shouldn’t be the issue. As you said, count() shouldn’t fail like
> this even if the table was empty. I was able to print out the schema of the
> DataFrame just fine with df.printSchema(), and I just wanted to see if data
> was populated correctly.
>
>
>
> *From:* Dean Wampler [mailto:deanwampler@gmail.com]
> *Sent:* Wednesday, April 01, 2015 6:05 PM
> *To:* Ashley Rose
> *Cc:* user@spark.apache.org
> *Subject:* Re: Spark 1.3.0 DataFrame count() method throwing
> java.io.EOFException
>
>
>
> Is it possible "tbBER" is empty? If so, it shouldn't fail like this, of
> course.
>
>
> Dean Wampler, Ph.D.
>
> Author: Programming Scala, 2nd Edition
> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
>
> Typesafe <http://typesafe.com>
> @deanwampler <http://twitter.com/deanwampler>
>
> http://polyglotprogramming.com
>
>
>
> On Wed, Apr 1, 2015 at 5:57 PM, ARose <As...@telarix.com> wrote:
>
> Note: I am running Spark on Windows 7 in standalone mode.
>
> In my app, I run the following:
>
> DataFrame df = sqlContext.sql("SELECT * FROM tbBER");
> System.out.println("Count: " + df.count());
>
> tbBER is registered as a temp table in my SQLContext. When I try to print
> the number of rows in the DataFrame, the job fails and I get the following
> error message:
>
> java.io.EOFException
> at
>
> java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:2747)
> at java.io.ObjectInputStream.readFully(ObjectInputStream.java:1033)
> at
>
> org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
> at
> org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
> at org.apache.hadoop.io.UTF8.readChars(UTF8.java:216)
> at org.apache.hadoop.io.UTF8.readString(UTF8.java:208)
> at org.apache.hadoop.mapred.FileSplit.readFields(FileSplit.java:87)
> at
> org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:237)
> at
> org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:66)
> at
>
> org.apache.spark.SerializableWritable$$anonfun$readObject$1.apply$mcV$sp(SerializableWritable.scala:43)
> at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1137)
> at
>
> org.apache.spark.SerializableWritable.readObject(SerializableWritable.scala:39)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
> at
>
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:68)
> at
>
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:94)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:185)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> This only happens when I try to call df.count(). The rest runs fine. Is the
> count() function not supported in standalone mode? The stack trace makes it
> appear to be Hadoop functionality...
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-3-0-DataFrame-count-method-throwing-java-io-EOFException-tp22344.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
>
RE: Spark 1.3.0 DataFrame count() method throwing
java.io.EOFException
Posted by Ashley Rose <As...@telarix.com>.
That’s precisely what I was trying to check. It should have 42577 records in it, because that’s how many there were in the text file I read in.
// Load a text file and convert each line to a JavaBean.
JavaRDD<String> lines = sc.textFile("file.txt");
JavaRDD<BERecord> tbBER = lines.map(s -> convertToBER(s));
// Apply a schema to an RDD of JavaBeans and register it as a table.
schemaBERecords = sqlContext.createDataFrame(tbBER, BERecord.class);
schemaBERecords.registerTempTable("tbBER");
The BERecord class is a standard Java Bean that implements Serializable, so that shouldn’t be the issue. As you said, count() shouldn’t fail like this even if the table was empty. I was able to print out the schema of the DataFrame just fine with df.printSchema(), and I just wanted to see if data was populated correctly.
From: Dean Wampler [mailto:deanwampler@gmail.com]
Sent: Wednesday, April 01, 2015 6:05 PM
To: Ashley Rose
Cc: user@spark.apache.org
Subject: Re: Spark 1.3.0 DataFrame count() method throwing java.io.EOFException
Is it possible "tbBER" is empty? If so, it shouldn't fail like this, of course.
Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition<http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
Typesafe<http://typesafe.com>
@deanwampler<http://twitter.com/deanwampler>
http://polyglotprogramming.com
On Wed, Apr 1, 2015 at 5:57 PM, ARose <As...@telarix.com>> wrote:
Note: I am running Spark on Windows 7 in standalone mode.
In my app, I run the following:
DataFrame df = sqlContext.sql("SELECT * FROM tbBER");
System.out.println("Count: " + df.count());
tbBER is registered as a temp table in my SQLContext. When I try to print
the number of rows in the DataFrame, the job fails and I get the following
error message:
java.io.EOFException
at
java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:2747)
at java.io.ObjectInputStream.readFully(ObjectInputStream.java:1033)
at
org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
at org.apache.hadoop.io.UTF8.readChars(UTF8.java:216)
at org.apache.hadoop.io.UTF8.readString(UTF8.java:208)
at org.apache.hadoop.mapred.FileSplit.readFields(FileSplit.java:87)
at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:237)
at org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:66)
at
org.apache.spark.SerializableWritable$$anonfun$readObject$1.apply$mcV$sp(SerializableWritable.scala:43)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1137)
at
org.apache.spark.SerializableWritable.readObject(SerializableWritable.scala:39)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:68)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:185)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
This only happens when I try to call df.count(). The rest runs fine. Is the
count() function not supported in standalone mode? The stack trace makes it
appear to be Hadoop functionality...
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-3-0-DataFrame-count-method-throwing-java-io-EOFException-tp22344.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<ma...@spark.apache.org>
Re: Spark 1.3.0 DataFrame count() method throwing java.io.EOFException
Posted by Dean Wampler <de...@gmail.com>.
Is it possible "tbBER" is empty? If so, it shouldn't fail like this, of
course.
Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
<http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
Typesafe <http://typesafe.com>
@deanwampler <http://twitter.com/deanwampler>
http://polyglotprogramming.com
On Wed, Apr 1, 2015 at 5:57 PM, ARose <As...@telarix.com> wrote:
> Note: I am running Spark on Windows 7 in standalone mode.
>
> In my app, I run the following:
>
> DataFrame df = sqlContext.sql("SELECT * FROM tbBER");
> System.out.println("Count: " + df.count());
>
> tbBER is registered as a temp table in my SQLContext. When I try to print
> the number of rows in the DataFrame, the job fails and I get the following
> error message:
>
> java.io.EOFException
> at
>
> java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:2747)
> at java.io.ObjectInputStream.readFully(ObjectInputStream.java:1033)
> at
>
> org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
> at
> org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
> at org.apache.hadoop.io.UTF8.readChars(UTF8.java:216)
> at org.apache.hadoop.io.UTF8.readString(UTF8.java:208)
> at org.apache.hadoop.mapred.FileSplit.readFields(FileSplit.java:87)
> at
> org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:237)
> at
> org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:66)
> at
>
> org.apache.spark.SerializableWritable$$anonfun$readObject$1.apply$mcV$sp(SerializableWritable.scala:43)
> at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1137)
> at
>
> org.apache.spark.SerializableWritable.readObject(SerializableWritable.scala:39)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
> at
>
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:68)
> at
>
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:94)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:185)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> This only happens when I try to call df.count(). The rest runs fine. Is the
> count() function not supported in standalone mode? The stack trace makes it
> appear to be Hadoop functionality...
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-3-0-DataFrame-count-method-throwing-java-io-EOFException-tp22344.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>