You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Michael O'Neill (JIRA)" <ji...@apache.org> on 2015/05/26 18:48:17 UTC
[jira] [Created] (SPARK-7872) Transaction stack trace with Spark
Streaming and Flume
Michael O'Neill created SPARK-7872:
--------------------------------------
Summary: Transaction stack trace with Spark Streaming and Flume
Key: SPARK-7872
URL: https://issues.apache.org/jira/browse/SPARK-7872
Project: Spark
Issue Type: Bug
Components: Streaming
Affects Versions: 1.3.0
Environment: Flume: 1.5.2
Scala: 2.11.6
Spark: 1.3.0
Reporter: Michael O'Neill
Using a Flume Polling Stream to read every 2 seconds (also tried with 1,3,5 second intervals) we're getting the following error:
{code}
15/05/26 17:26:37 WARN sink.SparkAvroCallbackHandler: Received an error batch - no events were received from channel!
15/05/26 17:26:37 WARN sink.SparkAvroCallbackHandler: Received an error batch - no events were received from channel!
15/05/26 17:26:37 WARN sink.SparkAvroCallbackHandler: Received an error batch - no events were received from channel!
15/05/26 17:26:37 WARN sink.SparkAvroCallbackHandler: Received an error batch - no events were received from channel!
15/05/26 17:26:37 WARN sink.SparkAvroCallbackHandler: Received an error batch - no events were received from channel!
15/05/26 17:26:37 WARN sink.SparkAvroCallbackHandler: Received an error batch - no events were received from channel!
15/05/26 17:26:37 WARN sink.SparkAvroCallbackHandler: Received an error batch - no events were received from channel!
15/05/26 17:26:37 WARN sink.SparkAvroCallbackHandler: Received an error batch - no events were received from channel!
15/05/26 17:26:37 WARN sink.SparkAvroCallbackHandler: Received an error batch - no events were received from channel!
15/05/26 17:26:37 WARN sink.SparkAvroCallbackHandler: Received an error batch - no events were received from channel!
15/05/26 17:26:37 WARN sink.TransactionProcessor: Error while processing transaction.
java.lang.IllegalStateException: begin() called when transaction is OPEN!
at com.google.common.base.Preconditions.checkState(Preconditions.java:173)
at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
at org.apache.spark.streaming.flume.sink.TransactionProcessor$$anonfun$populateEvents$1.apply(TransactionProcessor.scala:114)
at org.apache.spark.streaming.flume.sink.TransactionProcessor$$anonfun$populateEvents$1.apply(TransactionProcessor.scala:113)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.streaming.flume.sink.TransactionProcessor.populateEvents(TransactionProcessor.scala:113)
at org.apache.spark.streaming.flume.sink.TransactionProcessor.call(TransactionProcessor.scala:243)
at org.apache.spark.streaming.flume.sink.TransactionProcessor.call(TransactionProcessor.scala:43)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
agent.conf is as follows:
{code}
agent.sources = http-source
agent.channels = id2_channel
agent.sinks = id2_sink_spark
agent.sources.http-source.type = http
agent.sources.http-source.bind = localhost
agent.sources.http-source.port = 9050
agent.sources.http-source.channels = id2_channel
agent.channels.id2_channel.type = memory
agent.sinks.id2_sink_spark.type = org.apache.spark.streaming.flume.sink.SparkSink
agent.sinks.id2_sink_spark.hostname = localhost
agent.sinks.id2_sink_spark.port = 35001
agent.sinks.id2_sink_spark.channel = id2_channel
{code}
Just trying to read from the stream and print the received data.
Using a memory channel.
If I've missed any info let me know.
This happens when running locally (Mac OSX 10.9) and when submitted via Docker, Marathon and Mesos.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org