You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Tathagata Das (JIRA)" <ji...@apache.org> on 2015/09/12 02:56:45 UTC
[jira] [Commented] (SPARK-7872) Transaction stack trace with Spark
Streaming and Flume
[ https://issues.apache.org/jira/browse/SPARK-7872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741804#comment-14741804 ]
Tathagata Das commented on SPARK-7872:
--------------------------------------
Is this still a problem?
> Transaction stack trace with Spark Streaming and Flume
> ------------------------------------------------------
>
> Key: SPARK-7872
> URL: https://issues.apache.org/jira/browse/SPARK-7872
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.3.0
> Environment: Flume: 1.5.2
> Scala: 2.11.6
> Spark: 1.3.0
> Reporter: Michael O'Neill
>
> Using a Flume Polling Stream to read every 2 seconds (also tried with 1,3,5 second intervals) we're getting the following error:
> {code}
> 15/05/26 17:26:37 WARN sink.SparkAvroCallbackHandler: Received an error batch - no events were received from channel!
> 15/05/26 17:26:37 WARN sink.SparkAvroCallbackHandler: Received an error batch - no events were received from channel!
> 15/05/26 17:26:37 WARN sink.SparkAvroCallbackHandler: Received an error batch - no events were received from channel!
> 15/05/26 17:26:37 WARN sink.SparkAvroCallbackHandler: Received an error batch - no events were received from channel!
> 15/05/26 17:26:37 WARN sink.SparkAvroCallbackHandler: Received an error batch - no events were received from channel!
> 15/05/26 17:26:37 WARN sink.SparkAvroCallbackHandler: Received an error batch - no events were received from channel!
> 15/05/26 17:26:37 WARN sink.SparkAvroCallbackHandler: Received an error batch - no events were received from channel!
> 15/05/26 17:26:37 WARN sink.SparkAvroCallbackHandler: Received an error batch - no events were received from channel!
> 15/05/26 17:26:37 WARN sink.SparkAvroCallbackHandler: Received an error batch - no events were received from channel!
> 15/05/26 17:26:37 WARN sink.SparkAvroCallbackHandler: Received an error batch - no events were received from channel!
> 15/05/26 17:26:37 WARN sink.TransactionProcessor: Error while processing transaction.
> java.lang.IllegalStateException: begin() called when transaction is OPEN!
> at com.google.common.base.Preconditions.checkState(Preconditions.java:173)
> at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
> at org.apache.spark.streaming.flume.sink.TransactionProcessor$$anonfun$populateEvents$1.apply(TransactionProcessor.scala:114)
> at org.apache.spark.streaming.flume.sink.TransactionProcessor$$anonfun$populateEvents$1.apply(TransactionProcessor.scala:113)
> at scala.Option.foreach(Option.scala:236)
> at org.apache.spark.streaming.flume.sink.TransactionProcessor.populateEvents(TransactionProcessor.scala:113)
> at org.apache.spark.streaming.flume.sink.TransactionProcessor.call(TransactionProcessor.scala:243)
> at org.apache.spark.streaming.flume.sink.TransactionProcessor.call(TransactionProcessor.scala:43)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> agent.conf is as follows:
> {code}
> agent.sources = http-source
> agent.channels = id2_channel
> agent.sinks = id2_sink_spark
> agent.sources.http-source.type = http
> agent.sources.http-source.bind = localhost
> agent.sources.http-source.port = 9050
> agent.sources.http-source.channels = id2_channel
> agent.channels.id2_channel.type = memory
> agent.sinks.id2_sink_spark.type = org.apache.spark.streaming.flume.sink.SparkSink
> agent.sinks.id2_sink_spark.hostname = localhost
> agent.sinks.id2_sink_spark.port = 35001
> agent.sinks.id2_sink_spark.channel = id2_channel
> {code}
> Just trying to read from the stream and print the received data.
> Using a memory channel.
> If I've missed any info let me know.
> This happens when running locally (Mac OSX 10.9) and when submitted via Docker, Marathon and Mesos.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org