You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Jonathan Hsieh (JIRA)" <ji...@apache.org> on 2011/08/03 11:22:27 UTC
[jira] [Commented] (FLUME-719) Flume attribute field map from
Avro|ThriftEventConvertUtil.toFlumeEvent() should be mutable
[ https://issues.apache.org/jira/browse/FLUME-719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13078659#comment-13078659 ]
Jonathan Hsieh commented on FLUME-719:
--------------------------------------
Quoting Damien Hardy
{quote}
Flume was working well with CDH3 but now, after upgrade, fails on décorators every time I want to inject data.
Here is my conf for the node :
iflume02.int.adencf.local Wed Jul 13 11:09:37 CEST 2011 default-flow collectorSource digest("MD5", "digest",base64=true) haproxyLogExtractor() [ hbase("log", "%Y%m%d%H%M%S.%{nanos}-%{digest}", "default", "body", "%{body}",writeBufferSize=50000), amqpShuntingSink(), console]
haproxyLogExtractor is a tiny décorator witch extract values from haproxy logs
hbase is a tiny sink wich write data in hbase
amqpShuntingSink is a tiny sink writing data in amqp message broker.
This is the log of flume-node (failling on the builtin Digest decorator) but if I remove the digest décorator it fail on the haproxyLogExtractor() with the same exception on append().
2011-07-26 12:12:42,916 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop002.back.adencf.local:2181,hadoop001.back.adencf.local:2181 sessionTimeout=180000 watcher=hconnection
2011-07-26 12:12:42,926 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop002.back.adencf.local/192.168.130.93:2181
2011-07-26 12:12:42,931 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop002.back.adencf.local/192.168.130.93:2181, initiating session
2011-07-26 12:12:42,951 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop002.back.adencf.local/192.168.130.93:2181, sessionid = 0x23165e59a790000, negotiated timeout = 40000
2011-07-26 12:12:43,052 INFO com.figarocms.flume.HBaseSink: HBase sink successfully opened
2011-07-26 12:12:43,070 INFO org.springframework.context.support.ClassPathXmlApplicationContext: Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@5b6b9e62: startup date [Tue Jul 26 12:12:43 CEST 2011]; root of context hierarchy
2011-07-26 12:12:43,103 INFO org.springframework.beans.factory.xml.XmlBeanDefinitionReader: Loading XML bean definitions from class path resource [router.configuration.xml]
2011-07-26 12:12:43,184 INFO org.springframework.beans.factory.xml.XmlBeanDefinitionReader: Loading XML bean definitions from class path resource [routes.configuration.xml]
2011-07-26 12:12:43,234 INFO org.springframework.beans.factory.support.DefaultListableBeanFactory: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@303ec561: defining beans [amqp,routes,rabbitConnectionFactory,JsonMessageConverter,rabbit.template,amqpAdmin,router]; root of factory hierarchy
2011-07-26 12:12:43,274 INFO org.springframework.beans.factory.config.PropertiesFactoryBean: Loading properties file from class path resource [amqp.properties]
2011-07-26 12:12:43,354 INFO com.cloudera.flume.handlers.debug.ConsoleEventSink: ConsoleEventSink( debug ) opened
2011-07-26 12:14:11,981 ERROR com.cloudera.flume.core.connector.DirectDriver: Closing down due to exception during append calls
java.lang.UnsupportedOperationException
at java.util.Collections$UnmodifiableMap.put(Collections.java:1285)
at com.cloudera.flume.core.EventBaseImpl.set(EventBaseImpl.java:65)
at com.cloudera.flume.core.DigestDecorator.append(DigestDecorator.java:59)
at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110)
2011-07-26 12:14:11,982 INFO com.cloudera.flume.core.connector.DirectDriver: Connector logicalNode iflume01.int.adencf.local-19 exited with error: null
java.lang.UnsupportedOperationException
at java.util.Collections$UnmodifiableMap.put(Collections.java:1285)
at com.cloudera.flume.core.EventBaseImpl.set(EventBaseImpl.java:65)
at com.cloudera.flume.core.DigestDecorator.append(DigestDecorator.java:59)
at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110)
2011-07-26 12:14:11,982 INFO com.cloudera.flume.collector.CollectorSource: closed
2011-07-26 12:14:12,982 INFO com.cloudera.flume.handlers.thrift.ThriftEventSource: Closed server on port 35853...
2011-07-26 12:14:12,982 INFO com.cloudera.flume.handlers.thrift.ThriftEventSource: Queue still has 0 elements ...
2011-07-26 12:14:12,983 INFO com.figarocms.flume.HBaseSink: HBase sink successfully closed
{quote}
> Flume attribute field map from Avro|ThriftEventConvertUtil.toFlumeEvent() should be mutable
> -------------------------------------------------------------------------------------------
>
> Key: FLUME-719
> URL: https://issues.apache.org/jira/browse/FLUME-719
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v0.9.4
> Reporter: Jonathan Hsieh
> Priority: Critical
> Fix For: v0.9.5
>
>
> This fixes the root cause of the bug created by the FLUME-620 fix. The problem was that the getAttrs methods in both returned immutable hashtables. This makes the conversion methods return mutable tables, and tests for these cases.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira