You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flume.apache.org by "kaushik srinivas (JIRA)" <ji...@apache.org> on 2018/01/17 07:08:00 UTC
[jira] [Created] (FLUME-3213) TransactionCapacity for
spillableMemoryChannel with overflowCapacity set to '0' is limited to
1000000.
kaushik srinivas created FLUME-3213:
---------------------------------------
Summary: TransactionCapacity for spillableMemoryChannel with overflowCapacity set to '0' is limited to 1000000.
Key: FLUME-3213
URL: https://issues.apache.org/jira/browse/FLUME-3213
Project: Flume
Issue Type: Bug
Affects Versions: 1.7.0
Environment: Below is the setup info,
Channel :SPILLABLEMEMORY
byteCapacityBufferPercentage = 10
transactionCapacity = 2000000
memoryCapacity = 17000000
overflowCapacity = 0
Kafka source and kite data set sink is being used.
Reporter: kaushik srinivas
Intent is to make this spillabeMemory channel work as pure memory channel.
So the parameter,
overflowCapacity is set with value 0.
But then observing exceptions and by default capacity is being set to 1000000.
Error 1:
2018-01-17 05:57:26,783 (lifecycleSupervisor-1-0-EventThread) [WARN - org.apache.flume.channel.file.FileChannel.configure(FileChannel.java:167)] Invalid capacity specified, initializing channel to default capacity of 1000000
2018-01-17 05:57:26,787 (lifecycleSupervisor-1-0-EventThread) [ERROR - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:205)] Channel channel_store_sales has been removed due to an error during configuration
Impact of this is on the transaction capacity.
Error1 sets capacity to 10,00,000 and thus transactionCapacity is not allowed to be set more than the capacity value of 1000000 limit.
Error2:
java.lang.IllegalStateException: File Channel transaction capacity cannot be greater than the capacity of the channel.
at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
at org.apache.flume.channel.file.FileChannel.configure(FileChannel.java:186)
at org.apache.flume.channel.SpillableMemoryChannel.configure(SpillableMemoryChannel.java:745)
at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
at org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:199)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:100)
at org.apache.flume.node.PollingZooKeeperConfigurationProvider.refreshConfiguration(PollingZooKeeperConfigurationProvider.java:104)
at org.apache.flume.node.PollingZooKeeperConfigurationProvider.access$000(PollingZooKeeperConfigurationProvider.java:36)
at org.apache.flume.node.PollingZooKeeperConfigurationProvider$1.nodeChanged(PollingZooKeeperConfigurationProvider.java:78)
at org.apache.curator.framework.recipes.cache.NodeCache$4.apply(NodeCache.java:293)
at org.apache.curator.framework.recipes.cache.NodeCache$4.apply(NodeCache.java:287)
at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83)
at org.apache.curator.framework.recipes.cache.NodeCache.setNewData(NodeCache.java:284)
at org.apache.curator.framework.recipes.cache.NodeCache.processBackgroundResult(NodeCache.java:252)
at org.apache.curator.framework.recipes.cache.NodeCache.access$300(NodeCache.java:53)
at org.apache.curator.framework.recipes.cache.NodeCache$3.processResult(NodeCache.java:111)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:715)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:502)
at org.apache.curator.framework.imps.GetDataBuilderImpl$3.processResult(GetDataBuilderImpl.java:254)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:561)
So due to this, transactionCapacity for spillableMemoryChannel with overflowCapacity set to 0 is limited to 1000000.
So overflowCapacity has to be set with a high number in order to increase the transactionCapacity beyond1000000, which should not be the ideal scenario.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org
For additional commands, e-mail: issues-help@flume.apache.org