You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@camel.apache.org by "Claus Ibsen (JIRA)" <ji...@apache.org> on 2012/07/09 11:14:33 UTC
[jira] [Created] (CAMEL-5433) camel-hdfs - Allow to startup async
as hdfs file system has hardcoded 15 min retry in case no connection
Claus Ibsen created CAMEL-5433:
----------------------------------
Summary: camel-hdfs - Allow to startup async as hdfs file system has hardcoded 15 min retry in case no connection
Key: CAMEL-5433
URL: https://issues.apache.org/jira/browse/CAMEL-5433
Project: Camel
Issue Type: Improvement
Components: camel-hdfs
Affects Versions: 2.10.0
Reporter: Claus Ibsen
Assignee: Claus Ibsen
Fix For: 2.10.1, 2.11
When start a Camel app with camel-hdfs (either as producer or consumer), then getting a hdfs file system object may have Hadoop block for up till 15 minutes where it tries to obtain a connection to the cluster.
This makes the app appear as blocking and people dont want their apps to block for 15 min on startup.
We should let the hadoop hdfs connection happing in an async thread, so the app can startup. Configurable so you can turn back the old behavior.
Hadoop dosn't allow to configure a lower timeout etc this as its hardcoded.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAMEL-5433) camel-hdfs - Allow to startup async
as hdfs file system has hardcoded 15 min retry in case no connection
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13409308#comment-13409308 ]
Claus Ibsen commented on CAMEL-5433:
------------------------------------
There is a new {{connectOnStartup}} option.
Will need to add new option for async startup logic as well.
> camel-hdfs - Allow to startup async as hdfs file system has hardcoded 15 min retry in case no connection
> --------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-5433
> URL: https://issues.apache.org/jira/browse/CAMEL-5433
> Project: Camel
> Issue Type: Improvement
> Components: camel-hdfs
> Affects Versions: 2.10.0
> Reporter: Claus Ibsen
> Assignee: Claus Ibsen
> Fix For: 2.10.1, 2.11
>
>
> When start a Camel app with camel-hdfs (either as producer or consumer), then getting a hdfs file system object may have Hadoop block for up till 15 minutes where it tries to obtain a connection to the cluster.
> This makes the app appear as blocking and people dont want their apps to block for 15 min on startup.
> We should let the hadoop hdfs connection happing in an async thread, so the app can startup. Configurable so you can turn back the old behavior.
> Hadoop dosn't allow to configure a lower timeout etc this as its hardcoded.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAMEL-5433) camel-hdfs - Allow to startup async
as hdfs file system has hardcoded 15 min retry in case no connection
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13409279#comment-13409279 ]
Claus Ibsen commented on CAMEL-5433:
------------------------------------
Looking in the hadoop-core source code it seems that re-connect logic is hardcoded to 45 max attempts with 20 sec delay
{code}
/* The max number of retries is 45,
* which amounts to 20s*45 = 15 minutes retries.
*/
handleConnectionFailure(timeoutFailures++, 45, toe);
{code}
> camel-hdfs - Allow to startup async as hdfs file system has hardcoded 15 min retry in case no connection
> --------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-5433
> URL: https://issues.apache.org/jira/browse/CAMEL-5433
> Project: Camel
> Issue Type: Improvement
> Components: camel-hdfs
> Affects Versions: 2.10.0
> Reporter: Claus Ibsen
> Assignee: Claus Ibsen
> Fix For: 2.10.1, 2.11
>
>
> When start a Camel app with camel-hdfs (either as producer or consumer), then getting a hdfs file system object may have Hadoop block for up till 15 minutes where it tries to obtain a connection to the cluster.
> This makes the app appear as blocking and people dont want their apps to block for 15 min on startup.
> We should let the hadoop hdfs connection happing in an async thread, so the app can startup. Configurable so you can turn back the old behavior.
> Hadoop dosn't allow to configure a lower timeout etc this as its hardcoded.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CAMEL-5433) camel-hdfs - Allow to lazy connect as
hdfs file system has hardcoded 15 min retry in case no connection
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen updated CAMEL-5433:
-------------------------------
Summary: camel-hdfs - Allow to lazy connect as hdfs file system has hardcoded 15 min retry in case no connection (was: camel-hdfs - Allow to startup async as hdfs file system has hardcoded 15 min retry in case no connection)
> camel-hdfs - Allow to lazy connect as hdfs file system has hardcoded 15 min retry in case no connection
> -------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-5433
> URL: https://issues.apache.org/jira/browse/CAMEL-5433
> Project: Camel
> Issue Type: Improvement
> Components: camel-hdfs
> Affects Versions: 2.10.0
> Reporter: Claus Ibsen
> Assignee: Claus Ibsen
> Fix For: 2.9.3, 2.10.1, 2.11
>
>
> When start a Camel app with camel-hdfs (either as producer or consumer), then getting a hdfs file system object may have Hadoop block for up till 15 minutes where it tries to obtain a connection to the cluster.
> This makes the app appear as blocking and people dont want their apps to block for 15 min on startup.
> We should let the hadoop hdfs connection happing in an async thread, so the app can startup. Configurable so you can turn back the old behavior.
> Hadoop dosn't allow to configure a lower timeout etc this as its hardcoded.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAMEL-5433) camel-hdfs - Allow to startup async
as hdfs file system has hardcoded 15 min retry in case no connection
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13409283#comment-13409283 ]
Claus Ibsen commented on CAMEL-5433:
------------------------------------
And example of thread dump during that retry logic
{code}
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:136)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <7fb6998a8> (a sun.nio.ch.Util$2)
- locked <7fb699898> (a java.util.Collections$UnmodifiableSet)
- locked <7fb699690> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:203)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:656)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:434)
- locked <7fb679d80> (a org.apache.hadoop.ipc.Client$Connection)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:560)
- locked <7fb679d80> (a org.apache.hadoop.ipc.Client$Connection)
at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1202)
at org.apache.hadoop.ipc.Client.call(Client.java:1046)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at $Proxy87.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
at org.apache.camel.component.hdfs.HdfsInfo.<init>(HdfsInfo.java:34)
at org.apache.camel.component.hdfs.HdfsOutputStream.createOutputStream(HdfsOutputStream.java:48)
at org.apache.camel.component.hdfs.HdfsProducer.doStart(HdfsProducer.java:93)
at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:60)
at org.apache.camel.util.ServiceHelper.startService(ServiceHelper.java:62)
at org.apache.camel.impl.ProducerCache.doGetProducer(ProducerCache.java:387)
{code}
> camel-hdfs - Allow to startup async as hdfs file system has hardcoded 15 min retry in case no connection
> --------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-5433
> URL: https://issues.apache.org/jira/browse/CAMEL-5433
> Project: Camel
> Issue Type: Improvement
> Components: camel-hdfs
> Affects Versions: 2.10.0
> Reporter: Claus Ibsen
> Assignee: Claus Ibsen
> Fix For: 2.10.1, 2.11
>
>
> When start a Camel app with camel-hdfs (either as producer or consumer), then getting a hdfs file system object may have Hadoop block for up till 15 minutes where it tries to obtain a connection to the cluster.
> This makes the app appear as blocking and people dont want their apps to block for 15 min on startup.
> We should let the hadoop hdfs connection happing in an async thread, so the app can startup. Configurable so you can turn back the old behavior.
> Hadoop dosn't allow to configure a lower timeout etc this as its hardcoded.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CAMEL-5433) camel-hdfs - Allow to lazy connect
as hdfs file system has hardcoded 15 min retry in case no connection
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen resolved CAMEL-5433.
--------------------------------
Resolution: Fixed
Lets leave it as currently, setting that option = false on startup, allows people to start their camel apps
> camel-hdfs - Allow to lazy connect as hdfs file system has hardcoded 15 min retry in case no connection
> -------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-5433
> URL: https://issues.apache.org/jira/browse/CAMEL-5433
> Project: Camel
> Issue Type: Improvement
> Components: camel-hdfs
> Affects Versions: 2.10.0
> Reporter: Claus Ibsen
> Assignee: Claus Ibsen
> Fix For: 2.9.3, 2.10.1, 2.11
>
>
> When start a Camel app with camel-hdfs (either as producer or consumer), then getting a hdfs file system object may have Hadoop block for up till 15 minutes where it tries to obtain a connection to the cluster.
> This makes the app appear as blocking and people dont want their apps to block for 15 min on startup.
> We should let the hadoop hdfs connection happing in an async thread, so the app can startup. Configurable so you can turn back the old behavior.
> Hadoop dosn't allow to configure a lower timeout etc this as its hardcoded.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAMEL-5433) camel-hdfs - Allow to startup async
as hdfs file system has hardcoded 15 min retry in case no connection
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13409280#comment-13409280 ]
Claus Ibsen commented on CAMEL-5433:
------------------------------------
Logs where you can see the retry happening
{code}
2012-07-05 14:50:17,708 | INFO | FelixStartLevel | org.apache.hadoop.ipc.Client | | Retrying connect to server: 192.168.240.129/192.168.240.129:9000. Already tried 0 time(s).
2012-07-05 14:50:38,708 | INFO | FelixStartLevel | org.apache.hadoop.ipc.Client | | Retrying connect to server: 192.168.240.129/192.168.240.129:9000. Already tried 1 time(s).
2012-07-05 14:50:59,708 | INFO | FelixStartLevel | org.apache.hadoop.ipc.Client | | Retrying connect to server: 192.168.240.129/192.168.240.129:9000. Already tried 2 time(s).
2012-07-05 14:51:20,709 | INFO | FelixStartLevel | org.apache.hadoop.ipc.Client | | Retrying connect to server: 192.168.240.129/192.168.240.129:9000. Already tried 3 time(s).
2012-07-05 14:51:41,709 | INFO | FelixStartLevel | org.apache.hadoop.ipc.Client | | Retrying connect to server: 192.168.240.129/192.168.240.129:9000. Already tried 4 time(s).
2012-07-05 14:52:02,709 | INFO | FelixStartLevel | org.apache.hadoop.ipc.Client | | Retrying connect to server: 192.168.240.129/192.168.240.129:9000. Already tried 5 time(s).
2012-07-05 14:52:23,709 | INFO | FelixStartLevel | org.apache.hadoop.ipc.Client | | Retrying connect to server: 192.168.240.129/192.168.240.129:9000. Already tried 6 time(s).
2012-07-05 14:52:44,710 | INFO | FelixStartLevel | org.apache.hadoop.ipc.Client | | Retrying connect to server: 192.168.240.129/192.168.240.129:9000. Already tried 7 time(s).
2012-07-05 14:53:05,710 | INFO | FelixStartLevel | org.apache.hadoop.ipc.Client | | Retrying connect to server: 192.168.240.129/192.168.240.129:9000. Already tried 8 time(s).
2012-07-05 14:53:26,710 | INFO | FelixStartLevel | org.apache.hadoop.ipc.Client | | Retrying connect to server: 192.168.240.129/192.168.240.129:9000. Already tried 9 time(s).
2012-07-05 14:53:47,710 | INFO | FelixStartLevel | org.apache.hadoop.ipc.Client | | Retrying connect to server: 192.168.240.129/192.168.240.129:9000. Already tried 10 time(s).
2012-07-05 14:54:08,710 | INFO | FelixStartLevel | org.apache.hadoop.ipc.Client | | Retrying connect to server: 192.168.240.129/192.168.240.129:9000. Already tried 11 time(s).
2012-07-05 14:54:29,693 | INFO | FelixStartLevel | org.apache.hadoop.ipc.Client | | Retrying connect to server: 192.168.240.129/192.168.240.129:9000. Already tried 12 time(s).
2012-07-05 14:54:50,694 | INFO | FelixStartLevel | org.apache.hadoop.ipc.Client | | Retrying connect to server: 192.168.240.129/192.168.240.129:9000. Already tried 13 time(s).
2012-07-05 14:55:11,695 | INFO | FelixStartLevel | org.apache.hadoop.ipc.Client | | Retrying connect to server: 192.168.240.129/192.168.240.129:9000. Already tried 14 time(s).
2012-07-05 14:55:32,696 | INFO | FelixStartLevel | org.apache.hadoop.ipc.Client | | Retrying connect to server: 192.168.240.129/192.168.240.129:9000. Already tried 15 time(s).
2012-07-05 14:55:53,697 | INFO | FelixStartLevel | org.apache.hadoop.ipc.Client | | Retrying connect to server: 192.168.240.129/192.168.240.129:9000. Already tried 16 time(s).
2012-07-05 14:56:14,698 | INFO | FelixStartLevel | org.apache.hadoop.ipc.Client | | Retrying connect to server: 192.168.240.129/192.168.240.129:9000. Already tried 17 time(s).
2012-07-05 14:56:35,699 | INFO | FelixStartLevel | org.apache.hadoop.ipc.Client | | Retrying connect to server: 192.168.240.129/192.168.240.129:9000. Already tried 18 time(s).
2012-07-05 14:56:56,700 | INFO | FelixStartLevel | org.apache.hadoop.ipc.Client | | Retrying connect to server: 192.168.240.129/192.168.240.129:9000. Already tried 19 time(s).
{code}
> camel-hdfs - Allow to startup async as hdfs file system has hardcoded 15 min retry in case no connection
> --------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-5433
> URL: https://issues.apache.org/jira/browse/CAMEL-5433
> Project: Camel
> Issue Type: Improvement
> Components: camel-hdfs
> Affects Versions: 2.10.0
> Reporter: Claus Ibsen
> Assignee: Claus Ibsen
> Fix For: 2.10.1, 2.11
>
>
> When start a Camel app with camel-hdfs (either as producer or consumer), then getting a hdfs file system object may have Hadoop block for up till 15 minutes where it tries to obtain a connection to the cluster.
> This makes the app appear as blocking and people dont want their apps to block for 15 min on startup.
> We should let the hadoop hdfs connection happing in an async thread, so the app can startup. Configurable so you can turn back the old behavior.
> Hadoop dosn't allow to configure a lower timeout etc this as its hardcoded.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CAMEL-5433) camel-hdfs - Allow to startup async
as hdfs file system has hardcoded 15 min retry in case no connection
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen updated CAMEL-5433:
-------------------------------
Fix Version/s: 2.9.3
> camel-hdfs - Allow to startup async as hdfs file system has hardcoded 15 min retry in case no connection
> --------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-5433
> URL: https://issues.apache.org/jira/browse/CAMEL-5433
> Project: Camel
> Issue Type: Improvement
> Components: camel-hdfs
> Affects Versions: 2.10.0
> Reporter: Claus Ibsen
> Assignee: Claus Ibsen
> Fix For: 2.9.3, 2.10.1, 2.11
>
>
> When start a Camel app with camel-hdfs (either as producer or consumer), then getting a hdfs file system object may have Hadoop block for up till 15 minutes where it tries to obtain a connection to the cluster.
> This makes the app appear as blocking and people dont want their apps to block for 15 min on startup.
> We should let the hadoop hdfs connection happing in an async thread, so the app can startup. Configurable so you can turn back the old behavior.
> Hadoop dosn't allow to configure a lower timeout etc this as its hardcoded.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira