[jira] [Created] (METRON-527) Connection pool shut down seen when running threatintel_taxii_load.sh script

["Justin Leet (JIRA)" <ji...@apache.org>]

Justin Leet created METRON-527:
----------------------------------

             Summary: Connection pool shut down seen when running threatintel_taxii_load.sh script
                 Key: METRON-527
                 URL: https://issues.apache.org/jira/browse/METRON-527
             Project: Metron
          Issue Type: Bug
            Reporter: Justin Leet
            Assignee: Justin Leet


Discovered by [~anandsubbu]:

I have setup opentaxii server on a 12-node metron cluster. I was able to scan and pull records into the taxi server.

{code}
[root@metron-test2-8 ~]# service opentaxii status
Checking opentaxii...                             Running
guest.phishtank_com                                5891
guest.Abuse_ch                                     49
guest.CyberCrime_Tracker                           0
guest.EmergingThreats_rules                        0
guest.Lehigh_edu                                   2025
guest.MalwareDomainList_Hostlist                   50
guest.blutmagie_de_torExits                        14897
guest.dataForLast_7daysOnly                        23210
guest.dshield_BlockList                            0
{code}

When I ran the threatintel_taxii_load.sh script, it is able to push some of the records into hbase 'threatintel' table, but fails with a "Connection pool shut down" error shortly after. This was observed for all the subscriptions that I attempted. 

Here is the command that was used:
{code}
[root@metron-test2-8 ~]# /usr/metron/0.2.0BETA/bin//threatintel_taxii_load.sh -b "2015-01-01 00:00:00" -c ~/connection_config.json -e ~/extractor.json -p 300000
{code}

The error message is pasted below. 

{code}
16/08/30 09:24:14 ERROR taxii.TaxiiHandler: Connection pool shut down
java.lang.IllegalStateException: Connection pool shut down
	at org.apache.metron.httpcore.dataload.util.Asserts.check(Asserts.java:34)
	at org.apache.metron.httpcore.dataload.pool.AbstractConnPool.lease(AbstractConnPool.java:169)
	at org.apache.metron.httpcore.dataload.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:217)
	at org.apache.metron.httpcore.dataload.impl.execchain.MainClientExec.execute(MainClientExec.java:158)
	at org.apache.metron.httpcore.dataload.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
	at org.apache.metron.httpcore.dataload.impl.execchain.RetryExec.execute(RetryExec.java:85)
	at org.apache.metron.httpcore.dataload.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
	at org.apache.metron.httpcore.dataload.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
	at org.apache.metron.httpcore.dataload.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.mitre.taxii.client.HttpClient.callTaxiiService(HttpClient.java:297)
	at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.call(TaxiiHandler.java:336)
	at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.call(TaxiiHandler.java:242)
	at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.run(TaxiiHandler.java:171)
	at java.util.TimerThread.mainLoop(Timer.java:555)
	at java.util.TimerThread.run(Timer.java:505)
Exception in thread "Timer-0" java.lang.RuntimeException: Unable to make request
	at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.run(TaxiiHandler.java:214)
	at java.util.TimerThread.mainLoop(Timer.java:555)
	at java.util.TimerThread.run(Timer.java:505)
Caused by: java.lang.IllegalStateException: Connection pool shut down
	at org.apache.metron.httpcore.dataload.util.Asserts.check(Asserts.java:34)
	at org.apache.metron.httpcore.dataload.pool.AbstractConnPool.lease(AbstractConnPool.java:169)
	at org.apache.metron.httpcore.dataload.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:217)
	at org.apache.metron.httpcore.dataload.impl.execchain.MainClientExec.execute(MainClientExec.java:158)
	at org.apache.metron.httpcore.dataload.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
	at org.apache.metron.httpcore.dataload.impl.execchain.RetryExec.execute(RetryExec.java:85)
	at org.apache.metron.httpcore.dataload.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
	at org.apache.metron.httpcore.dataload.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
	at org.apache.metron.httpcore.dataload.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.mitre.taxii.client.HttpClient.callTaxiiService(HttpClient.java:297)
	at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.call(TaxiiHandler.java:336)
	at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.call(TaxiiHandler.java:242)
	at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.run(TaxiiHandler.java:171)
	... 2 more
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)