You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Jeff Jackson <Je...@faithlife.com> on 2015/10/13 18:58:30 UTC
Having trouble talking to elastic search from nutch 1.10
I'm trying to reindex my segments on a new elasticsearch server, and I'm having trouble. Sometimes, a segment will get indexed fine, but then on the next segment it will fail. I'm not seeing anything in elasticsearch's logs that would indicate a problem on that end (but I'm admittedly way out of area of expertise in dealing with this stuff0.
Below is what I'm seeing in nutch's hadoop.log. This is a fresh log file (I deleted the old one before running the bin/nutch index command). In this case it made it part way through indexing the segment before failing (I was watching the document count increase in marvel). Below that is the elasticsesarch log for the same timeframe.
Any idea what I might be doing wrong or how I might go about diagnosing the issue? Thanks,
Jeff Jackson
Hadoop.log:
2015-10-13 16:44:40,533 INFO indexer.IndexingJob - Indexer: starting at 2015-10-13 16:44:40
2015-10-13 16:44:40,645 INFO indexer.IndexingJob - Indexer: deleting gone documents: false
2015-10-13 16:44:40,645 INFO indexer.IndexingJob - Indexer: URL filtering: false
2015-10-13 16:44:40,645 INFO indexer.IndexingJob - Indexer: URL normalizing: false
2015-10-13 16:44:40,919 INFO indexer.IndexWriters - Adding org.apache.nutch.indexwriter.elastic.ElasticIndexWriter
2015-10-13 16:44:40,920 INFO indexer.IndexingJob - Active IndexWriters :
ElasticIndexWriter
elastic.cluster : elastic prefix cluster
elastic.host : hostname
elastic.port : port
elastic.index : elastic index command
elastic.max.bulk.docs : elastic bulk index doc counts. (default 250)
elastic.max.bulk.size : elastic bulk index length. (default 2500500 ~2.5MB)
2015-10-13 16:44:40,922 INFO indexer.IndexerMapReduce - IndexerMapReduce: crawldb: /root/apache-nutch-1.10/crawl/crawldb
2015-10-13 16:44:40,922 INFO indexer.IndexerMapReduce - IndexerMapReduce: linkdb: /root/apache-nutch-1.10/crawl/linkdb
2015-10-13 16:44:40,923 INFO indexer.IndexerMapReduce - IndexerMapReduces: adding segment: /root/apache-nutch-1.10/crawl/segments/20150526191748
2015-10-13 16:44:41,032 WARN util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-10-13 16:44:41,695 INFO anchor.AnchorIndexingFilter - Anchor deduplication is: off
2015-10-13 16:46:59,229 INFO indexer.IndexWriters - Adding org.apache.nutch.indexwriter.elastic.ElasticIndexWriter
2015-10-13 16:46:59,339 INFO elasticsearch.plugins - [Grandmaster] loaded [], sites []
2015-10-13 16:47:01,579 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 205, length = 2519186, total docs = 205, last doc in bulk = 'http://3forjc.blogspot.com/2010_11_01_archive.html']
2015-10-13 16:47:01,998 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 63, length = 2594569, total docs = 268, last doc in bulk = 'http://4womaninthewilderness.blogspot.com/2013_02_01_archive.html']
2015-10-13 16:47:02,170 INFO elastic.ElasticIndexWriter - Previous took in ms 384, including wait 171
2015-10-13 16:47:02,381 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 106, length = 2506430, total docs = 374, last doc in bulk = 'http://5621582817745579273_47da371f54bd3f164898f6392f5bdadc3d86df5e.blogspot.com/2015/05/vatican-officially-recognizes-state-of.html']
2015-10-13 16:47:02,824 INFO elastic.ElasticIndexWriter - Previous took in ms 541, including wait 443
2015-10-13 16:47:03,109 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 230, length = 2510289, total docs = 604, last doc in bulk = 'http://abc3miscellany.blogspot.com/2015_02_01_archive.html']
2015-10-13 16:47:03,622 INFO elastic.ElasticIndexWriter - Previous took in ms 604, including wait 513
2015-10-13 16:47:03,884 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 1938835, total docs = 854, last doc in bulk = 'http://activemindbodyandsoul.org/category/daily-climb/']
2015-10-13 16:47:04,287 INFO elastic.ElasticIndexWriter - Previous took in ms 610, including wait 403
2015-10-13 16:47:04,485 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 174, length = 2502740, total docs = 1028, last doc in bulk = 'http://aglow.com/resources/leader-development/prophetic-messages']
2015-10-13 16:47:05,089 INFO elastic.ElasticIndexWriter - Previous took in ms 713, including wait 604
2015-10-13 16:47:05,215 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 1347205, total docs = 1278, last doc in bulk = 'http://aglowinternational.org/give/a-company']
2015-10-13 16:47:05,867 INFO elastic.ElasticIndexWriter - Previous took in ms 718, including wait 652
2015-10-13 16:47:06,126 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 2298233, total docs = 1528, last doc in bulk = 'http://allsoulschristianchurch.com/mediaPlayer/']
2015-10-13 16:47:06,126 INFO elastic.ElasticIndexWriter - Previous took in ms 198, including wait 0
2015-10-13 16:47:06,270 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 195, length = 2509543, total docs = 1723, last doc in bulk = 'http://amazingfactsministries.com/index.php/publications/online-library/life-in-the-spirit']
2015-10-13 16:47:06,471 INFO elastic.ElasticIndexWriter - Previous took in ms 296, including wait 201
2015-10-13 16:47:06,654 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 92, length = 2563300, total docs = 1815, last doc in bulk = 'http://ancientchristiandefender.blogspot.com/2008_06_01_archive.html']
2015-10-13 16:47:07,069 INFO elastic.ElasticIndexWriter - Previous took in ms 544, including wait 414
2015-10-13 16:47:07,186 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 94, length = 2502386, total docs = 1909, last doc in bulk = 'http://andreayorkmuse.blogspot.com/2013_11_01_archive.html']
2015-10-13 16:47:07,650 INFO elastic.ElasticIndexWriter - Previous took in ms 461, including wait 463
2015-10-13 16:47:07,873 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 1096352, total docs = 2159, last doc in bulk = 'http://anunworthyservant.com/tag/churchianity/']
2015-10-13 16:47:08,026 INFO elastic.ElasticIndexWriter - Previous took in ms 320, including wait 153
2015-10-13 16:47:08,131 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 98, length = 2551881, total docs = 2257, last doc in bulk = 'http://apocalypse2010.blogspot.com/2012_12_01_archive.html']
2015-10-13 16:47:08,228 INFO elastic.ElasticIndexWriter - Previous took in ms 178, including wait 97
2015-10-13 16:47:08,424 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 199, length = 2513815, total docs = 2456, last doc in bulk = 'http://apostolicendtimescenario.blogspot.com/2009/08/is-third-temple-legitimate.html']
2015-10-13 16:47:13,989 INFO elastic.ElasticIndexWriter - Previous took in ms 5687, including wait 5565
2015-10-13 16:47:14,113 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 138, length = 2505616, total docs = 2594, last doc in bulk = 'http://apostolicvision.blogspot.com/2010/02/toxicology-of-complaining.html']
2015-10-13 16:47:14,339 INFO elastic.ElasticIndexWriter - Previous took in ms 284, including wait 226
2015-10-13 16:47:14,689 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 241, length = 2524444, total docs = 2835, last doc in bulk = 'http://armstrongismlibrary.blogspot.ca/2013_10_06_archive.html']
2015-10-13 16:47:14,811 INFO elastic.ElasticIndexWriter - Previous took in ms 416, including wait 121
2015-10-13 16:47:14,898 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 37, length = 2504364, total docs = 2872, last doc in bulk = 'http://armstrongismlibrary.blogspot.ca/2014_06_22_archive.html']
2015-10-13 16:47:15,411 INFO elastic.ElasticIndexWriter - Previous took in ms 544, including wait 513
2015-10-13 16:47:15,506 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 43, length = 2535017, total docs = 2915, last doc in bulk = 'http://armstrongismlibrary.blogspot.ca/2015_04_19_archive.html']
2015-10-13 16:47:15,869 INFO elastic.ElasticIndexWriter - Previous took in ms 402, including wait 363
2015-10-13 16:47:15,964 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 36, length = 2527853, total docs = 2951, last doc in bulk = 'http://armstrongismlibrary.blogspot.co.nz/2014_04_20_archive.html']
2015-10-13 16:47:16,302 INFO elastic.ElasticIndexWriter - Previous took in ms 349, including wait 338
2015-10-13 16:47:16,393 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 44, length = 2549719, total docs = 2995, last doc in bulk = 'http://armstrongismlibrary.blogspot.co.nz/2015_02_22_archive.html']
2015-10-13 16:47:23,374 INFO elastic.ElasticIndexWriter - Previous took in ms 7002, including wait 6981
2015-10-13 16:47:23,475 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 36, length = 2567438, total docs = 3031, last doc in bulk = 'http://armstrongismlibrary.blogspot.co.uk/2014_02_23_archive.html']
2015-10-13 16:47:23,994 INFO elastic.ElasticIndexWriter - Previous took in ms 493, including wait 519
2015-10-13 16:47:24,083 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 43, length = 2512172, total docs = 3074, last doc in bulk = 'http://armstrongismlibrary.blogspot.co.uk/2014_12_21_archive.html']
2015-10-13 16:47:25,074 INFO elastic.ElasticIndexWriter - Previous took in ms 948, including wait 991
2015-10-13 16:47:25,171 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 37, length = 2511370, total docs = 3111, last doc in bulk = 'http://armstrongismlibrary.blogspot.com.au/2013_12_29_archive.html']
2015-10-13 16:47:25,767 INFO elastic.ElasticIndexWriter - Previous took in ms 597, including wait 596
2015-10-13 16:47:25,861 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 41, length = 2535222, total docs = 3152, last doc in bulk = 'http://armstrongismlibrary.blogspot.com.au/2014_10_12_archive.html']
2015-10-13 16:47:26,902 INFO elastic.ElasticIndexWriter - Previous took in ms 1070, including wait 1041
2015-10-13 16:47:26,994 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 41, length = 2524383, total docs = 3193, last doc in bulk = 'http://armstrongismlibrary.blogspot.com/2013_12_01_archive.html']
2015-10-13 16:47:28,314 INFO elastic.ElasticIndexWriter - Previous took in ms 1185, including wait 1319
2015-10-13 16:47:28,405 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 41, length = 2532087, total docs = 3234, last doc in bulk = 'http://armstrongismlibrary.blogspot.com/2014_09_14_archive.html']
2015-10-13 16:47:29,500 INFO elastic.ElasticIndexWriter - Previous took in ms 1076, including wait 1095
2015-10-13 16:47:29,642 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 70, length = 2517694, total docs = 3304, last doc in bulk = 'http://ask.yuriyandinna.com/category/relationships/finding-a-spouse/']
2015-10-13 16:47:30,128 INFO elastic.ElasticIndexWriter - Previous took in ms 513, including wait 486
2015-10-13 16:47:30,353 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 2232716, total docs = 3554, last doc in bulk = 'http://babyloniansquirrel.blogspot.com/2010_05_01_archive.html']
2015-10-13 16:47:31,097 INFO elastic.ElasticIndexWriter - Previous took in ms 895, including wait 743
2015-10-13 16:47:31,223 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 118, length = 2518733, total docs = 3672, last doc in bulk = 'http://backtoluther.blogspot.com/2013_08_01_archive.html']
2015-10-13 16:47:31,771 INFO elastic.ElasticIndexWriter - Previous took in ms 573, including wait 548
2015-10-13 16:47:31,909 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 74, length = 2515412, total docs = 3746, last doc in bulk = 'http://baptist-distinctives.blogspot.com/2009/02/verbal-and-plenary-inspiration-of-bible.html']
2015-10-13 16:47:32,864 INFO elastic.ElasticIndexWriter - Previous took in ms 1035, including wait 955
2015-10-13 16:47:32,962 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 72, length = 2504157, total docs = 3818, last doc in bulk = 'http://baptist-rp.blogspot.com/2010/03/free-pdf-book-facebook-as-ministry-tool.html']
2015-10-13 16:47:33,588 INFO elastic.ElasticIndexWriter - Previous took in ms 540, including wait 626
2015-10-13 16:47:33,816 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 2417618, total docs = 4068, last doc in bulk = 'http://bearvalleychurch.org/slavic-gospel-the-mocks']
2015-10-13 16:47:34,653 INFO elastic.ElasticIndexWriter - Previous took in ms 881, including wait 836
2015-10-13 16:47:34,933 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 245, length = 2595875, total docs = 4313, last doc in bulk = 'http://bethanylutheranworship.blogspot.com/2008_11_01_archive.html']
2015-10-13 16:47:35,136 INFO elastic.ElasticIndexWriter - Previous took in ms 431, including wait 203
2015-10-13 16:47:35,212 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 17, length = 2515581, total docs = 4330, last doc in bulk = 'http://bethanylutheranworship.blogspot.com/2010_04_01_archive.html']
2015-10-13 16:47:35,940 INFO elastic.ElasticIndexWriter - Previous took in ms 746, including wait 728
2015-10-13 16:47:36,015 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 15, length = 2521102, total docs = 4345, last doc in bulk = 'http://bethanylutheranworship.blogspot.com/2011_07_01_archive.html']
2015-10-13 16:47:36,428 INFO elastic.ElasticIndexWriter - Previous took in ms 433, including wait 412
2015-10-13 16:47:36,511 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 19, length = 2548547, total docs = 4364, last doc in bulk = 'http://bethanylutheranworship.blogspot.com/2013_01_01_archive.html']
2015-10-13 16:47:37,171 INFO elastic.ElasticIndexWriter - Previous took in ms 687, including wait 660
2015-10-13 16:47:37,340 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 40, length = 2502627, total docs = 4404, last doc in bulk = 'http://bible-truths-revealed.com/RevelationOutline.html']
2015-10-13 16:47:37,674 INFO elastic.ElasticIndexWriter - Previous took in ms 399, including wait 334
2015-10-13 16:47:39,044 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 238, length = 2536423, total docs = 4642, last doc in bulk = 'http://biblenews1.com/grace/graced.htm']
2015-10-13 16:47:39,044 INFO elastic.ElasticIndexWriter - Previous took in ms 613, including wait 0
2015-10-13 16:47:39,317 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 246, length = 2502515, total docs = 4888, last doc in bulk = 'http://biblicalcreationandevangelism.blogspot.com/2015_02_01_archive.html']
2015-10-13 16:47:39,851 INFO elastic.ElasticIndexWriter - Previous took in ms 751, including wait 533
2015-10-13 16:47:40,158 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 163, length = 2583662, total docs = 5051, last doc in bulk = 'http://blog.chriskrycho.com/2010_10_01_archive.html']
2015-10-13 16:47:40,779 INFO elastic.ElasticIndexWriter - Previous took in ms 824, including wait 620
2015-10-13 16:47:41,056 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 176, length = 2505011, total docs = 5227, last doc in bulk = 'http://blog.poweredby4.org/challenge/2012/01/']
2015-10-13 16:47:41,550 INFO elastic.ElasticIndexWriter - Previous took in ms 527, including wait 494
2015-10-13 16:47:41,797 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 225, length = 2554774, total docs = 5452, last doc in bulk = 'http://bloggingscripturehisway.blogspot.com/2012_04_01_archive.html']
2015-10-13 16:47:42,308 INFO elastic.ElasticIndexWriter - Previous took in ms 702, including wait 510
2015-10-13 16:47:42,400 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 24, length = 2632288, total docs = 5476, last doc in bulk = 'http://bloggingscripturehisway.blogspot.com/2014_02_01_archive.html']
2015-10-13 16:47:42,881 INFO elastic.ElasticIndexWriter - Previous took in ms 463, including wait 480
2015-10-13 16:47:43,012 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 31, length = 2522883, total docs = 5507, last doc in bulk = 'http://blogotional.blogspot.com/2005_03_06_archive.html']
2015-10-13 16:47:43,778 INFO elastic.ElasticIndexWriter - Previous took in ms 827, including wait 766
2015-10-13 16:47:43,861 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 29, length = 2570359, total docs = 5536, last doc in bulk = 'http://blogotional.blogspot.com/2005_09_25_archive.html']
2015-10-13 16:47:44,418 INFO elastic.ElasticIndexWriter - Previous took in ms 539, including wait 557
2015-10-13 16:47:44,512 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 29, length = 2502887, total docs = 5565, last doc in bulk = 'http://blogotional.blogspot.com/2006_04_16_archive.html']
2015-10-13 16:47:45,348 INFO elastic.ElasticIndexWriter - Previous took in ms 855, including wait 836
2015-10-13 16:47:45,525 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 158, length = 2515000, total docs = 5723, last doc in bulk = 'http://brazilcarroll.org/page/2/']
2015-10-13 16:47:45,969 INFO elastic.ElasticIndexWriter - Previous took in ms 552, including wait 444
2015-10-13 16:47:46,211 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 1141609, total docs = 5973, last doc in bulk = 'http://calvarybaptistwarren.com/page/trivia']
2015-10-13 16:47:47,058 INFO elastic.ElasticIndexWriter - Previous took in ms 1039, including wait 847
2015-10-13 16:47:47,288 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 1308537, total docs = 6223, last doc in bulk = 'http://catalystcommunitychurch.org/people/seth-barber/']
2015-10-13 16:47:47,398 INFO elastic.ElasticIndexWriter - Previous took in ms 302, including wait 110
2015-10-13 16:47:47,579 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 122, length = 2500702, total docs = 6345, last doc in bulk = 'http://catholic-convert.com/resources/recommended/software/']
2015-10-13 16:47:48,057 INFO elastic.ElasticIndexWriter - Previous took in ms 611, including wait 478
2015-10-13 16:47:48,535 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 1727339, total docs = 6595, last doc in bulk = 'http://ccpville.com/2015/05/announcements-for-may-17-2015/']
2015-10-13 16:47:48,562 INFO elastic.ElasticIndexWriter - Previous took in ms 406, including wait 27
2015-10-13 16:47:48,878 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 1697119, total docs = 6845, last doc in bulk = 'http://cftministry.org/resources/bookmarks.html']
2015-10-13 16:47:49,356 INFO elastic.ElasticIndexWriter - Previous took in ms 747, including wait 478
2015-10-13 16:47:49,508 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 2183224, total docs = 7095, last doc in bulk = 'http://chicagoavenuechurchofchrist.org/new-years-resolution-christians/']
2015-10-13 16:47:49,771 INFO elastic.ElasticIndexWriter - Previous took in ms 315, including wait 263
2015-10-13 16:47:49,975 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 1717642, total docs = 7345, last doc in bulk = 'http://christevangelicalchurchmobmin.org/page/ministry_to_and_through_animals']
2015-10-13 16:47:50,455 INFO elastic.ElasticIndexWriter - Previous took in ms 599, including wait 480
2015-10-13 16:47:50,650 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 1600259, total docs = 7595, last doc in bulk = 'http://christianobserver.org/wp-includes/wlwmanifest.xml']
2015-10-13 16:47:51,021 INFO elastic.ElasticIndexWriter - Previous took in ms 507, including wait 371
2015-10-13 16:47:51,191 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 144, length = 2548642, total docs = 7739, last doc in bulk = 'http://christlifedailybible.blogspot.com.au/2012_07_01_archive.html']
2015-10-13 16:47:51,525 INFO elastic.ElasticIndexWriter - Previous took in ms 460, including wait 334
2015-10-13 16:47:51,661 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 156, length = 2501340, total docs = 7895, last doc in bulk = 'http://chuckanderson.blogspot.com/2013_01_01_archive.html']
2015-10-13 16:47:52,373 INFO elastic.ElasticIndexWriter - Previous took in ms 728, including wait 712
2015-10-13 16:47:52,645 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 205, length = 3001522, total docs = 8100, last doc in bulk = 'http://classicalchristianity.com/category/bysaint/blessed-augustine-ca-354-430/']
2015-10-13 16:47:53,310 INFO elastic.ElasticIndexWriter - Previous took in ms 881, including wait 664
2015-10-13 16:47:53,392 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 15, length = 2679457, total docs = 8115, last doc in bulk = 'http://classicalchristianity.com/category/bysaint/st-basil-of-caesarea-ca-330-379-%e3%80%80/']
2015-10-13 16:47:54,071 INFO elastic.ElasticIndexWriter - Previous took in ms 639, including wait 678
2015-10-13 16:47:54,157 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 20, length = 2506012, total docs = 8135, last doc in bulk = 'http://classicalchristianity.com/category/canon-law/']
2015-10-13 16:47:54,970 INFO elastic.ElasticIndexWriter - Previous took in ms 750, including wait 813
2015-10-13 16:47:55,067 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 13, length = 2964279, total docs = 8148, last doc in bulk = 'http://classicalchristianity.com/category/holyfathers/christology/']
2015-10-13 16:47:55,511 INFO elastic.ElasticIndexWriter - Previous took in ms 433, including wait 443
2015-10-13 16:47:55,598 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 17, length = 2755989, total docs = 8165, last doc in bulk = 'http://classicalchristianity.com/category/sacrament/']
2015-10-13 16:47:56,298 INFO elastic.ElasticIndexWriter - Previous took in ms 722, including wait 699
2015-10-13 16:47:56,487 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 107, length = 2598111, total docs = 8272, last doc in bulk = 'http://coffeehousebible.blogspot.com/2012_03_01_archive.html']
2015-10-13 16:47:56,781 INFO elastic.ElasticIndexWriter - Previous took in ms 384, including wait 294
2015-10-13 16:47:56,876 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 46, length = 2540888, total docs = 8318, last doc in bulk = 'http://coffeehousebible.blogspot.com/2015_04_01_archive.html']
2015-10-13 16:47:57,608 INFO elastic.ElasticIndexWriter - Previous took in ms 766, including wait 732
2015-10-13 16:47:57,894 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 221, length = 3106091, total docs = 8539, last doc in bulk = 'http://commfell.org/11009/ministry/ministry_id/301289/Men']
2015-10-13 16:47:58,091 INFO elastic.ElasticIndexWriter - Previous took in ms 359, including wait 197
2015-10-13 16:47:58,384 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 2241764, total docs = 8789, last doc in bulk = 'http://cornerstoneefree.org/mcintosh/']
2015-10-13 16:47:59,126 INFO elastic.ElasticIndexWriter - Previous took in ms 960, including wait 742
2015-10-13 16:47:59,374 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 2117950, total docs = 9039, last doc in bulk = 'http://crazycathie.ca/tag/quotes/']
2015-10-13 16:47:59,465 INFO elastic.ElasticIndexWriter - Previous took in ms 308, including wait 91
2015-10-13 16:47:59,760 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 2254727, total docs = 9289, last doc in bulk = 'http://cside.org/pastor-bryan-neal.aspx']
2015-10-13 16:47:59,970 INFO elastic.ElasticIndexWriter - Previous took in ms 458, including wait 209
2015-10-13 16:48:00,180 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 1911318, total docs = 9539, last doc in bulk = 'http://dailylightdevotional.org/01/0110.html']
2015-10-13 16:48:00,591 INFO elastic.ElasticIndexWriter - Previous took in ms 508, including wait 410
2015-10-13 16:48:00,746 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 1959251, total docs = 9789, last doc in bulk = 'http://davidmatthew.org.uk/wotwintro.html']
2015-10-13 16:48:01,061 INFO elastic.ElasticIndexWriter - Previous took in ms 433, including wait 315
2015-10-13 16:48:01,276 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 1000372, total docs = 10039, last doc in bulk = 'http://derekgriz.com/tag/student-ministry/']
2015-10-13 16:48:01,564 INFO elastic.ElasticIndexWriter - Previous took in ms 424, including wait 287
2015-10-13 16:48:01,795 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 217, length = 2526991, total docs = 10256, last doc in bulk = 'http://diatheke.blogspot.com/2013_07_01_archive.html']
2015-10-13 16:48:01,883 INFO elastic.ElasticIndexWriter - Previous took in ms 255, including wait 88
2015-10-13 16:48:01,990 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 47, length = 2545740, total docs = 10303, last doc in bulk = 'http://dictionaryofdoctrine.com/House-of-Cards.html']
2015-10-13 16:48:02,696 INFO elastic.ElasticIndexWriter - Previous took in ms 679, including wait 706
2015-10-13 16:48:02,894 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 180, length = 2512128, total docs = 10483, last doc in bulk = 'http://distinctivediscipleship.com/category/daily-distinctives/']
2015-10-13 16:48:04,086 INFO elastic.ElasticIndexWriter - Previous took in ms 1271, including wait 1192
2015-10-13 16:48:04,222 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 58, length = 2501110, total docs = 10541, last doc in bulk = 'http://doctrine.org/jesus-vs-paul/']
2015-10-13 16:48:05,479 INFO elastic.ElasticIndexWriter - Previous took in ms 1248, including wait 1257
2015-10-13 16:48:05,565 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 42, length = 2520935, total docs = 10583, last doc in bulk = 'http://doctrine.org/understanding-the-book-of-revelation/']
2015-10-13 16:48:06,175 INFO elastic.ElasticIndexWriter - Previous took in ms 593, including wait 610
2015-10-13 16:48:06,406 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 170, length = 2504080, total docs = 10753, last doc in bulk = 'http://doulogos.blogspot.com/2005/09/interview-what-happened.html']
2015-10-13 16:48:06,854 INFO elastic.ElasticIndexWriter - Previous took in ms 552, including wait 448
2015-10-13 16:48:06,939 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 46, length = 2528772, total docs = 10799, last doc in bulk = 'http://doulogos.blogspot.com/2008_08_01_archive.html']
2015-10-13 16:48:07,511 INFO elastic.ElasticIndexWriter - Previous took in ms 603, including wait 572
2015-10-13 16:48:07,699 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 141, length = 2505336, total docs = 10940, last doc in bulk = 'http://drmsh.com/category/archaeology/']
2015-10-13 16:48:08,146 INFO elastic.ElasticIndexWriter - Previous took in ms 567, including wait 447
2015-10-13 16:48:08,312 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 194, length = 2557695, total docs = 11134, last doc in bulk = 'http://eaandfaith.blogspot.ca/2009_09_01_archive.html']
2015-10-13 16:48:08,758 INFO elastic.ElasticIndexWriter - Previous took in ms 551, including wait 446
2015-10-13 16:48:08,866 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 78, length = 2573722, total docs = 11212, last doc in bulk = 'http://eaandfaith.blogspot.co.uk/2009_09_01_archive.html']
2015-10-13 16:48:09,361 INFO elastic.ElasticIndexWriter - Previous took in ms 534, including wait 495
2015-10-13 16:48:09,460 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 78, length = 2574267, total docs = 11290, last doc in bulk = 'http://eaandfaith.blogspot.com.au/2009_09_01_archive.html']
2015-10-13 16:48:09,938 INFO elastic.ElasticIndexWriter - Previous took in ms 509, including wait 477
2015-10-13 16:48:10,042 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 76, length = 2502863, total docs = 11366, last doc in bulk = 'http://eaandfaith.blogspot.com/2009_10_01_archive.html']
2015-10-13 16:48:10,343 INFO elastic.ElasticIndexWriter - Previous took in ms 328, including wait 300
2015-10-13 16:48:10,567 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 204, length = 2548890, total docs = 11570, last doc in bulk = 'http://echoofrestorationtruths.blogspot.com/2013/10/the-language-of-beasts-of-revelation.html']
2015-10-13 16:48:10,894 INFO elastic.ElasticIndexWriter - Previous took in ms 447, including wait 327
2015-10-13 16:48:11,124 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 2158970, total docs = 11820, last doc in bulk = 'http://elbaptist.org/about/our-beliefs/civil-government']
2015-10-13 16:48:14,103 INFO elastic.ElasticIndexWriter - Previous took in ms 610, including wait 2979
2015-10-13 16:48:14,321 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 239, length = 2543200, total docs = 12059, last doc in bulk = 'http://encountering-ahnsahnghong.blogspot.com/2012_02_01_archive.html']
2015-10-13 16:48:14,530 INFO elastic.ElasticIndexWriter - Previous took in ms 382, including wait 208
2015-10-13 16:48:14,682 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 145, length = 2577069, total docs = 12204, last doc in bulk = 'http://endtimepilgrim.org/puritans12.htm']
2015-10-13 16:48:15,212 INFO elastic.ElasticIndexWriter - Previous took in ms 628, including wait 530
2015-10-13 16:48:15,356 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 208, length = 2502224, total docs = 12412, last doc in bulk = 'http://english.genesis6.org/unmasking-sda-ellen-g-whites-satanic-hold-on-the-on-youtube/']
2015-10-13 16:48:23,467 INFO client.transport - [Grandmaster] failed to get node info for [#transport#-1][ci-dev-web06.lrscorp.net][inet[ci-dev-search04/10.70.15.17:9300]], disconnecting...
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[ci-dev-search04/10.70.15.17:9300]][cluster:monitor/nodes/info] request_id [101] timed out after [5002ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2015-10-13 16:48:33,475 INFO client.transport - [Grandmaster] failed to get node info for [#transport#-1][ci-dev-web06.lrscorp.net][inet[ci-dev-search04/10.70.15.17:9300]], disconnecting...
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[ci-dev-search04/10.70.15.17:9300]][cluster:monitor/nodes/info] request_id [102] timed out after [5000ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2015-10-13 16:48:37,246 INFO elastic.ElasticIndexWriter - Previous took in ms 21903, including wait 21890
2015-10-13 16:48:37,247 INFO elastic.ElasticIndexWriter - Processing remaining requests [docs = 208, length = 2502224, total docs = 12412]
2015-10-13 16:48:37,255 WARN mapred.LocalJobRunner - job_local1050818242_0001
org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: []
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:278)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:197)
at org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:106)
at org.elasticsearch.client.support.AbstractClient.bulk(AbstractClient.java:163)
at org.elasticsearch.client.transport.TransportClient.bulk(TransportClient.java:364)
at org.elasticsearch.action.bulk.BulkRequestBuilder.doExecute(BulkRequestBuilder.java:164)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65)
at org.apache.nutch.indexwriter.elastic.ElasticIndexWriter.commit(ElasticIndexWriter.java:211)
at org.apache.nutch.indexwriter.elastic.ElasticIndexWriter.write(ElasticIndexWriter.java:161)
at org.apache.nutch.indexer.IndexWriters.write(IndexWriters.java:85)
at org.apache.nutch.indexer.IndexerOutputFormat$1.write(IndexerOutputFormat.java:50)
at org.apache.nutch.indexer.IndexerOutputFormat$1.write(IndexerOutputFormat.java:41)
at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.write(ReduceTask.java:458)
at org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:500)
at org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:337)
at org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:53)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:522)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:398)
2015-10-13 16:48:37,542 ERROR indexer.IndexingJob - Indexer: java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1357)
at org.apache.nutch.indexer.IndexingJob.index(IndexingJob.java:113)
at org.apache.nutch.indexer.IndexingJob.run(IndexingJob.java:177)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.indexer.IndexingJob.main(IndexingJob.java:187)
Here's my elastic search log for the same timeframe:
[2015-10-13 01:14:07,445][INFO ][cluster.service ] [ci-dev-search04] removed {[La Lunatica][e0LVxHLbSKGYNhEavjIJXw][ws1938][inet[ws1890.lrscorp.net/10.200.208.38:9300]]{data=false, client=true},}, reason: zen-disco-node_failed([La Lunatica][e0LVxHLbSKGYNhEavjIJXw][ws1938][inet[ws1890.lrscorp.net/10.200.208.38:9300]]{data=false, client=true}), reason transport disconnected
[2015-10-13 16:18:09,749][WARN ][monitor.jvm ] [ci-dev-search04] [gc][young][54349][4966] duration [23.3s], collections [4]/[24.1s], total [23.3s]/[50.8s], memory [205.2mb]->[177.9mb]/[989.8mb], all_pools {[young] [56.9mb]->[112.6kb]/[273mb]}{[survivor] [8.5mb]->[8.5mb]/[34.1mb]}{[old] [139.7mb]->[169.3mb]/[682.6mb]}
[2015-10-13 16:23:09,914][WARN ][monitor.jvm ] [ci-dev-search04] [gc][young][54629][5066] duration [20.1s], collections [1]/[20.4s], total [20.1s]/[1.2m], memory [171.5mb]->[153.4mb]/[989.8mb], all_pools {[young] [25.2mb]->[1.7mb]/[273mb]}{[survivor] [4.1mb]->[8.5mb]/[34.1mb]}{[old] [142.1mb]->[143.5mb]/[682.6mb]}
[2015-10-13 16:47:13,468][WARN ][monitor.jvm ] [ci-dev-search04] [gc][young][56067][5203] duration [5s], collections [1]/[5.1s], total [5s]/[1.3m], memory [248.1mb]->[191.2mb]/[989.8mb], all_pools {[young] [66mb]->[894.5kb]/[273mb]}{[survivor] [8.5mb]->[8.5mb]/[34.1mb]}{[old] [173.6mb]->[181.9mb]/[682.6mb]}
[2015-10-13 16:47:23,360][WARN ][monitor.jvm ] [ci-dev-search04] [gc][young][56071][5213] duration [6.3s], collections [2]/[6.8s], total [6.3s]/[1.4m], memory [267.5mb]->[279.2mb]/[989.8mb], all_pools {[young] [196.7kb]->[165.7kb]/[273mb]}{[survivor] [8.5mb]->[7.8mb]/[34.1mb]}{[old] [258.9mb]->[271.2mb]/[682.6mb]}
[2015-10-13 16:48:13,461][WARN ][monitor.jvm ] [ci-dev-search04] [gc][young][56119][5353] duration [2.4s], collections [1]/[2.7s], total [2.4s]/[1.5m], memory [296.7mb]->[257.5mb]/[989.8mb], all_pools {[young] [48.2mb]->[785.7kb]/[273mb]}{[survivor] [8.5mb]->[8.5mb]/[34.1mb]}{[old] [239.9mb]->[248.4mb]/[682.6mb]}
[2015-10-13 16:48:36,621][WARN ][monitor.jvm ] [ci-dev-search04] [gc][young][56122][5360] duration [21s], collections [1]/[21.1s], total [21s]/[1.9m], memory [334.2mb]->[311.6mb]/[989.8mb], all_pools {[young] [29.1mb]->[991.5kb]/[273mb]}{[survivor] [3.1mb]->[8.5mb]/[34.1mb]}{[old] [302mb]->[302.3mb]/[682.6mb]}
Re: [MASSMAIL]Having trouble talking to elastic search from nutch
1.10
Posted by Jorge Luis Betancourt González <jl...@uci.cu>.
Looks like for some reason your elasticsearch cluster is becoming irresponsive or at least inaccesible to Nutch:
2015-10-13 16:48:23,467 INFO client.transport - [Grandmaster] failed to get node info for [#transport#-1][ci-dev-web06.lrscorp.net][inet[ci-dev-search04/10.70.15.17:9300]], disconnecting...
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[ci-dev-search04/10.70.15.17:9300]][cluster:monitor/nodes/info] request_id [101] timed out after [5002ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2015-10-13 16:48:33,475 INFO client.transport - [Grandmaster] failed to get node info for [#transport#-1][ci-dev-web06.lrscorp.net][inet[ci-dev-search04/10.70.15.17:9300]], disconnecting...
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[ci-dev-search04/10.70.15.17:9300]][cluster:monitor/nodes/info] request_id [102] timed out after [5000ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2015-10-13 16:48:37,246 INFO elastic.ElasticIndexWriter - Previous took in ms 21903, including wait 21890
2015-10-13 16:48:37,247 INFO elastic.ElasticIndexWriter - Processing remaining requests [docs = 208, length = 2502224, total docs = 12412]
2015-10-13 16:48:37,255 WARN mapred.LocalJobRunner - job_local1050818242_0001
org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: []
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:278)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:197)
at org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:106)
at org.elasticsearch.client.support.AbstractClient.bulk(AbstractClient.java:163)
at org.elasticsearch.client.transport.TransportClient.bulk(TransportClient.java:364)
at org.elasticsearch.action.bulk.BulkRequestBuilder.doExecute(BulkRequestBuilder.java:164)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65)
at org.apache.nutch.indexwriter.elastic.ElasticIndexWriter.commit(ElasticIndexWriter.java:211)
at org.apache.nutch.indexwriter.elastic.ElasticIndexWriter.write(ElasticIndexWriter.java:161)
at org.apache.nutch.indexer.IndexWriters.write(IndexWriters.java:85)
at org.apache.nutch.indexer.IndexerOutputFormat$1.write(IndexerOutputFormat.java:50)
at org.apache.nutch.indexer.IndexerOutputFormat$1.write(IndexerOutputFormat.java:41)
at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.write(ReduceTask.java:458)
at org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:500)
at org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:337)
at org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:53)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:522)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
This could be due a lot of reasons, does your elasticsearch cluster/node can be queried after the error in Nutch, are you monitoring (ej. Marvel) the cluster to get a more comprehensive view of what is going? Perhaps installing/enabling Marvel is a first step in the right direction.
Hope it helps,
----- Original Message -----
From: "Jeff Jackson" <Je...@faithlife.com>
To: user@nutch.apache.org
Sent: Tuesday, October 13, 2015 12:58:30 PM
Subject: [MASSMAIL]Having trouble talking to elastic search from nutch 1.10
I'm trying to reindex my segments on a new elasticsearch server, and I'm having trouble. Sometimes, a segment will get indexed fine, but then on the next segment it will fail. I'm not seeing anything in elasticsearch's logs that would indicate a problem on that end (but I'm admittedly way out of area of expertise in dealing with this stuff0.
Below is what I'm seeing in nutch's hadoop.log. This is a fresh log file (I deleted the old one before running the bin/nutch index command). In this case it made it part way through indexing the segment before failing (I was watching the document count increase in marvel). Below that is the elasticsesarch log for the same timeframe.
Any idea what I might be doing wrong or how I might go about diagnosing the issue? Thanks,
Jeff Jackson
Hadoop.log:
2015-10-13 16:44:40,533 INFO indexer.IndexingJob - Indexer: starting at 2015-10-13 16:44:40
2015-10-13 16:44:40,645 INFO indexer.IndexingJob - Indexer: deleting gone documents: false
2015-10-13 16:44:40,645 INFO indexer.IndexingJob - Indexer: URL filtering: false
2015-10-13 16:44:40,645 INFO indexer.IndexingJob - Indexer: URL normalizing: false
2015-10-13 16:44:40,919 INFO indexer.IndexWriters - Adding org.apache.nutch.indexwriter.elastic.ElasticIndexWriter
2015-10-13 16:44:40,920 INFO indexer.IndexingJob - Active IndexWriters :
ElasticIndexWriter
elastic.cluster : elastic prefix cluster
elastic.host : hostname
elastic.port : port
elastic.index : elastic index command
elastic.max.bulk.docs : elastic bulk index doc counts. (default 250)
elastic.max.bulk.size : elastic bulk index length. (default 2500500 ~2.5MB)
2015-10-13 16:44:40,922 INFO indexer.IndexerMapReduce - IndexerMapReduce: crawldb: /root/apache-nutch-1.10/crawl/crawldb
2015-10-13 16:44:40,922 INFO indexer.IndexerMapReduce - IndexerMapReduce: linkdb: /root/apache-nutch-1.10/crawl/linkdb
2015-10-13 16:44:40,923 INFO indexer.IndexerMapReduce - IndexerMapReduces: adding segment: /root/apache-nutch-1.10/crawl/segments/20150526191748
2015-10-13 16:44:41,032 WARN util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-10-13 16:44:41,695 INFO anchor.AnchorIndexingFilter - Anchor deduplication is: off
2015-10-13 16:46:59,229 INFO indexer.IndexWriters - Adding org.apache.nutch.indexwriter.elastic.ElasticIndexWriter
2015-10-13 16:46:59,339 INFO elasticsearch.plugins - [Grandmaster] loaded [], sites []
2015-10-13 16:47:01,579 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 205, length = 2519186, total docs = 205, last doc in bulk = 'http://3forjc.blogspot.com/2010_11_01_archive.html']
2015-10-13 16:47:01,998 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 63, length = 2594569, total docs = 268, last doc in bulk = 'http://4womaninthewilderness.blogspot.com/2013_02_01_archive.html']
2015-10-13 16:47:02,170 INFO elastic.ElasticIndexWriter - Previous took in ms 384, including wait 171
2015-10-13 16:47:02,381 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 106, length = 2506430, total docs = 374, last doc in bulk = 'http://5621582817745579273_47da371f54bd3f164898f6392f5bdadc3d86df5e.blogspot.com/2015/05/vatican-officially-recognizes-state-of.html']
2015-10-13 16:47:02,824 INFO elastic.ElasticIndexWriter - Previous took in ms 541, including wait 443
2015-10-13 16:47:03,109 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 230, length = 2510289, total docs = 604, last doc in bulk = 'http://abc3miscellany.blogspot.com/2015_02_01_archive.html']
2015-10-13 16:47:03,622 INFO elastic.ElasticIndexWriter - Previous took in ms 604, including wait 513
2015-10-13 16:47:03,884 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 1938835, total docs = 854, last doc in bulk = 'http://activemindbodyandsoul.org/category/daily-climb/']
2015-10-13 16:47:04,287 INFO elastic.ElasticIndexWriter - Previous took in ms 610, including wait 403
2015-10-13 16:47:04,485 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 174, length = 2502740, total docs = 1028, last doc in bulk = 'http://aglow.com/resources/leader-development/prophetic-messages']
2015-10-13 16:47:05,089 INFO elastic.ElasticIndexWriter - Previous took in ms 713, including wait 604
2015-10-13 16:47:05,215 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 1347205, total docs = 1278, last doc in bulk = 'http://aglowinternational.org/give/a-company']
2015-10-13 16:47:05,867 INFO elastic.ElasticIndexWriter - Previous took in ms 718, including wait 652
2015-10-13 16:47:06,126 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 2298233, total docs = 1528, last doc in bulk = 'http://allsoulschristianchurch.com/mediaPlayer/']
2015-10-13 16:47:06,126 INFO elastic.ElasticIndexWriter - Previous took in ms 198, including wait 0
2015-10-13 16:47:06,270 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 195, length = 2509543, total docs = 1723, last doc in bulk = 'http://amazingfactsministries.com/index.php/publications/online-library/life-in-the-spirit']
2015-10-13 16:47:06,471 INFO elastic.ElasticIndexWriter - Previous took in ms 296, including wait 201
2015-10-13 16:47:06,654 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 92, length = 2563300, total docs = 1815, last doc in bulk = 'http://ancientchristiandefender.blogspot.com/2008_06_01_archive.html']
2015-10-13 16:47:07,069 INFO elastic.ElasticIndexWriter - Previous took in ms 544, including wait 414
2015-10-13 16:47:07,186 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 94, length = 2502386, total docs = 1909, last doc in bulk = 'http://andreayorkmuse.blogspot.com/2013_11_01_archive.html']
2015-10-13 16:47:07,650 INFO elastic.ElasticIndexWriter - Previous took in ms 461, including wait 463
2015-10-13 16:47:07,873 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 1096352, total docs = 2159, last doc in bulk = 'http://anunworthyservant.com/tag/churchianity/']
2015-10-13 16:47:08,026 INFO elastic.ElasticIndexWriter - Previous took in ms 320, including wait 153
2015-10-13 16:47:08,131 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 98, length = 2551881, total docs = 2257, last doc in bulk = 'http://apocalypse2010.blogspot.com/2012_12_01_archive.html']
2015-10-13 16:47:08,228 INFO elastic.ElasticIndexWriter - Previous took in ms 178, including wait 97
2015-10-13 16:47:08,424 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 199, length = 2513815, total docs = 2456, last doc in bulk = 'http://apostolicendtimescenario.blogspot.com/2009/08/is-third-temple-legitimate.html']
2015-10-13 16:47:13,989 INFO elastic.ElasticIndexWriter - Previous took in ms 5687, including wait 5565
2015-10-13 16:47:14,113 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 138, length = 2505616, total docs = 2594, last doc in bulk = 'http://apostolicvision.blogspot.com/2010/02/toxicology-of-complaining.html']
2015-10-13 16:47:14,339 INFO elastic.ElasticIndexWriter - Previous took in ms 284, including wait 226
2015-10-13 16:47:14,689 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 241, length = 2524444, total docs = 2835, last doc in bulk = 'http://armstrongismlibrary.blogspot.ca/2013_10_06_archive.html']
2015-10-13 16:47:14,811 INFO elastic.ElasticIndexWriter - Previous took in ms 416, including wait 121
2015-10-13 16:47:14,898 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 37, length = 2504364, total docs = 2872, last doc in bulk = 'http://armstrongismlibrary.blogspot.ca/2014_06_22_archive.html']
2015-10-13 16:47:15,411 INFO elastic.ElasticIndexWriter - Previous took in ms 544, including wait 513
2015-10-13 16:47:15,506 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 43, length = 2535017, total docs = 2915, last doc in bulk = 'http://armstrongismlibrary.blogspot.ca/2015_04_19_archive.html']
2015-10-13 16:47:15,869 INFO elastic.ElasticIndexWriter - Previous took in ms 402, including wait 363
2015-10-13 16:47:15,964 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 36, length = 2527853, total docs = 2951, last doc in bulk = 'http://armstrongismlibrary.blogspot.co.nz/2014_04_20_archive.html']
2015-10-13 16:47:16,302 INFO elastic.ElasticIndexWriter - Previous took in ms 349, including wait 338
2015-10-13 16:47:16,393 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 44, length = 2549719, total docs = 2995, last doc in bulk = 'http://armstrongismlibrary.blogspot.co.nz/2015_02_22_archive.html']
2015-10-13 16:47:23,374 INFO elastic.ElasticIndexWriter - Previous took in ms 7002, including wait 6981
2015-10-13 16:47:23,475 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 36, length = 2567438, total docs = 3031, last doc in bulk = 'http://armstrongismlibrary.blogspot.co.uk/2014_02_23_archive.html']
2015-10-13 16:47:23,994 INFO elastic.ElasticIndexWriter - Previous took in ms 493, including wait 519
2015-10-13 16:47:24,083 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 43, length = 2512172, total docs = 3074, last doc in bulk = 'http://armstrongismlibrary.blogspot.co.uk/2014_12_21_archive.html']
2015-10-13 16:47:25,074 INFO elastic.ElasticIndexWriter - Previous took in ms 948, including wait 991
2015-10-13 16:47:25,171 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 37, length = 2511370, total docs = 3111, last doc in bulk = 'http://armstrongismlibrary.blogspot.com.au/2013_12_29_archive.html']
2015-10-13 16:47:25,767 INFO elastic.ElasticIndexWriter - Previous took in ms 597, including wait 596
2015-10-13 16:47:25,861 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 41, length = 2535222, total docs = 3152, last doc in bulk = 'http://armstrongismlibrary.blogspot.com.au/2014_10_12_archive.html']
2015-10-13 16:47:26,902 INFO elastic.ElasticIndexWriter - Previous took in ms 1070, including wait 1041
2015-10-13 16:47:26,994 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 41, length = 2524383, total docs = 3193, last doc in bulk = 'http://armstrongismlibrary.blogspot.com/2013_12_01_archive.html']
2015-10-13 16:47:28,314 INFO elastic.ElasticIndexWriter - Previous took in ms 1185, including wait 1319
2015-10-13 16:47:28,405 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 41, length = 2532087, total docs = 3234, last doc in bulk = 'http://armstrongismlibrary.blogspot.com/2014_09_14_archive.html']
2015-10-13 16:47:29,500 INFO elastic.ElasticIndexWriter - Previous took in ms 1076, including wait 1095
2015-10-13 16:47:29,642 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 70, length = 2517694, total docs = 3304, last doc in bulk = 'http://ask.yuriyandinna.com/category/relationships/finding-a-spouse/']
2015-10-13 16:47:30,128 INFO elastic.ElasticIndexWriter - Previous took in ms 513, including wait 486
2015-10-13 16:47:30,353 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 2232716, total docs = 3554, last doc in bulk = 'http://babyloniansquirrel.blogspot.com/2010_05_01_archive.html']
2015-10-13 16:47:31,097 INFO elastic.ElasticIndexWriter - Previous took in ms 895, including wait 743
2015-10-13 16:47:31,223 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 118, length = 2518733, total docs = 3672, last doc in bulk = 'http://backtoluther.blogspot.com/2013_08_01_archive.html']
2015-10-13 16:47:31,771 INFO elastic.ElasticIndexWriter - Previous took in ms 573, including wait 548
2015-10-13 16:47:31,909 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 74, length = 2515412, total docs = 3746, last doc in bulk = 'http://baptist-distinctives.blogspot.com/2009/02/verbal-and-plenary-inspiration-of-bible.html']
2015-10-13 16:47:32,864 INFO elastic.ElasticIndexWriter - Previous took in ms 1035, including wait 955
2015-10-13 16:47:32,962 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 72, length = 2504157, total docs = 3818, last doc in bulk = 'http://baptist-rp.blogspot.com/2010/03/free-pdf-book-facebook-as-ministry-tool.html']
2015-10-13 16:47:33,588 INFO elastic.ElasticIndexWriter - Previous took in ms 540, including wait 626
2015-10-13 16:47:33,816 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 2417618, total docs = 4068, last doc in bulk = 'http://bearvalleychurch.org/slavic-gospel-the-mocks']
2015-10-13 16:47:34,653 INFO elastic.ElasticIndexWriter - Previous took in ms 881, including wait 836
2015-10-13 16:47:34,933 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 245, length = 2595875, total docs = 4313, last doc in bulk = 'http://bethanylutheranworship.blogspot.com/2008_11_01_archive.html']
2015-10-13 16:47:35,136 INFO elastic.ElasticIndexWriter - Previous took in ms 431, including wait 203
2015-10-13 16:47:35,212 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 17, length = 2515581, total docs = 4330, last doc in bulk = 'http://bethanylutheranworship.blogspot.com/2010_04_01_archive.html']
2015-10-13 16:47:35,940 INFO elastic.ElasticIndexWriter - Previous took in ms 746, including wait 728
2015-10-13 16:47:36,015 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 15, length = 2521102, total docs = 4345, last doc in bulk = 'http://bethanylutheranworship.blogspot.com/2011_07_01_archive.html']
2015-10-13 16:47:36,428 INFO elastic.ElasticIndexWriter - Previous took in ms 433, including wait 412
2015-10-13 16:47:36,511 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 19, length = 2548547, total docs = 4364, last doc in bulk = 'http://bethanylutheranworship.blogspot.com/2013_01_01_archive.html']
2015-10-13 16:47:37,171 INFO elastic.ElasticIndexWriter - Previous took in ms 687, including wait 660
2015-10-13 16:47:37,340 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 40, length = 2502627, total docs = 4404, last doc in bulk = 'http://bible-truths-revealed.com/RevelationOutline.html']
2015-10-13 16:47:37,674 INFO elastic.ElasticIndexWriter - Previous took in ms 399, including wait 334
2015-10-13 16:47:39,044 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 238, length = 2536423, total docs = 4642, last doc in bulk = 'http://biblenews1.com/grace/graced.htm']
2015-10-13 16:47:39,044 INFO elastic.ElasticIndexWriter - Previous took in ms 613, including wait 0
2015-10-13 16:47:39,317 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 246, length = 2502515, total docs = 4888, last doc in bulk = 'http://biblicalcreationandevangelism.blogspot.com/2015_02_01_archive.html']
2015-10-13 16:47:39,851 INFO elastic.ElasticIndexWriter - Previous took in ms 751, including wait 533
2015-10-13 16:47:40,158 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 163, length = 2583662, total docs = 5051, last doc in bulk = 'http://blog.chriskrycho.com/2010_10_01_archive.html']
2015-10-13 16:47:40,779 INFO elastic.ElasticIndexWriter - Previous took in ms 824, including wait 620
2015-10-13 16:47:41,056 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 176, length = 2505011, total docs = 5227, last doc in bulk = 'http://blog.poweredby4.org/challenge/2012/01/']
2015-10-13 16:47:41,550 INFO elastic.ElasticIndexWriter - Previous took in ms 527, including wait 494
2015-10-13 16:47:41,797 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 225, length = 2554774, total docs = 5452, last doc in bulk = 'http://bloggingscripturehisway.blogspot.com/2012_04_01_archive.html']
2015-10-13 16:47:42,308 INFO elastic.ElasticIndexWriter - Previous took in ms 702, including wait 510
2015-10-13 16:47:42,400 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 24, length = 2632288, total docs = 5476, last doc in bulk = 'http://bloggingscripturehisway.blogspot.com/2014_02_01_archive.html']
2015-10-13 16:47:42,881 INFO elastic.ElasticIndexWriter - Previous took in ms 463, including wait 480
2015-10-13 16:47:43,012 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 31, length = 2522883, total docs = 5507, last doc in bulk = 'http://blogotional.blogspot.com/2005_03_06_archive.html']
2015-10-13 16:47:43,778 INFO elastic.ElasticIndexWriter - Previous took in ms 827, including wait 766
2015-10-13 16:47:43,861 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 29, length = 2570359, total docs = 5536, last doc in bulk = 'http://blogotional.blogspot.com/2005_09_25_archive.html']
2015-10-13 16:47:44,418 INFO elastic.ElasticIndexWriter - Previous took in ms 539, including wait 557
2015-10-13 16:47:44,512 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 29, length = 2502887, total docs = 5565, last doc in bulk = 'http://blogotional.blogspot.com/2006_04_16_archive.html']
2015-10-13 16:47:45,348 INFO elastic.ElasticIndexWriter - Previous took in ms 855, including wait 836
2015-10-13 16:47:45,525 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 158, length = 2515000, total docs = 5723, last doc in bulk = 'http://brazilcarroll.org/page/2/']
2015-10-13 16:47:45,969 INFO elastic.ElasticIndexWriter - Previous took in ms 552, including wait 444
2015-10-13 16:47:46,211 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 1141609, total docs = 5973, last doc in bulk = 'http://calvarybaptistwarren.com/page/trivia']
2015-10-13 16:47:47,058 INFO elastic.ElasticIndexWriter - Previous took in ms 1039, including wait 847
2015-10-13 16:47:47,288 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 1308537, total docs = 6223, last doc in bulk = 'http://catalystcommunitychurch.org/people/seth-barber/']
2015-10-13 16:47:47,398 INFO elastic.ElasticIndexWriter - Previous took in ms 302, including wait 110
2015-10-13 16:47:47,579 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 122, length = 2500702, total docs = 6345, last doc in bulk = 'http://catholic-convert.com/resources/recommended/software/']
2015-10-13 16:47:48,057 INFO elastic.ElasticIndexWriter - Previous took in ms 611, including wait 478
2015-10-13 16:47:48,535 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 1727339, total docs = 6595, last doc in bulk = 'http://ccpville.com/2015/05/announcements-for-may-17-2015/']
2015-10-13 16:47:48,562 INFO elastic.ElasticIndexWriter - Previous took in ms 406, including wait 27
2015-10-13 16:47:48,878 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 1697119, total docs = 6845, last doc in bulk = 'http://cftministry.org/resources/bookmarks.html']
2015-10-13 16:47:49,356 INFO elastic.ElasticIndexWriter - Previous took in ms 747, including wait 478
2015-10-13 16:47:49,508 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 2183224, total docs = 7095, last doc in bulk = 'http://chicagoavenuechurchofchrist.org/new-years-resolution-christians/']
2015-10-13 16:47:49,771 INFO elastic.ElasticIndexWriter - Previous took in ms 315, including wait 263
2015-10-13 16:47:49,975 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 1717642, total docs = 7345, last doc in bulk = 'http://christevangelicalchurchmobmin.org/page/ministry_to_and_through_animals']
2015-10-13 16:47:50,455 INFO elastic.ElasticIndexWriter - Previous took in ms 599, including wait 480
2015-10-13 16:47:50,650 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 1600259, total docs = 7595, last doc in bulk = 'http://christianobserver.org/wp-includes/wlwmanifest.xml']
2015-10-13 16:47:51,021 INFO elastic.ElasticIndexWriter - Previous took in ms 507, including wait 371
2015-10-13 16:47:51,191 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 144, length = 2548642, total docs = 7739, last doc in bulk = 'http://christlifedailybible.blogspot.com.au/2012_07_01_archive.html']
2015-10-13 16:47:51,525 INFO elastic.ElasticIndexWriter - Previous took in ms 460, including wait 334
2015-10-13 16:47:51,661 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 156, length = 2501340, total docs = 7895, last doc in bulk = 'http://chuckanderson.blogspot.com/2013_01_01_archive.html']
2015-10-13 16:47:52,373 INFO elastic.ElasticIndexWriter - Previous took in ms 728, including wait 712
2015-10-13 16:47:52,645 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 205, length = 3001522, total docs = 8100, last doc in bulk = 'http://classicalchristianity.com/category/bysaint/blessed-augustine-ca-354-430/']
2015-10-13 16:47:53,310 INFO elastic.ElasticIndexWriter - Previous took in ms 881, including wait 664
2015-10-13 16:47:53,392 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 15, length = 2679457, total docs = 8115, last doc in bulk = 'http://classicalchristianity.com/category/bysaint/st-basil-of-caesarea-ca-330-379-%e3%80%80/']
2015-10-13 16:47:54,071 INFO elastic.ElasticIndexWriter - Previous took in ms 639, including wait 678
2015-10-13 16:47:54,157 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 20, length = 2506012, total docs = 8135, last doc in bulk = 'http://classicalchristianity.com/category/canon-law/']
2015-10-13 16:47:54,970 INFO elastic.ElasticIndexWriter - Previous took in ms 750, including wait 813
2015-10-13 16:47:55,067 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 13, length = 2964279, total docs = 8148, last doc in bulk = 'http://classicalchristianity.com/category/holyfathers/christology/']
2015-10-13 16:47:55,511 INFO elastic.ElasticIndexWriter - Previous took in ms 433, including wait 443
2015-10-13 16:47:55,598 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 17, length = 2755989, total docs = 8165, last doc in bulk = 'http://classicalchristianity.com/category/sacrament/']
2015-10-13 16:47:56,298 INFO elastic.ElasticIndexWriter - Previous took in ms 722, including wait 699
2015-10-13 16:47:56,487 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 107, length = 2598111, total docs = 8272, last doc in bulk = 'http://coffeehousebible.blogspot.com/2012_03_01_archive.html']
2015-10-13 16:47:56,781 INFO elastic.ElasticIndexWriter - Previous took in ms 384, including wait 294
2015-10-13 16:47:56,876 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 46, length = 2540888, total docs = 8318, last doc in bulk = 'http://coffeehousebible.blogspot.com/2015_04_01_archive.html']
2015-10-13 16:47:57,608 INFO elastic.ElasticIndexWriter - Previous took in ms 766, including wait 732
2015-10-13 16:47:57,894 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 221, length = 3106091, total docs = 8539, last doc in bulk = 'http://commfell.org/11009/ministry/ministry_id/301289/Men']
2015-10-13 16:47:58,091 INFO elastic.ElasticIndexWriter - Previous took in ms 359, including wait 197
2015-10-13 16:47:58,384 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 2241764, total docs = 8789, last doc in bulk = 'http://cornerstoneefree.org/mcintosh/']
2015-10-13 16:47:59,126 INFO elastic.ElasticIndexWriter - Previous took in ms 960, including wait 742
2015-10-13 16:47:59,374 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 2117950, total docs = 9039, last doc in bulk = 'http://crazycathie.ca/tag/quotes/']
2015-10-13 16:47:59,465 INFO elastic.ElasticIndexWriter - Previous took in ms 308, including wait 91
2015-10-13 16:47:59,760 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 2254727, total docs = 9289, last doc in bulk = 'http://cside.org/pastor-bryan-neal.aspx']
2015-10-13 16:47:59,970 INFO elastic.ElasticIndexWriter - Previous took in ms 458, including wait 209
2015-10-13 16:48:00,180 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 1911318, total docs = 9539, last doc in bulk = 'http://dailylightdevotional.org/01/0110.html']
2015-10-13 16:48:00,591 INFO elastic.ElasticIndexWriter - Previous took in ms 508, including wait 410
2015-10-13 16:48:00,746 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 1959251, total docs = 9789, last doc in bulk = 'http://davidmatthew.org.uk/wotwintro.html']
2015-10-13 16:48:01,061 INFO elastic.ElasticIndexWriter - Previous took in ms 433, including wait 315
2015-10-13 16:48:01,276 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 1000372, total docs = 10039, last doc in bulk = 'http://derekgriz.com/tag/student-ministry/']
2015-10-13 16:48:01,564 INFO elastic.ElasticIndexWriter - Previous took in ms 424, including wait 287
2015-10-13 16:48:01,795 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 217, length = 2526991, total docs = 10256, last doc in bulk = 'http://diatheke.blogspot.com/2013_07_01_archive.html']
2015-10-13 16:48:01,883 INFO elastic.ElasticIndexWriter - Previous took in ms 255, including wait 88
2015-10-13 16:48:01,990 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 47, length = 2545740, total docs = 10303, last doc in bulk = 'http://dictionaryofdoctrine.com/House-of-Cards.html']
2015-10-13 16:48:02,696 INFO elastic.ElasticIndexWriter - Previous took in ms 679, including wait 706
2015-10-13 16:48:02,894 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 180, length = 2512128, total docs = 10483, last doc in bulk = 'http://distinctivediscipleship.com/category/daily-distinctives/']
2015-10-13 16:48:04,086 INFO elastic.ElasticIndexWriter - Previous took in ms 1271, including wait 1192
2015-10-13 16:48:04,222 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 58, length = 2501110, total docs = 10541, last doc in bulk = 'http://doctrine.org/jesus-vs-paul/']
2015-10-13 16:48:05,479 INFO elastic.ElasticIndexWriter - Previous took in ms 1248, including wait 1257
2015-10-13 16:48:05,565 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 42, length = 2520935, total docs = 10583, last doc in bulk = 'http://doctrine.org/understanding-the-book-of-revelation/']
2015-10-13 16:48:06,175 INFO elastic.ElasticIndexWriter - Previous took in ms 593, including wait 610
2015-10-13 16:48:06,406 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 170, length = 2504080, total docs = 10753, last doc in bulk = 'http://doulogos.blogspot.com/2005/09/interview-what-happened.html']
2015-10-13 16:48:06,854 INFO elastic.ElasticIndexWriter - Previous took in ms 552, including wait 448
2015-10-13 16:48:06,939 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 46, length = 2528772, total docs = 10799, last doc in bulk = 'http://doulogos.blogspot.com/2008_08_01_archive.html']
2015-10-13 16:48:07,511 INFO elastic.ElasticIndexWriter - Previous took in ms 603, including wait 572
2015-10-13 16:48:07,699 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 141, length = 2505336, total docs = 10940, last doc in bulk = 'http://drmsh.com/category/archaeology/']
2015-10-13 16:48:08,146 INFO elastic.ElasticIndexWriter - Previous took in ms 567, including wait 447
2015-10-13 16:48:08,312 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 194, length = 2557695, total docs = 11134, last doc in bulk = 'http://eaandfaith.blogspot.ca/2009_09_01_archive.html']
2015-10-13 16:48:08,758 INFO elastic.ElasticIndexWriter - Previous took in ms 551, including wait 446
2015-10-13 16:48:08,866 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 78, length = 2573722, total docs = 11212, last doc in bulk = 'http://eaandfaith.blogspot.co.uk/2009_09_01_archive.html']
2015-10-13 16:48:09,361 INFO elastic.ElasticIndexWriter - Previous took in ms 534, including wait 495
2015-10-13 16:48:09,460 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 78, length = 2574267, total docs = 11290, last doc in bulk = 'http://eaandfaith.blogspot.com.au/2009_09_01_archive.html']
2015-10-13 16:48:09,938 INFO elastic.ElasticIndexWriter - Previous took in ms 509, including wait 477
2015-10-13 16:48:10,042 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 76, length = 2502863, total docs = 11366, last doc in bulk = 'http://eaandfaith.blogspot.com/2009_10_01_archive.html']
2015-10-13 16:48:10,343 INFO elastic.ElasticIndexWriter - Previous took in ms 328, including wait 300
2015-10-13 16:48:10,567 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 204, length = 2548890, total docs = 11570, last doc in bulk = 'http://echoofrestorationtruths.blogspot.com/2013/10/the-language-of-beasts-of-revelation.html']
2015-10-13 16:48:10,894 INFO elastic.ElasticIndexWriter - Previous took in ms 447, including wait 327
2015-10-13 16:48:11,124 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 250, length = 2158970, total docs = 11820, last doc in bulk = 'http://elbaptist.org/about/our-beliefs/civil-government']
2015-10-13 16:48:14,103 INFO elastic.ElasticIndexWriter - Previous took in ms 610, including wait 2979
2015-10-13 16:48:14,321 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 239, length = 2543200, total docs = 12059, last doc in bulk = 'http://encountering-ahnsahnghong.blogspot.com/2012_02_01_archive.html']
2015-10-13 16:48:14,530 INFO elastic.ElasticIndexWriter - Previous took in ms 382, including wait 208
2015-10-13 16:48:14,682 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 145, length = 2577069, total docs = 12204, last doc in bulk = 'http://endtimepilgrim.org/puritans12.htm']
2015-10-13 16:48:15,212 INFO elastic.ElasticIndexWriter - Previous took in ms 628, including wait 530
2015-10-13 16:48:15,356 INFO elastic.ElasticIndexWriter - Processing bulk request [docs = 208, length = 2502224, total docs = 12412, last doc in bulk = 'http://english.genesis6.org/unmasking-sda-ellen-g-whites-satanic-hold-on-the-on-youtube/']
2015-10-13 16:48:23,467 INFO client.transport - [Grandmaster] failed to get node info for [#transport#-1][ci-dev-web06.lrscorp.net][inet[ci-dev-search04/10.70.15.17:9300]], disconnecting...
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[ci-dev-search04/10.70.15.17:9300]][cluster:monitor/nodes/info] request_id [101] timed out after [5002ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2015-10-13 16:48:33,475 INFO client.transport - [Grandmaster] failed to get node info for [#transport#-1][ci-dev-web06.lrscorp.net][inet[ci-dev-search04/10.70.15.17:9300]], disconnecting...
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[ci-dev-search04/10.70.15.17:9300]][cluster:monitor/nodes/info] request_id [102] timed out after [5000ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2015-10-13 16:48:37,246 INFO elastic.ElasticIndexWriter - Previous took in ms 21903, including wait 21890
2015-10-13 16:48:37,247 INFO elastic.ElasticIndexWriter - Processing remaining requests [docs = 208, length = 2502224, total docs = 12412]
2015-10-13 16:48:37,255 WARN mapred.LocalJobRunner - job_local1050818242_0001
org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: []
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:278)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:197)
at org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:106)
at org.elasticsearch.client.support.AbstractClient.bulk(AbstractClient.java:163)
at org.elasticsearch.client.transport.TransportClient.bulk(TransportClient.java:364)
at org.elasticsearch.action.bulk.BulkRequestBuilder.doExecute(BulkRequestBuilder.java:164)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65)
at org.apache.nutch.indexwriter.elastic.ElasticIndexWriter.commit(ElasticIndexWriter.java:211)
at org.apache.nutch.indexwriter.elastic.ElasticIndexWriter.write(ElasticIndexWriter.java:161)
at org.apache.nutch.indexer.IndexWriters.write(IndexWriters.java:85)
at org.apache.nutch.indexer.IndexerOutputFormat$1.write(IndexerOutputFormat.java:50)
at org.apache.nutch.indexer.IndexerOutputFormat$1.write(IndexerOutputFormat.java:41)
at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.write(ReduceTask.java:458)
at org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:500)
at org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:337)
at org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:53)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:522)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:398)
2015-10-13 16:48:37,542 ERROR indexer.IndexingJob - Indexer: java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1357)
at org.apache.nutch.indexer.IndexingJob.index(IndexingJob.java:113)
at org.apache.nutch.indexer.IndexingJob.run(IndexingJob.java:177)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.indexer.IndexingJob.main(IndexingJob.java:187)
Here's my elastic search log for the same timeframe:
[2015-10-13 01:14:07,445][INFO ][cluster.service ] [ci-dev-search04] removed {[La Lunatica][e0LVxHLbSKGYNhEavjIJXw][ws1938][inet[ws1890.lrscorp.net/10.200.208.38:9300]]{data=false, client=true},}, reason: zen-disco-node_failed([La Lunatica][e0LVxHLbSKGYNhEavjIJXw][ws1938][inet[ws1890.lrscorp.net/10.200.208.38:9300]]{data=false, client=true}), reason transport disconnected
[2015-10-13 16:18:09,749][WARN ][monitor.jvm ] [ci-dev-search04] [gc][young][54349][4966] duration [23.3s], collections [4]/[24.1s], total [23.3s]/[50.8s], memory [205.2mb]->[177.9mb]/[989.8mb], all_pools {[young] [56.9mb]->[112.6kb]/[273mb]}{[survivor] [8.5mb]->[8.5mb]/[34.1mb]}{[old] [139.7mb]->[169.3mb]/[682.6mb]}
[2015-10-13 16:23:09,914][WARN ][monitor.jvm ] [ci-dev-search04] [gc][young][54629][5066] duration [20.1s], collections [1]/[20.4s], total [20.1s]/[1.2m], memory [171.5mb]->[153.4mb]/[989.8mb], all_pools {[young] [25.2mb]->[1.7mb]/[273mb]}{[survivor] [4.1mb]->[8.5mb]/[34.1mb]}{[old] [142.1mb]->[143.5mb]/[682.6mb]}
[2015-10-13 16:47:13,468][WARN ][monitor.jvm ] [ci-dev-search04] [gc][young][56067][5203] duration [5s], collections [1]/[5.1s], total [5s]/[1.3m], memory [248.1mb]->[191.2mb]/[989.8mb], all_pools {[young] [66mb]->[894.5kb]/[273mb]}{[survivor] [8.5mb]->[8.5mb]/[34.1mb]}{[old] [173.6mb]->[181.9mb]/[682.6mb]}
[2015-10-13 16:47:23,360][WARN ][monitor.jvm ] [ci-dev-search04] [gc][young][56071][5213] duration [6.3s], collections [2]/[6.8s], total [6.3s]/[1.4m], memory [267.5mb]->[279.2mb]/[989.8mb], all_pools {[young] [196.7kb]->[165.7kb]/[273mb]}{[survivor] [8.5mb]->[7.8mb]/[34.1mb]}{[old] [258.9mb]->[271.2mb]/[682.6mb]}
[2015-10-13 16:48:13,461][WARN ][monitor.jvm ] [ci-dev-search04] [gc][young][56119][5353] duration [2.4s], collections [1]/[2.7s], total [2.4s]/[1.5m], memory [296.7mb]->[257.5mb]/[989.8mb], all_pools {[young] [48.2mb]->[785.7kb]/[273mb]}{[survivor] [8.5mb]->[8.5mb]/[34.1mb]}{[old] [239.9mb]->[248.4mb]/[682.6mb]}
[2015-10-13 16:48:36,621][WARN ][monitor.jvm ] [ci-dev-search04] [gc][young][56122][5360] duration [21s], collections [1]/[21.1s], total [21s]/[1.9m], memory [334.2mb]->[311.6mb]/[989.8mb], all_pools {[young] [29.1mb]->[991.5kb]/[273mb]}{[survivor] [3.1mb]->[8.5mb]/[34.1mb]}{[old] [302mb]->[302.3mb]/[682.6mb]}
17 de octubre: Final Cubana 2015 del Concurso de Programación ACM-ICPC.
http://coj.uci.cu/contest/contestview.xhtml?cid=1407