You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@manifoldcf.apache.org by Bisonti Mario <Ma...@vimar.com> on 2018/11/06 10:43:00 UTC
R: Job stuck without message
Hallo Karl.
When it hangs I see in the Queue status:
[cid:image001.png@01D475C5.DCE8D300]
And in the Document Status:
[cid:image002.png@01D475C5.DCE8D300]
Da: Karl Wright <da...@gmail.com>
Inviato: martedì 30 ottobre 2018 19:32
A: user@manifoldcf.apache.org
Oggetto: Re: Job stuck without message
What I am interested in now is the Document Status report for any one of the documents that is 'stuck'. The next crawl time value is the critical field. Can you include an example?
Karl
On Tue, Oct 30, 2018, 12:36 PM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot, Karl.
It happens that the job starts, it works and index for an hour and after it frezzes, I haven’t error or waiting status in Document Queue o Simple History, I have only “OK” status so, I haven’t failures.
I am not able to see other log errors other from the manifoldcf.log
Solr server is ok
Tika server is ok
Agent is ok
Tomcat with ManifoldCF is ok
I could search if I could to put in info log mode for example Tika servrer or Solr.
Thanks..
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 16:38
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
Hi Mario,
Please look at the Queue Status report to determine what is waiting and why it is waiting.
You can also look at the Simple History to see what has been happening. If you are getting 100% failures in fetching documents then you may need to address this because your infrastructure is unhappy. If the failure is something that indicates that the document is never going to be readable, that's a different problem and we might need to address that in the connector.
Karl
On Tue, Oct 30, 2018 at 10:33 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot Karl
Yes, I see many docs in the docs queue but they are inactive.
Infact i see that no more docs are indexed in Solr and I see that job is with the same number of docs Active (35012)
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 13:59
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
The reason the job is "stuck" is because:
' JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.'
This means that ManifoldCF will retry this document for a while before it gives up on it. It appears to be stuck but it is not. You can verify that by looking at the Document Queue report to see what is queued and what times the various documents will be retried.
Karl
On Tue, Oct 30, 2018 at 5:07 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo.
I started a job that works for some minutes, and after it stucks.
In the manifoldcf.log I see:
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-10-30T09:21:31,440 (Worker thread '2') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:33,502 (Worker thread '14') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:37,725 (Worker thread '30') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:44,406 (Worker thread '49') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:47,310 (Worker thread '15') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:52,000 (Worker thread '27') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:53,526 (Worker thread '15') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:04,511 (Worker thread '3') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2438) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1221) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-10-30T09:22:10,359 (Worker thread '27') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:13,932 (Worker thread '12') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:14,274 (Worker thread '23') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:19,933 (Worker thread '8') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:23:59,920 (Worker thread '39') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:24:09,059 (Worker thread '43') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
What could I check?
Tika server works as standalone.
Could you help me?
Thanks a lot
Mario
Re: Job stuck without message
Posted by Karl Wright <da...@gmail.com>.
I am sorry, it is not clear what you are doing.
184 worker threads is probably *way* more than you want, especially since
you have already discovered you cannot really have more than a few
connections to your SMB servers active at the same time. Plus, you run the
risk of running out of memory that way.
Karl
On Thu, Nov 29, 2018 at 12:23 PM Bisonti Mario <Ma...@vimar.com>
wrote:
> Thanks, Karl.
>
>
>
> But, the worker threads are set by these parameters?
>
>
>
> <property name="org.apache.manifoldcf.database.maxhandles" value="250"/>
>
> <property name="org.apache.manifoldcf.crawler.threads" value="184"/>
>
>
>
> I set that valuie into my properties.xml.
>
>
>
> But if now, I execute:
>
> pstree 1369
>
> java───686*[{java}]
>
>
>
> so, 686 process child of the agent.
>
>
>
> Is there any relation about these values 686 and 184 ?
>
>
>
> Thanks.
>
> Mario
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* giovedì 29 novembre 2018 17:43
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> Hi Mario,
>
>
>
> I agree with your assessment that this could have been due to an out of
> memory condition.
>
> When ManifoldCF detects such a situation, it tries to shut itself down.
> If there are further problems during that time, it may wind up not being
> able to shut down completely. Your thread dump shows that the Stuffer
> Thread, which is what queues documents for processing, is gone, but the
> zookeeper threads related to it are still there. So the entire process was
> hosed.
>
> If you want to control the amount of memory you use, pay some attention to
> keeping the number of worker threads limited. Each worker thread's
> consumption of memory is bounded, but if you have a lot of them, then the
> odds of a bad situation developing memory-wise go up.
>
>
>
> Karl
>
>
>
>
>
> On Thu, Nov 29, 2018 at 6:45 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> I think that the problemi s an out of memory error of the agent.
>
>
>
> sudo -u tomcat jcmd 1371 VM.info | more
>
>
>
> I obtain:
> 1371:
>
> #
>
> # JRE version: OpenJDK Runtime Environment (10.0.2+13) (build
> 10.0.2+13-Ubuntu-1ubuntu0.18.04.3)
>
> # Java VM: OpenJDK 64-Bit Server VM (10.0.2+13-Ubuntu-1ubuntu0.18.04.3,
> mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
>
>
>
> --------------- S U M M A R Y ------------
>
>
>
> Command Line: -Xms2048m -Xmx2048m
> -Dorg.apache.manifoldcf.configfile=./properties.xml
> -Dorg.apache.manifoldcf.processid=A org.apache.manifoldcf.agents.AgentRun
>
>
>
> Host: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz, 8 cores, 70G, Ubuntu
> 18.04.1 LTS
>
> Time: Thu Nov 29 12:18:11 2018 CET elapsed time: 152407 seconds (1d 18h
> 20m 7s)
>
>
>
> --------------- P R O C E S S ---------------
>
>
>
> OutOfMemory and StackOverflow Exception counts:
>
> OutOfMemoryError java_heap_errors=1
>
>
>
> Heap address: 0x0000000080000000, size: 2048 MB, Compressed Oops mode:
> 32-bit
>
> Narrow klass base: 0x0000000000000000, Narrow klass shift: 3
>
> Compressed class space size: 1073741824 Address: 0x0000000100000000
>
>
>
> Heap:
>
> garbage-first heap total 2097152K, used 1143846K [0x0000000080000000,
> 0x0000000100000000)
>
> region size 1024K, 254 young (260096K), 2 survivors (2048K)
>
> Metaspace used 36821K, capacity 37327K, committed 38272K, reserved
> 1083392K
>
> class space used 3251K, capacity 3432K, committed 3712K, reserved
> 1048576K
>
> Heap Regions: E=young(eden), S=young(survivor), O=old,
> HS=humongous(starts), HC=humongous(continues), CS=collection set, F=free,
> A=archive, TS=gc time stamp, AC=allocation context,
>
> TAMS=top-at-mark-start (previous, next)
>
> | 0|0x0000000080000000, 0x0000000080100000, 0x0000000080100000|100%| O|
> |TS 0|AC 0|TAMS 0x0000000080000000, 0x0000000080000000|
>
> | 1|0x0000000080100000, 0x00000000801f6dd0, 0x0000000080200000| 96%| O|
> |TS 0|AC 0|TAMS 0x0
>
>
>
>
>
> So :
>
> OutOfMemory and StackOverflow Exception counts:
>
> OutOfMemoryError java_heap_errors=1
>
>
>
> Could be useful for me..
>
>
>
> I try to set :
> sudo -u tomcat nano options.env.unix
>
> -Xms32768m
>
> -Xmx32768m
>
>
>
> And restart the process
>
> And restart the job..
>
>
>
>
>
>
>
>
>
> *Da:* Bisonti Mario <Ma...@vimar.com>
> *Inviato:* giovedì 29 novembre 2018 12:05
> *A:* user@manifoldcf.apache.org
> *Oggetto:* R: Job stuck without message
>
>
>
> *Update
>
>
>
>
>
> I found that pid 1371 is the agent pid
>
> I used the command, to retrieve this:
> ps -Flww -p 1371
>
> F S UID PID PPID C PRI NI ADDR SZ WCHAN RSS PSR STIME
> TTY TIME CMD
>
> 0 S tomcat 1371 1282 11 80 0 - 2082955 - 2621680 7 Nov27
> ? 04:40:54 /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Xms2048m
> -Xmx2048m -Dorg.apache.manifoldcf.configfile=./properties.xml -cp
> .:../lib/mcf-core.jar:../lib/mcf-agents.jar:../lib/mcf-pull-agent.jar:../lib/hsqldb-2.3.2.jar:../lib/postgresql-42.1.3.jar:../lib/commons-codec-1.10.jar:../lib/commons-collections-3.2.1.jar:../lib/commons-collections4-4.1.jar:../lib/commons-discovery-0.5.jar:../lib/commons-el-1.0.jar:../lib/commons-exec-1.3.jar:../lib/commons-fileupload-1.3.3.jar:../lib/commons-io-2.5.jar:../lib/commons-lang-2.6.jar:../lib/commons-lang3-3.6.jar:../lib/commons-logging-1.2.jar:../lib/ecj-4.3.1.jar:../lib/gson-2.8.0.jar:../lib/guava-25.1-jre.jar:../lib/httpclient-4.5.6.jar:../lib/httpcore-4.4.10.jar:../lib/jasper-6.0.35.jar:../lib/jasper-el-6.0.35.jar:../lib/javax.servlet-api-3.1.0.jar:../lib/jna-4.3.0.jar:../lib/jna-platform-4.3.0.jar:../lib/json-simple-1.1.1.jar:../lib/jsp-api-2.1-glassfish-2.1.v20091210.jar:../lib/juli-6.0.35.jar:../lib/log4j-1.2-api-2.4.1.jar:../lib/log4j-api-2.4.1.jar:../lib/log4j-core-2.4.1.jar:../lib/mail-1.4.5.jar:../lib/serializer-2.7.1.jar:../lib/slf4j-api-1.7.25.jar:../lib/slf4j-simple-1.7.25.jar:../lib/velocity-1.7.jar:../lib/xalan-2.7.1.jar:../lib/xercesImpl-2.10.0.jar:../lib/xml-apis-1.4.01.jar:../lib/zookeeper-3.4.10.jar:../lib-proprietary/jtds-1.2.4.jar:../lib-proprietary/mariadb-java-client-1.1.7.jar:../lib-proprietary/mysql-connector-java-5.1.33.jar:
> -Dorg.apache.manifoldcf.processid=A org.apache.manifoldcf.agents.AgentRun
>
>
>
> So I am able to heap the pid 1371 !
>
> sudo -u tomcat jstack -l 1371 > /tmp/jstack_agent.log
>
>
>
>
>
> I attached to this mail
>
>
>
> Thanks a lot, Karl
>
>
>
>
>
>
>
>
>
>
>
>
>
> *Da:* Bisonti Mario <Ma...@vimar.com>
> *Inviato:* giovedì 29 novembre 2018 11:12
> *A:* user@manifoldcf.apache.org
> *Oggetto:* R: Job stuck without message
>
>
>
> Thanks a lot for your big help, Karl.
>
>
>
> I would like to give you the dump but in my system (ubuntu 18.04) I am not
> able to dump the process of the start-agent.sh
>
>
>
> I tried but I always obtain:
>
> sudo -u tomcat jhsdb jinfo --pid 1233
>
> Attaching to process ID 1233, please wait...
>
> Error attaching to process: Doesn't appear to be a HotSpot VM (could not
> find symbol "gHotSpotVMTypes" in remote process)
>
> sun.jvm.hotspot.debugger.DebuggerException: Doesn't appear to be a HotSpot
> VM (could not find symbol "gHotSpotVMTypes" in remote process)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupVM(HotSpotAgent.java:413)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:306)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:141)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.tools.JInfo.runWithArgs(JInfo.java:139)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.tools.JInfo.main(JInfo.java:144)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJINFO(SALauncher.java:364)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:460)
>
>
>
>
>
>
>
> sudo -u tomcat jstack -l 1233
>
> 1233: Unable to open socket file /proc/1233/cwd/.attach_pid1233: target
> process 1233 doesn't respond within 10500ms or HotSpot VM not loaded
>
>
>
>
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* mercoledì 28 novembre 2018 21:52
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> The database row indicates there is no reason that the document should not
> be queued and processed.
>
> As for getting a thread dump, there's a "force" option (-F).
>
> The only other reason stuff may not run is if the query plan for
> identifying documents to process has gone horribly wrong. We should see
> that in the thread dump however.
>
>
>
> I will unfortunately need to be offline for the next 24 hours due to an
> emergency situation, but if it turns out that your agents process is busy
> executing a long-running query, then I suggest analyzing the jobqueue table
> to get a better plan. This happens automatically but there are conditions
> under which it doesn't happen frequently enough. If the job is waiting for
> locks, then the stack trace will tell me where.
>
>
>
> Thanks,
>
> Karl
>
> Karl
>
>
>
>
>
> On Wed, Nov 28, 2018 at 11:07 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> I attatched a row that correspond to a row of one of these documents in
> this mail
>
>
>
>
>
>
>
> I obtain the pid of:
> "/bin/bash -e
> /opt/manifoldcf/multiprocess-zk-example-proprietary/start-agents.sh"
>
> The pid is 1233
>
>
>
> I tried to use
>
> sudo jstack -l 1233 > /tmp/jstack_start_agent.log
>
>
>
> but I obtain:
>
> 1233: Unable to open socket file /proc/1233/cwd/.attach_pid1233: target
> process 1233 doesn't respond within 10500ms or HotSpot VM not loaded
>
>
>
> Perhaps isn’t it the right way to obtain a thread dump?
>
> Excuse me but I am not a Linux expert..
>
>
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* mercoledì 28 novembre 2018 16:36
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> Another thing you could do is get a thread dump of the agents process.
>
>
>
> Karl
>
>
>
>
>
> On Wed, Nov 28, 2018 at 10:35 AM Karl Wright <da...@gmail.com> wrote:
>
> Can you look into the database jobqueue table and provide a row that
> corresponds to one of these documents?
>
>
>
> Thanks,
>
> Karl
>
>
>
>
>
> On Wed, Nov 28, 2018 at 10:26 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo.
>
> Repository has Max connection=10
>
>
>
> In the Document Status report” I see many item with :
>
> State=“Not yet processed”
>
> Status=”Ready for processing”
>
> Scheduled=01-01-1970 01:00:00.000”
>
> Scheduled Action=”Process”
>
>
>
>
>
>
>
>
>
> But the job no more walk..
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* mercoledì 28 novembre 2018 16:03
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> "Pipe instances are busy" occurs because you are overloading the SMB
> access to your servers. How many connections do you have allocated for
> your repository connection? You probably want to limit this to 2-3 if you
> see this error a lot, and it appears you do.
>
> " Tika Server: Tika Server rejects: Tika Server rejected document with
> the following reason: Unprocessable Entity" means the document is not
> properly formed XML. The rejection will mean the document isn't indexed,
> but this will not stop the job.
>
> If nothing is happening and you don't know why, I'd suggest looking at the
> Document Status report to figure out what documents are not being processed
> and why. It is quite possible they are all in the process of being retried
> because of the "Pipe instances" issue above.
>
>
>
> Karl
>
>
>
> On Wed, Nov 28, 2018 at 9:46 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo Karl.
>
> I take this ticket because,now, after I use zookeeper, my job works for 7
> hours and now it is in hang status.
>
> I see running but it seems hanging, no log from 1 hour
>
>
>
> This is the last manifoldcf.log lines:
>
>
>
>
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:21,524 (Worker thread '59') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:25,232 (Worker thread '45') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-11-28T14:46:25,277 (Worker thread '70') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-11-28T14:46:28,825 (Worker thread '27') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:40,473 (Worker thread '79') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:54,855 (Worker thread '65') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:55,175 (Worker thread '83') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
>
>
>
>
>
>
> I don’t know what to check.
>
>
>
> Tika server is ok, and it doesn’t more restart it .
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 6 novembre 2018 15:27
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> I added a couple of questions to the ticket. Please reply.
>
>
>
> Thanks,
>
> Karl
>
>
>
>
>
> On Tue, Nov 6, 2018 at 8:56 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Thanks a lot, Karl.
>
> I created a ticket.
>
> https://issues.apache.org/jira/browse/CONNECTORS-1554
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCONNECTORS-1554&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ce3888a90f18046e77f3308d65619bcc1%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=cIlgBNF2dTzyc4TA4LMiCUVyOMkvh3t8cjkltxAjAaE%3D&reserved=0>
>
>
>
>
>
> Thanks
>
>
>
> Mario
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 6 novembre 2018 14:28
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> ok, can you create a ticket? Also, I'd appreciate it if you can look at
> the simple history for one of these documents; I need to see what happened
> to it last.
>
>
>
> Thanks,
>
> Karl
>
>
>
>
>
> On Tue, Nov 6, 2018 at 7:32 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> My version is 2.11
>
>
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 6 novembre 2018 13:07
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> Thanks.
>
> What version of ManifoldCF are you using? We fixed a problem a while back
> having to do with documents that (because of error processing) get put into
> a "ready for processing" state which don't have any document priority set.
> But this should have been addressed, certainly, by the most recent release
> and probably by 2.10 as well.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Nov 6, 2018 at 5:43 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo Karl.
>
> When it hangs I see in the Queue status:
>
>
>
> And in the Document Status:
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 19:32
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> What I am interested in now is the Document Status report for any one of
> the documents that is 'stuck'. The next crawl time value is the critical
> field. Can you include an example?
>
>
>
> Karl
>
>
>
> On Tue, Oct 30, 2018, 12:36 PM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Thanks a lot, Karl.
>
>
>
> It happens that the job starts, it works and index for an hour and after
> it frezzes, I haven’t error or waiting status in Document Queue o Simple
> History, I have only “OK” status so, I haven’t failures.
>
>
>
> I am not able to see other log errors other from the manifoldcf.log
>
>
>
> Solr server is ok
>
> Tika server is ok
>
> Agent is ok
>
> Tomcat with ManifoldCF is ok
>
>
>
> I could search if I could to put in info log mode for example Tika servrer
> or Solr.
>
>
>
> Thanks..
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 16:38
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> Hi Mario,
>
> Please look at the Queue Status report to determine what is waiting and
> why it is waiting.
> You can also look at the Simple History to see what has been happening.
> If you are getting 100% failures in fetching documents then you may need to
> address this because your infrastructure is unhappy. If the failure is
> something that indicates that the document is never going to be readable,
> that's a different problem and we might need to address that in the
> connector.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Oct 30, 2018 at 10:33 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
>
>
> Thanks a lot Karl
>
>
>
> Yes, I see many docs in the docs queue but they are inactive.
>
>
>
> Infact i see that no more docs are indexed in Solr and I see that job is
> with the same number of docs Active (35012)
>
>
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 13:59
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> The reason the job is "stuck" is because:
>
> ' JCIFS: Possibly transient exception detected on attempt 1 while getting
> share security: All pipe instances are busy.'
>
> This means that ManifoldCF will retry this document for a while before it
> gives up on it. It appears to be stuck but it is not. You can verify that
> by looking at the Document Queue report to see what is queued and what
> times the various documents will be retried.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Oct 30, 2018 at 5:07 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo.
>
>
>
> I started a job that works for some minutes, and after it stucks.
>
>
>
> In the manifoldcf.log I see:
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-10-30T09:21:31,440 (Worker thread '2') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:33,502 (Worker thread '14') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:37,725 (Worker thread '30') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:44,406 (Worker thread '49') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:47,310 (Worker thread '15') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:52,000 (Worker thread '27') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:53,526 (Worker thread '15') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:04,511 (Worker thread '3') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2438)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1221)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-10-30T09:22:10,359 (Worker thread '27') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:13,932 (Worker thread '12') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:14,274 (Worker thread '23') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:19,933 (Worker thread '8') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:23:59,920 (Worker thread '39') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:24:09,059 (Worker thread '43') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
>
>
>
>
>
>
> What could I check?
>
>
>
> Tika server works as standalone.
>
>
>
> Could you help me?
>
>
>
> Thanks a lot
>
>
>
> Mario
>
>
>
>
R: Job stuck without message
Posted by Bisonti Mario <Ma...@vimar.com>.
Thanks, Karl.
But, the worker threads are set by these parameters?
<property name="org.apache.manifoldcf.database.maxhandles" value="250"/>
<property name="org.apache.manifoldcf.crawler.threads" value="184"/>
I set that valuie into my properties.xml.
But if now, I execute:
pstree 1369
java───686*[{java}]
so, 686 process child of the agent.
Is there any relation about these values 686 and 184 ?
Thanks.
Mario
Da: Karl Wright <da...@gmail.com>
Inviato: giovedì 29 novembre 2018 17:43
A: user@manifoldcf.apache.org
Oggetto: Re: Job stuck without message
Hi Mario,
I agree with your assessment that this could have been due to an out of memory condition.
When ManifoldCF detects such a situation, it tries to shut itself down. If there are further problems during that time, it may wind up not being able to shut down completely. Your thread dump shows that the Stuffer Thread, which is what queues documents for processing, is gone, but the zookeeper threads related to it are still there. So the entire process was hosed.
If you want to control the amount of memory you use, pay some attention to keeping the number of worker threads limited. Each worker thread's consumption of memory is bounded, but if you have a lot of them, then the odds of a bad situation developing memory-wise go up.
Karl
On Thu, Nov 29, 2018 at 6:45 AM Bisonti Mario <Ma...@vimar.com>> wrote:
I think that the problemi s an out of memory error of the agent.
sudo -u tomcat jcmd 1371 VM.info | more
I obtain:
1371:
#
# JRE version: OpenJDK Runtime Environment (10.0.2+13) (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.3)
# Java VM: OpenJDK 64-Bit Server VM (10.0.2+13-Ubuntu-1ubuntu0.18.04.3, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
--------------- S U M M A R Y ------------
Command Line: -Xms2048m -Xmx2048m -Dorg.apache.manifoldcf.configfile=./properties.xml -Dorg.apache.manifoldcf.processid=A org.apache.manifoldcf.agents.AgentRun
Host: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz, 8 cores, 70G, Ubuntu 18.04.1 LTS
Time: Thu Nov 29 12:18:11 2018 CET elapsed time: 152407 seconds (1d 18h 20m 7s)
--------------- P R O C E S S ---------------
OutOfMemory and StackOverflow Exception counts:
OutOfMemoryError java_heap_errors=1
Heap address: 0x0000000080000000, size: 2048 MB, Compressed Oops mode: 32-bit
Narrow klass base: 0x0000000000000000, Narrow klass shift: 3
Compressed class space size: 1073741824 Address: 0x0000000100000000
Heap:
garbage-first heap total 2097152K, used 1143846K [0x0000000080000000, 0x0000000100000000)
region size 1024K, 254 young (260096K), 2 survivors (2048K)
Metaspace used 36821K, capacity 37327K, committed 38272K, reserved 1083392K
class space used 3251K, capacity 3432K, committed 3712K, reserved 1048576K
Heap Regions: E=young(eden), S=young(survivor), O=old, HS=humongous(starts), HC=humongous(continues), CS=collection set, F=free, A=archive, TS=gc time stamp, AC=allocation context,
TAMS=top-at-mark-start (previous, next)
| 0|0x0000000080000000, 0x0000000080100000, 0x0000000080100000|100%| O| |TS 0|AC 0|TAMS 0x0000000080000000, 0x0000000080000000|
| 1|0x0000000080100000, 0x00000000801f6dd0, 0x0000000080200000| 96%| O| |TS 0|AC 0|TAMS 0x0
So :
OutOfMemory and StackOverflow Exception counts:
OutOfMemoryError java_heap_errors=1
Could be useful for me..
I try to set :
sudo -u tomcat nano options.env.unix
-Xms32768m
-Xmx32768m
And restart the process
And restart the job..
Da: Bisonti Mario <Ma...@vimar.com>>
Inviato: giovedì 29 novembre 2018 12:05
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: R: Job stuck without message
*Update
I found that pid 1371 is the agent pid
I used the command, to retrieve this:
ps -Flww -p 1371
F S UID PID PPID C PRI NI ADDR SZ WCHAN RSS PSR STIME TTY TIME CMD
0 S tomcat 1371 1282 11 80 0 - 2082955 - 2621680 7 Nov27 ? 04:40:54 /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Xms2048m -Xmx2048m -Dorg.apache.manifoldcf.configfile=./properties.xml -cp .:../lib/mcf-core.jar:../lib/mcf-agents.jar:../lib/mcf-pull-agent.jar:../lib/hsqldb-2.3.2.jar:../lib/postgresql-42.1.3.jar:../lib/commons-codec-1.10.jar:../lib/commons-collections-3.2.1.jar:../lib/commons-collections4-4.1.jar:../lib/commons-discovery-0.5.jar:../lib/commons-el-1.0.jar:../lib/commons-exec-1.3.jar:../lib/commons-fileupload-1.3.3.jar:../lib/commons-io-2.5.jar:../lib/commons-lang-2.6.jar:../lib/commons-lang3-3.6.jar:../lib/commons-logging-1.2.jar:../lib/ecj-4.3.1.jar:../lib/gson-2.8.0.jar:../lib/guava-25.1-jre.jar:../lib/httpclient-4.5.6.jar:../lib/httpcore-4.4.10.jar:../lib/jasper-6.0.35.jar:../lib/jasper-el-6.0.35.jar:../lib/javax.servlet-api-3.1.0.jar:../lib/jna-4.3.0.jar:../lib/jna-platform-4.3.0.jar:../lib/json-simple-1.1.1.jar:../lib/jsp-api-2.1-glassfish-2.1.v20091210.jar:../lib/juli-6.0.35.jar:../lib/log4j-1.2-api-2.4.1.jar:../lib/log4j-api-2.4.1.jar:../lib/log4j-core-2.4.1.jar:../lib/mail-1.4.5.jar:../lib/serializer-2.7.1.jar:../lib/slf4j-api-1.7.25.jar:../lib/slf4j-simple-1.7.25.jar:../lib/velocity-1.7.jar:../lib/xalan-2.7.1.jar:../lib/xercesImpl-2.10.0.jar:../lib/xml-apis-1.4.01.jar:../lib/zookeeper-3.4.10.jar:../lib-proprietary/jtds-1.2.4.jar:../lib-proprietary/mariadb-java-client-1.1.7.jar:../lib-proprietary/mysql-connector-java-5.1.33.jar: -Dorg.apache.manifoldcf.processid=A org.apache.manifoldcf.agents.AgentRun
So I am able to heap the pid 1371 !
sudo -u tomcat jstack -l 1371 > /tmp/jstack_agent.log
I attached to this mail
Thanks a lot, Karl
Da: Bisonti Mario <Ma...@vimar.com>>
Inviato: giovedì 29 novembre 2018 11:12
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: R: Job stuck without message
Thanks a lot for your big help, Karl.
I would like to give you the dump but in my system (ubuntu 18.04) I am not able to dump the process of the start-agent.sh
I tried but I always obtain:
sudo -u tomcat jhsdb jinfo --pid 1233
Attaching to process ID 1233, please wait...
Error attaching to process: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMTypes" in remote process)
sun.jvm.hotspot.debugger.DebuggerException: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMTypes" in remote process)
at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupVM(HotSpotAgent.java:413)
at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:306)
at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:141)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.JInfo.runWithArgs(JInfo.java:139)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.JInfo.main(JInfo.java:144)
at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJINFO(SALauncher.java:364)
at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:460)
sudo -u tomcat jstack -l 1233
1233: Unable to open socket file /proc/1233/cwd/.attach_pid1233: target process 1233 doesn't respond within 10500ms or HotSpot VM not loaded
Da: Karl Wright <da...@gmail.com>>
Inviato: mercoledì 28 novembre 2018 21:52
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
The database row indicates there is no reason that the document should not be queued and processed.
As for getting a thread dump, there's a "force" option (-F).
The only other reason stuff may not run is if the query plan for identifying documents to process has gone horribly wrong. We should see that in the thread dump however.
I will unfortunately need to be offline for the next 24 hours due to an emergency situation, but if it turns out that your agents process is busy executing a long-running query, then I suggest analyzing the jobqueue table to get a better plan. This happens automatically but there are conditions under which it doesn't happen frequently enough. If the job is waiting for locks, then the stack trace will tell me where.
Thanks,
Karl
Karl
On Wed, Nov 28, 2018 at 11:07 AM Bisonti Mario <Ma...@vimar.com>> wrote:
I attatched a row that correspond to a row of one of these documents in this mail
I obtain the pid of:
"/bin/bash -e /opt/manifoldcf/multiprocess-zk-example-proprietary/start-agents.sh"
The pid is 1233
I tried to use
sudo jstack -l 1233 > /tmp/jstack_start_agent.log
but I obtain:
1233: Unable to open socket file /proc/1233/cwd/.attach_pid1233: target process 1233 doesn't respond within 10500ms or HotSpot VM not loaded
Perhaps isn’t it the right way to obtain a thread dump?
Excuse me but I am not a Linux expert..
Da: Karl Wright <da...@gmail.com>>
Inviato: mercoledì 28 novembre 2018 16:36
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
Another thing you could do is get a thread dump of the agents process.
Karl
On Wed, Nov 28, 2018 at 10:35 AM Karl Wright <da...@gmail.com>> wrote:
Can you look into the database jobqueue table and provide a row that corresponds to one of these documents?
Thanks,
Karl
On Wed, Nov 28, 2018 at 10:26 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo.
Repository has Max connection=10
In the Document Status report” I see many item with :
State=“Not yet processed”
Status=”Ready for processing”
Scheduled=01-01-1970 01:00:00.000”
Scheduled Action=”Process”
But the job no more walk..
Da: Karl Wright <da...@gmail.com>>
Inviato: mercoledì 28 novembre 2018 16:03
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
"Pipe instances are busy" occurs because you are overloading the SMB access to your servers. How many connections do you have allocated for your repository connection? You probably want to limit this to 2-3 if you see this error a lot, and it appears you do.
" Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity" means the document is not properly formed XML. The rejection will mean the document isn't indexed, but this will not stop the job.
If nothing is happening and you don't know why, I'd suggest looking at the Document Status report to figure out what documents are not being processed and why. It is quite possible they are all in the process of being retried because of the "Pipe instances" issue above.
Karl
On Wed, Nov 28, 2018 at 9:46 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo Karl.
I take this ticket because,now, after I use zookeeper, my job works for 7 hours and now it is in hang status.
I see running but it seems hanging, no log from 1 hour
This is the last manifoldcf.log lines:
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:21,524 (Worker thread '59') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:25,232 (Worker thread '45') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-11-28T14:46:25,277 (Worker thread '70') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-11-28T14:46:28,825 (Worker thread '27') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:40,473 (Worker thread '79') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:54,855 (Worker thread '65') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:55,175 (Worker thread '83') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
I don’t know what to check.
Tika server is ok, and it doesn’t more restart it .
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 6 novembre 2018 15:27
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
I added a couple of questions to the ticket. Please reply.
Thanks,
Karl
On Tue, Nov 6, 2018 at 8:56 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot, Karl.
I created a ticket.
https://issues.apache.org/jira/browse/CONNECTORS-1554<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCONNECTORS-1554&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ce3888a90f18046e77f3308d65619bcc1%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=cIlgBNF2dTzyc4TA4LMiCUVyOMkvh3t8cjkltxAjAaE%3D&reserved=0>
Thanks
Mario
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 6 novembre 2018 14:28
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
ok, can you create a ticket? Also, I'd appreciate it if you can look at the simple history for one of these documents; I need to see what happened to it last.
Thanks,
Karl
On Tue, Nov 6, 2018 at 7:32 AM Bisonti Mario <Ma...@vimar.com>> wrote:
My version is 2.11
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 6 novembre 2018 13:07
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
Thanks.
What version of ManifoldCF are you using? We fixed a problem a while back having to do with documents that (because of error processing) get put into a "ready for processing" state which don't have any document priority set. But this should have been addressed, certainly, by the most recent release and probably by 2.10 as well.
Karl
On Tue, Nov 6, 2018 at 5:43 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo Karl.
When it hangs I see in the Queue status:
And in the Document Status:
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 19:32
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
What I am interested in now is the Document Status report for any one of the documents that is 'stuck'. The next crawl time value is the critical field. Can you include an example?
Karl
On Tue, Oct 30, 2018, 12:36 PM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot, Karl.
It happens that the job starts, it works and index for an hour and after it frezzes, I haven’t error or waiting status in Document Queue o Simple History, I have only “OK” status so, I haven’t failures.
I am not able to see other log errors other from the manifoldcf.log
Solr server is ok
Tika server is ok
Agent is ok
Tomcat with ManifoldCF is ok
I could search if I could to put in info log mode for example Tika servrer or Solr.
Thanks..
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 16:38
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
Hi Mario,
Please look at the Queue Status report to determine what is waiting and why it is waiting.
You can also look at the Simple History to see what has been happening. If you are getting 100% failures in fetching documents then you may need to address this because your infrastructure is unhappy. If the failure is something that indicates that the document is never going to be readable, that's a different problem and we might need to address that in the connector.
Karl
On Tue, Oct 30, 2018 at 10:33 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot Karl
Yes, I see many docs in the docs queue but they are inactive.
Infact i see that no more docs are indexed in Solr and I see that job is with the same number of docs Active (35012)
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 13:59
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
The reason the job is "stuck" is because:
' JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.'
This means that ManifoldCF will retry this document for a while before it gives up on it. It appears to be stuck but it is not. You can verify that by looking at the Document Queue report to see what is queued and what times the various documents will be retried.
Karl
On Tue, Oct 30, 2018 at 5:07 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo.
I started a job that works for some minutes, and after it stucks.
In the manifoldcf.log I see:
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-10-30T09:21:31,440 (Worker thread '2') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:33,502 (Worker thread '14') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:37,725 (Worker thread '30') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:44,406 (Worker thread '49') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:47,310 (Worker thread '15') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:52,000 (Worker thread '27') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:53,526 (Worker thread '15') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:04,511 (Worker thread '3') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2438) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1221) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-10-30T09:22:10,359 (Worker thread '27') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:13,932 (Worker thread '12') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:14,274 (Worker thread '23') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:19,933 (Worker thread '8') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:23:59,920 (Worker thread '39') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:24:09,059 (Worker thread '43') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
What could I check?
Tika server works as standalone.
Could you help me?
Thanks a lot
Mario
Re: Job stuck without message
Posted by Karl Wright <da...@gmail.com>.
Hi Mario,
I am glad your job is running now.
The manual section you describe gives you the number of postgresql handles
you need. That computation is correct. But the number of worker threads
is what is at issue.
Just because your system has tons of processors available doesn't mean you
can benefit from them, because (as I pointed out) there are a number of
places where throttling can occur in any job, and ManifoldCF is not
fundamentally process intensive. Postgresql needs the processors more than
ManifoldCF does. Setting a high number of worker threads doesn't buy you
any throughput, but will require lots of memory because each thread can use
up to a certain amount.
Thanks,
Karl
On Fri, Nov 30, 2018 at 3:44 AM Bisonti Mario <Ma...@vimar.com>
wrote:
> Hi Karl,
>
>
>
> I followed:
>
> https://manifoldcf.apache.org/release/release-2.10/en_US/performance-tuning.html#Configuring+PostgreSQL+correctly
>
>
>
> I choosed 23 Threads per processor, so, because I have 8 CPU, then
> 23*8=184
>
>
>
> And, I read, worker_thread_count + delete_thread_count +
> expiration_thread_count + cleanup_thread_count + 10 <
> manifoldcf_db_pool_size
>
> then:
> 184+10+10+10+10 < manifoldcf_db_pool_size so I choosed
> manifoldcf_db_pool_size= 250
>
>
>
>
>
>
>
> Furthermore, after I modified options.env.unix
> -Xms32768m
> -Xmx32768m
>
> I restarted my system, I deleted the documents indexed and I restart my
> big job.
>
> My job is running from yesterday at 4 p.m. without interruption 😊
>
> It has indexed 261000 docs now.
>
> I suppose that i twill finish in two days.
>
> I will update you.
>
> Thanks a lot!
>
> Mario
>
>
>
>
>
>
>
> .
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* giovedì 29 novembre 2018 17:43
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> Hi Mario,
>
>
>
> I agree with your assessment that this could have been due to an out of
> memory condition.
>
> When ManifoldCF detects such a situation, it tries to shut itself down.
> If there are further problems during that time, it may wind up not being
> able to shut down completely. Your thread dump shows that the Stuffer
> Thread, which is what queues documents for processing, is gone, but the
> zookeeper threads related to it are still there. So the entire process was
> hosed.
>
> If you want to control the amount of memory you use, pay some attention to
> keeping the number of worker threads limited. Each worker thread's
> consumption of memory is bounded, but if you have a lot of them, then the
> odds of a bad situation developing memory-wise go up.
>
>
>
> Karl
>
>
>
>
>
> On Thu, Nov 29, 2018 at 6:45 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> I think that the problemi s an out of memory error of the agent.
>
>
>
> sudo -u tomcat jcmd 1371 VM.info | more
>
>
>
> I obtain:
> 1371:
>
> #
>
> # JRE version: OpenJDK Runtime Environment (10.0.2+13) (build
> 10.0.2+13-Ubuntu-1ubuntu0.18.04.3)
>
> # Java VM: OpenJDK 64-Bit Server VM (10.0.2+13-Ubuntu-1ubuntu0.18.04.3,
> mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
>
>
>
> --------------- S U M M A R Y ------------
>
>
>
> Command Line: -Xms2048m -Xmx2048m
> -Dorg.apache.manifoldcf.configfile=./properties.xml
> -Dorg.apache.manifoldcf.processid=A org.apache.manifoldcf.agents.AgentRun
>
>
>
> Host: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz, 8 cores, 70G, Ubuntu
> 18.04.1 LTS
>
> Time: Thu Nov 29 12:18:11 2018 CET elapsed time: 152407 seconds (1d 18h
> 20m 7s)
>
>
>
> --------------- P R O C E S S ---------------
>
>
>
> OutOfMemory and StackOverflow Exception counts:
>
> OutOfMemoryError java_heap_errors=1
>
>
>
> Heap address: 0x0000000080000000, size: 2048 MB, Compressed Oops mode:
> 32-bit
>
> Narrow klass base: 0x0000000000000000, Narrow klass shift: 3
>
> Compressed class space size: 1073741824 Address: 0x0000000100000000
>
>
>
> Heap:
>
> garbage-first heap total 2097152K, used 1143846K [0x0000000080000000,
> 0x0000000100000000)
>
> region size 1024K, 254 young (260096K), 2 survivors (2048K)
>
> Metaspace used 36821K, capacity 37327K, committed 38272K, reserved
> 1083392K
>
> class space used 3251K, capacity 3432K, committed 3712K, reserved
> 1048576K
>
> Heap Regions: E=young(eden), S=young(survivor), O=old,
> HS=humongous(starts), HC=humongous(continues), CS=collection set, F=free,
> A=archive, TS=gc time stamp, AC=allocation context,
>
> TAMS=top-at-mark-start (previous, next)
>
> | 0|0x0000000080000000, 0x0000000080100000, 0x0000000080100000|100%| O|
> |TS 0|AC 0|TAMS 0x0000000080000000, 0x0000000080000000|
>
> | 1|0x0000000080100000, 0x00000000801f6dd0, 0x0000000080200000| 96%| O|
> |TS 0|AC 0|TAMS 0x0
>
>
>
>
>
> So :
>
> OutOfMemory and StackOverflow Exception counts:
>
> OutOfMemoryError java_heap_errors=1
>
>
>
> Could be useful for me..
>
>
>
> I try to set :
> sudo -u tomcat nano options.env.unix
>
> -Xms32768m
>
> -Xmx32768m
>
>
>
> And restart the process
>
> And restart the job..
>
>
>
>
>
>
>
>
>
> *Da:* Bisonti Mario <Ma...@vimar.com>
> *Inviato:* giovedì 29 novembre 2018 12:05
> *A:* user@manifoldcf.apache.org
> *Oggetto:* R: Job stuck without message
>
>
>
> *Update
>
>
>
>
>
> I found that pid 1371 is the agent pid
>
> I used the command, to retrieve this:
> ps -Flww -p 1371
>
> F S UID PID PPID C PRI NI ADDR SZ WCHAN RSS PSR STIME
> TTY TIME CMD
>
> 0 S tomcat 1371 1282 11 80 0 - 2082955 - 2621680 7 Nov27
> ? 04:40:54 /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Xms2048m
> -Xmx2048m -Dorg.apache.manifoldcf.configfile=./properties.xml -cp
> .:../lib/mcf-core.jar:../lib/mcf-agents.jar:../lib/mcf-pull-agent.jar:../lib/hsqldb-2.3.2.jar:../lib/postgresql-42.1.3.jar:../lib/commons-codec-1.10.jar:../lib/commons-collections-3.2.1.jar:../lib/commons-collections4-4.1.jar:../lib/commons-discovery-0.5.jar:../lib/commons-el-1.0.jar:../lib/commons-exec-1.3.jar:../lib/commons-fileupload-1.3.3.jar:../lib/commons-io-2.5.jar:../lib/commons-lang-2.6.jar:../lib/commons-lang3-3.6.jar:../lib/commons-logging-1.2.jar:../lib/ecj-4.3.1.jar:../lib/gson-2.8.0.jar:../lib/guava-25.1-jre.jar:../lib/httpclient-4.5.6.jar:../lib/httpcore-4.4.10.jar:../lib/jasper-6.0.35.jar:../lib/jasper-el-6.0.35.jar:../lib/javax.servlet-api-3.1.0.jar:../lib/jna-4.3.0.jar:../lib/jna-platform-4.3.0.jar:../lib/json-simple-1.1.1.jar:../lib/jsp-api-2.1-glassfish-2.1.v20091210.jar:../lib/juli-6.0.35.jar:../lib/log4j-1.2-api-2.4.1.jar:../lib/log4j-api-2.4.1.jar:../lib/log4j-core-2.4.1.jar:../lib/mail-1.4.5.jar:../lib/serializer-2.7.1.jar:../lib/slf4j-api-1.7.25.jar:../lib/slf4j-simple-1.7.25.jar:../lib/velocity-1.7.jar:../lib/xalan-2.7.1.jar:../lib/xercesImpl-2.10.0.jar:../lib/xml-apis-1.4.01.jar:../lib/zookeeper-3.4.10.jar:../lib-proprietary/jtds-1.2.4.jar:../lib-proprietary/mariadb-java-client-1.1.7.jar:../lib-proprietary/mysql-connector-java-5.1.33.jar:
> -Dorg.apache.manifoldcf.processid=A org.apache.manifoldcf.agents.AgentRun
>
>
>
> So I am able to heap the pid 1371 !
>
> sudo -u tomcat jstack -l 1371 > /tmp/jstack_agent.log
>
>
>
>
>
> I attached to this mail
>
>
>
> Thanks a lot, Karl
>
>
>
>
>
>
>
>
>
>
>
>
>
> *Da:* Bisonti Mario <Ma...@vimar.com>
> *Inviato:* giovedì 29 novembre 2018 11:12
> *A:* user@manifoldcf.apache.org
> *Oggetto:* R: Job stuck without message
>
>
>
> Thanks a lot for your big help, Karl.
>
>
>
> I would like to give you the dump but in my system (ubuntu 18.04) I am not
> able to dump the process of the start-agent.sh
>
>
>
> I tried but I always obtain:
>
> sudo -u tomcat jhsdb jinfo --pid 1233
>
> Attaching to process ID 1233, please wait...
>
> Error attaching to process: Doesn't appear to be a HotSpot VM (could not
> find symbol "gHotSpotVMTypes" in remote process)
>
> sun.jvm.hotspot.debugger.DebuggerException: Doesn't appear to be a HotSpot
> VM (could not find symbol "gHotSpotVMTypes" in remote process)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupVM(HotSpotAgent.java:413)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:306)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:141)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.tools.JInfo.runWithArgs(JInfo.java:139)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.tools.JInfo.main(JInfo.java:144)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJINFO(SALauncher.java:364)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:460)
>
>
>
>
>
>
>
> sudo -u tomcat jstack -l 1233
>
> 1233: Unable to open socket file /proc/1233/cwd/.attach_pid1233: target
> process 1233 doesn't respond within 10500ms or HotSpot VM not loaded
>
>
>
>
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* mercoledì 28 novembre 2018 21:52
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> The database row indicates there is no reason that the document should not
> be queued and processed.
>
> As for getting a thread dump, there's a "force" option (-F).
>
> The only other reason stuff may not run is if the query plan for
> identifying documents to process has gone horribly wrong. We should see
> that in the thread dump however.
>
>
>
> I will unfortunately need to be offline for the next 24 hours due to an
> emergency situation, but if it turns out that your agents process is busy
> executing a long-running query, then I suggest analyzing the jobqueue table
> to get a better plan. This happens automatically but there are conditions
> under which it doesn't happen frequently enough. If the job is waiting for
> locks, then the stack trace will tell me where.
>
>
>
> Thanks,
>
> Karl
>
> Karl
>
>
>
>
>
> On Wed, Nov 28, 2018 at 11:07 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> I attatched a row that correspond to a row of one of these documents in
> this mail
>
>
>
>
>
>
>
> I obtain the pid of:
> "/bin/bash -e
> /opt/manifoldcf/multiprocess-zk-example-proprietary/start-agents.sh"
>
> The pid is 1233
>
>
>
> I tried to use
>
> sudo jstack -l 1233 > /tmp/jstack_start_agent.log
>
>
>
> but I obtain:
>
> 1233: Unable to open socket file /proc/1233/cwd/.attach_pid1233: target
> process 1233 doesn't respond within 10500ms or HotSpot VM not loaded
>
>
>
> Perhaps isn’t it the right way to obtain a thread dump?
>
> Excuse me but I am not a Linux expert..
>
>
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* mercoledì 28 novembre 2018 16:36
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> Another thing you could do is get a thread dump of the agents process.
>
>
>
> Karl
>
>
>
>
>
> On Wed, Nov 28, 2018 at 10:35 AM Karl Wright <da...@gmail.com> wrote:
>
> Can you look into the database jobqueue table and provide a row that
> corresponds to one of these documents?
>
>
>
> Thanks,
>
> Karl
>
>
>
>
>
> On Wed, Nov 28, 2018 at 10:26 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo.
>
> Repository has Max connection=10
>
>
>
> In the Document Status report” I see many item with :
>
> State=“Not yet processed”
>
> Status=”Ready for processing”
>
> Scheduled=01-01-1970 01:00:00.000”
>
> Scheduled Action=”Process”
>
>
>
>
>
>
>
>
>
> But the job no more walk..
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* mercoledì 28 novembre 2018 16:03
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> "Pipe instances are busy" occurs because you are overloading the SMB
> access to your servers. How many connections do you have allocated for
> your repository connection? You probably want to limit this to 2-3 if you
> see this error a lot, and it appears you do.
>
> " Tika Server: Tika Server rejects: Tika Server rejected document with
> the following reason: Unprocessable Entity" means the document is not
> properly formed XML. The rejection will mean the document isn't indexed,
> but this will not stop the job.
>
> If nothing is happening and you don't know why, I'd suggest looking at the
> Document Status report to figure out what documents are not being processed
> and why. It is quite possible they are all in the process of being retried
> because of the "Pipe instances" issue above.
>
>
>
> Karl
>
>
>
> On Wed, Nov 28, 2018 at 9:46 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo Karl.
>
> I take this ticket because,now, after I use zookeeper, my job works for 7
> hours and now it is in hang status.
>
> I see running but it seems hanging, no log from 1 hour
>
>
>
> This is the last manifoldcf.log lines:
>
>
>
>
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:21,524 (Worker thread '59') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:25,232 (Worker thread '45') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-11-28T14:46:25,277 (Worker thread '70') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-11-28T14:46:28,825 (Worker thread '27') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:40,473 (Worker thread '79') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:54,855 (Worker thread '65') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:55,175 (Worker thread '83') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
>
>
>
>
>
>
> I don’t know what to check.
>
>
>
> Tika server is ok, and it doesn’t more restart it .
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 6 novembre 2018 15:27
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> I added a couple of questions to the ticket. Please reply.
>
>
>
> Thanks,
>
> Karl
>
>
>
>
>
> On Tue, Nov 6, 2018 at 8:56 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Thanks a lot, Karl.
>
> I created a ticket.
>
> https://issues.apache.org/jira/browse/CONNECTORS-1554
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCONNECTORS-1554&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ce3888a90f18046e77f3308d65619bcc1%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=cIlgBNF2dTzyc4TA4LMiCUVyOMkvh3t8cjkltxAjAaE%3D&reserved=0>
>
>
>
>
>
> Thanks
>
>
>
> Mario
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 6 novembre 2018 14:28
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> ok, can you create a ticket? Also, I'd appreciate it if you can look at
> the simple history for one of these documents; I need to see what happened
> to it last.
>
>
>
> Thanks,
>
> Karl
>
>
>
>
>
> On Tue, Nov 6, 2018 at 7:32 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> My version is 2.11
>
>
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 6 novembre 2018 13:07
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> Thanks.
>
> What version of ManifoldCF are you using? We fixed a problem a while back
> having to do with documents that (because of error processing) get put into
> a "ready for processing" state which don't have any document priority set.
> But this should have been addressed, certainly, by the most recent release
> and probably by 2.10 as well.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Nov 6, 2018 at 5:43 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo Karl.
>
> When it hangs I see in the Queue status:
>
>
>
> And in the Document Status:
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 19:32
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> What I am interested in now is the Document Status report for any one of
> the documents that is 'stuck'. The next crawl time value is the critical
> field. Can you include an example?
>
>
>
> Karl
>
>
>
> On Tue, Oct 30, 2018, 12:36 PM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Thanks a lot, Karl.
>
>
>
> It happens that the job starts, it works and index for an hour and after
> it frezzes, I haven’t error or waiting status in Document Queue o Simple
> History, I have only “OK” status so, I haven’t failures.
>
>
>
> I am not able to see other log errors other from the manifoldcf.log
>
>
>
> Solr server is ok
>
> Tika server is ok
>
> Agent is ok
>
> Tomcat with ManifoldCF is ok
>
>
>
> I could search if I could to put in info log mode for example Tika servrer
> or Solr.
>
>
>
> Thanks..
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 16:38
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> Hi Mario,
>
> Please look at the Queue Status report to determine what is waiting and
> why it is waiting.
> You can also look at the Simple History to see what has been happening.
> If you are getting 100% failures in fetching documents then you may need to
> address this because your infrastructure is unhappy. If the failure is
> something that indicates that the document is never going to be readable,
> that's a different problem and we might need to address that in the
> connector.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Oct 30, 2018 at 10:33 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
>
>
> Thanks a lot Karl
>
>
>
> Yes, I see many docs in the docs queue but they are inactive.
>
>
>
> Infact i see that no more docs are indexed in Solr and I see that job is
> with the same number of docs Active (35012)
>
>
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 13:59
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> The reason the job is "stuck" is because:
>
> ' JCIFS: Possibly transient exception detected on attempt 1 while getting
> share security: All pipe instances are busy.'
>
> This means that ManifoldCF will retry this document for a while before it
> gives up on it. It appears to be stuck but it is not. You can verify that
> by looking at the Document Queue report to see what is queued and what
> times the various documents will be retried.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Oct 30, 2018 at 5:07 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo.
>
>
>
> I started a job that works for some minutes, and after it stucks.
>
>
>
> In the manifoldcf.log I see:
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-10-30T09:21:31,440 (Worker thread '2') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:33,502 (Worker thread '14') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:37,725 (Worker thread '30') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:44,406 (Worker thread '49') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:47,310 (Worker thread '15') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:52,000 (Worker thread '27') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:53,526 (Worker thread '15') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:04,511 (Worker thread '3') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2438)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1221)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-10-30T09:22:10,359 (Worker thread '27') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:13,932 (Worker thread '12') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:14,274 (Worker thread '23') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:19,933 (Worker thread '8') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:23:59,920 (Worker thread '39') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:24:09,059 (Worker thread '43') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
>
>
>
>
>
>
> What could I check?
>
>
>
> Tika server works as standalone.
>
>
>
> Could you help me?
>
>
>
> Thanks a lot
>
>
>
> Mario
>
>
>
>
R: Job stuck without message
Posted by Bisonti Mario <Ma...@vimar.com>.
Hi Karl,
I followed:
https://manifoldcf.apache.org/release/release-2.10/en_US/performance-tuning.html#Configuring+PostgreSQL+correctly
I choosed 23 Threads per processor, so, because I have 8 CPU, then 23*8=184
And, I read, worker_thread_count + delete_thread_count + expiration_thread_count + cleanup_thread_count + 10 < manifoldcf_db_pool_size
then:
184+10+10+10+10 < manifoldcf_db_pool_size so I choosed manifoldcf_db_pool_size= 250
Furthermore, after I modified options.env.unix
-Xms32768m
-Xmx32768m
I restarted my system, I deleted the documents indexed and I restart my big job.
My job is running from yesterday at 4 p.m. without interruption 😊
It has indexed 261000 docs now.
I suppose that i twill finish in two days.
I will update you.
Thanks a lot!
Mario
.
Da: Karl Wright <da...@gmail.com>
Inviato: giovedì 29 novembre 2018 17:43
A: user@manifoldcf.apache.org
Oggetto: Re: Job stuck without message
Hi Mario,
I agree with your assessment that this could have been due to an out of memory condition.
When ManifoldCF detects such a situation, it tries to shut itself down. If there are further problems during that time, it may wind up not being able to shut down completely. Your thread dump shows that the Stuffer Thread, which is what queues documents for processing, is gone, but the zookeeper threads related to it are still there. So the entire process was hosed.
If you want to control the amount of memory you use, pay some attention to keeping the number of worker threads limited. Each worker thread's consumption of memory is bounded, but if you have a lot of them, then the odds of a bad situation developing memory-wise go up.
Karl
On Thu, Nov 29, 2018 at 6:45 AM Bisonti Mario <Ma...@vimar.com>> wrote:
I think that the problemi s an out of memory error of the agent.
sudo -u tomcat jcmd 1371 VM.info | more
I obtain:
1371:
#
# JRE version: OpenJDK Runtime Environment (10.0.2+13) (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.3)
# Java VM: OpenJDK 64-Bit Server VM (10.0.2+13-Ubuntu-1ubuntu0.18.04.3, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
--------------- S U M M A R Y ------------
Command Line: -Xms2048m -Xmx2048m -Dorg.apache.manifoldcf.configfile=./properties.xml -Dorg.apache.manifoldcf.processid=A org.apache.manifoldcf.agents.AgentRun
Host: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz, 8 cores, 70G, Ubuntu 18.04.1 LTS
Time: Thu Nov 29 12:18:11 2018 CET elapsed time: 152407 seconds (1d 18h 20m 7s)
--------------- P R O C E S S ---------------
OutOfMemory and StackOverflow Exception counts:
OutOfMemoryError java_heap_errors=1
Heap address: 0x0000000080000000, size: 2048 MB, Compressed Oops mode: 32-bit
Narrow klass base: 0x0000000000000000, Narrow klass shift: 3
Compressed class space size: 1073741824 Address: 0x0000000100000000
Heap:
garbage-first heap total 2097152K, used 1143846K [0x0000000080000000, 0x0000000100000000)
region size 1024K, 254 young (260096K), 2 survivors (2048K)
Metaspace used 36821K, capacity 37327K, committed 38272K, reserved 1083392K
class space used 3251K, capacity 3432K, committed 3712K, reserved 1048576K
Heap Regions: E=young(eden), S=young(survivor), O=old, HS=humongous(starts), HC=humongous(continues), CS=collection set, F=free, A=archive, TS=gc time stamp, AC=allocation context,
TAMS=top-at-mark-start (previous, next)
| 0|0x0000000080000000, 0x0000000080100000, 0x0000000080100000|100%| O| |TS 0|AC 0|TAMS 0x0000000080000000, 0x0000000080000000|
| 1|0x0000000080100000, 0x00000000801f6dd0, 0x0000000080200000| 96%| O| |TS 0|AC 0|TAMS 0x0
So :
OutOfMemory and StackOverflow Exception counts:
OutOfMemoryError java_heap_errors=1
Could be useful for me..
I try to set :
sudo -u tomcat nano options.env.unix
-Xms32768m
-Xmx32768m
And restart the process
And restart the job..
Da: Bisonti Mario <Ma...@vimar.com>>
Inviato: giovedì 29 novembre 2018 12:05
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: R: Job stuck without message
*Update
I found that pid 1371 is the agent pid
I used the command, to retrieve this:
ps -Flww -p 1371
F S UID PID PPID C PRI NI ADDR SZ WCHAN RSS PSR STIME TTY TIME CMD
0 S tomcat 1371 1282 11 80 0 - 2082955 - 2621680 7 Nov27 ? 04:40:54 /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Xms2048m -Xmx2048m -Dorg.apache.manifoldcf.configfile=./properties.xml -cp .:../lib/mcf-core.jar:../lib/mcf-agents.jar:../lib/mcf-pull-agent.jar:../lib/hsqldb-2.3.2.jar:../lib/postgresql-42.1.3.jar:../lib/commons-codec-1.10.jar:../lib/commons-collections-3.2.1.jar:../lib/commons-collections4-4.1.jar:../lib/commons-discovery-0.5.jar:../lib/commons-el-1.0.jar:../lib/commons-exec-1.3.jar:../lib/commons-fileupload-1.3.3.jar:../lib/commons-io-2.5.jar:../lib/commons-lang-2.6.jar:../lib/commons-lang3-3.6.jar:../lib/commons-logging-1.2.jar:../lib/ecj-4.3.1.jar:../lib/gson-2.8.0.jar:../lib/guava-25.1-jre.jar:../lib/httpclient-4.5.6.jar:../lib/httpcore-4.4.10.jar:../lib/jasper-6.0.35.jar:../lib/jasper-el-6.0.35.jar:../lib/javax.servlet-api-3.1.0.jar:../lib/jna-4.3.0.jar:../lib/jna-platform-4.3.0.jar:../lib/json-simple-1.1.1.jar:../lib/jsp-api-2.1-glassfish-2.1.v20091210.jar:../lib/juli-6.0.35.jar:../lib/log4j-1.2-api-2.4.1.jar:../lib/log4j-api-2.4.1.jar:../lib/log4j-core-2.4.1.jar:../lib/mail-1.4.5.jar:../lib/serializer-2.7.1.jar:../lib/slf4j-api-1.7.25.jar:../lib/slf4j-simple-1.7.25.jar:../lib/velocity-1.7.jar:../lib/xalan-2.7.1.jar:../lib/xercesImpl-2.10.0.jar:../lib/xml-apis-1.4.01.jar:../lib/zookeeper-3.4.10.jar:../lib-proprietary/jtds-1.2.4.jar:../lib-proprietary/mariadb-java-client-1.1.7.jar:../lib-proprietary/mysql-connector-java-5.1.33.jar: -Dorg.apache.manifoldcf.processid=A org.apache.manifoldcf.agents.AgentRun
So I am able to heap the pid 1371 !
sudo -u tomcat jstack -l 1371 > /tmp/jstack_agent.log
I attached to this mail
Thanks a lot, Karl
Da: Bisonti Mario <Ma...@vimar.com>>
Inviato: giovedì 29 novembre 2018 11:12
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: R: Job stuck without message
Thanks a lot for your big help, Karl.
I would like to give you the dump but in my system (ubuntu 18.04) I am not able to dump the process of the start-agent.sh
I tried but I always obtain:
sudo -u tomcat jhsdb jinfo --pid 1233
Attaching to process ID 1233, please wait...
Error attaching to process: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMTypes" in remote process)
sun.jvm.hotspot.debugger.DebuggerException: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMTypes" in remote process)
at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupVM(HotSpotAgent.java:413)
at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:306)
at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:141)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.JInfo.runWithArgs(JInfo.java:139)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.JInfo.main(JInfo.java:144)
at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJINFO(SALauncher.java:364)
at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:460)
sudo -u tomcat jstack -l 1233
1233: Unable to open socket file /proc/1233/cwd/.attach_pid1233: target process 1233 doesn't respond within 10500ms or HotSpot VM not loaded
Da: Karl Wright <da...@gmail.com>>
Inviato: mercoledì 28 novembre 2018 21:52
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
The database row indicates there is no reason that the document should not be queued and processed.
As for getting a thread dump, there's a "force" option (-F).
The only other reason stuff may not run is if the query plan for identifying documents to process has gone horribly wrong. We should see that in the thread dump however.
I will unfortunately need to be offline for the next 24 hours due to an emergency situation, but if it turns out that your agents process is busy executing a long-running query, then I suggest analyzing the jobqueue table to get a better plan. This happens automatically but there are conditions under which it doesn't happen frequently enough. If the job is waiting for locks, then the stack trace will tell me where.
Thanks,
Karl
Karl
On Wed, Nov 28, 2018 at 11:07 AM Bisonti Mario <Ma...@vimar.com>> wrote:
I attatched a row that correspond to a row of one of these documents in this mail
I obtain the pid of:
"/bin/bash -e /opt/manifoldcf/multiprocess-zk-example-proprietary/start-agents.sh"
The pid is 1233
I tried to use
sudo jstack -l 1233 > /tmp/jstack_start_agent.log
but I obtain:
1233: Unable to open socket file /proc/1233/cwd/.attach_pid1233: target process 1233 doesn't respond within 10500ms or HotSpot VM not loaded
Perhaps isn’t it the right way to obtain a thread dump?
Excuse me but I am not a Linux expert..
Da: Karl Wright <da...@gmail.com>>
Inviato: mercoledì 28 novembre 2018 16:36
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
Another thing you could do is get a thread dump of the agents process.
Karl
On Wed, Nov 28, 2018 at 10:35 AM Karl Wright <da...@gmail.com>> wrote:
Can you look into the database jobqueue table and provide a row that corresponds to one of these documents?
Thanks,
Karl
On Wed, Nov 28, 2018 at 10:26 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo.
Repository has Max connection=10
In the Document Status report” I see many item with :
State=“Not yet processed”
Status=”Ready for processing”
Scheduled=01-01-1970 01:00:00.000”
Scheduled Action=”Process”
But the job no more walk..
Da: Karl Wright <da...@gmail.com>>
Inviato: mercoledì 28 novembre 2018 16:03
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
"Pipe instances are busy" occurs because you are overloading the SMB access to your servers. How many connections do you have allocated for your repository connection? You probably want to limit this to 2-3 if you see this error a lot, and it appears you do.
" Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity" means the document is not properly formed XML. The rejection will mean the document isn't indexed, but this will not stop the job.
If nothing is happening and you don't know why, I'd suggest looking at the Document Status report to figure out what documents are not being processed and why. It is quite possible they are all in the process of being retried because of the "Pipe instances" issue above.
Karl
On Wed, Nov 28, 2018 at 9:46 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo Karl.
I take this ticket because,now, after I use zookeeper, my job works for 7 hours and now it is in hang status.
I see running but it seems hanging, no log from 1 hour
This is the last manifoldcf.log lines:
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:21,524 (Worker thread '59') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:25,232 (Worker thread '45') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-11-28T14:46:25,277 (Worker thread '70') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-11-28T14:46:28,825 (Worker thread '27') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:40,473 (Worker thread '79') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:54,855 (Worker thread '65') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:55,175 (Worker thread '83') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
I don’t know what to check.
Tika server is ok, and it doesn’t more restart it .
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 6 novembre 2018 15:27
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
I added a couple of questions to the ticket. Please reply.
Thanks,
Karl
On Tue, Nov 6, 2018 at 8:56 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot, Karl.
I created a ticket.
https://issues.apache.org/jira/browse/CONNECTORS-1554<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCONNECTORS-1554&data=01%7C01%7CMario.Bisonti%40vimar.com%7Ce3888a90f18046e77f3308d65619bcc1%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=cIlgBNF2dTzyc4TA4LMiCUVyOMkvh3t8cjkltxAjAaE%3D&reserved=0>
Thanks
Mario
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 6 novembre 2018 14:28
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
ok, can you create a ticket? Also, I'd appreciate it if you can look at the simple history for one of these documents; I need to see what happened to it last.
Thanks,
Karl
On Tue, Nov 6, 2018 at 7:32 AM Bisonti Mario <Ma...@vimar.com>> wrote:
My version is 2.11
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 6 novembre 2018 13:07
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
Thanks.
What version of ManifoldCF are you using? We fixed a problem a while back having to do with documents that (because of error processing) get put into a "ready for processing" state which don't have any document priority set. But this should have been addressed, certainly, by the most recent release and probably by 2.10 as well.
Karl
On Tue, Nov 6, 2018 at 5:43 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo Karl.
When it hangs I see in the Queue status:
And in the Document Status:
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 19:32
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
What I am interested in now is the Document Status report for any one of the documents that is 'stuck'. The next crawl time value is the critical field. Can you include an example?
Karl
On Tue, Oct 30, 2018, 12:36 PM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot, Karl.
It happens that the job starts, it works and index for an hour and after it frezzes, I haven’t error or waiting status in Document Queue o Simple History, I have only “OK” status so, I haven’t failures.
I am not able to see other log errors other from the manifoldcf.log
Solr server is ok
Tika server is ok
Agent is ok
Tomcat with ManifoldCF is ok
I could search if I could to put in info log mode for example Tika servrer or Solr.
Thanks..
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 16:38
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
Hi Mario,
Please look at the Queue Status report to determine what is waiting and why it is waiting.
You can also look at the Simple History to see what has been happening. If you are getting 100% failures in fetching documents then you may need to address this because your infrastructure is unhappy. If the failure is something that indicates that the document is never going to be readable, that's a different problem and we might need to address that in the connector.
Karl
On Tue, Oct 30, 2018 at 10:33 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot Karl
Yes, I see many docs in the docs queue but they are inactive.
Infact i see that no more docs are indexed in Solr and I see that job is with the same number of docs Active (35012)
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 13:59
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
The reason the job is "stuck" is because:
' JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.'
This means that ManifoldCF will retry this document for a while before it gives up on it. It appears to be stuck but it is not. You can verify that by looking at the Document Queue report to see what is queued and what times the various documents will be retried.
Karl
On Tue, Oct 30, 2018 at 5:07 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo.
I started a job that works for some minutes, and after it stucks.
In the manifoldcf.log I see:
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-10-30T09:21:31,440 (Worker thread '2') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:33,502 (Worker thread '14') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:37,725 (Worker thread '30') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:44,406 (Worker thread '49') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:47,310 (Worker thread '15') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:52,000 (Worker thread '27') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:53,526 (Worker thread '15') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:04,511 (Worker thread '3') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2438) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1221) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-10-30T09:22:10,359 (Worker thread '27') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:13,932 (Worker thread '12') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:14,274 (Worker thread '23') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:19,933 (Worker thread '8') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:23:59,920 (Worker thread '39') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:24:09,059 (Worker thread '43') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
What could I check?
Tika server works as standalone.
Could you help me?
Thanks a lot
Mario
Re: Job stuck without message
Posted by Karl Wright <da...@gmail.com>.
Hi Mario,
I agree with your assessment that this could have been due to an out of
memory condition.
When ManifoldCF detects such a situation, it tries to shut itself down. If
there are further problems during that time, it may wind up not being able
to shut down completely. Your thread dump shows that the Stuffer Thread,
which is what queues documents for processing, is gone, but the zookeeper
threads related to it are still there. So the entire process was hosed.
If you want to control the amount of memory you use, pay some attention to
keeping the number of worker threads limited. Each worker thread's
consumption of memory is bounded, but if you have a lot of them, then the
odds of a bad situation developing memory-wise go up.
Karl
On Thu, Nov 29, 2018 at 6:45 AM Bisonti Mario <Ma...@vimar.com>
wrote:
> I think that the problemi s an out of memory error of the agent.
>
>
>
> sudo -u tomcat jcmd 1371 VM.info | more
>
>
>
> I obtain:
> 1371:
>
> #
>
> # JRE version: OpenJDK Runtime Environment (10.0.2+13) (build
> 10.0.2+13-Ubuntu-1ubuntu0.18.04.3)
>
> # Java VM: OpenJDK 64-Bit Server VM (10.0.2+13-Ubuntu-1ubuntu0.18.04.3,
> mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
>
>
>
> --------------- S U M M A R Y ------------
>
>
>
> Command Line: -Xms2048m -Xmx2048m
> -Dorg.apache.manifoldcf.configfile=./properties.xml
> -Dorg.apache.manifoldcf.processid=A org.apache.manifoldcf.agents.AgentRun
>
>
>
> Host: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz, 8 cores, 70G, Ubuntu
> 18.04.1 LTS
>
> Time: Thu Nov 29 12:18:11 2018 CET elapsed time: 152407 seconds (1d 18h
> 20m 7s)
>
>
>
> --------------- P R O C E S S ---------------
>
>
>
> OutOfMemory and StackOverflow Exception counts:
>
> OutOfMemoryError java_heap_errors=1
>
>
>
> Heap address: 0x0000000080000000, size: 2048 MB, Compressed Oops mode:
> 32-bit
>
> Narrow klass base: 0x0000000000000000, Narrow klass shift: 3
>
> Compressed class space size: 1073741824 Address: 0x0000000100000000
>
>
>
> Heap:
>
> garbage-first heap total 2097152K, used 1143846K [0x0000000080000000,
> 0x0000000100000000)
>
> region size 1024K, 254 young (260096K), 2 survivors (2048K)
>
> Metaspace used 36821K, capacity 37327K, committed 38272K, reserved
> 1083392K
>
> class space used 3251K, capacity 3432K, committed 3712K, reserved
> 1048576K
>
> Heap Regions: E=young(eden), S=young(survivor), O=old,
> HS=humongous(starts), HC=humongous(continues), CS=collection set, F=free,
> A=archive, TS=gc time stamp, AC=allocation context,
>
> TAMS=top-at-mark-start (previous, next)
>
> | 0|0x0000000080000000, 0x0000000080100000, 0x0000000080100000|100%| O|
> |TS 0|AC 0|TAMS 0x0000000080000000, 0x0000000080000000|
>
> | 1|0x0000000080100000, 0x00000000801f6dd0, 0x0000000080200000| 96%| O|
> |TS 0|AC 0|TAMS 0x0
>
>
>
>
>
> So :
>
> OutOfMemory and StackOverflow Exception counts:
>
> OutOfMemoryError java_heap_errors=1
>
>
>
> Could be useful for me..
>
>
>
> I try to set :
> sudo -u tomcat nano options.env.unix
>
> -Xms32768m
>
> -Xmx32768m
>
>
>
> And restart the process
>
> And restart the job..
>
>
>
>
>
>
>
>
>
> *Da:* Bisonti Mario <Ma...@vimar.com>
> *Inviato:* giovedì 29 novembre 2018 12:05
> *A:* user@manifoldcf.apache.org
> *Oggetto:* R: Job stuck without message
>
>
>
> *Update
>
>
>
>
>
> I found that pid 1371 is the agent pid
>
> I used the command, to retrieve this:
> ps -Flww -p 1371
>
> F S UID PID PPID C PRI NI ADDR SZ WCHAN RSS PSR STIME
> TTY TIME CMD
>
> 0 S tomcat 1371 1282 11 80 0 - 2082955 - 2621680 7 Nov27
> ? 04:40:54 /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Xms2048m
> -Xmx2048m -Dorg.apache.manifoldcf.configfile=./properties.xml -cp
> .:../lib/mcf-core.jar:../lib/mcf-agents.jar:../lib/mcf-pull-agent.jar:../lib/hsqldb-2.3.2.jar:../lib/postgresql-42.1.3.jar:../lib/commons-codec-1.10.jar:../lib/commons-collections-3.2.1.jar:../lib/commons-collections4-4.1.jar:../lib/commons-discovery-0.5.jar:../lib/commons-el-1.0.jar:../lib/commons-exec-1.3.jar:../lib/commons-fileupload-1.3.3.jar:../lib/commons-io-2.5.jar:../lib/commons-lang-2.6.jar:../lib/commons-lang3-3.6.jar:../lib/commons-logging-1.2.jar:../lib/ecj-4.3.1.jar:../lib/gson-2.8.0.jar:../lib/guava-25.1-jre.jar:../lib/httpclient-4.5.6.jar:../lib/httpcore-4.4.10.jar:../lib/jasper-6.0.35.jar:../lib/jasper-el-6.0.35.jar:../lib/javax.servlet-api-3.1.0.jar:../lib/jna-4.3.0.jar:../lib/jna-platform-4.3.0.jar:../lib/json-simple-1.1.1.jar:../lib/jsp-api-2.1-glassfish-2.1.v20091210.jar:../lib/juli-6.0.35.jar:../lib/log4j-1.2-api-2.4.1.jar:../lib/log4j-api-2.4.1.jar:../lib/log4j-core-2.4.1.jar:../lib/mail-1.4.5.jar:../lib/serializer-2.7.1.jar:../lib/slf4j-api-1.7.25.jar:../lib/slf4j-simple-1.7.25.jar:../lib/velocity-1.7.jar:../lib/xalan-2.7.1.jar:../lib/xercesImpl-2.10.0.jar:../lib/xml-apis-1.4.01.jar:../lib/zookeeper-3.4.10.jar:../lib-proprietary/jtds-1.2.4.jar:../lib-proprietary/mariadb-java-client-1.1.7.jar:../lib-proprietary/mysql-connector-java-5.1.33.jar:
> -Dorg.apache.manifoldcf.processid=A org.apache.manifoldcf.agents.AgentRun
>
>
>
> So I am able to heap the pid 1371 !
>
> sudo -u tomcat jstack -l 1371 > /tmp/jstack_agent.log
>
>
>
>
>
> I attached to this mail
>
>
>
> Thanks a lot, Karl
>
>
>
>
>
>
>
>
>
>
>
>
>
> *Da:* Bisonti Mario <Ma...@vimar.com>
> *Inviato:* giovedì 29 novembre 2018 11:12
> *A:* user@manifoldcf.apache.org
> *Oggetto:* R: Job stuck without message
>
>
>
> Thanks a lot for your big help, Karl.
>
>
>
> I would like to give you the dump but in my system (ubuntu 18.04) I am not
> able to dump the process of the start-agent.sh
>
>
>
> I tried but I always obtain:
>
> sudo -u tomcat jhsdb jinfo --pid 1233
>
> Attaching to process ID 1233, please wait...
>
> Error attaching to process: Doesn't appear to be a HotSpot VM (could not
> find symbol "gHotSpotVMTypes" in remote process)
>
> sun.jvm.hotspot.debugger.DebuggerException: Doesn't appear to be a HotSpot
> VM (could not find symbol "gHotSpotVMTypes" in remote process)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupVM(HotSpotAgent.java:413)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:306)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:141)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.tools.JInfo.runWithArgs(JInfo.java:139)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.tools.JInfo.main(JInfo.java:144)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJINFO(SALauncher.java:364)
>
> at
> jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:460)
>
>
>
>
>
>
>
> sudo -u tomcat jstack -l 1233
>
> 1233: Unable to open socket file /proc/1233/cwd/.attach_pid1233: target
> process 1233 doesn't respond within 10500ms or HotSpot VM not loaded
>
>
>
>
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* mercoledì 28 novembre 2018 21:52
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> The database row indicates there is no reason that the document should not
> be queued and processed.
>
> As for getting a thread dump, there's a "force" option (-F).
>
> The only other reason stuff may not run is if the query plan for
> identifying documents to process has gone horribly wrong. We should see
> that in the thread dump however.
>
>
>
> I will unfortunately need to be offline for the next 24 hours due to an
> emergency situation, but if it turns out that your agents process is busy
> executing a long-running query, then I suggest analyzing the jobqueue table
> to get a better plan. This happens automatically but there are conditions
> under which it doesn't happen frequently enough. If the job is waiting for
> locks, then the stack trace will tell me where.
>
>
>
> Thanks,
>
> Karl
>
> Karl
>
>
>
>
>
> On Wed, Nov 28, 2018 at 11:07 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> I attatched a row that correspond to a row of one of these documents in
> this mail
>
>
>
>
>
>
>
> I obtain the pid of:
> "/bin/bash -e
> /opt/manifoldcf/multiprocess-zk-example-proprietary/start-agents.sh"
>
> The pid is 1233
>
>
>
> I tried to use
>
> sudo jstack -l 1233 > /tmp/jstack_start_agent.log
>
>
>
> but I obtain:
>
> 1233: Unable to open socket file /proc/1233/cwd/.attach_pid1233: target
> process 1233 doesn't respond within 10500ms or HotSpot VM not loaded
>
>
>
> Perhaps isn’t it the right way to obtain a thread dump?
>
> Excuse me but I am not a Linux expert..
>
>
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* mercoledì 28 novembre 2018 16:36
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> Another thing you could do is get a thread dump of the agents process.
>
>
>
> Karl
>
>
>
>
>
> On Wed, Nov 28, 2018 at 10:35 AM Karl Wright <da...@gmail.com> wrote:
>
> Can you look into the database jobqueue table and provide a row that
> corresponds to one of these documents?
>
>
>
> Thanks,
>
> Karl
>
>
>
>
>
> On Wed, Nov 28, 2018 at 10:26 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo.
>
> Repository has Max connection=10
>
>
>
> In the Document Status report” I see many item with :
>
> State=“Not yet processed”
>
> Status=”Ready for processing”
>
> Scheduled=01-01-1970 01:00:00.000”
>
> Scheduled Action=”Process”
>
>
>
>
>
>
>
>
>
> But the job no more walk..
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* mercoledì 28 novembre 2018 16:03
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> "Pipe instances are busy" occurs because you are overloading the SMB
> access to your servers. How many connections do you have allocated for
> your repository connection? You probably want to limit this to 2-3 if you
> see this error a lot, and it appears you do.
>
> " Tika Server: Tika Server rejects: Tika Server rejected document with
> the following reason: Unprocessable Entity" means the document is not
> properly formed XML. The rejection will mean the document isn't indexed,
> but this will not stop the job.
>
> If nothing is happening and you don't know why, I'd suggest looking at the
> Document Status report to figure out what documents are not being processed
> and why. It is quite possible they are all in the process of being retried
> because of the "Pipe instances" issue above.
>
>
>
> Karl
>
>
>
> On Wed, Nov 28, 2018 at 9:46 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo Karl.
>
> I take this ticket because,now, after I use zookeeper, my job works for 7
> hours and now it is in hang status.
>
> I see running but it seems hanging, no log from 1 hour
>
>
>
> This is the last manifoldcf.log lines:
>
>
>
>
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:21,524 (Worker thread '59') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:25,232 (Worker thread '45') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-11-28T14:46:25,277 (Worker thread '70') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-11-28T14:46:28,825 (Worker thread '27') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:40,473 (Worker thread '79') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:54,855 (Worker thread '65') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:55,175 (Worker thread '83') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
>
>
>
>
>
>
> I don’t know what to check.
>
>
>
> Tika server is ok, and it doesn’t more restart it .
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 6 novembre 2018 15:27
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> I added a couple of questions to the ticket. Please reply.
>
>
>
> Thanks,
>
> Karl
>
>
>
>
>
> On Tue, Nov 6, 2018 at 8:56 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Thanks a lot, Karl.
>
> I created a ticket.
>
> https://issues.apache.org/jira/browse/CONNECTORS-1554
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCONNECTORS-1554&data=01%7C01%7CMario.Bisonti%40vimar.com%7C98877f463c1a46a299e408d655ebce48%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=zaDTmL7D5m9znylm%2FIQDy857SEd8TERvjSLCjDfDjik%3D&reserved=0>
>
>
>
>
>
> Thanks
>
>
>
> Mario
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 6 novembre 2018 14:28
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> ok, can you create a ticket? Also, I'd appreciate it if you can look at
> the simple history for one of these documents; I need to see what happened
> to it last.
>
>
>
> Thanks,
>
> Karl
>
>
>
>
>
> On Tue, Nov 6, 2018 at 7:32 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> My version is 2.11
>
>
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 6 novembre 2018 13:07
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> Thanks.
>
> What version of ManifoldCF are you using? We fixed a problem a while back
> having to do with documents that (because of error processing) get put into
> a "ready for processing" state which don't have any document priority set.
> But this should have been addressed, certainly, by the most recent release
> and probably by 2.10 as well.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Nov 6, 2018 at 5:43 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo Karl.
>
> When it hangs I see in the Queue status:
>
>
>
> And in the Document Status:
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 19:32
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> What I am interested in now is the Document Status report for any one of
> the documents that is 'stuck'. The next crawl time value is the critical
> field. Can you include an example?
>
>
>
> Karl
>
>
>
> On Tue, Oct 30, 2018, 12:36 PM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Thanks a lot, Karl.
>
>
>
> It happens that the job starts, it works and index for an hour and after
> it frezzes, I haven’t error or waiting status in Document Queue o Simple
> History, I have only “OK” status so, I haven’t failures.
>
>
>
> I am not able to see other log errors other from the manifoldcf.log
>
>
>
> Solr server is ok
>
> Tika server is ok
>
> Agent is ok
>
> Tomcat with ManifoldCF is ok
>
>
>
> I could search if I could to put in info log mode for example Tika servrer
> or Solr.
>
>
>
> Thanks..
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 16:38
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> Hi Mario,
>
> Please look at the Queue Status report to determine what is waiting and
> why it is waiting.
> You can also look at the Simple History to see what has been happening.
> If you are getting 100% failures in fetching documents then you may need to
> address this because your infrastructure is unhappy. If the failure is
> something that indicates that the document is never going to be readable,
> that's a different problem and we might need to address that in the
> connector.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Oct 30, 2018 at 10:33 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
>
>
> Thanks a lot Karl
>
>
>
> Yes, I see many docs in the docs queue but they are inactive.
>
>
>
> Infact i see that no more docs are indexed in Solr and I see that job is
> with the same number of docs Active (35012)
>
>
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 13:59
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> The reason the job is "stuck" is because:
>
> ' JCIFS: Possibly transient exception detected on attempt 1 while getting
> share security: All pipe instances are busy.'
>
> This means that ManifoldCF will retry this document for a while before it
> gives up on it. It appears to be stuck but it is not. You can verify that
> by looking at the Document Queue report to see what is queued and what
> times the various documents will be retried.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Oct 30, 2018 at 5:07 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo.
>
>
>
> I started a job that works for some minutes, and after it stucks.
>
>
>
> In the manifoldcf.log I see:
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-10-30T09:21:31,440 (Worker thread '2') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:33,502 (Worker thread '14') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:37,725 (Worker thread '30') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:44,406 (Worker thread '49') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:47,310 (Worker thread '15') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:52,000 (Worker thread '27') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:53,526 (Worker thread '15') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:04,511 (Worker thread '3') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2438)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1221)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-10-30T09:22:10,359 (Worker thread '27') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:13,932 (Worker thread '12') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:14,274 (Worker thread '23') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:19,933 (Worker thread '8') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:23:59,920 (Worker thread '39') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:24:09,059 (Worker thread '43') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
>
>
>
>
>
>
> What could I check?
>
>
>
> Tika server works as standalone.
>
>
>
> Could you help me?
>
>
>
> Thanks a lot
>
>
>
> Mario
>
>
>
>
R: Job stuck without message
Posted by Bisonti Mario <Ma...@vimar.com>.
I think that the problemi s an out of memory error of the agent.
sudo -u tomcat jcmd 1371 VM.info | more
I obtain:
1371:
#
# JRE version: OpenJDK Runtime Environment (10.0.2+13) (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.3)
# Java VM: OpenJDK 64-Bit Server VM (10.0.2+13-Ubuntu-1ubuntu0.18.04.3, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
--------------- S U M M A R Y ------------
Command Line: -Xms2048m -Xmx2048m -Dorg.apache.manifoldcf.configfile=./properties.xml -Dorg.apache.manifoldcf.processid=A org.apache.manifoldcf.agents.AgentRun
Host: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz, 8 cores, 70G, Ubuntu 18.04.1 LTS
Time: Thu Nov 29 12:18:11 2018 CET elapsed time: 152407 seconds (1d 18h 20m 7s)
--------------- P R O C E S S ---------------
OutOfMemory and StackOverflow Exception counts:
OutOfMemoryError java_heap_errors=1
Heap address: 0x0000000080000000, size: 2048 MB, Compressed Oops mode: 32-bit
Narrow klass base: 0x0000000000000000, Narrow klass shift: 3
Compressed class space size: 1073741824 Address: 0x0000000100000000
Heap:
garbage-first heap total 2097152K, used 1143846K [0x0000000080000000, 0x0000000100000000)
region size 1024K, 254 young (260096K), 2 survivors (2048K)
Metaspace used 36821K, capacity 37327K, committed 38272K, reserved 1083392K
class space used 3251K, capacity 3432K, committed 3712K, reserved 1048576K
Heap Regions: E=young(eden), S=young(survivor), O=old, HS=humongous(starts), HC=humongous(continues), CS=collection set, F=free, A=archive, TS=gc time stamp, AC=allocation context,
TAMS=top-at-mark-start (previous, next)
| 0|0x0000000080000000, 0x0000000080100000, 0x0000000080100000|100%| O| |TS 0|AC 0|TAMS 0x0000000080000000, 0x0000000080000000|
| 1|0x0000000080100000, 0x00000000801f6dd0, 0x0000000080200000| 96%| O| |TS 0|AC 0|TAMS 0x0
So :
OutOfMemory and StackOverflow Exception counts:
OutOfMemoryError java_heap_errors=1
Could be useful for me..
I try to set :
sudo -u tomcat nano options.env.unix
-Xms32768m
-Xmx32768m
And restart the process
And restart the job..
Da: Bisonti Mario <Ma...@vimar.com>
Inviato: giovedì 29 novembre 2018 12:05
A: user@manifoldcf.apache.org
Oggetto: R: Job stuck without message
*Update
I found that pid 1371 is the agent pid
I used the command, to retrieve this:
ps -Flww -p 1371
F S UID PID PPID C PRI NI ADDR SZ WCHAN RSS PSR STIME TTY TIME CMD
0 S tomcat 1371 1282 11 80 0 - 2082955 - 2621680 7 Nov27 ? 04:40:54 /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Xms2048m -Xmx2048m -Dorg.apache.manifoldcf.configfile=./properties.xml -cp .:../lib/mcf-core.jar:../lib/mcf-agents.jar:../lib/mcf-pull-agent.jar:../lib/hsqldb-2.3.2.jar:../lib/postgresql-42.1.3.jar:../lib/commons-codec-1.10.jar:../lib/commons-collections-3.2.1.jar:../lib/commons-collections4-4.1.jar:../lib/commons-discovery-0.5.jar:../lib/commons-el-1.0.jar:../lib/commons-exec-1.3.jar:../lib/commons-fileupload-1.3.3.jar:../lib/commons-io-2.5.jar:../lib/commons-lang-2.6.jar:../lib/commons-lang3-3.6.jar:../lib/commons-logging-1.2.jar:../lib/ecj-4.3.1.jar:../lib/gson-2.8.0.jar:../lib/guava-25.1-jre.jar:../lib/httpclient-4.5.6.jar:../lib/httpcore-4.4.10.jar:../lib/jasper-6.0.35.jar:../lib/jasper-el-6.0.35.jar:../lib/javax.servlet-api-3.1.0.jar:../lib/jna-4.3.0.jar:../lib/jna-platform-4.3.0.jar:../lib/json-simple-1.1.1.jar:../lib/jsp-api-2.1-glassfish-2.1.v20091210.jar:../lib/juli-6.0.35.jar:../lib/log4j-1.2-api-2.4.1.jar:../lib/log4j-api-2.4.1.jar:../lib/log4j-core-2.4.1.jar:../lib/mail-1.4.5.jar:../lib/serializer-2.7.1.jar:../lib/slf4j-api-1.7.25.jar:../lib/slf4j-simple-1.7.25.jar:../lib/velocity-1.7.jar:../lib/xalan-2.7.1.jar:../lib/xercesImpl-2.10.0.jar:../lib/xml-apis-1.4.01.jar:../lib/zookeeper-3.4.10.jar:../lib-proprietary/jtds-1.2.4.jar:../lib-proprietary/mariadb-java-client-1.1.7.jar:../lib-proprietary/mysql-connector-java-5.1.33.jar: -Dorg.apache.manifoldcf.processid=A org.apache.manifoldcf.agents.AgentRun
So I am able to heap the pid 1371 !
sudo -u tomcat jstack -l 1371 > /tmp/jstack_agent.log
I attached to this mail
Thanks a lot, Karl
Da: Bisonti Mario <Ma...@vimar.com>>
Inviato: giovedì 29 novembre 2018 11:12
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: R: Job stuck without message
Thanks a lot for your big help, Karl.
I would like to give you the dump but in my system (ubuntu 18.04) I am not able to dump the process of the start-agent.sh
I tried but I always obtain:
sudo -u tomcat jhsdb jinfo --pid 1233
Attaching to process ID 1233, please wait...
Error attaching to process: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMTypes" in remote process)
sun.jvm.hotspot.debugger.DebuggerException: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMTypes" in remote process)
at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupVM(HotSpotAgent.java:413)
at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:306)
at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:141)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.JInfo.runWithArgs(JInfo.java:139)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.JInfo.main(JInfo.java:144)
at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJINFO(SALauncher.java:364)
at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:460)
sudo -u tomcat jstack -l 1233
1233: Unable to open socket file /proc/1233/cwd/.attach_pid1233: target process 1233 doesn't respond within 10500ms or HotSpot VM not loaded
Da: Karl Wright <da...@gmail.com>>
Inviato: mercoledì 28 novembre 2018 21:52
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
The database row indicates there is no reason that the document should not be queued and processed.
As for getting a thread dump, there's a "force" option (-F).
The only other reason stuff may not run is if the query plan for identifying documents to process has gone horribly wrong. We should see that in the thread dump however.
I will unfortunately need to be offline for the next 24 hours due to an emergency situation, but if it turns out that your agents process is busy executing a long-running query, then I suggest analyzing the jobqueue table to get a better plan. This happens automatically but there are conditions under which it doesn't happen frequently enough. If the job is waiting for locks, then the stack trace will tell me where.
Thanks,
Karl
Karl
On Wed, Nov 28, 2018 at 11:07 AM Bisonti Mario <Ma...@vimar.com>> wrote:
I attatched a row that correspond to a row of one of these documents in this mail
I obtain the pid of:
"/bin/bash -e /opt/manifoldcf/multiprocess-zk-example-proprietary/start-agents.sh"
The pid is 1233
I tried to use
sudo jstack -l 1233 > /tmp/jstack_start_agent.log
but I obtain:
1233: Unable to open socket file /proc/1233/cwd/.attach_pid1233: target process 1233 doesn't respond within 10500ms or HotSpot VM not loaded
Perhaps isn’t it the right way to obtain a thread dump?
Excuse me but I am not a Linux expert..
Da: Karl Wright <da...@gmail.com>>
Inviato: mercoledì 28 novembre 2018 16:36
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
Another thing you could do is get a thread dump of the agents process.
Karl
On Wed, Nov 28, 2018 at 10:35 AM Karl Wright <da...@gmail.com>> wrote:
Can you look into the database jobqueue table and provide a row that corresponds to one of these documents?
Thanks,
Karl
On Wed, Nov 28, 2018 at 10:26 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo.
Repository has Max connection=10
In the Document Status report” I see many item with :
State=“Not yet processed”
Status=”Ready for processing”
Scheduled=01-01-1970 01:00:00.000”
Scheduled Action=”Process”
But the job no more walk..
Da: Karl Wright <da...@gmail.com>>
Inviato: mercoledì 28 novembre 2018 16:03
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
"Pipe instances are busy" occurs because you are overloading the SMB access to your servers. How many connections do you have allocated for your repository connection? You probably want to limit this to 2-3 if you see this error a lot, and it appears you do.
" Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity" means the document is not properly formed XML. The rejection will mean the document isn't indexed, but this will not stop the job.
If nothing is happening and you don't know why, I'd suggest looking at the Document Status report to figure out what documents are not being processed and why. It is quite possible they are all in the process of being retried because of the "Pipe instances" issue above.
Karl
On Wed, Nov 28, 2018 at 9:46 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo Karl.
I take this ticket because,now, after I use zookeeper, my job works for 7 hours and now it is in hang status.
I see running but it seems hanging, no log from 1 hour
This is the last manifoldcf.log lines:
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:21,524 (Worker thread '59') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:25,232 (Worker thread '45') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-11-28T14:46:25,277 (Worker thread '70') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-11-28T14:46:28,825 (Worker thread '27') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:40,473 (Worker thread '79') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:54,855 (Worker thread '65') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:55,175 (Worker thread '83') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
I don’t know what to check.
Tika server is ok, and it doesn’t more restart it .
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 6 novembre 2018 15:27
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
I added a couple of questions to the ticket. Please reply.
Thanks,
Karl
On Tue, Nov 6, 2018 at 8:56 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot, Karl.
I created a ticket.
https://issues.apache.org/jira/browse/CONNECTORS-1554<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCONNECTORS-1554&data=01%7C01%7CMario.Bisonti%40vimar.com%7C98877f463c1a46a299e408d655ebce48%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=zaDTmL7D5m9znylm%2FIQDy857SEd8TERvjSLCjDfDjik%3D&reserved=0>
Thanks
Mario
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 6 novembre 2018 14:28
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
ok, can you create a ticket? Also, I'd appreciate it if you can look at the simple history for one of these documents; I need to see what happened to it last.
Thanks,
Karl
On Tue, Nov 6, 2018 at 7:32 AM Bisonti Mario <Ma...@vimar.com>> wrote:
My version is 2.11
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 6 novembre 2018 13:07
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
Thanks.
What version of ManifoldCF are you using? We fixed a problem a while back having to do with documents that (because of error processing) get put into a "ready for processing" state which don't have any document priority set. But this should have been addressed, certainly, by the most recent release and probably by 2.10 as well.
Karl
On Tue, Nov 6, 2018 at 5:43 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo Karl.
When it hangs I see in the Queue status:
And in the Document Status:
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 19:32
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
What I am interested in now is the Document Status report for any one of the documents that is 'stuck'. The next crawl time value is the critical field. Can you include an example?
Karl
On Tue, Oct 30, 2018, 12:36 PM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot, Karl.
It happens that the job starts, it works and index for an hour and after it frezzes, I haven’t error or waiting status in Document Queue o Simple History, I have only “OK” status so, I haven’t failures.
I am not able to see other log errors other from the manifoldcf.log
Solr server is ok
Tika server is ok
Agent is ok
Tomcat with ManifoldCF is ok
I could search if I could to put in info log mode for example Tika servrer or Solr.
Thanks..
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 16:38
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
Hi Mario,
Please look at the Queue Status report to determine what is waiting and why it is waiting.
You can also look at the Simple History to see what has been happening. If you are getting 100% failures in fetching documents then you may need to address this because your infrastructure is unhappy. If the failure is something that indicates that the document is never going to be readable, that's a different problem and we might need to address that in the connector.
Karl
On Tue, Oct 30, 2018 at 10:33 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot Karl
Yes, I see many docs in the docs queue but they are inactive.
Infact i see that no more docs are indexed in Solr and I see that job is with the same number of docs Active (35012)
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 13:59
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
The reason the job is "stuck" is because:
' JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.'
This means that ManifoldCF will retry this document for a while before it gives up on it. It appears to be stuck but it is not. You can verify that by looking at the Document Queue report to see what is queued and what times the various documents will be retried.
Karl
On Tue, Oct 30, 2018 at 5:07 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo.
I started a job that works for some minutes, and after it stucks.
In the manifoldcf.log I see:
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-10-30T09:21:31,440 (Worker thread '2') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:33,502 (Worker thread '14') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:37,725 (Worker thread '30') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:44,406 (Worker thread '49') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:47,310 (Worker thread '15') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:52,000 (Worker thread '27') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:53,526 (Worker thread '15') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:04,511 (Worker thread '3') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2438) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1221) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-10-30T09:22:10,359 (Worker thread '27') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:13,932 (Worker thread '12') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:14,274 (Worker thread '23') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:19,933 (Worker thread '8') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:23:59,920 (Worker thread '39') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:24:09,059 (Worker thread '43') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
What could I check?
Tika server works as standalone.
Could you help me?
Thanks a lot
Mario
R: Job stuck without message
Posted by Bisonti Mario <Ma...@vimar.com>.
*Update
I found that pid 1371 is the agent pid
I used the command, to retrieve this:
ps -Flww -p 1371
F S UID PID PPID C PRI NI ADDR SZ WCHAN RSS PSR STIME TTY TIME CMD
0 S tomcat 1371 1282 11 80 0 - 2082955 - 2621680 7 Nov27 ? 04:40:54 /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Xms2048m -Xmx2048m -Dorg.apache.manifoldcf.configfile=./properties.xml -cp .:../lib/mcf-core.jar:../lib/mcf-agents.jar:../lib/mcf-pull-agent.jar:../lib/hsqldb-2.3.2.jar:../lib/postgresql-42.1.3.jar:../lib/commons-codec-1.10.jar:../lib/commons-collections-3.2.1.jar:../lib/commons-collections4-4.1.jar:../lib/commons-discovery-0.5.jar:../lib/commons-el-1.0.jar:../lib/commons-exec-1.3.jar:../lib/commons-fileupload-1.3.3.jar:../lib/commons-io-2.5.jar:../lib/commons-lang-2.6.jar:../lib/commons-lang3-3.6.jar:../lib/commons-logging-1.2.jar:../lib/ecj-4.3.1.jar:../lib/gson-2.8.0.jar:../lib/guava-25.1-jre.jar:../lib/httpclient-4.5.6.jar:../lib/httpcore-4.4.10.jar:../lib/jasper-6.0.35.jar:../lib/jasper-el-6.0.35.jar:../lib/javax.servlet-api-3.1.0.jar:../lib/jna-4.3.0.jar:../lib/jna-platform-4.3.0.jar:../lib/json-simple-1.1.1.jar:../lib/jsp-api-2.1-glassfish-2.1.v20091210.jar:../lib/juli-6.0.35.jar:../lib/log4j-1.2-api-2.4.1.jar:../lib/log4j-api-2.4.1.jar:../lib/log4j-core-2.4.1.jar:../lib/mail-1.4.5.jar:../lib/serializer-2.7.1.jar:../lib/slf4j-api-1.7.25.jar:../lib/slf4j-simple-1.7.25.jar:../lib/velocity-1.7.jar:../lib/xalan-2.7.1.jar:../lib/xercesImpl-2.10.0.jar:../lib/xml-apis-1.4.01.jar:../lib/zookeeper-3.4.10.jar:../lib-proprietary/jtds-1.2.4.jar:../lib-proprietary/mariadb-java-client-1.1.7.jar:../lib-proprietary/mysql-connector-java-5.1.33.jar: -Dorg.apache.manifoldcf.processid=A org.apache.manifoldcf.agents.AgentRun
So I am able to heap the pid 1371 !
sudo -u tomcat jstack -l 1371 > /tmp/jstack_agent.log
I attached to this mail
Thanks a lot, Karl
Da: Bisonti Mario <Ma...@vimar.com>
Inviato: giovedì 29 novembre 2018 11:12
A: user@manifoldcf.apache.org
Oggetto: R: Job stuck without message
Thanks a lot for your big help, Karl.
I would like to give you the dump but in my system (ubuntu 18.04) I am not able to dump the process of the start-agent.sh
I tried but I always obtain:
sudo -u tomcat jhsdb jinfo --pid 1233
Attaching to process ID 1233, please wait...
Error attaching to process: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMTypes" in remote process)
sun.jvm.hotspot.debugger.DebuggerException: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMTypes" in remote process)
at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupVM(HotSpotAgent.java:413)
at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:306)
at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:141)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.JInfo.runWithArgs(JInfo.java:139)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.JInfo.main(JInfo.java:144)
at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJINFO(SALauncher.java:364)
at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:460)
sudo -u tomcat jstack -l 1233
1233: Unable to open socket file /proc/1233/cwd/.attach_pid1233: target process 1233 doesn't respond within 10500ms or HotSpot VM not loaded
Da: Karl Wright <da...@gmail.com>>
Inviato: mercoledì 28 novembre 2018 21:52
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
The database row indicates there is no reason that the document should not be queued and processed.
As for getting a thread dump, there's a "force" option (-F).
The only other reason stuff may not run is if the query plan for identifying documents to process has gone horribly wrong. We should see that in the thread dump however.
I will unfortunately need to be offline for the next 24 hours due to an emergency situation, but if it turns out that your agents process is busy executing a long-running query, then I suggest analyzing the jobqueue table to get a better plan. This happens automatically but there are conditions under which it doesn't happen frequently enough. If the job is waiting for locks, then the stack trace will tell me where.
Thanks,
Karl
Karl
On Wed, Nov 28, 2018 at 11:07 AM Bisonti Mario <Ma...@vimar.com>> wrote:
I attatched a row that correspond to a row of one of these documents in this mail
I obtain the pid of:
"/bin/bash -e /opt/manifoldcf/multiprocess-zk-example-proprietary/start-agents.sh"
The pid is 1233
I tried to use
sudo jstack -l 1233 > /tmp/jstack_start_agent.log
but I obtain:
1233: Unable to open socket file /proc/1233/cwd/.attach_pid1233: target process 1233 doesn't respond within 10500ms or HotSpot VM not loaded
Perhaps isn’t it the right way to obtain a thread dump?
Excuse me but I am not a Linux expert..
Da: Karl Wright <da...@gmail.com>>
Inviato: mercoledì 28 novembre 2018 16:36
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
Another thing you could do is get a thread dump of the agents process.
Karl
On Wed, Nov 28, 2018 at 10:35 AM Karl Wright <da...@gmail.com>> wrote:
Can you look into the database jobqueue table and provide a row that corresponds to one of these documents?
Thanks,
Karl
On Wed, Nov 28, 2018 at 10:26 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo.
Repository has Max connection=10
In the Document Status report” I see many item with :
State=“Not yet processed”
Status=”Ready for processing”
Scheduled=01-01-1970 01:00:00.000”
Scheduled Action=”Process”
But the job no more walk..
Da: Karl Wright <da...@gmail.com>>
Inviato: mercoledì 28 novembre 2018 16:03
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
"Pipe instances are busy" occurs because you are overloading the SMB access to your servers. How many connections do you have allocated for your repository connection? You probably want to limit this to 2-3 if you see this error a lot, and it appears you do.
" Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity" means the document is not properly formed XML. The rejection will mean the document isn't indexed, but this will not stop the job.
If nothing is happening and you don't know why, I'd suggest looking at the Document Status report to figure out what documents are not being processed and why. It is quite possible they are all in the process of being retried because of the "Pipe instances" issue above.
Karl
On Wed, Nov 28, 2018 at 9:46 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo Karl.
I take this ticket because,now, after I use zookeeper, my job works for 7 hours and now it is in hang status.
I see running but it seems hanging, no log from 1 hour
This is the last manifoldcf.log lines:
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:21,524 (Worker thread '59') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:25,232 (Worker thread '45') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-11-28T14:46:25,277 (Worker thread '70') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-11-28T14:46:28,825 (Worker thread '27') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:40,473 (Worker thread '79') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:54,855 (Worker thread '65') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:55,175 (Worker thread '83') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
I don’t know what to check.
Tika server is ok, and it doesn’t more restart it .
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 6 novembre 2018 15:27
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
I added a couple of questions to the ticket. Please reply.
Thanks,
Karl
On Tue, Nov 6, 2018 at 8:56 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot, Karl.
I created a ticket.
https://issues.apache.org/jira/browse/CONNECTORS-1554<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCONNECTORS-1554&data=01%7C01%7CMario.Bisonti%40vimar.com%7C6f21ea8e17944ff0dc6408d655e32269%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=bJxt11iMgrp6JE6OpzuVJUWs%2BhA8N3kSNkV8Z3wIngk%3D&reserved=0>
Thanks
Mario
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 6 novembre 2018 14:28
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
ok, can you create a ticket? Also, I'd appreciate it if you can look at the simple history for one of these documents; I need to see what happened to it last.
Thanks,
Karl
On Tue, Nov 6, 2018 at 7:32 AM Bisonti Mario <Ma...@vimar.com>> wrote:
My version is 2.11
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 6 novembre 2018 13:07
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
Thanks.
What version of ManifoldCF are you using? We fixed a problem a while back having to do with documents that (because of error processing) get put into a "ready for processing" state which don't have any document priority set. But this should have been addressed, certainly, by the most recent release and probably by 2.10 as well.
Karl
On Tue, Nov 6, 2018 at 5:43 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo Karl.
When it hangs I see in the Queue status:
And in the Document Status:
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 19:32
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
What I am interested in now is the Document Status report for any one of the documents that is 'stuck'. The next crawl time value is the critical field. Can you include an example?
Karl
On Tue, Oct 30, 2018, 12:36 PM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot, Karl.
It happens that the job starts, it works and index for an hour and after it frezzes, I haven’t error or waiting status in Document Queue o Simple History, I have only “OK” status so, I haven’t failures.
I am not able to see other log errors other from the manifoldcf.log
Solr server is ok
Tika server is ok
Agent is ok
Tomcat with ManifoldCF is ok
I could search if I could to put in info log mode for example Tika servrer or Solr.
Thanks..
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 16:38
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
Hi Mario,
Please look at the Queue Status report to determine what is waiting and why it is waiting.
You can also look at the Simple History to see what has been happening. If you are getting 100% failures in fetching documents then you may need to address this because your infrastructure is unhappy. If the failure is something that indicates that the document is never going to be readable, that's a different problem and we might need to address that in the connector.
Karl
On Tue, Oct 30, 2018 at 10:33 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot Karl
Yes, I see many docs in the docs queue but they are inactive.
Infact i see that no more docs are indexed in Solr and I see that job is with the same number of docs Active (35012)
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 13:59
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
The reason the job is "stuck" is because:
' JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.'
This means that ManifoldCF will retry this document for a while before it gives up on it. It appears to be stuck but it is not. You can verify that by looking at the Document Queue report to see what is queued and what times the various documents will be retried.
Karl
On Tue, Oct 30, 2018 at 5:07 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo.
I started a job that works for some minutes, and after it stucks.
In the manifoldcf.log I see:
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-10-30T09:21:31,440 (Worker thread '2') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:33,502 (Worker thread '14') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:37,725 (Worker thread '30') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:44,406 (Worker thread '49') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:47,310 (Worker thread '15') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:52,000 (Worker thread '27') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:53,526 (Worker thread '15') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:04,511 (Worker thread '3') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2438) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1221) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-10-30T09:22:10,359 (Worker thread '27') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:13,932 (Worker thread '12') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:14,274 (Worker thread '23') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:19,933 (Worker thread '8') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:23:59,920 (Worker thread '39') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:24:09,059 (Worker thread '43') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
What could I check?
Tika server works as standalone.
Could you help me?
Thanks a lot
Mario
R: Job stuck without message
Posted by Bisonti Mario <Ma...@vimar.com>.
Thanks a lot for your big help, Karl.
I would like to give you the dump but in my system (ubuntu 18.04) I am not able to dump the process of the start-agent.sh
I tried but I always obtain:
sudo -u tomcat jhsdb jinfo --pid 1233
Attaching to process ID 1233, please wait...
Error attaching to process: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMTypes" in remote process)
sun.jvm.hotspot.debugger.DebuggerException: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMTypes" in remote process)
at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupVM(HotSpotAgent.java:413)
at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:306)
at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:141)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.JInfo.runWithArgs(JInfo.java:139)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.JInfo.main(JInfo.java:144)
at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJINFO(SALauncher.java:364)
at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:460)
sudo -u tomcat jstack -l 1233
1233: Unable to open socket file /proc/1233/cwd/.attach_pid1233: target process 1233 doesn't respond within 10500ms or HotSpot VM not loaded
Da: Karl Wright <da...@gmail.com>
Inviato: mercoledì 28 novembre 2018 21:52
A: user@manifoldcf.apache.org
Oggetto: Re: Job stuck without message
The database row indicates there is no reason that the document should not be queued and processed.
As for getting a thread dump, there's a "force" option (-F).
The only other reason stuff may not run is if the query plan for identifying documents to process has gone horribly wrong. We should see that in the thread dump however.
I will unfortunately need to be offline for the next 24 hours due to an emergency situation, but if it turns out that your agents process is busy executing a long-running query, then I suggest analyzing the jobqueue table to get a better plan. This happens automatically but there are conditions under which it doesn't happen frequently enough. If the job is waiting for locks, then the stack trace will tell me where.
Thanks,
Karl
Karl
On Wed, Nov 28, 2018 at 11:07 AM Bisonti Mario <Ma...@vimar.com>> wrote:
I attatched a row that correspond to a row of one of these documents in this mail
I obtain the pid of:
"/bin/bash -e /opt/manifoldcf/multiprocess-zk-example-proprietary/start-agents.sh"
The pid is 1233
I tried to use
sudo jstack -l 1233 > /tmp/jstack_start_agent.log
but I obtain:
1233: Unable to open socket file /proc/1233/cwd/.attach_pid1233: target process 1233 doesn't respond within 10500ms or HotSpot VM not loaded
Perhaps isn’t it the right way to obtain a thread dump?
Excuse me but I am not a Linux expert..
Da: Karl Wright <da...@gmail.com>>
Inviato: mercoledì 28 novembre 2018 16:36
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
Another thing you could do is get a thread dump of the agents process.
Karl
On Wed, Nov 28, 2018 at 10:35 AM Karl Wright <da...@gmail.com>> wrote:
Can you look into the database jobqueue table and provide a row that corresponds to one of these documents?
Thanks,
Karl
On Wed, Nov 28, 2018 at 10:26 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo.
Repository has Max connection=10
In the Document Status report” I see many item with :
State=“Not yet processed”
Status=”Ready for processing”
Scheduled=01-01-1970 01:00:00.000”
Scheduled Action=”Process”
But the job no more walk..
Da: Karl Wright <da...@gmail.com>>
Inviato: mercoledì 28 novembre 2018 16:03
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
"Pipe instances are busy" occurs because you are overloading the SMB access to your servers. How many connections do you have allocated for your repository connection? You probably want to limit this to 2-3 if you see this error a lot, and it appears you do.
" Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity" means the document is not properly formed XML. The rejection will mean the document isn't indexed, but this will not stop the job.
If nothing is happening and you don't know why, I'd suggest looking at the Document Status report to figure out what documents are not being processed and why. It is quite possible they are all in the process of being retried because of the "Pipe instances" issue above.
Karl
On Wed, Nov 28, 2018 at 9:46 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo Karl.
I take this ticket because,now, after I use zookeeper, my job works for 7 hours and now it is in hang status.
I see running but it seems hanging, no log from 1 hour
This is the last manifoldcf.log lines:
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:21,524 (Worker thread '59') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:25,232 (Worker thread '45') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-11-28T14:46:25,277 (Worker thread '70') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-11-28T14:46:28,825 (Worker thread '27') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:40,473 (Worker thread '79') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:54,855 (Worker thread '65') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:55,175 (Worker thread '83') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
I don’t know what to check.
Tika server is ok, and it doesn’t more restart it .
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 6 novembre 2018 15:27
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
I added a couple of questions to the ticket. Please reply.
Thanks,
Karl
On Tue, Nov 6, 2018 at 8:56 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot, Karl.
I created a ticket.
https://issues.apache.org/jira/browse/CONNECTORS-1554<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCONNECTORS-1554&data=01%7C01%7CMario.Bisonti%40vimar.com%7Cbba19ce1456745bc9f0608d655736961%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=kGyBdfifrb1%2FCmk%2BeEuBqWY7EC%2BNh1btRzuiMNjd8a8%3D&reserved=0>
Thanks
Mario
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 6 novembre 2018 14:28
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
ok, can you create a ticket? Also, I'd appreciate it if you can look at the simple history for one of these documents; I need to see what happened to it last.
Thanks,
Karl
On Tue, Nov 6, 2018 at 7:32 AM Bisonti Mario <Ma...@vimar.com>> wrote:
My version is 2.11
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 6 novembre 2018 13:07
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
Thanks.
What version of ManifoldCF are you using? We fixed a problem a while back having to do with documents that (because of error processing) get put into a "ready for processing" state which don't have any document priority set. But this should have been addressed, certainly, by the most recent release and probably by 2.10 as well.
Karl
On Tue, Nov 6, 2018 at 5:43 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo Karl.
When it hangs I see in the Queue status:
And in the Document Status:
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 19:32
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
What I am interested in now is the Document Status report for any one of the documents that is 'stuck'. The next crawl time value is the critical field. Can you include an example?
Karl
On Tue, Oct 30, 2018, 12:36 PM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot, Karl.
It happens that the job starts, it works and index for an hour and after it frezzes, I haven’t error or waiting status in Document Queue o Simple History, I have only “OK” status so, I haven’t failures.
I am not able to see other log errors other from the manifoldcf.log
Solr server is ok
Tika server is ok
Agent is ok
Tomcat with ManifoldCF is ok
I could search if I could to put in info log mode for example Tika servrer or Solr.
Thanks..
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 16:38
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
Hi Mario,
Please look at the Queue Status report to determine what is waiting and why it is waiting.
You can also look at the Simple History to see what has been happening. If you are getting 100% failures in fetching documents then you may need to address this because your infrastructure is unhappy. If the failure is something that indicates that the document is never going to be readable, that's a different problem and we might need to address that in the connector.
Karl
On Tue, Oct 30, 2018 at 10:33 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot Karl
Yes, I see many docs in the docs queue but they are inactive.
Infact i see that no more docs are indexed in Solr and I see that job is with the same number of docs Active (35012)
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 13:59
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
The reason the job is "stuck" is because:
' JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.'
This means that ManifoldCF will retry this document for a while before it gives up on it. It appears to be stuck but it is not. You can verify that by looking at the Document Queue report to see what is queued and what times the various documents will be retried.
Karl
On Tue, Oct 30, 2018 at 5:07 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo.
I started a job that works for some minutes, and after it stucks.
In the manifoldcf.log I see:
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-10-30T09:21:31,440 (Worker thread '2') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:33,502 (Worker thread '14') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:37,725 (Worker thread '30') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:44,406 (Worker thread '49') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:47,310 (Worker thread '15') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:52,000 (Worker thread '27') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:53,526 (Worker thread '15') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:04,511 (Worker thread '3') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2438) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1221) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-10-30T09:22:10,359 (Worker thread '27') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:13,932 (Worker thread '12') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:14,274 (Worker thread '23') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:19,933 (Worker thread '8') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:23:59,920 (Worker thread '39') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:24:09,059 (Worker thread '43') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
What could I check?
Tika server works as standalone.
Could you help me?
Thanks a lot
Mario
Re: Job stuck without message
Posted by Karl Wright <da...@gmail.com>.
The database row indicates there is no reason that the document should not
be queued and processed.
As for getting a thread dump, there's a "force" option (-F).
The only other reason stuff may not run is if the query plan for
identifying documents to process has gone horribly wrong. We should see
that in the thread dump however.
I will unfortunately need to be offline for the next 24 hours due to an
emergency situation, but if it turns out that your agents process is busy
executing a long-running query, then I suggest analyzing the jobqueue table
to get a better plan. This happens automatically but there are conditions
under which it doesn't happen frequently enough. If the job is waiting for
locks, then the stack trace will tell me where.
Thanks,
Karl
Karl
On Wed, Nov 28, 2018 at 11:07 AM Bisonti Mario <Ma...@vimar.com>
wrote:
> I attatched a row that correspond to a row of one of these documents in
> this mail
>
>
>
>
>
>
>
> I obtain the pid of:
> "/bin/bash -e
> /opt/manifoldcf/multiprocess-zk-example-proprietary/start-agents.sh"
>
> The pid is 1233
>
>
>
> I tried to use
>
> sudo jstack -l 1233 > /tmp/jstack_start_agent.log
>
>
>
> but I obtain:
>
> 1233: Unable to open socket file /proc/1233/cwd/.attach_pid1233: target
> process 1233 doesn't respond within 10500ms or HotSpot VM not loaded
>
>
>
> Perhaps isn’t it the right way to obtain a thread dump?
>
> Excuse me but I am not a Linux expert..
>
>
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* mercoledì 28 novembre 2018 16:36
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> Another thing you could do is get a thread dump of the agents process.
>
>
>
> Karl
>
>
>
>
>
> On Wed, Nov 28, 2018 at 10:35 AM Karl Wright <da...@gmail.com> wrote:
>
> Can you look into the database jobqueue table and provide a row that
> corresponds to one of these documents?
>
>
>
> Thanks,
>
> Karl
>
>
>
>
>
> On Wed, Nov 28, 2018 at 10:26 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo.
>
> Repository has Max connection=10
>
>
>
> In the Document Status report” I see many item with :
>
> State=“Not yet processed”
>
> Status=”Ready for processing”
>
> Scheduled=01-01-1970 01:00:00.000”
>
> Scheduled Action=”Process”
>
>
>
>
>
>
>
>
>
> But the job no more walk..
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* mercoledì 28 novembre 2018 16:03
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> "Pipe instances are busy" occurs because you are overloading the SMB
> access to your servers. How many connections do you have allocated for
> your repository connection? You probably want to limit this to 2-3 if you
> see this error a lot, and it appears you do.
>
> " Tika Server: Tika Server rejects: Tika Server rejected document with
> the following reason: Unprocessable Entity" means the document is not
> properly formed XML. The rejection will mean the document isn't indexed,
> but this will not stop the job.
>
> If nothing is happening and you don't know why, I'd suggest looking at the
> Document Status report to figure out what documents are not being processed
> and why. It is quite possible they are all in the process of being retried
> because of the "Pipe instances" issue above.
>
>
>
> Karl
>
>
>
> On Wed, Nov 28, 2018 at 9:46 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo Karl.
>
> I take this ticket because,now, after I use zookeeper, my job works for 7
> hours and now it is in hang status.
>
> I see running but it seems hanging, no log from 1 hour
>
>
>
> This is the last manifoldcf.log lines:
>
>
>
>
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:21,524 (Worker thread '59') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:25,232 (Worker thread '45') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-11-28T14:46:25,277 (Worker thread '70') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-11-28T14:46:28,825 (Worker thread '27') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:40,473 (Worker thread '79') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:54,855 (Worker thread '65') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:55,175 (Worker thread '83') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
>
>
>
>
>
>
> I don’t know what to check.
>
>
>
> Tika server is ok, and it doesn’t more restart it .
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 6 novembre 2018 15:27
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> I added a couple of questions to the ticket. Please reply.
>
>
>
> Thanks,
>
> Karl
>
>
>
>
>
> On Tue, Nov 6, 2018 at 8:56 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Thanks a lot, Karl.
>
> I created a ticket.
>
> https://issues.apache.org/jira/browse/CONNECTORS-1554
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCONNECTORS-1554&data=01%7C01%7CMario.Bisonti%40vimar.com%7C04a2b2ff508d4ac0254908d655473c51%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=xFzROisXXTNNBkGZwOE3%2F4ypIROkxz82CgW%2BvS%2Bwqgw%3D&reserved=0>
>
>
>
>
>
> Thanks
>
>
>
> Mario
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 6 novembre 2018 14:28
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> ok, can you create a ticket? Also, I'd appreciate it if you can look at
> the simple history for one of these documents; I need to see what happened
> to it last.
>
>
>
> Thanks,
>
> Karl
>
>
>
>
>
> On Tue, Nov 6, 2018 at 7:32 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> My version is 2.11
>
>
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 6 novembre 2018 13:07
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> Thanks.
>
> What version of ManifoldCF are you using? We fixed a problem a while back
> having to do with documents that (because of error processing) get put into
> a "ready for processing" state which don't have any document priority set.
> But this should have been addressed, certainly, by the most recent release
> and probably by 2.10 as well.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Nov 6, 2018 at 5:43 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo Karl.
>
> When it hangs I see in the Queue status:
>
>
>
> And in the Document Status:
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 19:32
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> What I am interested in now is the Document Status report for any one of
> the documents that is 'stuck'. The next crawl time value is the critical
> field. Can you include an example?
>
>
>
> Karl
>
>
>
> On Tue, Oct 30, 2018, 12:36 PM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Thanks a lot, Karl.
>
>
>
> It happens that the job starts, it works and index for an hour and after
> it frezzes, I haven’t error or waiting status in Document Queue o Simple
> History, I have only “OK” status so, I haven’t failures.
>
>
>
> I am not able to see other log errors other from the manifoldcf.log
>
>
>
> Solr server is ok
>
> Tika server is ok
>
> Agent is ok
>
> Tomcat with ManifoldCF is ok
>
>
>
> I could search if I could to put in info log mode for example Tika servrer
> or Solr.
>
>
>
> Thanks..
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 16:38
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> Hi Mario,
>
> Please look at the Queue Status report to determine what is waiting and
> why it is waiting.
> You can also look at the Simple History to see what has been happening.
> If you are getting 100% failures in fetching documents then you may need to
> address this because your infrastructure is unhappy. If the failure is
> something that indicates that the document is never going to be readable,
> that's a different problem and we might need to address that in the
> connector.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Oct 30, 2018 at 10:33 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
>
>
> Thanks a lot Karl
>
>
>
> Yes, I see many docs in the docs queue but they are inactive.
>
>
>
> Infact i see that no more docs are indexed in Solr and I see that job is
> with the same number of docs Active (35012)
>
>
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 13:59
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> The reason the job is "stuck" is because:
>
> ' JCIFS: Possibly transient exception detected on attempt 1 while getting
> share security: All pipe instances are busy.'
>
> This means that ManifoldCF will retry this document for a while before it
> gives up on it. It appears to be stuck but it is not. You can verify that
> by looking at the Document Queue report to see what is queued and what
> times the various documents will be retried.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Oct 30, 2018 at 5:07 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo.
>
>
>
> I started a job that works for some minutes, and after it stucks.
>
>
>
> In the manifoldcf.log I see:
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-10-30T09:21:31,440 (Worker thread '2') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:33,502 (Worker thread '14') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:37,725 (Worker thread '30') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:44,406 (Worker thread '49') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:47,310 (Worker thread '15') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:52,000 (Worker thread '27') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:53,526 (Worker thread '15') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:04,511 (Worker thread '3') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2438)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1221)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-10-30T09:22:10,359 (Worker thread '27') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:13,932 (Worker thread '12') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:14,274 (Worker thread '23') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:19,933 (Worker thread '8') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:23:59,920 (Worker thread '39') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:24:09,059 (Worker thread '43') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
>
>
>
>
>
>
> What could I check?
>
>
>
> Tika server works as standalone.
>
>
>
> Could you help me?
>
>
>
> Thanks a lot
>
>
>
> Mario
>
>
>
>
R: Job stuck without message
Posted by Bisonti Mario <Ma...@vimar.com>.
I attatched a row that correspond to a row of one of these documents in this mail
I obtain the pid of:
"/bin/bash -e /opt/manifoldcf/multiprocess-zk-example-proprietary/start-agents.sh"
The pid is 1233
I tried to use
sudo jstack -l 1233 > /tmp/jstack_start_agent.log
but I obtain:
1233: Unable to open socket file /proc/1233/cwd/.attach_pid1233: target process 1233 doesn't respond within 10500ms or HotSpot VM not loaded
Perhaps isn’t it the right way to obtain a thread dump?
Excuse me but I am not a Linux expert..
Da: Karl Wright <da...@gmail.com>
Inviato: mercoledì 28 novembre 2018 16:36
A: user@manifoldcf.apache.org
Oggetto: Re: Job stuck without message
Another thing you could do is get a thread dump of the agents process.
Karl
On Wed, Nov 28, 2018 at 10:35 AM Karl Wright <da...@gmail.com>> wrote:
Can you look into the database jobqueue table and provide a row that corresponds to one of these documents?
Thanks,
Karl
On Wed, Nov 28, 2018 at 10:26 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo.
Repository has Max connection=10
In the Document Status report” I see many item with :
State=“Not yet processed”
Status=”Ready for processing”
Scheduled=01-01-1970 01:00:00.000”
Scheduled Action=”Process”
But the job no more walk..
Da: Karl Wright <da...@gmail.com>>
Inviato: mercoledì 28 novembre 2018 16:03
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
"Pipe instances are busy" occurs because you are overloading the SMB access to your servers. How many connections do you have allocated for your repository connection? You probably want to limit this to 2-3 if you see this error a lot, and it appears you do.
" Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity" means the document is not properly formed XML. The rejection will mean the document isn't indexed, but this will not stop the job.
If nothing is happening and you don't know why, I'd suggest looking at the Document Status report to figure out what documents are not being processed and why. It is quite possible they are all in the process of being retried because of the "Pipe instances" issue above.
Karl
On Wed, Nov 28, 2018 at 9:46 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo Karl.
I take this ticket because,now, after I use zookeeper, my job works for 7 hours and now it is in hang status.
I see running but it seems hanging, no log from 1 hour
This is the last manifoldcf.log lines:
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:21,524 (Worker thread '59') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:25,232 (Worker thread '45') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-11-28T14:46:25,277 (Worker thread '70') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-11-28T14:46:28,825 (Worker thread '27') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:40,473 (Worker thread '79') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:54,855 (Worker thread '65') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:55,175 (Worker thread '83') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
I don’t know what to check.
Tika server is ok, and it doesn’t more restart it .
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 6 novembre 2018 15:27
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
I added a couple of questions to the ticket. Please reply.
Thanks,
Karl
On Tue, Nov 6, 2018 at 8:56 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot, Karl.
I created a ticket.
https://issues.apache.org/jira/browse/CONNECTORS-1554<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCONNECTORS-1554&data=01%7C01%7CMario.Bisonti%40vimar.com%7C04a2b2ff508d4ac0254908d655473c51%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=xFzROisXXTNNBkGZwOE3%2F4ypIROkxz82CgW%2BvS%2Bwqgw%3D&reserved=0>
Thanks
Mario
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 6 novembre 2018 14:28
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
ok, can you create a ticket? Also, I'd appreciate it if you can look at the simple history for one of these documents; I need to see what happened to it last.
Thanks,
Karl
On Tue, Nov 6, 2018 at 7:32 AM Bisonti Mario <Ma...@vimar.com>> wrote:
My version is 2.11
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 6 novembre 2018 13:07
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
Thanks.
What version of ManifoldCF are you using? We fixed a problem a while back having to do with documents that (because of error processing) get put into a "ready for processing" state which don't have any document priority set. But this should have been addressed, certainly, by the most recent release and probably by 2.10 as well.
Karl
On Tue, Nov 6, 2018 at 5:43 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo Karl.
When it hangs I see in the Queue status:
And in the Document Status:
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 19:32
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
What I am interested in now is the Document Status report for any one of the documents that is 'stuck'. The next crawl time value is the critical field. Can you include an example?
Karl
On Tue, Oct 30, 2018, 12:36 PM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot, Karl.
It happens that the job starts, it works and index for an hour and after it frezzes, I haven’t error or waiting status in Document Queue o Simple History, I have only “OK” status so, I haven’t failures.
I am not able to see other log errors other from the manifoldcf.log
Solr server is ok
Tika server is ok
Agent is ok
Tomcat with ManifoldCF is ok
I could search if I could to put in info log mode for example Tika servrer or Solr.
Thanks..
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 16:38
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
Hi Mario,
Please look at the Queue Status report to determine what is waiting and why it is waiting.
You can also look at the Simple History to see what has been happening. If you are getting 100% failures in fetching documents then you may need to address this because your infrastructure is unhappy. If the failure is something that indicates that the document is never going to be readable, that's a different problem and we might need to address that in the connector.
Karl
On Tue, Oct 30, 2018 at 10:33 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot Karl
Yes, I see many docs in the docs queue but they are inactive.
Infact i see that no more docs are indexed in Solr and I see that job is with the same number of docs Active (35012)
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 13:59
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
The reason the job is "stuck" is because:
' JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.'
This means that ManifoldCF will retry this document for a while before it gives up on it. It appears to be stuck but it is not. You can verify that by looking at the Document Queue report to see what is queued and what times the various documents will be retried.
Karl
On Tue, Oct 30, 2018 at 5:07 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo.
I started a job that works for some minutes, and after it stucks.
In the manifoldcf.log I see:
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-10-30T09:21:31,440 (Worker thread '2') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:33,502 (Worker thread '14') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:37,725 (Worker thread '30') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:44,406 (Worker thread '49') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:47,310 (Worker thread '15') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:52,000 (Worker thread '27') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:53,526 (Worker thread '15') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:04,511 (Worker thread '3') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2438) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1221) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-10-30T09:22:10,359 (Worker thread '27') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:13,932 (Worker thread '12') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:14,274 (Worker thread '23') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:19,933 (Worker thread '8') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:23:59,920 (Worker thread '39') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:24:09,059 (Worker thread '43') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
What could I check?
Tika server works as standalone.
Could you help me?
Thanks a lot
Mario
Re: Job stuck without message
Posted by Karl Wright <da...@gmail.com>.
Another thing you could do is get a thread dump of the agents process.
Karl
On Wed, Nov 28, 2018 at 10:35 AM Karl Wright <da...@gmail.com> wrote:
> Can you look into the database jobqueue table and provide a row that
> corresponds to one of these documents?
>
> Thanks,
> Karl
>
>
> On Wed, Nov 28, 2018 at 10:26 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
>> Hallo.
>>
>> Repository has Max connection=10
>>
>>
>>
>> In the Document Status report” I see many item with :
>>
>> State=“Not yet processed”
>>
>> Status=”Ready for processing”
>>
>> Scheduled=01-01-1970 01:00:00.000”
>>
>> Scheduled Action=”Process”
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> But the job no more walk..
>>
>>
>>
>>
>>
>> *Da:* Karl Wright <da...@gmail.com>
>> *Inviato:* mercoledì 28 novembre 2018 16:03
>> *A:* user@manifoldcf.apache.org
>> *Oggetto:* Re: Job stuck without message
>>
>>
>>
>> "Pipe instances are busy" occurs because you are overloading the SMB
>> access to your servers. How many connections do you have allocated for
>> your repository connection? You probably want to limit this to 2-3 if you
>> see this error a lot, and it appears you do.
>>
>> " Tika Server: Tika Server rejects: Tika Server rejected document with
>> the following reason: Unprocessable Entity" means the document is not
>> properly formed XML. The rejection will mean the document isn't indexed,
>> but this will not stop the job.
>>
>> If nothing is happening and you don't know why, I'd suggest looking at
>> the Document Status report to figure out what documents are not being
>> processed and why. It is quite possible they are all in the process of
>> being retried because of the "Pipe instances" issue above.
>>
>>
>>
>> Karl
>>
>>
>>
>> On Wed, Nov 28, 2018 at 9:46 AM Bisonti Mario <Ma...@vimar.com>
>> wrote:
>>
>> Hallo Karl.
>>
>> I take this ticket because,now, after I use zookeeper, my job works for 7
>> hours and now it is in hang status.
>>
>> I see running but it seems hanging, no log from 1 hour
>>
>>
>>
>> This is the last manifoldcf.log lines:
>>
>>
>>
>>
>>
>> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
>> [mcf-jcifs-connector.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
>> [mcf-jcifs-connector.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
>> [mcf-jcifs-connector.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
>> [mcf-pull-agent.jar:?]
>>
>> WARN 2018-11-28T14:46:21,524 (Worker thread '59') - JCIFS: Possibly
>> transient exception detected on attempt 1 while getting share security: All
>> pipe instances are busy.
>>
>> jcifs.smb.SmbException: All pipe instances are busy.
>>
>> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbSession.send(SmbSession.java:238)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbTree.send(SmbTree.java:119)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbFile.send(SmbFile.java:776)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
>> [mcf-jcifs-connector.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
>> [mcf-jcifs-connector.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
>> [mcf-jcifs-connector.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
>> [mcf-pull-agent.jar:?]
>>
>> WARN 2018-11-28T14:46:25,232 (Worker thread '45') - Tika Server: Tika
>> Server rejects: Tika Server rejected document with the following reason:
>> Unprocessable Entity
>>
>> WARN 2018-11-28T14:46:25,277 (Worker thread '70') - Tika Server: Tika
>> Server rejects: Tika Server rejected document with the following reason:
>> Unprocessable Entity
>>
>> WARN 2018-11-28T14:46:28,825 (Worker thread '27') - JCIFS: Possibly
>> transient exception detected on attempt 1 while getting share security: All
>> pipe instances are busy.
>>
>> jcifs.smb.SmbException: All pipe instances are busy.
>>
>> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbSession.send(SmbSession.java:238)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbTree.send(SmbTree.java:119)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbFile.send(SmbFile.java:776)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
>> [mcf-jcifs-connector.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
>> [mcf-jcifs-connector.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
>> [mcf-jcifs-connector.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
>> [mcf-pull-agent.jar:?]
>>
>> WARN 2018-11-28T14:46:40,473 (Worker thread '79') - JCIFS: Possibly
>> transient exception detected on attempt 1 while getting share security: All
>> pipe instances are busy.
>>
>> jcifs.smb.SmbException: All pipe instances are busy.
>>
>> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbSession.send(SmbSession.java:238)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbTree.send(SmbTree.java:119)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbFile.send(SmbFile.java:776)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
>> [mcf-jcifs-connector.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
>> [mcf-jcifs-connector.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
>> [mcf-jcifs-connector.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
>> [mcf-pull-agent.jar:?]
>>
>> WARN 2018-11-28T14:46:54,855 (Worker thread '65') - JCIFS: Possibly
>> transient exception detected on attempt 1 while getting share security: All
>> pipe instances are busy.
>>
>> jcifs.smb.SmbException: All pipe instances are busy.
>>
>> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbSession.send(SmbSession.java:238)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbTree.send(SmbTree.java:119)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbFile.send(SmbFile.java:776)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
>> [mcf-jcifs-connector.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
>> [mcf-jcifs-connector.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
>> [mcf-jcifs-connector.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
>> [mcf-pull-agent.jar:?]
>>
>> WARN 2018-11-28T14:46:55,175 (Worker thread '83') - JCIFS: Possibly
>> transient exception detected on attempt 1 while getting share security: All
>> pipe instances are busy.
>>
>> jcifs.smb.SmbException: All pipe instances are busy.
>>
>> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbSession.send(SmbSession.java:238)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbTree.send(SmbTree.java:119)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbFile.send(SmbFile.java:776)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
>> [mcf-jcifs-connector.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
>> [mcf-jcifs-connector.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
>> [mcf-jcifs-connector.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
>> [mcf-pull-agent.jar:?]
>>
>>
>>
>>
>>
>>
>>
>> I don’t know what to check.
>>
>>
>>
>> Tika server is ok, and it doesn’t more restart it .
>>
>>
>>
>>
>>
>>
>>
>> *Da:* Karl Wright <da...@gmail.com>
>> *Inviato:* martedì 6 novembre 2018 15:27
>> *A:* user@manifoldcf.apache.org
>> *Oggetto:* Re: Job stuck without message
>>
>>
>>
>> I added a couple of questions to the ticket. Please reply.
>>
>>
>>
>> Thanks,
>>
>> Karl
>>
>>
>>
>>
>>
>> On Tue, Nov 6, 2018 at 8:56 AM Bisonti Mario <Ma...@vimar.com>
>> wrote:
>>
>> Thanks a lot, Karl.
>>
>> I created a ticket.
>>
>> https://issues.apache.org/jira/browse/CONNECTORS-1554
>> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCONNECTORS-1554&data=01%7C01%7CMario.Bisonti%40vimar.com%7C76564dfb521a4a175ef608d655429b6b%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=H2Ps6N6MQHUo72mKEs%2BX0xSHtonLkXTNG%2FEA3c5qbUc%3D&reserved=0>
>>
>>
>>
>>
>>
>> Thanks
>>
>>
>>
>> Mario
>>
>>
>>
>>
>>
>>
>>
>> *Da:* Karl Wright <da...@gmail.com>
>> *Inviato:* martedì 6 novembre 2018 14:28
>> *A:* user@manifoldcf.apache.org
>> *Oggetto:* Re: Job stuck without message
>>
>>
>>
>> ok, can you create a ticket? Also, I'd appreciate it if you can look at
>> the simple history for one of these documents; I need to see what happened
>> to it last.
>>
>>
>>
>> Thanks,
>>
>> Karl
>>
>>
>>
>>
>>
>> On Tue, Nov 6, 2018 at 7:32 AM Bisonti Mario <Ma...@vimar.com>
>> wrote:
>>
>> My version is 2.11
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *Da:* Karl Wright <da...@gmail.com>
>> *Inviato:* martedì 6 novembre 2018 13:07
>> *A:* user@manifoldcf.apache.org
>> *Oggetto:* Re: Job stuck without message
>>
>>
>>
>> Thanks.
>>
>> What version of ManifoldCF are you using? We fixed a problem a while
>> back having to do with documents that (because of error processing) get put
>> into a "ready for processing" state which don't have any document priority
>> set. But this should have been addressed, certainly, by the most recent
>> release and probably by 2.10 as well.
>>
>>
>>
>> Karl
>>
>>
>>
>>
>>
>> On Tue, Nov 6, 2018 at 5:43 AM Bisonti Mario <Ma...@vimar.com>
>> wrote:
>>
>> Hallo Karl.
>>
>> When it hangs I see in the Queue status:
>>
>>
>>
>> And in the Document Status:
>>
>>
>>
>>
>>
>> *Da:* Karl Wright <da...@gmail.com>
>> *Inviato:* martedì 30 ottobre 2018 19:32
>> *A:* user@manifoldcf.apache.org
>> *Oggetto:* Re: Job stuck without message
>>
>>
>>
>> What I am interested in now is the Document Status report for any one of
>> the documents that is 'stuck'. The next crawl time value is the critical
>> field. Can you include an example?
>>
>>
>>
>> Karl
>>
>>
>>
>> On Tue, Oct 30, 2018, 12:36 PM Bisonti Mario <Ma...@vimar.com>
>> wrote:
>>
>> Thanks a lot, Karl.
>>
>>
>>
>> It happens that the job starts, it works and index for an hour and after
>> it frezzes, I haven’t error or waiting status in Document Queue o Simple
>> History, I have only “OK” status so, I haven’t failures.
>>
>>
>>
>> I am not able to see other log errors other from the manifoldcf.log
>>
>>
>>
>> Solr server is ok
>>
>> Tika server is ok
>>
>> Agent is ok
>>
>> Tomcat with ManifoldCF is ok
>>
>>
>>
>> I could search if I could to put in info log mode for example Tika
>> servrer or Solr.
>>
>>
>>
>> Thanks..
>>
>>
>>
>>
>>
>> *Da:* Karl Wright <da...@gmail.com>
>> *Inviato:* martedì 30 ottobre 2018 16:38
>> *A:* user@manifoldcf.apache.org
>> *Oggetto:* Re: Job stuck without message
>>
>>
>>
>> Hi Mario,
>>
>> Please look at the Queue Status report to determine what is waiting and
>> why it is waiting.
>> You can also look at the Simple History to see what has been happening.
>> If you are getting 100% failures in fetching documents then you may need to
>> address this because your infrastructure is unhappy. If the failure is
>> something that indicates that the document is never going to be readable,
>> that's a different problem and we might need to address that in the
>> connector.
>>
>>
>>
>> Karl
>>
>>
>>
>>
>>
>> On Tue, Oct 30, 2018 at 10:33 AM Bisonti Mario <Ma...@vimar.com>
>> wrote:
>>
>>
>>
>> Thanks a lot Karl
>>
>>
>>
>> Yes, I see many docs in the docs queue but they are inactive.
>>
>>
>>
>> Infact i see that no more docs are indexed in Solr and I see that job is
>> with the same number of docs Active (35012)
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *Da:* Karl Wright <da...@gmail.com>
>> *Inviato:* martedì 30 ottobre 2018 13:59
>> *A:* user@manifoldcf.apache.org
>> *Oggetto:* Re: Job stuck without message
>>
>>
>>
>> The reason the job is "stuck" is because:
>>
>> ' JCIFS: Possibly transient exception detected on attempt 1 while
>> getting share security: All pipe instances are busy.'
>>
>> This means that ManifoldCF will retry this document for a while before it
>> gives up on it. It appears to be stuck but it is not. You can verify that
>> by looking at the Document Queue report to see what is queued and what
>> times the various documents will be retried.
>>
>>
>>
>> Karl
>>
>>
>>
>>
>>
>> On Tue, Oct 30, 2018 at 5:07 AM Bisonti Mario <Ma...@vimar.com>
>> wrote:
>>
>> Hallo.
>>
>>
>>
>> I started a job that works for some minutes, and after it stucks.
>>
>>
>>
>> In the manifoldcf.log I see:
>> at
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
>> [mcf-jcifs-connector.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
>> [mcf-pull-agent.jar:?]
>>
>> WARN 2018-10-30T09:21:31,440 (Worker thread '2') - Tika Server: Tika
>> Server rejects: Tika Server rejected document with the following reason:
>> Unprocessable Entity
>>
>> WARN 2018-10-30T09:21:33,502 (Worker thread '14') - Tika Server: Tika
>> Server rejects: Tika Server rejected document with the following reason:
>> Unprocessable Entity
>>
>> WARN 2018-10-30T09:21:37,725 (Worker thread '30') - Tika Server: Tika
>> Server rejects: Tika Server rejected document with the following reason:
>> Unprocessable Entity
>>
>> WARN 2018-10-30T09:21:44,406 (Worker thread '49') - Tika Server: Tika
>> Server rejects: Tika Server rejected document with the following reason:
>> Unprocessable Entity
>>
>> WARN 2018-10-30T09:21:47,310 (Worker thread '15') - Tika Server: Tika
>> Server rejects: Tika Server rejected document with the following reason:
>> Unprocessable Entity
>>
>> WARN 2018-10-30T09:21:52,000 (Worker thread '27') - Tika Server: Tika
>> Server rejects: Tika Server rejected document with the following reason:
>> Unprocessable Entity
>>
>> WARN 2018-10-30T09:21:53,526 (Worker thread '15') - Tika Server: Tika
>> Server rejects: Tika Server rejected document with the following reason:
>> Unprocessable Entity
>>
>> WARN 2018-10-30T09:22:04,511 (Worker thread '3') - JCIFS: Possibly
>> transient exception detected on attempt 1 while getting share security: All
>> pipe instances are busy.
>>
>> jcifs.smb.SmbException: All pipe instances are busy.
>>
>> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbSession.send(SmbSession.java:238)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbTree.send(SmbTree.java:119)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbFile.send(SmbFile.java:776)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
>> ~[jcifs-1.3.18.3.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2438)
>> [mcf-jcifs-connector.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1221)
>> [mcf-jcifs-connector.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
>> [mcf-jcifs-connector.jar:?]
>>
>> at
>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
>> [mcf-pull-agent.jar:?]
>>
>> WARN 2018-10-30T09:22:10,359 (Worker thread '27') - Tika Server: Tika
>> Server rejects: Tika Server rejected document with the following reason:
>> Unprocessable Entity
>>
>> WARN 2018-10-30T09:22:13,932 (Worker thread '12') - Tika Server: Tika
>> Server rejects: Tika Server rejected document with the following reason:
>> Unprocessable Entity
>>
>> WARN 2018-10-30T09:22:14,274 (Worker thread '23') - Tika Server: Tika
>> Server rejects: Tika Server rejected document with the following reason:
>> Unprocessable Entity
>>
>> WARN 2018-10-30T09:22:19,933 (Worker thread '8') - Tika Server: Tika
>> Server rejects: Tika Server rejected document with the following reason:
>> Unprocessable Entity
>>
>> WARN 2018-10-30T09:23:59,920 (Worker thread '39') - Tika Server: Tika
>> Server rejects: Tika Server rejected document with the following reason:
>> Unprocessable Entity
>>
>> WARN 2018-10-30T09:24:09,059 (Worker thread '43') - Tika Server: Tika
>> Server rejects: Tika Server rejected document with the following reason:
>> Unprocessable Entity
>>
>>
>>
>>
>>
>>
>>
>> What could I check?
>>
>>
>>
>> Tika server works as standalone.
>>
>>
>>
>> Could you help me?
>>
>>
>>
>> Thanks a lot
>>
>>
>>
>> Mario
>>
>>
>>
>>
Re: Job stuck without message
Posted by Karl Wright <da...@gmail.com>.
Can you look into the database jobqueue table and provide a row that
corresponds to one of these documents?
Thanks,
Karl
On Wed, Nov 28, 2018 at 10:26 AM Bisonti Mario <Ma...@vimar.com>
wrote:
> Hallo.
>
> Repository has Max connection=10
>
>
>
> In the Document Status report” I see many item with :
>
> State=“Not yet processed”
>
> Status=”Ready for processing”
>
> Scheduled=01-01-1970 01:00:00.000”
>
> Scheduled Action=”Process”
>
>
>
>
>
>
>
>
>
> But the job no more walk..
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* mercoledì 28 novembre 2018 16:03
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> "Pipe instances are busy" occurs because you are overloading the SMB
> access to your servers. How many connections do you have allocated for
> your repository connection? You probably want to limit this to 2-3 if you
> see this error a lot, and it appears you do.
>
> " Tika Server: Tika Server rejects: Tika Server rejected document with
> the following reason: Unprocessable Entity" means the document is not
> properly formed XML. The rejection will mean the document isn't indexed,
> but this will not stop the job.
>
> If nothing is happening and you don't know why, I'd suggest looking at the
> Document Status report to figure out what documents are not being processed
> and why. It is quite possible they are all in the process of being retried
> because of the "Pipe instances" issue above.
>
>
>
> Karl
>
>
>
> On Wed, Nov 28, 2018 at 9:46 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo Karl.
>
> I take this ticket because,now, after I use zookeeper, my job works for 7
> hours and now it is in hang status.
>
> I see running but it seems hanging, no log from 1 hour
>
>
>
> This is the last manifoldcf.log lines:
>
>
>
>
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:21,524 (Worker thread '59') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:25,232 (Worker thread '45') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-11-28T14:46:25,277 (Worker thread '70') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-11-28T14:46:28,825 (Worker thread '27') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:40,473 (Worker thread '79') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:54,855 (Worker thread '65') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:55,175 (Worker thread '83') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
>
>
>
>
>
>
> I don’t know what to check.
>
>
>
> Tika server is ok, and it doesn’t more restart it .
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 6 novembre 2018 15:27
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> I added a couple of questions to the ticket. Please reply.
>
>
>
> Thanks,
>
> Karl
>
>
>
>
>
> On Tue, Nov 6, 2018 at 8:56 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Thanks a lot, Karl.
>
> I created a ticket.
>
> https://issues.apache.org/jira/browse/CONNECTORS-1554
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCONNECTORS-1554&data=01%7C01%7CMario.Bisonti%40vimar.com%7C76564dfb521a4a175ef608d655429b6b%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=H2Ps6N6MQHUo72mKEs%2BX0xSHtonLkXTNG%2FEA3c5qbUc%3D&reserved=0>
>
>
>
>
>
> Thanks
>
>
>
> Mario
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 6 novembre 2018 14:28
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> ok, can you create a ticket? Also, I'd appreciate it if you can look at
> the simple history for one of these documents; I need to see what happened
> to it last.
>
>
>
> Thanks,
>
> Karl
>
>
>
>
>
> On Tue, Nov 6, 2018 at 7:32 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> My version is 2.11
>
>
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 6 novembre 2018 13:07
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> Thanks.
>
> What version of ManifoldCF are you using? We fixed a problem a while back
> having to do with documents that (because of error processing) get put into
> a "ready for processing" state which don't have any document priority set.
> But this should have been addressed, certainly, by the most recent release
> and probably by 2.10 as well.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Nov 6, 2018 at 5:43 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo Karl.
>
> When it hangs I see in the Queue status:
>
>
>
> And in the Document Status:
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 19:32
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> What I am interested in now is the Document Status report for any one of
> the documents that is 'stuck'. The next crawl time value is the critical
> field. Can you include an example?
>
>
>
> Karl
>
>
>
> On Tue, Oct 30, 2018, 12:36 PM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Thanks a lot, Karl.
>
>
>
> It happens that the job starts, it works and index for an hour and after
> it frezzes, I haven’t error or waiting status in Document Queue o Simple
> History, I have only “OK” status so, I haven’t failures.
>
>
>
> I am not able to see other log errors other from the manifoldcf.log
>
>
>
> Solr server is ok
>
> Tika server is ok
>
> Agent is ok
>
> Tomcat with ManifoldCF is ok
>
>
>
> I could search if I could to put in info log mode for example Tika servrer
> or Solr.
>
>
>
> Thanks..
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 16:38
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> Hi Mario,
>
> Please look at the Queue Status report to determine what is waiting and
> why it is waiting.
> You can also look at the Simple History to see what has been happening.
> If you are getting 100% failures in fetching documents then you may need to
> address this because your infrastructure is unhappy. If the failure is
> something that indicates that the document is never going to be readable,
> that's a different problem and we might need to address that in the
> connector.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Oct 30, 2018 at 10:33 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
>
>
> Thanks a lot Karl
>
>
>
> Yes, I see many docs in the docs queue but they are inactive.
>
>
>
> Infact i see that no more docs are indexed in Solr and I see that job is
> with the same number of docs Active (35012)
>
>
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 13:59
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> The reason the job is "stuck" is because:
>
> ' JCIFS: Possibly transient exception detected on attempt 1 while getting
> share security: All pipe instances are busy.'
>
> This means that ManifoldCF will retry this document for a while before it
> gives up on it. It appears to be stuck but it is not. You can verify that
> by looking at the Document Queue report to see what is queued and what
> times the various documents will be retried.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Oct 30, 2018 at 5:07 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo.
>
>
>
> I started a job that works for some minutes, and after it stucks.
>
>
>
> In the manifoldcf.log I see:
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-10-30T09:21:31,440 (Worker thread '2') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:33,502 (Worker thread '14') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:37,725 (Worker thread '30') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:44,406 (Worker thread '49') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:47,310 (Worker thread '15') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:52,000 (Worker thread '27') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:53,526 (Worker thread '15') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:04,511 (Worker thread '3') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2438)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1221)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-10-30T09:22:10,359 (Worker thread '27') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:13,932 (Worker thread '12') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:14,274 (Worker thread '23') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:19,933 (Worker thread '8') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:23:59,920 (Worker thread '39') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:24:09,059 (Worker thread '43') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
>
>
>
>
>
>
> What could I check?
>
>
>
> Tika server works as standalone.
>
>
>
> Could you help me?
>
>
>
> Thanks a lot
>
>
>
> Mario
>
>
>
>
R: Job stuck without message
Posted by Bisonti Mario <Ma...@vimar.com>.
Hallo.
Repository has Max connection=10
In the Document Status report” I see many item with :
State=“Not yet processed”
Status=”Ready for processing”
Scheduled=01-01-1970 01:00:00.000”
Scheduled Action=”Process”
[cid:image002.jpg@01D48737.20908440]
But the job no more walk..
Da: Karl Wright <da...@gmail.com>
Inviato: mercoledì 28 novembre 2018 16:03
A: user@manifoldcf.apache.org
Oggetto: Re: Job stuck without message
"Pipe instances are busy" occurs because you are overloading the SMB access to your servers. How many connections do you have allocated for your repository connection? You probably want to limit this to 2-3 if you see this error a lot, and it appears you do.
" Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity" means the document is not properly formed XML. The rejection will mean the document isn't indexed, but this will not stop the job.
If nothing is happening and you don't know why, I'd suggest looking at the Document Status report to figure out what documents are not being processed and why. It is quite possible they are all in the process of being retried because of the "Pipe instances" issue above.
Karl
On Wed, Nov 28, 2018 at 9:46 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo Karl.
I take this ticket because,now, after I use zookeeper, my job works for 7 hours and now it is in hang status.
[cid:image001.png@01D48731.7DA5C470]
I see running but it seems hanging, no log from 1 hour
This is the last manifoldcf.log lines:
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:21,524 (Worker thread '59') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:25,232 (Worker thread '45') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-11-28T14:46:25,277 (Worker thread '70') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-11-28T14:46:28,825 (Worker thread '27') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:40,473 (Worker thread '79') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:54,855 (Worker thread '65') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:55,175 (Worker thread '83') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
I don’t know what to check.
Tika server is ok, and it doesn’t more restart it .
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 6 novembre 2018 15:27
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
I added a couple of questions to the ticket. Please reply.
Thanks,
Karl
On Tue, Nov 6, 2018 at 8:56 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot, Karl.
I created a ticket.
https://issues.apache.org/jira/browse/CONNECTORS-1554<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCONNECTORS-1554&data=01%7C01%7CMario.Bisonti%40vimar.com%7C76564dfb521a4a175ef608d655429b6b%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=H2Ps6N6MQHUo72mKEs%2BX0xSHtonLkXTNG%2FEA3c5qbUc%3D&reserved=0>
Thanks
Mario
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 6 novembre 2018 14:28
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
ok, can you create a ticket? Also, I'd appreciate it if you can look at the simple history for one of these documents; I need to see what happened to it last.
Thanks,
Karl
On Tue, Nov 6, 2018 at 7:32 AM Bisonti Mario <Ma...@vimar.com>> wrote:
My version is 2.11
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 6 novembre 2018 13:07
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
Thanks.
What version of ManifoldCF are you using? We fixed a problem a while back having to do with documents that (because of error processing) get put into a "ready for processing" state which don't have any document priority set. But this should have been addressed, certainly, by the most recent release and probably by 2.10 as well.
Karl
On Tue, Nov 6, 2018 at 5:43 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo Karl.
When it hangs I see in the Queue status:
And in the Document Status:
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 19:32
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
What I am interested in now is the Document Status report for any one of the documents that is 'stuck'. The next crawl time value is the critical field. Can you include an example?
Karl
On Tue, Oct 30, 2018, 12:36 PM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot, Karl.
It happens that the job starts, it works and index for an hour and after it frezzes, I haven’t error or waiting status in Document Queue o Simple History, I have only “OK” status so, I haven’t failures.
I am not able to see other log errors other from the manifoldcf.log
Solr server is ok
Tika server is ok
Agent is ok
Tomcat with ManifoldCF is ok
I could search if I could to put in info log mode for example Tika servrer or Solr.
Thanks..
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 16:38
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
Hi Mario,
Please look at the Queue Status report to determine what is waiting and why it is waiting.
You can also look at the Simple History to see what has been happening. If you are getting 100% failures in fetching documents then you may need to address this because your infrastructure is unhappy. If the failure is something that indicates that the document is never going to be readable, that's a different problem and we might need to address that in the connector.
Karl
On Tue, Oct 30, 2018 at 10:33 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot Karl
Yes, I see many docs in the docs queue but they are inactive.
Infact i see that no more docs are indexed in Solr and I see that job is with the same number of docs Active (35012)
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 13:59
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
The reason the job is "stuck" is because:
' JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.'
This means that ManifoldCF will retry this document for a while before it gives up on it. It appears to be stuck but it is not. You can verify that by looking at the Document Queue report to see what is queued and what times the various documents will be retried.
Karl
On Tue, Oct 30, 2018 at 5:07 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo.
I started a job that works for some minutes, and after it stucks.
In the manifoldcf.log I see:
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-10-30T09:21:31,440 (Worker thread '2') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:33,502 (Worker thread '14') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:37,725 (Worker thread '30') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:44,406 (Worker thread '49') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:47,310 (Worker thread '15') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:52,000 (Worker thread '27') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:53,526 (Worker thread '15') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:04,511 (Worker thread '3') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2438) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1221) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-10-30T09:22:10,359 (Worker thread '27') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:13,932 (Worker thread '12') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:14,274 (Worker thread '23') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:19,933 (Worker thread '8') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:23:59,920 (Worker thread '39') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:24:09,059 (Worker thread '43') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
What could I check?
Tika server works as standalone.
Could you help me?
Thanks a lot
Mario
Re: Job stuck without message
Posted by Karl Wright <da...@gmail.com>.
"Pipe instances are busy" occurs because you are overloading the SMB access
to your servers. How many connections do you have allocated for your
repository connection? You probably want to limit this to 2-3 if you see
this error a lot, and it appears you do.
" Tika Server: Tika Server rejects: Tika Server rejected document with the
following reason: Unprocessable Entity" means the document is not properly
formed XML. The rejection will mean the document isn't indexed, but this
will not stop the job.
If nothing is happening and you don't know why, I'd suggest looking at the
Document Status report to figure out what documents are not being processed
and why. It is quite possible they are all in the process of being retried
because of the "Pipe instances" issue above.
Karl
On Wed, Nov 28, 2018 at 9:46 AM Bisonti Mario <Ma...@vimar.com>
wrote:
> Hallo Karl.
>
> I take this ticket because,now, after I use zookeeper, my job works for 7
> hours and now it is in hang status.
>
> I see running but it seems hanging, no log from 1 hour
>
>
>
> This is the last manifoldcf.log lines:
>
>
>
>
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:21,524 (Worker thread '59') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:25,232 (Worker thread '45') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-11-28T14:46:25,277 (Worker thread '70') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-11-28T14:46:28,825 (Worker thread '27') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:40,473 (Worker thread '79') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:54,855 (Worker thread '65') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-11-28T14:46:55,175 (Worker thread '83') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
>
>
>
>
>
>
> I don’t know what to check.
>
>
>
> Tika server is ok, and it doesn’t more restart it .
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 6 novembre 2018 15:27
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> I added a couple of questions to the ticket. Please reply.
>
>
>
> Thanks,
>
> Karl
>
>
>
>
>
> On Tue, Nov 6, 2018 at 8:56 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Thanks a lot, Karl.
>
> I created a ticket.
>
> https://issues.apache.org/jira/browse/CONNECTORS-1554
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCONNECTORS-1554&data=01%7C01%7CMario.Bisonti%40vimar.com%7Cdad620c5f4514577be9108d643f402cc%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=jVvnNY%2FHnfc6K805bt1YdO3nsIkpoD9An4BhkMRYOU0%3D&reserved=0>
>
>
>
>
>
> Thanks
>
>
>
> Mario
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 6 novembre 2018 14:28
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> ok, can you create a ticket? Also, I'd appreciate it if you can look at
> the simple history for one of these documents; I need to see what happened
> to it last.
>
>
>
> Thanks,
>
> Karl
>
>
>
>
>
> On Tue, Nov 6, 2018 at 7:32 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> My version is 2.11
>
>
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 6 novembre 2018 13:07
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> Thanks.
>
> What version of ManifoldCF are you using? We fixed a problem a while back
> having to do with documents that (because of error processing) get put into
> a "ready for processing" state which don't have any document priority set.
> But this should have been addressed, certainly, by the most recent release
> and probably by 2.10 as well.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Nov 6, 2018 at 5:43 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo Karl.
>
> When it hangs I see in the Queue status:
>
>
>
> And in the Document Status:
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 19:32
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> What I am interested in now is the Document Status report for any one of
> the documents that is 'stuck'. The next crawl time value is the critical
> field. Can you include an example?
>
>
>
> Karl
>
>
>
> On Tue, Oct 30, 2018, 12:36 PM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Thanks a lot, Karl.
>
>
>
> It happens that the job starts, it works and index for an hour and after
> it frezzes, I haven’t error or waiting status in Document Queue o Simple
> History, I have only “OK” status so, I haven’t failures.
>
>
>
> I am not able to see other log errors other from the manifoldcf.log
>
>
>
> Solr server is ok
>
> Tika server is ok
>
> Agent is ok
>
> Tomcat with ManifoldCF is ok
>
>
>
> I could search if I could to put in info log mode for example Tika servrer
> or Solr.
>
>
>
> Thanks..
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 16:38
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> Hi Mario,
>
> Please look at the Queue Status report to determine what is waiting and
> why it is waiting.
> You can also look at the Simple History to see what has been happening.
> If you are getting 100% failures in fetching documents then you may need to
> address this because your infrastructure is unhappy. If the failure is
> something that indicates that the document is never going to be readable,
> that's a different problem and we might need to address that in the
> connector.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Oct 30, 2018 at 10:33 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
>
>
> Thanks a lot Karl
>
>
>
> Yes, I see many docs in the docs queue but they are inactive.
>
>
>
> Infact i see that no more docs are indexed in Solr and I see that job is
> with the same number of docs Active (35012)
>
>
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 13:59
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> The reason the job is "stuck" is because:
>
> ' JCIFS: Possibly transient exception detected on attempt 1 while getting
> share security: All pipe instances are busy.'
>
> This means that ManifoldCF will retry this document for a while before it
> gives up on it. It appears to be stuck but it is not. You can verify that
> by looking at the Document Queue report to see what is queued and what
> times the various documents will be retried.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Oct 30, 2018 at 5:07 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo.
>
>
>
> I started a job that works for some minutes, and after it stucks.
>
>
>
> In the manifoldcf.log I see:
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-10-30T09:21:31,440 (Worker thread '2') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:33,502 (Worker thread '14') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:37,725 (Worker thread '30') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:44,406 (Worker thread '49') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:47,310 (Worker thread '15') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:52,000 (Worker thread '27') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:53,526 (Worker thread '15') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:04,511 (Worker thread '3') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2438)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1221)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-10-30T09:22:10,359 (Worker thread '27') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:13,932 (Worker thread '12') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:14,274 (Worker thread '23') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:19,933 (Worker thread '8') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:23:59,920 (Worker thread '39') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:24:09,059 (Worker thread '43') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
>
>
>
>
>
>
> What could I check?
>
>
>
> Tika server works as standalone.
>
>
>
> Could you help me?
>
>
>
> Thanks a lot
>
>
>
> Mario
>
>
>
>
R: Job stuck without message
Posted by Bisonti Mario <Ma...@vimar.com>.
Hallo Karl.
I take this ticket because,now, after I use zookeeper, my job works for 7 hours and now it is in hang status.
[cid:image001.png@01D48731.7DA5C470]
I see running but it seems hanging, no log from 1 hour
This is the last manifoldcf.log lines:
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:21,524 (Worker thread '59') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:25,232 (Worker thread '45') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-11-28T14:46:25,277 (Worker thread '70') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-11-28T14:46:28,825 (Worker thread '27') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:40,473 (Worker thread '79') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:54,855 (Worker thread '65') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-11-28T14:46:55,175 (Worker thread '83') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2446) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1222) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
I don’t know what to check.
Tika server is ok, and it doesn’t more restart it .
Da: Karl Wright <da...@gmail.com>
Inviato: martedì 6 novembre 2018 15:27
A: user@manifoldcf.apache.org
Oggetto: Re: Job stuck without message
I added a couple of questions to the ticket. Please reply.
Thanks,
Karl
On Tue, Nov 6, 2018 at 8:56 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot, Karl.
I created a ticket.
https://issues.apache.org/jira/browse/CONNECTORS-1554<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCONNECTORS-1554&data=01%7C01%7CMario.Bisonti%40vimar.com%7Cdad620c5f4514577be9108d643f402cc%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1&sdata=jVvnNY%2FHnfc6K805bt1YdO3nsIkpoD9An4BhkMRYOU0%3D&reserved=0>
Thanks
Mario
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 6 novembre 2018 14:28
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
ok, can you create a ticket? Also, I'd appreciate it if you can look at the simple history for one of these documents; I need to see what happened to it last.
Thanks,
Karl
On Tue, Nov 6, 2018 at 7:32 AM Bisonti Mario <Ma...@vimar.com>> wrote:
My version is 2.11
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 6 novembre 2018 13:07
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
Thanks.
What version of ManifoldCF are you using? We fixed a problem a while back having to do with documents that (because of error processing) get put into a "ready for processing" state which don't have any document priority set. But this should have been addressed, certainly, by the most recent release and probably by 2.10 as well.
Karl
On Tue, Nov 6, 2018 at 5:43 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo Karl.
When it hangs I see in the Queue status:
And in the Document Status:
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 19:32
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
What I am interested in now is the Document Status report for any one of the documents that is 'stuck'. The next crawl time value is the critical field. Can you include an example?
Karl
On Tue, Oct 30, 2018, 12:36 PM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot, Karl.
It happens that the job starts, it works and index for an hour and after it frezzes, I haven’t error or waiting status in Document Queue o Simple History, I have only “OK” status so, I haven’t failures.
I am not able to see other log errors other from the manifoldcf.log
Solr server is ok
Tika server is ok
Agent is ok
Tomcat with ManifoldCF is ok
I could search if I could to put in info log mode for example Tika servrer or Solr.
Thanks..
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 16:38
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
Hi Mario,
Please look at the Queue Status report to determine what is waiting and why it is waiting.
You can also look at the Simple History to see what has been happening. If you are getting 100% failures in fetching documents then you may need to address this because your infrastructure is unhappy. If the failure is something that indicates that the document is never going to be readable, that's a different problem and we might need to address that in the connector.
Karl
On Tue, Oct 30, 2018 at 10:33 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot Karl
Yes, I see many docs in the docs queue but they are inactive.
Infact i see that no more docs are indexed in Solr and I see that job is with the same number of docs Active (35012)
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 13:59
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
The reason the job is "stuck" is because:
' JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.'
This means that ManifoldCF will retry this document for a while before it gives up on it. It appears to be stuck but it is not. You can verify that by looking at the Document Queue report to see what is queued and what times the various documents will be retried.
Karl
On Tue, Oct 30, 2018 at 5:07 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo.
I started a job that works for some minutes, and after it stucks.
In the manifoldcf.log I see:
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-10-30T09:21:31,440 (Worker thread '2') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:33,502 (Worker thread '14') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:37,725 (Worker thread '30') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:44,406 (Worker thread '49') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:47,310 (Worker thread '15') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:52,000 (Worker thread '27') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:53,526 (Worker thread '15') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:04,511 (Worker thread '3') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2438) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1221) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-10-30T09:22:10,359 (Worker thread '27') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:13,932 (Worker thread '12') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:14,274 (Worker thread '23') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:19,933 (Worker thread '8') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:23:59,920 (Worker thread '39') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:24:09,059 (Worker thread '43') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
What could I check?
Tika server works as standalone.
Could you help me?
Thanks a lot
Mario
Re: Job stuck without message
Posted by Karl Wright <da...@gmail.com>.
I added a couple of questions to the ticket. Please reply.
Thanks,
Karl
On Tue, Nov 6, 2018 at 8:56 AM Bisonti Mario <Ma...@vimar.com>
wrote:
> Thanks a lot, Karl.
>
> I created a ticket.
>
> https://issues.apache.org/jira/browse/CONNECTORS-1554
>
>
>
>
>
> Thanks
>
>
>
> Mario
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 6 novembre 2018 14:28
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> ok, can you create a ticket? Also, I'd appreciate it if you can look at
> the simple history for one of these documents; I need to see what happened
> to it last.
>
>
>
> Thanks,
>
> Karl
>
>
>
>
>
> On Tue, Nov 6, 2018 at 7:32 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> My version is 2.11
>
>
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 6 novembre 2018 13:07
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> Thanks.
>
> What version of ManifoldCF are you using? We fixed a problem a while back
> having to do with documents that (because of error processing) get put into
> a "ready for processing" state which don't have any document priority set.
> But this should have been addressed, certainly, by the most recent release
> and probably by 2.10 as well.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Nov 6, 2018 at 5:43 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo Karl.
>
> When it hangs I see in the Queue status:
>
>
>
> And in the Document Status:
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 19:32
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> What I am interested in now is the Document Status report for any one of
> the documents that is 'stuck'. The next crawl time value is the critical
> field. Can you include an example?
>
>
>
> Karl
>
>
>
> On Tue, Oct 30, 2018, 12:36 PM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Thanks a lot, Karl.
>
>
>
> It happens that the job starts, it works and index for an hour and after
> it frezzes, I haven’t error or waiting status in Document Queue o Simple
> History, I have only “OK” status so, I haven’t failures.
>
>
>
> I am not able to see other log errors other from the manifoldcf.log
>
>
>
> Solr server is ok
>
> Tika server is ok
>
> Agent is ok
>
> Tomcat with ManifoldCF is ok
>
>
>
> I could search if I could to put in info log mode for example Tika servrer
> or Solr.
>
>
>
> Thanks..
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 16:38
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> Hi Mario,
>
> Please look at the Queue Status report to determine what is waiting and
> why it is waiting.
> You can also look at the Simple History to see what has been happening.
> If you are getting 100% failures in fetching documents then you may need to
> address this because your infrastructure is unhappy. If the failure is
> something that indicates that the document is never going to be readable,
> that's a different problem and we might need to address that in the
> connector.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Oct 30, 2018 at 10:33 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
>
>
> Thanks a lot Karl
>
>
>
> Yes, I see many docs in the docs queue but they are inactive.
>
>
>
> Infact i see that no more docs are indexed in Solr and I see that job is
> with the same number of docs Active (35012)
>
>
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 13:59
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> The reason the job is "stuck" is because:
>
> ' JCIFS: Possibly transient exception detected on attempt 1 while getting
> share security: All pipe instances are busy.'
>
> This means that ManifoldCF will retry this document for a while before it
> gives up on it. It appears to be stuck but it is not. You can verify that
> by looking at the Document Queue report to see what is queued and what
> times the various documents will be retried.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Oct 30, 2018 at 5:07 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo.
>
>
>
> I started a job that works for some minutes, and after it stucks.
>
>
>
> In the manifoldcf.log I see:
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-10-30T09:21:31,440 (Worker thread '2') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:33,502 (Worker thread '14') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:37,725 (Worker thread '30') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:44,406 (Worker thread '49') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:47,310 (Worker thread '15') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:52,000 (Worker thread '27') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:53,526 (Worker thread '15') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:04,511 (Worker thread '3') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2438)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1221)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-10-30T09:22:10,359 (Worker thread '27') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:13,932 (Worker thread '12') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:14,274 (Worker thread '23') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:19,933 (Worker thread '8') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:23:59,920 (Worker thread '39') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:24:09,059 (Worker thread '43') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
>
>
>
>
>
>
> What could I check?
>
>
>
> Tika server works as standalone.
>
>
>
> Could you help me?
>
>
>
> Thanks a lot
>
>
>
> Mario
>
>
>
>
R: Job stuck without message
Posted by Bisonti Mario <Ma...@vimar.com>.
Thanks a lot, Karl.
I created a ticket.
https://issues.apache.org/jira/browse/CONNECTORS-1554
Thanks
Mario
Da: Karl Wright <da...@gmail.com>
Inviato: martedì 6 novembre 2018 14:28
A: user@manifoldcf.apache.org
Oggetto: Re: Job stuck without message
ok, can you create a ticket? Also, I'd appreciate it if you can look at the simple history for one of these documents; I need to see what happened to it last.
Thanks,
Karl
On Tue, Nov 6, 2018 at 7:32 AM Bisonti Mario <Ma...@vimar.com>> wrote:
My version is 2.11
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 6 novembre 2018 13:07
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
Thanks.
What version of ManifoldCF are you using? We fixed a problem a while back having to do with documents that (because of error processing) get put into a "ready for processing" state which don't have any document priority set. But this should have been addressed, certainly, by the most recent release and probably by 2.10 as well.
Karl
On Tue, Nov 6, 2018 at 5:43 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo Karl.
When it hangs I see in the Queue status:
And in the Document Status:
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 19:32
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
What I am interested in now is the Document Status report for any one of the documents that is 'stuck'. The next crawl time value is the critical field. Can you include an example?
Karl
On Tue, Oct 30, 2018, 12:36 PM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot, Karl.
It happens that the job starts, it works and index for an hour and after it frezzes, I haven’t error or waiting status in Document Queue o Simple History, I have only “OK” status so, I haven’t failures.
I am not able to see other log errors other from the manifoldcf.log
Solr server is ok
Tika server is ok
Agent is ok
Tomcat with ManifoldCF is ok
I could search if I could to put in info log mode for example Tika servrer or Solr.
Thanks..
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 16:38
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
Hi Mario,
Please look at the Queue Status report to determine what is waiting and why it is waiting.
You can also look at the Simple History to see what has been happening. If you are getting 100% failures in fetching documents then you may need to address this because your infrastructure is unhappy. If the failure is something that indicates that the document is never going to be readable, that's a different problem and we might need to address that in the connector.
Karl
On Tue, Oct 30, 2018 at 10:33 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot Karl
Yes, I see many docs in the docs queue but they are inactive.
Infact i see that no more docs are indexed in Solr and I see that job is with the same number of docs Active (35012)
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 13:59
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
The reason the job is "stuck" is because:
' JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.'
This means that ManifoldCF will retry this document for a while before it gives up on it. It appears to be stuck but it is not. You can verify that by looking at the Document Queue report to see what is queued and what times the various documents will be retried.
Karl
On Tue, Oct 30, 2018 at 5:07 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo.
I started a job that works for some minutes, and after it stucks.
In the manifoldcf.log I see:
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-10-30T09:21:31,440 (Worker thread '2') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:33,502 (Worker thread '14') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:37,725 (Worker thread '30') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:44,406 (Worker thread '49') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:47,310 (Worker thread '15') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:52,000 (Worker thread '27') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:53,526 (Worker thread '15') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:04,511 (Worker thread '3') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2438) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1221) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-10-30T09:22:10,359 (Worker thread '27') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:13,932 (Worker thread '12') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:14,274 (Worker thread '23') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:19,933 (Worker thread '8') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:23:59,920 (Worker thread '39') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:24:09,059 (Worker thread '43') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
What could I check?
Tika server works as standalone.
Could you help me?
Thanks a lot
Mario
Re: Job stuck without message
Posted by Karl Wright <da...@gmail.com>.
ok, can you create a ticket? Also, I'd appreciate it if you can look at
the simple history for one of these documents; I need to see what happened
to it last.
Thanks,
Karl
On Tue, Nov 6, 2018 at 7:32 AM Bisonti Mario <Ma...@vimar.com>
wrote:
> My version is 2.11
>
>
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 6 novembre 2018 13:07
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> Thanks.
>
> What version of ManifoldCF are you using? We fixed a problem a while back
> having to do with documents that (because of error processing) get put into
> a "ready for processing" state which don't have any document priority set.
> But this should have been addressed, certainly, by the most recent release
> and probably by 2.10 as well.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Nov 6, 2018 at 5:43 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo Karl.
>
> When it hangs I see in the Queue status:
>
>
>
> And in the Document Status:
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 19:32
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> What I am interested in now is the Document Status report for any one of
> the documents that is 'stuck'. The next crawl time value is the critical
> field. Can you include an example?
>
>
>
> Karl
>
>
>
> On Tue, Oct 30, 2018, 12:36 PM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Thanks a lot, Karl.
>
>
>
> It happens that the job starts, it works and index for an hour and after
> it frezzes, I haven’t error or waiting status in Document Queue o Simple
> History, I have only “OK” status so, I haven’t failures.
>
>
>
> I am not able to see other log errors other from the manifoldcf.log
>
>
>
> Solr server is ok
>
> Tika server is ok
>
> Agent is ok
>
> Tomcat with ManifoldCF is ok
>
>
>
> I could search if I could to put in info log mode for example Tika servrer
> or Solr.
>
>
>
> Thanks..
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 16:38
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> Hi Mario,
>
> Please look at the Queue Status report to determine what is waiting and
> why it is waiting.
> You can also look at the Simple History to see what has been happening.
> If you are getting 100% failures in fetching documents then you may need to
> address this because your infrastructure is unhappy. If the failure is
> something that indicates that the document is never going to be readable,
> that's a different problem and we might need to address that in the
> connector.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Oct 30, 2018 at 10:33 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
>
>
> Thanks a lot Karl
>
>
>
> Yes, I see many docs in the docs queue but they are inactive.
>
>
>
> Infact i see that no more docs are indexed in Solr and I see that job is
> with the same number of docs Active (35012)
>
>
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 13:59
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> The reason the job is "stuck" is because:
>
> ' JCIFS: Possibly transient exception detected on attempt 1 while getting
> share security: All pipe instances are busy.'
>
> This means that ManifoldCF will retry this document for a while before it
> gives up on it. It appears to be stuck but it is not. You can verify that
> by looking at the Document Queue report to see what is queued and what
> times the various documents will be retried.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Oct 30, 2018 at 5:07 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo.
>
>
>
> I started a job that works for some minutes, and after it stucks.
>
>
>
> In the manifoldcf.log I see:
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-10-30T09:21:31,440 (Worker thread '2') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:33,502 (Worker thread '14') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:37,725 (Worker thread '30') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:44,406 (Worker thread '49') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:47,310 (Worker thread '15') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:52,000 (Worker thread '27') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:53,526 (Worker thread '15') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:04,511 (Worker thread '3') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2438)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1221)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-10-30T09:22:10,359 (Worker thread '27') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:13,932 (Worker thread '12') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:14,274 (Worker thread '23') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:19,933 (Worker thread '8') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:23:59,920 (Worker thread '39') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:24:09,059 (Worker thread '43') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
>
>
>
>
>
>
> What could I check?
>
>
>
> Tika server works as standalone.
>
>
>
> Could you help me?
>
>
>
> Thanks a lot
>
>
>
> Mario
>
>
>
>
R: Job stuck without message
Posted by Bisonti Mario <Ma...@vimar.com>.
My version is 2.11
Da: Karl Wright <da...@gmail.com>
Inviato: martedì 6 novembre 2018 13:07
A: user@manifoldcf.apache.org
Oggetto: Re: Job stuck without message
Thanks.
What version of ManifoldCF are you using? We fixed a problem a while back having to do with documents that (because of error processing) get put into a "ready for processing" state which don't have any document priority set. But this should have been addressed, certainly, by the most recent release and probably by 2.10 as well.
Karl
On Tue, Nov 6, 2018 at 5:43 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo Karl.
When it hangs I see in the Queue status:
[cid:image001.png@01D475C5.DCE8D300]
And in the Document Status:
[cid:image002.png@01D475C5.DCE8D300]
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 19:32
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
What I am interested in now is the Document Status report for any one of the documents that is 'stuck'. The next crawl time value is the critical field. Can you include an example?
Karl
On Tue, Oct 30, 2018, 12:36 PM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot, Karl.
It happens that the job starts, it works and index for an hour and after it frezzes, I haven’t error or waiting status in Document Queue o Simple History, I have only “OK” status so, I haven’t failures.
I am not able to see other log errors other from the manifoldcf.log
Solr server is ok
Tika server is ok
Agent is ok
Tomcat with ManifoldCF is ok
I could search if I could to put in info log mode for example Tika servrer or Solr.
Thanks..
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 16:38
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
Hi Mario,
Please look at the Queue Status report to determine what is waiting and why it is waiting.
You can also look at the Simple History to see what has been happening. If you are getting 100% failures in fetching documents then you may need to address this because your infrastructure is unhappy. If the failure is something that indicates that the document is never going to be readable, that's a different problem and we might need to address that in the connector.
Karl
On Tue, Oct 30, 2018 at 10:33 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Thanks a lot Karl
Yes, I see many docs in the docs queue but they are inactive.
Infact i see that no more docs are indexed in Solr and I see that job is with the same number of docs Active (35012)
Da: Karl Wright <da...@gmail.com>>
Inviato: martedì 30 ottobre 2018 13:59
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: Job stuck without message
The reason the job is "stuck" is because:
' JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.'
This means that ManifoldCF will retry this document for a while before it gives up on it. It appears to be stuck but it is not. You can verify that by looking at the Document Queue report to see what is queued and what times the various documents will be retried.
Karl
On Tue, Oct 30, 2018 at 5:07 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hallo.
I started a job that works for some minutes, and after it stucks.
In the manifoldcf.log I see:
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-10-30T09:21:31,440 (Worker thread '2') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:33,502 (Worker thread '14') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:37,725 (Worker thread '30') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:44,406 (Worker thread '49') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:47,310 (Worker thread '15') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:52,000 (Worker thread '27') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:21:53,526 (Worker thread '15') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:04,511 (Worker thread '3') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTransport.send(SmbTransport.java:669) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbSession.send(SmbSession.java:238) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open0(SmbFile.java:993) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.open(SmbFile.java:1010) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126) ~[jcifs-1.3.18.3.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140) ~[jcifs-1.3.18.3.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951) ~[jcifs-1.3.18.3.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2438) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1221) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [mcf-pull-agent.jar:?]
WARN 2018-10-30T09:22:10,359 (Worker thread '27') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:13,932 (Worker thread '12') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:14,274 (Worker thread '23') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:22:19,933 (Worker thread '8') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:23:59,920 (Worker thread '39') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
WARN 2018-10-30T09:24:09,059 (Worker thread '43') - Tika Server: Tika Server rejects: Tika Server rejected document with the following reason: Unprocessable Entity
What could I check?
Tika server works as standalone.
Could you help me?
Thanks a lot
Mario
Re: Job stuck without message
Posted by Karl Wright <da...@gmail.com>.
Thanks.
What version of ManifoldCF are you using? We fixed a problem a while back
having to do with documents that (because of error processing) get put into
a "ready for processing" state which don't have any document priority set.
But this should have been addressed, certainly, by the most recent release
and probably by 2.10 as well.
Karl
On Tue, Nov 6, 2018 at 5:43 AM Bisonti Mario <Ma...@vimar.com>
wrote:
> Hallo Karl.
>
> When it hangs I see in the Queue status:
>
>
>
> And in the Document Status:
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 19:32
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> What I am interested in now is the Document Status report for any one of
> the documents that is 'stuck'. The next crawl time value is the critical
> field. Can you include an example?
>
>
>
> Karl
>
>
>
> On Tue, Oct 30, 2018, 12:36 PM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Thanks a lot, Karl.
>
>
>
> It happens that the job starts, it works and index for an hour and after
> it frezzes, I haven’t error or waiting status in Document Queue o Simple
> History, I have only “OK” status so, I haven’t failures.
>
>
>
> I am not able to see other log errors other from the manifoldcf.log
>
>
>
> Solr server is ok
>
> Tika server is ok
>
> Agent is ok
>
> Tomcat with ManifoldCF is ok
>
>
>
> I could search if I could to put in info log mode for example Tika servrer
> or Solr.
>
>
>
> Thanks..
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 16:38
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> Hi Mario,
>
> Please look at the Queue Status report to determine what is waiting and
> why it is waiting.
> You can also look at the Simple History to see what has been happening.
> If you are getting 100% failures in fetching documents then you may need to
> address this because your infrastructure is unhappy. If the failure is
> something that indicates that the document is never going to be readable,
> that's a different problem and we might need to address that in the
> connector.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Oct 30, 2018 at 10:33 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
>
>
> Thanks a lot Karl
>
>
>
> Yes, I see many docs in the docs queue but they are inactive.
>
>
>
> Infact i see that no more docs are indexed in Solr and I see that job is
> with the same number of docs Active (35012)
>
>
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* martedì 30 ottobre 2018 13:59
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: Job stuck without message
>
>
>
> The reason the job is "stuck" is because:
>
> ' JCIFS: Possibly transient exception detected on attempt 1 while getting
> share security: All pipe instances are busy.'
>
> This means that ManifoldCF will retry this document for a while before it
> gives up on it. It appears to be stuck but it is not. You can verify that
> by looking at the Document Queue report to see what is queued and what
> times the various documents will be retried.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Oct 30, 2018 at 5:07 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hallo.
>
>
>
> I started a job that works for some minutes, and after it stucks.
>
>
>
> In the manifoldcf.log I see:
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-10-30T09:21:31,440 (Worker thread '2') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:33,502 (Worker thread '14') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:37,725 (Worker thread '30') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:44,406 (Worker thread '49') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:47,310 (Worker thread '15') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:52,000 (Worker thread '27') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:21:53,526 (Worker thread '15') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:04,511 (Worker thread '3') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:569)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTransport.send(SmbTransport.java:669)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbSession.send(SmbSession.java:238)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbTree.send(SmbTree.java:119) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.send(SmbFile.java:776) ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open0(SmbFile.java:993)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.open(SmbFile.java:1010)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbFileOutputStream.<init>(SmbFileOutputStream.java:142)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.TransactNamedPipeOutputStream.<init>(TransactNamedPipeOutputStream.java:32)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.smb.SmbNamedPipe.getNamedPipeOutputStream(SmbNamedPipe.java:187)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendFragment(DcerpcPipeHandle.java:68)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:190)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:126)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:140)
> ~[jcifs-1.3.18.3.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2951)
> ~[jcifs-1.3.18.3.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2438)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1221)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:627)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> [mcf-pull-agent.jar:?]
>
> WARN 2018-10-30T09:22:10,359 (Worker thread '27') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:13,932 (Worker thread '12') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:14,274 (Worker thread '23') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:22:19,933 (Worker thread '8') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:23:59,920 (Worker thread '39') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
> WARN 2018-10-30T09:24:09,059 (Worker thread '43') - Tika Server: Tika
> Server rejects: Tika Server rejected document with the following reason:
> Unprocessable Entity
>
>
>
>
>
>
>
> What could I check?
>
>
>
> Tika server works as standalone.
>
>
>
> Could you help me?
>
>
>
> Thanks a lot
>
>
>
> Mario
>
>
>
>