You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@manifoldcf.apache.org by Bisonti Mario <Ma...@vimar.com> on 2023/02/01 13:04:49 UTC
R: JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy
This is my hs_err_pid_xxxx.log
Command Line: -Xms32768m -Xmx32768m -Dorg.apache.manifoldcf.configfile=./properties.xml -Djava.security.auth.login.con
fig= -Dorg.apache.manifoldcf.processid=A org.apache.manifoldcf.agents.AgentRun
.
.
.
CodeHeap 'non-profiled nmethods': size=120032Kb used=23677Kb max_used=23677Kb free=96354Kb
CodeHeap 'profiled nmethods': size=120028Kb used=20405Kb max_used=27584Kb free=99622Kb
CodeHeap 'non-nmethods': size=5700Kb used=1278Kb max_used=1417Kb free=4421Kb
Memory: 4k page, physical 72057128k(7300332k free), swap 4039676k(4039676k free)
.
.
Perhaps could be a RAM problem?
Thanks a lot
Da: Bisonti Mario
Inviato: venerdì 20 gennaio 2023 10:28
A: user@manifoldcf.apache.org
Oggetto: R: JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy
I see that the agent crashed:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (g1ConcurrentMark.cpp:1665), pid=2537463, tid=2537470
# fatal error: Overflow during reference processing, can not continue. Please increase MarkStackSizeMax (current value: 16777216) and restart.
#
# JRE version: OpenJDK Runtime Environment (11.0.16+8) (build 11.0.16+8-post-Ubuntu-0ubuntu120.04)
# Java VM: OpenJDK 64-Bit Server VM (11.0.16+8-post-Ubuntu-0ubuntu120.04, mixed mode, tiered, g1 gc, linux-amd64)
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /opt/manifoldcf/multiprocess-zk-example-proprietary/core.2537463)
#
# If you would like to submit a bug report, please visit:
# https://bugs.launchpad.net/ubuntu/+source/openjdk-lts
#
--------------- S U M M A R Y ------------
Command Line: -Xms32768m -Xmx32768m -Dorg.apache.manifoldcf.configfile=./properties.xml -Djava.security.auth.login.config= -Dorg.apache.manifoldcf.processid=A org.apache.manifoldcf.agents.AgentRun
Host: Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz, 8 cores, 68G, Ubuntu 20.04.4 LTS
Time: Fri Jan 20 09:38:54 2023 CET elapsed time: 54532.106681 seconds (0d 15h 8m 52s)
--------------- T H R E A D ---------------
Current thread (0x00007f051940a000): VMThread "VM Thread" [stack: 0x00007f051c50a000,0x00007f051c60a000] [id=2537470]
Stack: [0x00007f051c50a000,0x00007f051c60a000], sp=0x00007f051c608080, free space=1016k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xe963a9]
V [libjvm.so+0x67b504]
V [libjvm.so+0x7604e6]
So, where could I change that parameter?
Is it an Agent configuration?
Thanks a lot
Mario
Da: Karl Wright <da...@gmail.com>>
Inviato: mercoledì 18 gennaio 2023 14:59
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy
When you get a hang like this, getting a thread dump of the agents process is essential to figure out what the issue is. You can't assume that a transient error would block anything because that's not how ManifoldCF works, at all. Errors push the document in question back onto the queue with a retry time.
Karl
On Wed, Jan 18, 2023 at 6:15 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hi Karl.
But I noted that the job was hanging, the document processed was stucked on the same number, no further document processing from the 6 a.m until I restart Agent
Da: Karl Wright <da...@gmail.com>>
Inviato: mercoledì 18 gennaio 2023 12:10
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy
Hi, "Possibly transient issue" means that the error will be retried anyway, according to a schedule. There should not need to be any requirement to shut down the agents process and restart.
Karl
On Wed, Jan 18, 2023 at 5:08 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hi.
Often, I obtain the error:
WARN 2023-01-18T06:18:19,316 (Worker thread '89') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransportImpl.checkStatus2(SmbTransportImpl.java:1441) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTransportImpl.checkStatus(SmbTransportImpl.java:1552) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTransportImpl.sendrecv(SmbTransportImpl.java:1007) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTransportImpl.send(SmbTransportImpl.java:1523) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:409) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTreeImpl.send(SmbTreeImpl.java:472) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTreeConnection.send0(SmbTreeConnection.java:399) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTreeConnection.send(SmbTreeConnection.java:314) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTreeConnection.send(SmbTreeConnection.java:294) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTreeHandleImpl.send(SmbTreeHandleImpl.java:130) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTreeHandleImpl.send(SmbTreeHandleImpl.java:117) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbFile.openUnshared(SmbFile.java:665) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbPipeHandleImpl.ensureOpen(SmbPipeHandleImpl.java:169) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbPipeHandleImpl.sendrecv(SmbPipeHandleImpl.java:250) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendReceiveFragment(DcerpcPipeHandle.java:113) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:243) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:216) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:234) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2337) ~[jcifs-ng-2.1.2.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2468) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1243) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:647) [mcf-jcifs-connector.jar:?]
So, I have to stop the agent, restart it, and the crwling continues.
How could I solve my issue?
Thanks a lot.
Mario
R: JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy
Posted by Bisonti Mario <Ma...@vimar.com>.
Hi, could you give me any suggestion to solve my issue?
I note that to index 1 million documents (office, pdf, ect) so I use Tika, it finishes after near 18 hours.
My host is an Ubuntu server with 8cpu and 68GB RAM
Thanks a lot
Mario
Da: Bisonti Mario
Inviato: mercoledì 1 febbraio 2023 17:50
A: user@manifoldcf.apache.org
Oggetto: R: JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy
I don't understand.
Would you like to explain me what "running with a profiler" mean, please?
I start agent running a start-agents.sh script, and zookeeper too.
/opt/manifoldcf/multiprocess-zk-example-proprietary/runzookeeper.sh
/opt/manifoldcf/multiprocess-zk-example-proprietary/start-agents.sh
Where start-agents.sh is:
#!/bin/bash -e
cd /opt/manifoldcf/multiprocess-zk-example-proprietary/
if [ -e "$JAVA_HOME"/bin/java ] ; then
if [ -f ./properties.xml ] ; then
./executecommand.sh -Dorg.apache.manifoldcf.processid=A org.apache.manifoldcf.agents.AgentRun
exit $?
else
echo "Working directory contains no properties.xml file." 1>&2
exit 1
fi
else
echo "Environment variable JAVA_HOME is not properly set." 1>&2
exit 1
fi
Thanks a lot Karl.
Da: Karl Wright <da...@gmail.com>>
Inviato: mercoledì 1 febbraio 2023 17:38
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy
It looks like you are running with a profiler? That uses a lot of memory.
Karl
On Wed, Feb 1, 2023 at 8:06 AM Bisonti Mario <Ma...@vimar.com>> wrote:
This is my hs_err_pid_xxxx.log
Command Line: -Xms32768m -Xmx32768m -Dorg.apache.manifoldcf.configfile=./properties.xml -Djava.security.auth.login.con
fig= -Dorg.apache.manifoldcf.processid=A org.apache.manifoldcf.agents.AgentRun
.
.
.
CodeHeap 'non-profiled nmethods': size=120032Kb used=23677Kb max_used=23677Kb free=96354Kb
CodeHeap 'profiled nmethods': size=120028Kb used=20405Kb max_used=27584Kb free=99622Kb
CodeHeap 'non-nmethods': size=5700Kb used=1278Kb max_used=1417Kb free=4421Kb
Memory: 4k page, physical 72057128k(7300332k free), swap 4039676k(4039676k free)
.
.
Perhaps could be a RAM problem?
Thanks a lot
Da: Bisonti Mario
Inviato: venerdì 20 gennaio 2023 10:28
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: R: JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy
I see that the agent crashed:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (g1ConcurrentMark.cpp:1665), pid=2537463, tid=2537470
# fatal error: Overflow during reference processing, can not continue. Please increase MarkStackSizeMax (current value: 16777216) and restart.
#
# JRE version: OpenJDK Runtime Environment (11.0.16+8) (build 11.0.16+8-post-Ubuntu-0ubuntu120.04)
# Java VM: OpenJDK 64-Bit Server VM (11.0.16+8-post-Ubuntu-0ubuntu120.04, mixed mode, tiered, g1 gc, linux-amd64)
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /opt/manifoldcf/multiprocess-zk-example-proprietary/core.2537463)
#
# If you would like to submit a bug report, please visit:
# https://bugs.launchpad.net/ubuntu/+source/openjdk-lts<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.launchpad.net%2Fubuntu%2F%2Bsource%2Fopenjdk-lts&data=05%7C01%7CMario.Bisonti%40vimar.com%7C45cb8a6ebe294cdba68208db0472b0f2%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1%7C0%7C638108662863184656%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=i3XHLbhk2ij02pOCwde3Pj5bYZRJ7KyjI6eEXDnPRSY%3D&reserved=0>
#
--------------- S U M M A R Y ------------
Command Line: -Xms32768m -Xmx32768m -Dorg.apache.manifoldcf.configfile=./properties.xml -Djava.security.auth.login.config= -Dorg.apache.manifoldcf.processid=A org.apache.manifoldcf.agents.AgentRun
Host: Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz, 8 cores, 68G, Ubuntu 20.04.4 LTS
Time: Fri Jan 20 09:38:54 2023 CET elapsed time: 54532.106681 seconds (0d 15h 8m 52s)
--------------- T H R E A D ---------------
Current thread (0x00007f051940a000): VMThread "VM Thread" [stack: 0x00007f051c50a000,0x00007f051c60a000] [id=2537470]
Stack: [0x00007f051c50a000,0x00007f051c60a000], sp=0x00007f051c608080, free space=1016k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xe963a9]
V [libjvm.so+0x67b504]
V [libjvm.so+0x7604e6]
So, where could I change that parameter?
Is it an Agent configuration?
Thanks a lot
Mario
Da: Karl Wright <da...@gmail.com>>
Inviato: mercoledì 18 gennaio 2023 14:59
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy
When you get a hang like this, getting a thread dump of the agents process is essential to figure out what the issue is. You can't assume that a transient error would block anything because that's not how ManifoldCF works, at all. Errors push the document in question back onto the queue with a retry time.
Karl
On Wed, Jan 18, 2023 at 6:15 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hi Karl.
But I noted that the job was hanging, the document processed was stucked on the same number, no further document processing from the 6 a.m until I restart Agent
Da: Karl Wright <da...@gmail.com>>
Inviato: mercoledì 18 gennaio 2023 12:10
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy
Hi, "Possibly transient issue" means that the error will be retried anyway, according to a schedule. There should not need to be any requirement to shut down the agents process and restart.
Karl
On Wed, Jan 18, 2023 at 5:08 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hi.
Often, I obtain the error:
WARN 2023-01-18T06:18:19,316 (Worker thread '89') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransportImpl.checkStatus2(SmbTransportImpl.java:1441) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTransportImpl.checkStatus(SmbTransportImpl.java:1552) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTransportImpl.sendrecv(SmbTransportImpl.java:1007) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTransportImpl.send(SmbTransportImpl.java:1523) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:409) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTreeImpl.send(SmbTreeImpl.java:472) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTreeConnection.send0(SmbTreeConnection.java:399) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTreeConnection.send(SmbTreeConnection.java:314) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTreeConnection.send(SmbTreeConnection.java:294) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTreeHandleImpl.send(SmbTreeHandleImpl.java:130) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTreeHandleImpl.send(SmbTreeHandleImpl.java:117) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbFile.openUnshared(SmbFile.java:665) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbPipeHandleImpl.ensureOpen(SmbPipeHandleImpl.java:169) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbPipeHandleImpl.sendrecv(SmbPipeHandleImpl.java:250) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendReceiveFragment(DcerpcPipeHandle.java:113) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:243) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:216) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:234) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2337) ~[jcifs-ng-2.1.2.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2468) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1243) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:647) [mcf-jcifs-connector.jar:?]
So, I have to stop the agent, restart it, and the crwling continues.
How could I solve my issue?
Thanks a lot.
Mario
R: JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy
Posted by Bisonti Mario <Ma...@vimar.com>.
I don't understand.
Would you like to explain me what "running with a profiler" mean, please?
I start agent running a start-agents.sh script, and zookeeper too.
/opt/manifoldcf/multiprocess-zk-example-proprietary/runzookeeper.sh
/opt/manifoldcf/multiprocess-zk-example-proprietary/start-agents.sh
Where start-agents.sh is:
#!/bin/bash -e
cd /opt/manifoldcf/multiprocess-zk-example-proprietary/
if [ -e "$JAVA_HOME"/bin/java ] ; then
if [ -f ./properties.xml ] ; then
./executecommand.sh -Dorg.apache.manifoldcf.processid=A org.apache.manifoldcf.agents.AgentRun
exit $?
else
echo "Working directory contains no properties.xml file." 1>&2
exit 1
fi
else
echo "Environment variable JAVA_HOME is not properly set." 1>&2
exit 1
fi
Thanks a lot Karl.
Da: Karl Wright <da...@gmail.com>
Inviato: mercoledì 1 febbraio 2023 17:38
A: user@manifoldcf.apache.org
Oggetto: Re: JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy
It looks like you are running with a profiler? That uses a lot of memory.
Karl
On Wed, Feb 1, 2023 at 8:06 AM Bisonti Mario <Ma...@vimar.com>> wrote:
This is my hs_err_pid_xxxx.log
Command Line: -Xms32768m -Xmx32768m -Dorg.apache.manifoldcf.configfile=./properties.xml -Djava.security.auth.login.con
fig= -Dorg.apache.manifoldcf.processid=A org.apache.manifoldcf.agents.AgentRun
.
.
.
CodeHeap 'non-profiled nmethods': size=120032Kb used=23677Kb max_used=23677Kb free=96354Kb
CodeHeap 'profiled nmethods': size=120028Kb used=20405Kb max_used=27584Kb free=99622Kb
CodeHeap 'non-nmethods': size=5700Kb used=1278Kb max_used=1417Kb free=4421Kb
Memory: 4k page, physical 72057128k(7300332k free), swap 4039676k(4039676k free)
.
.
Perhaps could be a RAM problem?
Thanks a lot
Da: Bisonti Mario
Inviato: venerdì 20 gennaio 2023 10:28
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: R: JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy
I see that the agent crashed:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (g1ConcurrentMark.cpp:1665), pid=2537463, tid=2537470
# fatal error: Overflow during reference processing, can not continue. Please increase MarkStackSizeMax (current value: 16777216) and restart.
#
# JRE version: OpenJDK Runtime Environment (11.0.16+8) (build 11.0.16+8-post-Ubuntu-0ubuntu120.04)
# Java VM: OpenJDK 64-Bit Server VM (11.0.16+8-post-Ubuntu-0ubuntu120.04, mixed mode, tiered, g1 gc, linux-amd64)
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /opt/manifoldcf/multiprocess-zk-example-proprietary/core.2537463)
#
# If you would like to submit a bug report, please visit:
# https://bugs.launchpad.net/ubuntu/+source/openjdk-lts<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.launchpad.net%2Fubuntu%2F%2Bsource%2Fopenjdk-lts&data=05%7C01%7CMario.Bisonti%40vimar.com%7C45cb8a6ebe294cdba68208db0472b0f2%7Ca1f008bcd59b4c668f8760fd9af15c7f%7C1%7C0%7C638108662863184656%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=i3XHLbhk2ij02pOCwde3Pj5bYZRJ7KyjI6eEXDnPRSY%3D&reserved=0>
#
--------------- S U M M A R Y ------------
Command Line: -Xms32768m -Xmx32768m -Dorg.apache.manifoldcf.configfile=./properties.xml -Djava.security.auth.login.config= -Dorg.apache.manifoldcf.processid=A org.apache.manifoldcf.agents.AgentRun
Host: Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz, 8 cores, 68G, Ubuntu 20.04.4 LTS
Time: Fri Jan 20 09:38:54 2023 CET elapsed time: 54532.106681 seconds (0d 15h 8m 52s)
--------------- T H R E A D ---------------
Current thread (0x00007f051940a000): VMThread "VM Thread" [stack: 0x00007f051c50a000,0x00007f051c60a000] [id=2537470]
Stack: [0x00007f051c50a000,0x00007f051c60a000], sp=0x00007f051c608080, free space=1016k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xe963a9]
V [libjvm.so+0x67b504]
V [libjvm.so+0x7604e6]
So, where could I change that parameter?
Is it an Agent configuration?
Thanks a lot
Mario
Da: Karl Wright <da...@gmail.com>>
Inviato: mercoledì 18 gennaio 2023 14:59
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy
When you get a hang like this, getting a thread dump of the agents process is essential to figure out what the issue is. You can't assume that a transient error would block anything because that's not how ManifoldCF works, at all. Errors push the document in question back onto the queue with a retry time.
Karl
On Wed, Jan 18, 2023 at 6:15 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hi Karl.
But I noted that the job was hanging, the document processed was stucked on the same number, no further document processing from the 6 a.m until I restart Agent
Da: Karl Wright <da...@gmail.com>>
Inviato: mercoledì 18 gennaio 2023 12:10
A: user@manifoldcf.apache.org<ma...@manifoldcf.apache.org>
Oggetto: Re: JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy
Hi, "Possibly transient issue" means that the error will be retried anyway, according to a schedule. There should not need to be any requirement to shut down the agents process and restart.
Karl
On Wed, Jan 18, 2023 at 5:08 AM Bisonti Mario <Ma...@vimar.com>> wrote:
Hi.
Often, I obtain the error:
WARN 2023-01-18T06:18:19,316 (Worker thread '89') - JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy.
jcifs.smb.SmbException: All pipe instances are busy.
at jcifs.smb.SmbTransportImpl.checkStatus2(SmbTransportImpl.java:1441) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTransportImpl.checkStatus(SmbTransportImpl.java:1552) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTransportImpl.sendrecv(SmbTransportImpl.java:1007) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTransportImpl.send(SmbTransportImpl.java:1523) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:409) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTreeImpl.send(SmbTreeImpl.java:472) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTreeConnection.send0(SmbTreeConnection.java:399) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTreeConnection.send(SmbTreeConnection.java:314) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTreeConnection.send(SmbTreeConnection.java:294) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTreeHandleImpl.send(SmbTreeHandleImpl.java:130) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbTreeHandleImpl.send(SmbTreeHandleImpl.java:117) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbFile.openUnshared(SmbFile.java:665) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbPipeHandleImpl.ensureOpen(SmbPipeHandleImpl.java:169) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbPipeHandleImpl.sendrecv(SmbPipeHandleImpl.java:250) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.dcerpc.DcerpcPipeHandle.doSendReceiveFragment(DcerpcPipeHandle.java:113) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:243) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:216) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:234) ~[jcifs-ng-2.1.2.jar:?]
at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2337) ~[jcifs-ng-2.1.2.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2468) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1243) [mcf-jcifs-connector.jar:?]
at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:647) [mcf-jcifs-connector.jar:?]
So, I have to stop the agent, restart it, and the crwling continues.
How could I solve my issue?
Thanks a lot.
Mario
Re: JCIFS: Possibly transient exception detected on attempt 1 while getting share security: All pipe instances are busy
Posted by Karl Wright <da...@gmail.com>.
It looks like you are running with a profiler? That uses a lot of memory.
Karl
On Wed, Feb 1, 2023 at 8:06 AM Bisonti Mario <Ma...@vimar.com>
wrote:
> This is my hs_err_pid_xxxx.log
>
>
>
> Command Line: -Xms32768m -Xmx32768m
> -Dorg.apache.manifoldcf.configfile=./properties.xml
> -Djava.security.auth.login.con
>
> fig= -Dorg.apache.manifoldcf.processid=A
> org.apache.manifoldcf.agents.AgentRun
>
>
>
> .
>
> .
>
> .
>
> CodeHeap 'non-profiled nmethods': size=120032Kb used=23677Kb
> max_used=23677Kb free=96354Kb
>
> CodeHeap 'profiled nmethods': size=120028Kb used=20405Kb max_used=27584Kb
> free=99622Kb
>
> CodeHeap 'non-nmethods': size=5700Kb used=1278Kb max_used=1417Kb
> free=4421Kb
>
> Memory: 4k page, physical 72057128k(7300332k free), swap 4039676k(4039676k
> free)
>
> .
>
> .
>
>
>
> Perhaps could be a RAM problem?
>
>
>
> Thanks a lot
>
>
>
>
>
>
>
>
>
> *Da:* Bisonti Mario
> *Inviato:* venerdì 20 gennaio 2023 10:28
> *A:* user@manifoldcf.apache.org
> *Oggetto:* R: JCIFS: Possibly transient exception detected on attempt 1
> while getting share security: All pipe instances are busy
>
>
>
> I see that the agent crashed:
>
> #
>
> # A fatal error has been detected by the Java Runtime Environment:
>
> #
>
> # Internal Error (g1ConcurrentMark.cpp:1665), pid=2537463, tid=2537470
>
> # fatal error: Overflow during reference processing, can not continue.
> Please increase MarkStackSizeMax (current value: 16777216) and restart.
>
> #
>
> # JRE version: OpenJDK Runtime Environment (11.0.16+8) (build
> 11.0.16+8-post-Ubuntu-0ubuntu120.04)
>
> # Java VM: OpenJDK 64-Bit Server VM (11.0.16+8-post-Ubuntu-0ubuntu120.04,
> mixed mode, tiered, g1 gc, linux-amd64)
>
> # Core dump will be written. Default location: Core dumps may be processed
> with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E"
> (or dumping to
> /opt/manifoldcf/multiprocess-zk-example-proprietary/core.2537463)
>
> #
>
> # If you would like to submit a bug report, please visit:
>
> # https://bugs.launchpad.net/ubuntu/+source/openjdk-lts
>
> #
>
>
>
> --------------- S U M M A R Y ------------
>
>
>
> Command Line: -Xms32768m -Xmx32768m
> -Dorg.apache.manifoldcf.configfile=./properties.xml
> -Djava.security.auth.login.config= -Dorg.apache.manifoldcf.processid=A
> org.apache.manifoldcf.agents.AgentRun
>
>
>
> Host: Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz, 8 cores, 68G, Ubuntu
> 20.04.4 LTS
>
> Time: Fri Jan 20 09:38:54 2023 CET elapsed time: 54532.106681 seconds (0d
> 15h 8m 52s)
>
>
>
> --------------- T H R E A D ---------------
>
>
>
> Current thread (0x00007f051940a000): VMThread "VM Thread" [stack:
> 0x00007f051c50a000,0x00007f051c60a000] [id=2537470]
>
>
>
> Stack: [0x00007f051c50a000,0x00007f051c60a000], sp=0x00007f051c608080,
> free space=1016k
>
> Native frames: (J=compiled Java code, A=aot compiled Java code,
> j=interpreted, Vv=VM code, C=native code)
>
> V [libjvm.so+0xe963a9]
>
> V [libjvm.so+0x67b504]
>
> V [libjvm.so+0x7604e6]
>
>
>
>
>
> So, where could I change that parameter?
>
> Is it an Agent configuration?
>
> Thanks a lot
>
> Mario
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* mercoledì 18 gennaio 2023 14:59
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: JCIFS: Possibly transient exception detected on attempt 1
> while getting share security: All pipe instances are busy
>
>
>
> When you get a hang like this, getting a thread dump of the agents process
> is essential to figure out what the issue is. You can't assume that a
> transient error would block anything because that's not how ManifoldCF
> works, at all. Errors push the document in question back onto the queue
> with a retry time.
>
>
>
> Karl
>
>
>
>
>
> On Wed, Jan 18, 2023 at 6:15 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hi Karl.
>
> But I noted that the job was hanging, the document processed was stucked
> on the same number, no further document processing from the 6 a.m until I
> restart Agent
>
>
>
>
>
>
>
>
>
> *Da:* Karl Wright <da...@gmail.com>
> *Inviato:* mercoledì 18 gennaio 2023 12:10
> *A:* user@manifoldcf.apache.org
> *Oggetto:* Re: JCIFS: Possibly transient exception detected on attempt 1
> while getting share security: All pipe instances are busy
>
>
>
> Hi, "Possibly transient issue" means that the error will be retried
> anyway, according to a schedule. There should not need to be any
> requirement to shut down the agents process and restart.
>
> Karl
>
>
>
> On Wed, Jan 18, 2023 at 5:08 AM Bisonti Mario <Ma...@vimar.com>
> wrote:
>
> Hi.
>
> Often, I obtain the error:
>
> WARN 2023-01-18T06:18:19,316 (Worker thread '89') - JCIFS: Possibly
> transient exception detected on attempt 1 while getting share security: All
> pipe instances are busy.
>
> jcifs.smb.SmbException: All pipe instances are busy.
>
> at
> jcifs.smb.SmbTransportImpl.checkStatus2(SmbTransportImpl.java:1441)
> ~[jcifs-ng-2.1.2.jar:?]
>
> at
> jcifs.smb.SmbTransportImpl.checkStatus(SmbTransportImpl.java:1552)
> ~[jcifs-ng-2.1.2.jar:?]
>
> at jcifs.smb.SmbTransportImpl.sendrecv(SmbTransportImpl.java:1007)
> ~[jcifs-ng-2.1.2.jar:?]
>
> at jcifs.smb.SmbTransportImpl.send(SmbTransportImpl.java:1523)
> ~[jcifs-ng-2.1.2.jar:?]
>
> at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:409)
> ~[jcifs-ng-2.1.2.jar:?]
>
> at jcifs.smb.SmbTreeImpl.send(SmbTreeImpl.java:472)
> ~[jcifs-ng-2.1.2.jar:?]
>
> at jcifs.smb.SmbTreeConnection.send0(SmbTreeConnection.java:399)
> ~[jcifs-ng-2.1.2.jar:?]
>
> at jcifs.smb.SmbTreeConnection.send(SmbTreeConnection.java:314)
> ~[jcifs-ng-2.1.2.jar:?]
>
> at jcifs.smb.SmbTreeConnection.send(SmbTreeConnection.java:294)
> ~[jcifs-ng-2.1.2.jar:?]
>
> at jcifs.smb.SmbTreeHandleImpl.send(SmbTreeHandleImpl.java:130)
> ~[jcifs-ng-2.1.2.jar:?]
>
> at jcifs.smb.SmbTreeHandleImpl.send(SmbTreeHandleImpl.java:117)
> ~[jcifs-ng-2.1.2.jar:?]
>
> at jcifs.smb.SmbFile.openUnshared(SmbFile.java:665)
> ~[jcifs-ng-2.1.2.jar:?]
>
> at
> jcifs.smb.SmbPipeHandleImpl.ensureOpen(SmbPipeHandleImpl.java:169)
> ~[jcifs-ng-2.1.2.jar:?]
>
> at
> jcifs.smb.SmbPipeHandleImpl.sendrecv(SmbPipeHandleImpl.java:250)
> ~[jcifs-ng-2.1.2.jar:?]
>
> at
> jcifs.dcerpc.DcerpcPipeHandle.doSendReceiveFragment(DcerpcPipeHandle.java:113)
> ~[jcifs-ng-2.1.2.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:243)
> ~[jcifs-ng-2.1.2.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.bind(DcerpcHandle.java:216)
> ~[jcifs-ng-2.1.2.jar:?]
>
> at jcifs.dcerpc.DcerpcHandle.sendrecv(DcerpcHandle.java:234)
> ~[jcifs-ng-2.1.2.jar:?]
>
> at jcifs.smb.SmbFile.getShareSecurity(SmbFile.java:2337)
> ~[jcifs-ng-2.1.2.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecurity(SharedDriveConnector.java:2468)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getFileShareSecuritySet(SharedDriveConnector.java:1243)
> [mcf-jcifs-connector.jar:?]
>
> at
> org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:647)
> [mcf-jcifs-connector.jar:?]
>
>
>
> So, I have to stop the agent, restart it, and the crwling continues.
>
>
>
> How could I solve my issue?
> Thanks a lot.
>
> Mario
>
>