You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by Konstantinos Mavrommatis <km...@celgene.com> on 2014/07/10 11:22:31 UTC
UPDATE-PROBABLY SOLVED: File manager keeps connections to SOLR in
CLOSE_WAIT state for hours
Hi,
Following the suggestion at
http://blogs.nuxeo.com/development/2013/02/using-httpclient-properly-avoid-closewait-tcp-connections/
I modified the code in
src/main/java/org/apache/oodt/cas/filemgr/catalog/solr/SolrClient.java
and added the line highlighted in red (method.setRequestHeader(“Connection”,”close”).
This seems to have resolved the problem at least until now ingestion of few thousand files have not produced any stale connection.
Please let me know your thoughts on this solution.
Best,
Konstantinos
======================================================
private String doHttp(HttpMethod method) throws Exception {
StringBuilder response = new StringBuilder();
BufferedReader br = null;
try {
// send request
HttpClient httpClient = new HttpClient();
// 10 July 2014. attempting to avoid problems with CLOSE_WAIT connections Based on
//http://blogs.nuxeo.com/development/2013/02/using-httpclient-properly-avoid-closewait-tcp-connections/
method.setRequestHeader("Connection", "close");
int statusCode = httpClient.executeMethod(method);
// read response
if (statusCode != HttpStatus.SC_OK) {
// still consume the response
method.getResponseBodyAsString();
throw new CatalogException("HTTP method failed: " + method.getStatusLine());
} else {
// read the response body.
br = new BufferedReader(new InputStreamReader(method.getResponseBodyAsStream()));
String readLine;
while(((readLine = br.readLine()) != null)) {
response.append(readLine);
}
=======================================================
Information about the system:
OS is MacOS Mavericks,
SOLR version : 4.6.1 , SOLR runs on a single server, no replication at all.
Tomcat verision: 7.0.50
OODT version is 0.6
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
> -----Original Message-----
> From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
> Sent: Wednesday, July 09, 2014 4:57 AM
> To: dev@oodt.apache.org
> Subject: Re: File manager keeps connections to SOLR in CLOSE_WAIT state
> for hours
>
> It's just occoured to me that everything in
> org.apache.oodt.cas.filemgr.catalog.solr.SolrCatalog (well,
> specifically
> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient) is done through
> HTTPClient so all of the above may not be relevant.
> Can you please scope all the same?
> Thanks
> Lewis
>
>
> On Tue, Jul 8, 2014 at 10:54 PM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com<ma...@gmail.com>> wrote:
>
> > Hi Konstantinos,
> > OK, I was ablel to scope this one and I have a few questions for you.
> > 1) Which version of Solr are you using? Is it 3.5, 3.6, 4.0-ALPHA?
> If
> > so please scope this issue [0], the solution would be to upgrade if
> > you are not too long ahead with ingestion as fixes in Solr are worth
> > having based on recent release cycles.
> > 2) How many cores do you have on Solr server? Also what kind of setUp
> > do you have? Replication at all?
> > In recent versions of Solr 4.X SolrJ clients should now call
> > shutdown() on their SolrServer object to let it know they don't want
> > to re-use any existing connections anymore, and when Solr internally
> > uses SolrJ to talk to other nodes in SolrCloud it should be doing
> this
> > (as of 4.0-ALPHA)so this is why I ask.
> > Lewis
> >
> > [0] https://issues.apache.org/jira/browse/SOLR-3280
> >
> >
> > On Tue, Jul 8, 2014 at 7:14 AM, Konstantinos Mavrommatis <
> > kmavrommatis@celgene.com<ma...@celgene.com>> wrote:
> >
> >> Hi,
> >> I have setup OODT filemanager on port 9000, using SOLR as the
> >> indexing service on port 8081. They are both setup on the same
> >> computer, while crawler runs on a number of different compute nodes
> >> spread across the local network and the cloud.
> >>
> >> When the crawler runs and ingests files I notice that there are
> >> several connections that open to solr and remain in CLOSE_WAIT state
> for hours.
> >> any idea why this happens? Moving forward I am planning to use
> >> several hundreds of crawler instances, each running on different
> >> computer, that will create thousands of such connections and will
> >> probably create problems to the system.
> >> Thanks in advance for any help
> >> Kostas
> >>
> >> $lsof -i :8081
> >> COMMAND PID USER FD TYPE DEVICE SIZE/OFF
> NODE
> >> NAME
> >> java 92065 kmavrommatis 75u IPv6 0x392c3fa3b63b29cf 0t0
> TCP
> >> localhost:49205->localhost:sunproxyadmin (CLOSE_WAIT)
> >> java 92065 kmavrommatis 76u IPv6 0x392c3fa3b6dcbbcf 0t0
> TCP
> >> localhost:49206->localhost:sunproxyadmin (CLOSE_WAIT)
> >> java 92065 kmavrommatis 77u IPv6 0x392c3fa39fd12e0f 0t0
> TCP
> >> localhost:49207->localhost:sunproxyadmin (CLOSE_WAIT)
> >> java 92065 kmavrommatis 78u IPv6 0x392c3fa39fdcdbcf 0t0
> TCP
> >> localhost:49208->localhost:sunproxyadmin (CLOSE_WAIT)
> >> java 92065 kmavrommatis 79u IPv6 0x392c3fa3b62cde0f 0t0
> TCP
> >> localhost:49209->localhost:sunproxyadmin (CLOSE_WAIT)
> >> java 92065 kmavrommatis 80u IPv6 0x392c3fa39fa2714f 0t0
> TCP
> >> localhost:49210->localhost:sunproxyadmin (CLOSE_WAIT)
> >> java 92065 kmavrommatis 81u IPv6 0x392c3fa3b6c32acf 0t0
> TCP
> >> localhost:49211->localhost:sunproxyadmin (CLOSE_WAIT)
> >> java 92065 kmavrommatis 82u IPv6 0x392c3fa3b6aa714f 0t0
> TCP
> >> localhost:49212->localhost:sunproxyadmin (CLOSE_WAIT)
> >>
> >>
> >> process 92065 is:
> >> /usr/bin/java -Djava.ext.dirs=../lib
> >> -Djava.util.logging.config.file=../etc/logging.properties
> >> -Dorg.apache.oodt.cas.filemgr.properties=../etc/filemgr.properties
> >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager --portNum 9000
> >>
> >> *********************************************************
> >> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND
> >> MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE USE
> >> OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
> >> If the reader is not the intended recipient, or the employee or
> agent
> >> responsible to deliver it to the intended recipient, you are hereby
> >> notified that any dissemination, distribution or copying of this
> >> communication is strictly prohibited. If you have received this
> >> communication in error, please reply to the sender to notify us of
> >> the error and delete the original message. Thank You.
> >>
> >
> >
> >
> > --
> > *Lewis*
> >
>
>
>
> --
> *Lewis*
*********************************************************
THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS
CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED
INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL
OR INDIVIDUALS NAMED ABOVE.
If the reader is not the intended recipient, or the
employee or agent responsible to deliver it to the
intended recipient, you are hereby notified that any
dissemination, distribution or copying of this
communication is strictly prohibited. If you have
received this communication in error, please reply to the
sender to notify us of the error and delete the original
message. Thank You.
Re: UPDATE-PROBABLY SOLVED: File manager keeps connections to SOLR in
CLOSE_WAIT state for hours
Posted by Chris Mattmann <ch...@gmail.com>.
Hey Konstantinos can you sign up for a JIRA account?
Then I can grant you permissions to add/update/edit issues.
------------------------
Chris Mattmann
chris.mattmann@gmail.com
-----Original Message-----
From: Konstantinos Mavrommatis <km...@celgene.com>
Reply-To: <de...@oodt.apache.org>
Date: Thursday, July 10, 2014 7:11 AM
To: "dev@oodt.apache.org" <de...@oodt.apache.org>
Subject: RE: UPDATE-PROBABLY SOLVED: File manager keeps connections to
SOLR in CLOSE_WAIT state for hours
>Unless I am missing something I am not able to create an issue in JIRA :(
>Thanks
>K
>
>> -----Original Message-----
>> From: Ramirez, Paul M (398J) [mailto:paul.m.ramirez@jpl.nasa.gov]
>> Sent: Thursday, July 10, 2014 4:06 PM
>> To: <de...@oodt.apache.org>
>> Subject: Re: UPDATE-PROBABLY SOLVED: File manager keeps connections to
>> SOLR in CLOSE_WAIT state for hours
>>
>> Konstantinos,
>>
>> Could you open a Jira issue for this and attach a patch? This would be
>> helpful for the project.
>>
>> https://issues.apache.org/jira/browse/OODT/?selectedTab=com.atlassian.j
>> ira.jira-projects-plugin:summary-panel
>>
>>
>> Thanks,
>> Paul Ramirez
>>
>> > On Jul 10, 2014, at 5:23 AM, "Konstantinos Mavrommatis"
>> <km...@celgene.com> wrote:
>> >
>> > Hi,
>> >
>> > Following the suggestion at
>> >
>> > http://blogs.nuxeo.com/development/2013/02/using-httpclient-properly-
>> a
>> > void-closewait-tcp-connections/
>> >
>> >
>> >
>> > I modified the code in
>> >
>> >
>> src/main/java/org/apache/oodt/cas/filemgr/catalog/solr/SolrClient.java
>> >
>> >
>> >
>> > and added the line highlighted in red
>> (method.setRequestHeader("Connection","close").
>> >
>> > This seems to have resolved the problem at least until now ingestion
>> of few thousand files have not produced any stale connection.
>> >
>> > Please let me know your thoughts on this solution.
>> >
>> > Best,
>> >
>> > Konstantinos
>> >
>> >
>> >
>> > ======================================================
>> >
>> > private String doHttp(HttpMethod method) throws Exception {
>> >
>> >
>> >
>> > StringBuilder response = new StringBuilder();
>> >
>> > BufferedReader br = null;
>> >
>> > try {
>> >
>> >
>> >
>> > // send request
>> >
>> > HttpClient httpClient = new HttpClient();
>> >
>> > // 10 July 2014. attempting to avoid problems
>> > with CLOSE_WAIT connections Based on
>> >
>> >
>> > //http://blogs.nuxeo.com/development/2013/02/using-httpclient-
>> properly
>> > -avoid-closewait-tcp-connections/
>> >
>> > method.setRequestHeader("Connection",
>> "close");
>> >
>> > int statusCode = httpClient.executeMethod(method);
>> >
>> >
>> >
>> > // read response
>> >
>> > if (statusCode != HttpStatus.SC_OK) {
>> >
>> >
>> >
>> > // still consume the response
>> >
>> > method.getResponseBodyAsString();
>> >
>> > throw new CatalogException("HTTP method failed: " +
>> > method.getStatusLine());
>> >
>> >
>> >
>> > } else {
>> >
>> >
>> >
>> > // read the response body.
>> >
>> > br = new BufferedReader(new
>> > InputStreamReader(method.getResponseBodyAsStream()));
>> >
>> > String readLine;
>> >
>> > while(((readLine = br.readLine()) != null)) {
>> >
>> > response.append(readLine);
>> >
>> > }
>> >
>> >
>> >
>> > =======================================================
>> >
>> >
>> >
>> > Information about the system:
>> >
>> > OS is MacOS Mavericks,
>> >
>> > SOLR version : 4.6.1 , SOLR runs on a single server, no replication
>> at all.
>> >
>> > Tomcat verision: 7.0.50
>> >
>> > OODT version is 0.6
>> >
>> > Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >> -----Original Message-----
>> >
>> >> From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
>> >
>> >> Sent: Wednesday, July 09, 2014 4:57 AM
>> >
>> >> To: dev@oodt.apache.org
>> >
>> >> Subject: Re: File manager keeps connections to SOLR in CLOSE_WAIT
>> >> state
>> >
>> >> for hours
>> >
>> >
>> >> It's just occoured to me that everything in
>> >
>> >> org.apache.oodt.cas.filemgr.catalog.solr.SolrCatalog (well,
>> >
>> >> specifically
>> >
>> >> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient) is done through
>> >
>> >> HTTPClient so all of the above may not be relevant.
>> >
>> >> Can you please scope all the same?
>> >
>> >> Thanks
>> >
>> >> Lewis
>> >
>> >
>> >
>> >> On Tue, Jul 8, 2014 at 10:54 PM, Lewis John Mcgibbney <
>> >
>> >> lewis.mcgibbney@gmail.com<ma...@gmail.com>> wrote:
>> >
>> >
>> >>> Hi Konstantinos,
>> >
>> >>> OK, I was ablel to scope this one and I have a few questions for
>> you.
>> >
>> >>> 1) Which version of Solr are you using? Is it 3.5, 3.6, 4.0-ALPHA?
>> >
>> >> If
>> >
>> >>> so please scope this issue [0], the solution would be to upgrade if
>> >
>> >>> you are not too long ahead with ingestion as fixes in Solr are
>> worth
>> >
>> >>> having based on recent release cycles.
>> >
>> >>> 2) How many cores do you have on Solr server? Also what kind of
>> >>> setUp
>> >
>> >>> do you have? Replication at all?
>> >
>> >>> In recent versions of Solr 4.X SolrJ clients should now call
>> >
>> >>> shutdown() on their SolrServer object to let it know they don't
>> want
>> >
>> >>> to re-use any existing connections anymore, and when Solr
>> internally
>> >
>> >>> uses SolrJ to talk to other nodes in SolrCloud it should be doing
>> >
>> >> this
>> >
>> >>> (as of 4.0-ALPHA)so this is why I ask.
>> >
>> >>> Lewis
>> >
>> >
>> >>> [0] https://issues.apache.org/jira/browse/SOLR-3280
>> >
>> >
>> >
>> >>> On Tue, Jul 8, 2014 at 7:14 AM, Konstantinos Mavrommatis <
>> >
>> >>> kmavrommatis@celgene.com<ma...@celgene.com>> wrote:
>> >
>> >
>> >>>> Hi,
>> >
>> >>>> I have setup OODT filemanager on port 9000, using SOLR as the
>> >
>> >>>> indexing service on port 8081. They are both setup on the same
>> >
>> >>>> computer, while crawler runs on a number of different compute
>> nodes
>> >
>> >>>> spread across the local network and the cloud.
>> >
>> >
>> >>>> When the crawler runs and ingests files I notice that there are
>> >
>> >>>> several connections that open to solr and remain in CLOSE_WAIT
>> >>>> state
>> >
>> >> for hours.
>> >
>> >>>> any idea why this happens? Moving forward I am planning to use
>> >
>> >>>> several hundreds of crawler instances, each running on different
>> >
>> >>>> computer, that will create thousands of such connections and will
>> >
>> >>>> probably create problems to the system.
>> >
>> >>>> Thanks in advance for any help
>> >
>> >>>> Kostas
>> >
>> >
>> >>>> $lsof -i :8081
>> >
>> >>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF
>> >
>> >> NODE
>> >
>> >>>> NAME
>> >
>> >>>> java 92065 kmavrommatis 75u IPv6 0x392c3fa3b63b29cf 0t0
>> >
>> >> TCP
>> >
>> >>>> localhost:49205->localhost:sunproxyadmin (CLOSE_WAIT)
>> >
>> >>>> java 92065 kmavrommatis 76u IPv6 0x392c3fa3b6dcbbcf 0t0
>> >
>> >> TCP
>> >
>> >>>> localhost:49206->localhost:sunproxyadmin (CLOSE_WAIT)
>> >
>> >>>> java 92065 kmavrommatis 77u IPv6 0x392c3fa39fd12e0f 0t0
>> >
>> >> TCP
>> >
>> >>>> localhost:49207->localhost:sunproxyadmin (CLOSE_WAIT)
>> >
>> >>>> java 92065 kmavrommatis 78u IPv6 0x392c3fa39fdcdbcf 0t0
>> >
>> >> TCP
>> >
>> >>>> localhost:49208->localhost:sunproxyadmin (CLOSE_WAIT)
>> >
>> >>>> java 92065 kmavrommatis 79u IPv6 0x392c3fa3b62cde0f 0t0
>> >
>> >> TCP
>> >
>> >>>> localhost:49209->localhost:sunproxyadmin (CLOSE_WAIT)
>> >
>> >>>> java 92065 kmavrommatis 80u IPv6 0x392c3fa39fa2714f 0t0
>> >
>> >> TCP
>> >
>> >>>> localhost:49210->localhost:sunproxyadmin (CLOSE_WAIT)
>> >
>> >>>> java 92065 kmavrommatis 81u IPv6 0x392c3fa3b6c32acf 0t0
>> >
>> >> TCP
>> >
>> >>>> localhost:49211->localhost:sunproxyadmin (CLOSE_WAIT)
>> >
>> >>>> java 92065 kmavrommatis 82u IPv6 0x392c3fa3b6aa714f 0t0
>> >
>> >> TCP
>> >
>> >>>> localhost:49212->localhost:sunproxyadmin (CLOSE_WAIT)
>> >
>> >
>> >
>> >>>> process 92065 is:
>> >
>> >>>> /usr/bin/java -Djava.ext.dirs=../lib
>> >
>> >>>> -Djava.util.logging.config.file=../etc/logging.properties
>> >
>> >>>> -Dorg.apache.oodt.cas.filemgr.properties=../etc/filemgr.properties
>> >
>> >>>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager --portNum
>> 9000
>> >
>> >
>> >>>> *********************************************************
>> >
>> >>>> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL
>> AND
>> >
>> >>>> MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE
>> >>>> USE
>> >
>> >>>> OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
>> >
>> >>>> If the reader is not the intended recipient, or the employee or
>> >
>> >> agent
>> >
>> >>>> responsible to deliver it to the intended recipient, you are
>> hereby
>> >
>> >>>> notified that any dissemination, distribution or copying of this
>> >
>> >>>> communication is strictly prohibited. If you have received this
>> >
>> >>>> communication in error, please reply to the sender to notify us of
>> >
>> >>>> the error and delete the original message. Thank You.
>> >
>> >
>> >
>> >
>> >
>> >>> --
>> >
>> >>> *Lewis*
>> >
>> >
>> >
>> >
>> >
>> >> --
>> >
>> >> *Lewis*
>> >
>> > *********************************************************
>> > THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND
>> > MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE USE
>> > OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
>> > If the reader is not the intended recipient, or the employee or agent
>> > responsible to deliver it to the intended recipient, you are hereby
>> > notified that any dissemination, distribution or copying of this
>> > communication is strictly prohibited. If you have received this
>> > communication in error, please reply to the sender to notify us of
>> the
>> > error and delete the original message. Thank You.
>
>*********************************************************
>THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS
>CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED
>INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL
>OR INDIVIDUALS NAMED ABOVE.
>If the reader is not the intended recipient, or the
>employee or agent responsible to deliver it to the
>intended recipient, you are hereby notified that any
>dissemination, distribution or copying of this
>communication is strictly prohibited. If you have
>received this communication in error, please reply to the
>sender to notify us of the error and delete the original
>message. Thank You.
Re: UPDATE-PROBABLY SOLVED: File manager keeps connections to SOLR in
CLOSE_WAIT state for hours
Posted by Lewis John Mcgibbney <le...@gmail.com>.
https://issues.apache.org/jira/browse/OODT-719
I'll ad Kos ASAP.
Thanks
Lewis
On Thu, Jul 10, 2014 at 10:50 AM, Mattmann, Chris A (3980) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Can someone log into jira and grant kostas perms?
>
> Sent from my iPhone
>
> > On Jul 10, 2014, at 7:12 AM, "Konstantinos Mavrommatis" <
> kmavrommatis@celgene.com> wrote:
> >
> > Unless I am missing something I am not able to create an issue in JIRA :(
> > Thanks
> > K
> >
> >> -----Original Message-----
> >> From: Ramirez, Paul M (398J) [mailto:paul.m.ramirez@jpl.nasa.gov]
> >> Sent: Thursday, July 10, 2014 4:06 PM
> >> To: <de...@oodt.apache.org>
> >> Subject: Re: UPDATE-PROBABLY SOLVED: File manager keeps connections to
> >> SOLR in CLOSE_WAIT state for hours
> >>
> >> Konstantinos,
> >>
> >> Could you open a Jira issue for this and attach a patch? This would be
> >> helpful for the project.
> >>
> >> https://issues.apache.org/jira/browse/OODT/?selectedTab=com.atlassian.j
> >> ira.jira-projects-plugin:summary-panel
> >>
> >>
> >> Thanks,
> >> Paul Ramirez
> >>
> >>>> On Jul 10, 2014, at 5:23 AM, "Konstantinos Mavrommatis"
> >>> <km...@celgene.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> Following the suggestion at
> >>>
> >>> http://blogs.nuxeo.com/development/2013/02/using-httpclient-properly-
> >> a
> >>> void-closewait-tcp-connections/
> >>>
> >>>
> >>>
> >>> I modified the code in
> >> src/main/java/org/apache/oodt/cas/filemgr/catalog/solr/SolrClient.java
> >>>
> >>>
> >>>
> >>> and added the line highlighted in red
> >> (method.setRequestHeader("Connection","close").
> >>>
> >>> This seems to have resolved the problem at least until now ingestion
> >> of few thousand files have not produced any stale connection.
> >>>
> >>> Please let me know your thoughts on this solution.
> >>>
> >>> Best,
> >>>
> >>> Konstantinos
> >>>
> >>>
> >>>
> >>> ======================================================
> >>>
> >>> private String doHttp(HttpMethod method) throws Exception {
> >>>
> >>>
> >>>
> >>> StringBuilder response = new StringBuilder();
> >>>
> >>> BufferedReader br = null;
> >>>
> >>> try {
> >>>
> >>>
> >>>
> >>> // send request
> >>>
> >>> HttpClient httpClient = new HttpClient();
> >>>
> >>> // 10 July 2014. attempting to avoid problems
> >>> with CLOSE_WAIT connections Based on
> >>>
> >>>
> >>> //http://blogs.nuxeo.com/development/2013/02/using-httpclient-
> >> properly
> >>> -avoid-closewait-tcp-connections/
> >>>
> >>> method.setRequestHeader("Connection",
> >> "close");
> >>>
> >>> int statusCode = httpClient.executeMethod(method);
> >>>
> >>>
> >>>
> >>> // read response
> >>>
> >>> if (statusCode != HttpStatus.SC_OK) {
> >>>
> >>>
> >>>
> >>> // still consume the response
> >>>
> >>> method.getResponseBodyAsString();
> >>>
> >>> throw new CatalogException("HTTP method failed: " +
> >>> method.getStatusLine());
> >>>
> >>>
> >>>
> >>> } else {
> >>>
> >>>
> >>>
> >>> // read the response body.
> >>>
> >>> br = new BufferedReader(new
> >>> InputStreamReader(method.getResponseBodyAsStream()));
> >>>
> >>> String readLine;
> >>>
> >>> while(((readLine = br.readLine()) != null)) {
> >>>
> >>> response.append(readLine);
> >>>
> >>> }
> >>>
> >>>
> >>>
> >>> =======================================================
> >>>
> >>>
> >>>
> >>> Information about the system:
> >>>
> >>> OS is MacOS Mavericks,
> >>>
> >>> SOLR version : 4.6.1 , SOLR runs on a single server, no replication
> >> at all.
> >>>
> >>> Tomcat verision: 7.0.50
> >>>
> >>> OODT version is 0.6
> >>>
> >>> Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>> -----Original Message-----
> >>>
> >>>> From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
> >>>
> >>>> Sent: Wednesday, July 09, 2014 4:57 AM
> >>>
> >>>> To: dev@oodt.apache.org
> >>>
> >>>> Subject: Re: File manager keeps connections to SOLR in CLOSE_WAIT
> >>>> state
> >>>
> >>>> for hours
> >>>
> >>>
> >>>> It's just occoured to me that everything in
> >>>
> >>>> org.apache.oodt.cas.filemgr.catalog.solr.SolrCatalog (well,
> >>>
> >>>> specifically
> >>>
> >>>> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient) is done through
> >>>
> >>>> HTTPClient so all of the above may not be relevant.
> >>>
> >>>> Can you please scope all the same?
> >>>
> >>>> Thanks
> >>>
> >>>> Lewis
> >>>
> >>>
> >>>
> >>>> On Tue, Jul 8, 2014 at 10:54 PM, Lewis John Mcgibbney <
> >>>
> >>>> lewis.mcgibbney@gmail.com<ma...@gmail.com>> wrote:
> >>>
> >>>
> >>>>> Hi Konstantinos,
> >>>
> >>>>> OK, I was ablel to scope this one and I have a few questions for
> >> you.
> >>>
> >>>>> 1) Which version of Solr are you using? Is it 3.5, 3.6, 4.0-ALPHA?
> >>>
> >>>> If
> >>>
> >>>>> so please scope this issue [0], the solution would be to upgrade if
> >>>
> >>>>> you are not too long ahead with ingestion as fixes in Solr are
> >> worth
> >>>
> >>>>> having based on recent release cycles.
> >>>
> >>>>> 2) How many cores do you have on Solr server? Also what kind of
> >>>>> setUp
> >>>
> >>>>> do you have? Replication at all?
> >>>
> >>>>> In recent versions of Solr 4.X SolrJ clients should now call
> >>>
> >>>>> shutdown() on their SolrServer object to let it know they don't
> >> want
> >>>
> >>>>> to re-use any existing connections anymore, and when Solr
> >> internally
> >>>
> >>>>> uses SolrJ to talk to other nodes in SolrCloud it should be doing
> >>>
> >>>> this
> >>>
> >>>>> (as of 4.0-ALPHA)so this is why I ask.
> >>>
> >>>>> Lewis
> >>>
> >>>
> >>>>> [0] https://issues.apache.org/jira/browse/SOLR-3280
> >>>
> >>>
> >>>
> >>>>> On Tue, Jul 8, 2014 at 7:14 AM, Konstantinos Mavrommatis <
> >>>
> >>>>> kmavrommatis@celgene.com<ma...@celgene.com>> wrote:
> >>>
> >>>
> >>>>>> Hi,
> >>>
> >>>>>> I have setup OODT filemanager on port 9000, using SOLR as the
> >>>
> >>>>>> indexing service on port 8081. They are both setup on the same
> >>>
> >>>>>> computer, while crawler runs on a number of different compute
> >> nodes
> >>>
> >>>>>> spread across the local network and the cloud.
> >>>
> >>>
> >>>>>> When the crawler runs and ingests files I notice that there are
> >>>
> >>>>>> several connections that open to solr and remain in CLOSE_WAIT
> >>>>>> state
> >>>
> >>>> for hours.
> >>>
> >>>>>> any idea why this happens? Moving forward I am planning to use
> >>>
> >>>>>> several hundreds of crawler instances, each running on different
> >>>
> >>>>>> computer, that will create thousands of such connections and will
> >>>
> >>>>>> probably create problems to the system.
> >>>
> >>>>>> Thanks in advance for any help
> >>>
> >>>>>> Kostas
> >>>
> >>>
> >>>>>> $lsof -i :8081
> >>>
> >>>>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF
> >>>
> >>>> NODE
> >>>
> >>>>>> NAME
> >>>
> >>>>>> java 92065 kmavrommatis 75u IPv6 0x392c3fa3b63b29cf 0t0
> >>>
> >>>> TCP
> >>>
> >>>>>> localhost:49205->localhost:sunproxyadmin (CLOSE_WAIT)
> >>>
> >>>>>> java 92065 kmavrommatis 76u IPv6 0x392c3fa3b6dcbbcf 0t0
> >>>
> >>>> TCP
> >>>
> >>>>>> localhost:49206->localhost:sunproxyadmin (CLOSE_WAIT)
> >>>
> >>>>>> java 92065 kmavrommatis 77u IPv6 0x392c3fa39fd12e0f 0t0
> >>>
> >>>> TCP
> >>>
> >>>>>> localhost:49207->localhost:sunproxyadmin (CLOSE_WAIT)
> >>>
> >>>>>> java 92065 kmavrommatis 78u IPv6 0x392c3fa39fdcdbcf 0t0
> >>>
> >>>> TCP
> >>>
> >>>>>> localhost:49208->localhost:sunproxyadmin (CLOSE_WAIT)
> >>>
> >>>>>> java 92065 kmavrommatis 79u IPv6 0x392c3fa3b62cde0f 0t0
> >>>
> >>>> TCP
> >>>
> >>>>>> localhost:49209->localhost:sunproxyadmin (CLOSE_WAIT)
> >>>
> >>>>>> java 92065 kmavrommatis 80u IPv6 0x392c3fa39fa2714f 0t0
> >>>
> >>>> TCP
> >>>
> >>>>>> localhost:49210->localhost:sunproxyadmin (CLOSE_WAIT)
> >>>
> >>>>>> java 92065 kmavrommatis 81u IPv6 0x392c3fa3b6c32acf 0t0
> >>>
> >>>> TCP
> >>>
> >>>>>> localhost:49211->localhost:sunproxyadmin (CLOSE_WAIT)
> >>>
> >>>>>> java 92065 kmavrommatis 82u IPv6 0x392c3fa3b6aa714f 0t0
> >>>
> >>>> TCP
> >>>
> >>>>>> localhost:49212->localhost:sunproxyadmin (CLOSE_WAIT)
> >>>
> >>>
> >>>
> >>>>>> process 92065 is:
> >>>
> >>>>>> /usr/bin/java -Djava.ext.dirs=../lib
> >>>
> >>>>>> -Djava.util.logging.config.file=../etc/logging.properties
> >>>
> >>>>>> -Dorg.apache.oodt.cas.filemgr.properties=../etc/filemgr.properties
> >>>
> >>>>>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager --portNum
> >> 9000
> >>>
> >>>
> >>>>>> *********************************************************
> >>>
> >>>>>> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL
> >> AND
> >>>
> >>>>>> MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE
> >>>>>> USE
> >>>
> >>>>>> OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
> >>>
> >>>>>> If the reader is not the intended recipient, or the employee or
> >>>
> >>>> agent
> >>>
> >>>>>> responsible to deliver it to the intended recipient, you are
> >> hereby
> >>>
> >>>>>> notified that any dissemination, distribution or copying of this
> >>>
> >>>>>> communication is strictly prohibited. If you have received this
> >>>
> >>>>>> communication in error, please reply to the sender to notify us of
> >>>
> >>>>>> the error and delete the original message. Thank You.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>>> --
> >>>
> >>>>> *Lewis*
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>> --
> >>>
> >>>> *Lewis*
> >>>
> >>> *********************************************************
> >>> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND
> >>> MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE USE
> >>> OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
> >>> If the reader is not the intended recipient, or the employee or agent
> >>> responsible to deliver it to the intended recipient, you are hereby
> >>> notified that any dissemination, distribution or copying of this
> >>> communication is strictly prohibited. If you have received this
> >>> communication in error, please reply to the sender to notify us of
> >> the
> >>> error and delete the original message. Thank You.
> >
> > *********************************************************
> > THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS
> > CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED
> > INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL
> > OR INDIVIDUALS NAMED ABOVE.
> > If the reader is not the intended recipient, or the
> > employee or agent responsible to deliver it to the
> > intended recipient, you are hereby notified that any
> > dissemination, distribution or copying of this
> > communication is strictly prohibited. If you have
> > received this communication in error, please reply to the
> > sender to notify us of the error and delete the original
> > message. Thank You.
>
--
*Lewis*
Re: UPDATE-PROBABLY SOLVED: File manager keeps connections to SOLR
in CLOSE_WAIT state for hours
Posted by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>.
Can someone log into jira and grant kostas perms?
Sent from my iPhone
> On Jul 10, 2014, at 7:12 AM, "Konstantinos Mavrommatis" <km...@celgene.com> wrote:
>
> Unless I am missing something I am not able to create an issue in JIRA :(
> Thanks
> K
>
>> -----Original Message-----
>> From: Ramirez, Paul M (398J) [mailto:paul.m.ramirez@jpl.nasa.gov]
>> Sent: Thursday, July 10, 2014 4:06 PM
>> To: <de...@oodt.apache.org>
>> Subject: Re: UPDATE-PROBABLY SOLVED: File manager keeps connections to
>> SOLR in CLOSE_WAIT state for hours
>>
>> Konstantinos,
>>
>> Could you open a Jira issue for this and attach a patch? This would be
>> helpful for the project.
>>
>> https://issues.apache.org/jira/browse/OODT/?selectedTab=com.atlassian.j
>> ira.jira-projects-plugin:summary-panel
>>
>>
>> Thanks,
>> Paul Ramirez
>>
>>>> On Jul 10, 2014, at 5:23 AM, "Konstantinos Mavrommatis"
>>> <km...@celgene.com> wrote:
>>>
>>> Hi,
>>>
>>> Following the suggestion at
>>>
>>> http://blogs.nuxeo.com/development/2013/02/using-httpclient-properly-
>> a
>>> void-closewait-tcp-connections/
>>>
>>>
>>>
>>> I modified the code in
>> src/main/java/org/apache/oodt/cas/filemgr/catalog/solr/SolrClient.java
>>>
>>>
>>>
>>> and added the line highlighted in red
>> (method.setRequestHeader("Connection","close").
>>>
>>> This seems to have resolved the problem at least until now ingestion
>> of few thousand files have not produced any stale connection.
>>>
>>> Please let me know your thoughts on this solution.
>>>
>>> Best,
>>>
>>> Konstantinos
>>>
>>>
>>>
>>> ======================================================
>>>
>>> private String doHttp(HttpMethod method) throws Exception {
>>>
>>>
>>>
>>> StringBuilder response = new StringBuilder();
>>>
>>> BufferedReader br = null;
>>>
>>> try {
>>>
>>>
>>>
>>> // send request
>>>
>>> HttpClient httpClient = new HttpClient();
>>>
>>> // 10 July 2014. attempting to avoid problems
>>> with CLOSE_WAIT connections Based on
>>>
>>>
>>> //http://blogs.nuxeo.com/development/2013/02/using-httpclient-
>> properly
>>> -avoid-closewait-tcp-connections/
>>>
>>> method.setRequestHeader("Connection",
>> "close");
>>>
>>> int statusCode = httpClient.executeMethod(method);
>>>
>>>
>>>
>>> // read response
>>>
>>> if (statusCode != HttpStatus.SC_OK) {
>>>
>>>
>>>
>>> // still consume the response
>>>
>>> method.getResponseBodyAsString();
>>>
>>> throw new CatalogException("HTTP method failed: " +
>>> method.getStatusLine());
>>>
>>>
>>>
>>> } else {
>>>
>>>
>>>
>>> // read the response body.
>>>
>>> br = new BufferedReader(new
>>> InputStreamReader(method.getResponseBodyAsStream()));
>>>
>>> String readLine;
>>>
>>> while(((readLine = br.readLine()) != null)) {
>>>
>>> response.append(readLine);
>>>
>>> }
>>>
>>>
>>>
>>> =======================================================
>>>
>>>
>>>
>>> Information about the system:
>>>
>>> OS is MacOS Mavericks,
>>>
>>> SOLR version : 4.6.1 , SOLR runs on a single server, no replication
>> at all.
>>>
>>> Tomcat verision: 7.0.50
>>>
>>> OODT version is 0.6
>>>
>>> Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>> -----Original Message-----
>>>
>>>> From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
>>>
>>>> Sent: Wednesday, July 09, 2014 4:57 AM
>>>
>>>> To: dev@oodt.apache.org
>>>
>>>> Subject: Re: File manager keeps connections to SOLR in CLOSE_WAIT
>>>> state
>>>
>>>> for hours
>>>
>>>
>>>> It's just occoured to me that everything in
>>>
>>>> org.apache.oodt.cas.filemgr.catalog.solr.SolrCatalog (well,
>>>
>>>> specifically
>>>
>>>> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient) is done through
>>>
>>>> HTTPClient so all of the above may not be relevant.
>>>
>>>> Can you please scope all the same?
>>>
>>>> Thanks
>>>
>>>> Lewis
>>>
>>>
>>>
>>>> On Tue, Jul 8, 2014 at 10:54 PM, Lewis John Mcgibbney <
>>>
>>>> lewis.mcgibbney@gmail.com<ma...@gmail.com>> wrote:
>>>
>>>
>>>>> Hi Konstantinos,
>>>
>>>>> OK, I was ablel to scope this one and I have a few questions for
>> you.
>>>
>>>>> 1) Which version of Solr are you using? Is it 3.5, 3.6, 4.0-ALPHA?
>>>
>>>> If
>>>
>>>>> so please scope this issue [0], the solution would be to upgrade if
>>>
>>>>> you are not too long ahead with ingestion as fixes in Solr are
>> worth
>>>
>>>>> having based on recent release cycles.
>>>
>>>>> 2) How many cores do you have on Solr server? Also what kind of
>>>>> setUp
>>>
>>>>> do you have? Replication at all?
>>>
>>>>> In recent versions of Solr 4.X SolrJ clients should now call
>>>
>>>>> shutdown() on their SolrServer object to let it know they don't
>> want
>>>
>>>>> to re-use any existing connections anymore, and when Solr
>> internally
>>>
>>>>> uses SolrJ to talk to other nodes in SolrCloud it should be doing
>>>
>>>> this
>>>
>>>>> (as of 4.0-ALPHA)so this is why I ask.
>>>
>>>>> Lewis
>>>
>>>
>>>>> [0] https://issues.apache.org/jira/browse/SOLR-3280
>>>
>>>
>>>
>>>>> On Tue, Jul 8, 2014 at 7:14 AM, Konstantinos Mavrommatis <
>>>
>>>>> kmavrommatis@celgene.com<ma...@celgene.com>> wrote:
>>>
>>>
>>>>>> Hi,
>>>
>>>>>> I have setup OODT filemanager on port 9000, using SOLR as the
>>>
>>>>>> indexing service on port 8081. They are both setup on the same
>>>
>>>>>> computer, while crawler runs on a number of different compute
>> nodes
>>>
>>>>>> spread across the local network and the cloud.
>>>
>>>
>>>>>> When the crawler runs and ingests files I notice that there are
>>>
>>>>>> several connections that open to solr and remain in CLOSE_WAIT
>>>>>> state
>>>
>>>> for hours.
>>>
>>>>>> any idea why this happens? Moving forward I am planning to use
>>>
>>>>>> several hundreds of crawler instances, each running on different
>>>
>>>>>> computer, that will create thousands of such connections and will
>>>
>>>>>> probably create problems to the system.
>>>
>>>>>> Thanks in advance for any help
>>>
>>>>>> Kostas
>>>
>>>
>>>>>> $lsof -i :8081
>>>
>>>>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF
>>>
>>>> NODE
>>>
>>>>>> NAME
>>>
>>>>>> java 92065 kmavrommatis 75u IPv6 0x392c3fa3b63b29cf 0t0
>>>
>>>> TCP
>>>
>>>>>> localhost:49205->localhost:sunproxyadmin (CLOSE_WAIT)
>>>
>>>>>> java 92065 kmavrommatis 76u IPv6 0x392c3fa3b6dcbbcf 0t0
>>>
>>>> TCP
>>>
>>>>>> localhost:49206->localhost:sunproxyadmin (CLOSE_WAIT)
>>>
>>>>>> java 92065 kmavrommatis 77u IPv6 0x392c3fa39fd12e0f 0t0
>>>
>>>> TCP
>>>
>>>>>> localhost:49207->localhost:sunproxyadmin (CLOSE_WAIT)
>>>
>>>>>> java 92065 kmavrommatis 78u IPv6 0x392c3fa39fdcdbcf 0t0
>>>
>>>> TCP
>>>
>>>>>> localhost:49208->localhost:sunproxyadmin (CLOSE_WAIT)
>>>
>>>>>> java 92065 kmavrommatis 79u IPv6 0x392c3fa3b62cde0f 0t0
>>>
>>>> TCP
>>>
>>>>>> localhost:49209->localhost:sunproxyadmin (CLOSE_WAIT)
>>>
>>>>>> java 92065 kmavrommatis 80u IPv6 0x392c3fa39fa2714f 0t0
>>>
>>>> TCP
>>>
>>>>>> localhost:49210->localhost:sunproxyadmin (CLOSE_WAIT)
>>>
>>>>>> java 92065 kmavrommatis 81u IPv6 0x392c3fa3b6c32acf 0t0
>>>
>>>> TCP
>>>
>>>>>> localhost:49211->localhost:sunproxyadmin (CLOSE_WAIT)
>>>
>>>>>> java 92065 kmavrommatis 82u IPv6 0x392c3fa3b6aa714f 0t0
>>>
>>>> TCP
>>>
>>>>>> localhost:49212->localhost:sunproxyadmin (CLOSE_WAIT)
>>>
>>>
>>>
>>>>>> process 92065 is:
>>>
>>>>>> /usr/bin/java -Djava.ext.dirs=../lib
>>>
>>>>>> -Djava.util.logging.config.file=../etc/logging.properties
>>>
>>>>>> -Dorg.apache.oodt.cas.filemgr.properties=../etc/filemgr.properties
>>>
>>>>>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager --portNum
>> 9000
>>>
>>>
>>>>>> *********************************************************
>>>
>>>>>> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL
>> AND
>>>
>>>>>> MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE
>>>>>> USE
>>>
>>>>>> OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
>>>
>>>>>> If the reader is not the intended recipient, or the employee or
>>>
>>>> agent
>>>
>>>>>> responsible to deliver it to the intended recipient, you are
>> hereby
>>>
>>>>>> notified that any dissemination, distribution or copying of this
>>>
>>>>>> communication is strictly prohibited. If you have received this
>>>
>>>>>> communication in error, please reply to the sender to notify us of
>>>
>>>>>> the error and delete the original message. Thank You.
>>>
>>>
>>>
>>>
>>>
>>>>> --
>>>
>>>>> *Lewis*
>>>
>>>
>>>
>>>
>>>
>>>> --
>>>
>>>> *Lewis*
>>>
>>> *********************************************************
>>> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND
>>> MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE USE
>>> OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
>>> If the reader is not the intended recipient, or the employee or agent
>>> responsible to deliver it to the intended recipient, you are hereby
>>> notified that any dissemination, distribution or copying of this
>>> communication is strictly prohibited. If you have received this
>>> communication in error, please reply to the sender to notify us of
>> the
>>> error and delete the original message. Thank You.
>
> *********************************************************
> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS
> CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED
> INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL
> OR INDIVIDUALS NAMED ABOVE.
> If the reader is not the intended recipient, or the
> employee or agent responsible to deliver it to the
> intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this
> communication is strictly prohibited. If you have
> received this communication in error, please reply to the
> sender to notify us of the error and delete the original
> message. Thank You.
RE: UPDATE-PROBABLY SOLVED: File manager keeps connections to SOLR
in CLOSE_WAIT state for hours
Posted by Konstantinos Mavrommatis <km...@celgene.com>.
Unless I am missing something I am not able to create an issue in JIRA :(
Thanks
K
> -----Original Message-----
> From: Ramirez, Paul M (398J) [mailto:paul.m.ramirez@jpl.nasa.gov]
> Sent: Thursday, July 10, 2014 4:06 PM
> To: <de...@oodt.apache.org>
> Subject: Re: UPDATE-PROBABLY SOLVED: File manager keeps connections to
> SOLR in CLOSE_WAIT state for hours
>
> Konstantinos,
>
> Could you open a Jira issue for this and attach a patch? This would be
> helpful for the project.
>
> https://issues.apache.org/jira/browse/OODT/?selectedTab=com.atlassian.j
> ira.jira-projects-plugin:summary-panel
>
>
> Thanks,
> Paul Ramirez
>
> > On Jul 10, 2014, at 5:23 AM, "Konstantinos Mavrommatis"
> <km...@celgene.com> wrote:
> >
> > Hi,
> >
> > Following the suggestion at
> >
> > http://blogs.nuxeo.com/development/2013/02/using-httpclient-properly-
> a
> > void-closewait-tcp-connections/
> >
> >
> >
> > I modified the code in
> >
> >
> src/main/java/org/apache/oodt/cas/filemgr/catalog/solr/SolrClient.java
> >
> >
> >
> > and added the line highlighted in red
> (method.setRequestHeader("Connection","close").
> >
> > This seems to have resolved the problem at least until now ingestion
> of few thousand files have not produced any stale connection.
> >
> > Please let me know your thoughts on this solution.
> >
> > Best,
> >
> > Konstantinos
> >
> >
> >
> > ======================================================
> >
> > private String doHttp(HttpMethod method) throws Exception {
> >
> >
> >
> > StringBuilder response = new StringBuilder();
> >
> > BufferedReader br = null;
> >
> > try {
> >
> >
> >
> > // send request
> >
> > HttpClient httpClient = new HttpClient();
> >
> > // 10 July 2014. attempting to avoid problems
> > with CLOSE_WAIT connections Based on
> >
> >
> > //http://blogs.nuxeo.com/development/2013/02/using-httpclient-
> properly
> > -avoid-closewait-tcp-connections/
> >
> > method.setRequestHeader("Connection",
> "close");
> >
> > int statusCode = httpClient.executeMethod(method);
> >
> >
> >
> > // read response
> >
> > if (statusCode != HttpStatus.SC_OK) {
> >
> >
> >
> > // still consume the response
> >
> > method.getResponseBodyAsString();
> >
> > throw new CatalogException("HTTP method failed: " +
> > method.getStatusLine());
> >
> >
> >
> > } else {
> >
> >
> >
> > // read the response body.
> >
> > br = new BufferedReader(new
> > InputStreamReader(method.getResponseBodyAsStream()));
> >
> > String readLine;
> >
> > while(((readLine = br.readLine()) != null)) {
> >
> > response.append(readLine);
> >
> > }
> >
> >
> >
> > =======================================================
> >
> >
> >
> > Information about the system:
> >
> > OS is MacOS Mavericks,
> >
> > SOLR version : 4.6.1 , SOLR runs on a single server, no replication
> at all.
> >
> > Tomcat verision: 7.0.50
> >
> > OODT version is 0.6
> >
> > Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
> >
> >
> >
> >
> >
> >
> >
> >> -----Original Message-----
> >
> >> From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
> >
> >> Sent: Wednesday, July 09, 2014 4:57 AM
> >
> >> To: dev@oodt.apache.org
> >
> >> Subject: Re: File manager keeps connections to SOLR in CLOSE_WAIT
> >> state
> >
> >> for hours
> >
> >
> >> It's just occoured to me that everything in
> >
> >> org.apache.oodt.cas.filemgr.catalog.solr.SolrCatalog (well,
> >
> >> specifically
> >
> >> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient) is done through
> >
> >> HTTPClient so all of the above may not be relevant.
> >
> >> Can you please scope all the same?
> >
> >> Thanks
> >
> >> Lewis
> >
> >
> >
> >> On Tue, Jul 8, 2014 at 10:54 PM, Lewis John Mcgibbney <
> >
> >> lewis.mcgibbney@gmail.com<ma...@gmail.com>> wrote:
> >
> >
> >>> Hi Konstantinos,
> >
> >>> OK, I was ablel to scope this one and I have a few questions for
> you.
> >
> >>> 1) Which version of Solr are you using? Is it 3.5, 3.6, 4.0-ALPHA?
> >
> >> If
> >
> >>> so please scope this issue [0], the solution would be to upgrade if
> >
> >>> you are not too long ahead with ingestion as fixes in Solr are
> worth
> >
> >>> having based on recent release cycles.
> >
> >>> 2) How many cores do you have on Solr server? Also what kind of
> >>> setUp
> >
> >>> do you have? Replication at all?
> >
> >>> In recent versions of Solr 4.X SolrJ clients should now call
> >
> >>> shutdown() on their SolrServer object to let it know they don't
> want
> >
> >>> to re-use any existing connections anymore, and when Solr
> internally
> >
> >>> uses SolrJ to talk to other nodes in SolrCloud it should be doing
> >
> >> this
> >
> >>> (as of 4.0-ALPHA)so this is why I ask.
> >
> >>> Lewis
> >
> >
> >>> [0] https://issues.apache.org/jira/browse/SOLR-3280
> >
> >
> >
> >>> On Tue, Jul 8, 2014 at 7:14 AM, Konstantinos Mavrommatis <
> >
> >>> kmavrommatis@celgene.com<ma...@celgene.com>> wrote:
> >
> >
> >>>> Hi,
> >
> >>>> I have setup OODT filemanager on port 9000, using SOLR as the
> >
> >>>> indexing service on port 8081. They are both setup on the same
> >
> >>>> computer, while crawler runs on a number of different compute
> nodes
> >
> >>>> spread across the local network and the cloud.
> >
> >
> >>>> When the crawler runs and ingests files I notice that there are
> >
> >>>> several connections that open to solr and remain in CLOSE_WAIT
> >>>> state
> >
> >> for hours.
> >
> >>>> any idea why this happens? Moving forward I am planning to use
> >
> >>>> several hundreds of crawler instances, each running on different
> >
> >>>> computer, that will create thousands of such connections and will
> >
> >>>> probably create problems to the system.
> >
> >>>> Thanks in advance for any help
> >
> >>>> Kostas
> >
> >
> >>>> $lsof -i :8081
> >
> >>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF
> >
> >> NODE
> >
> >>>> NAME
> >
> >>>> java 92065 kmavrommatis 75u IPv6 0x392c3fa3b63b29cf 0t0
> >
> >> TCP
> >
> >>>> localhost:49205->localhost:sunproxyadmin (CLOSE_WAIT)
> >
> >>>> java 92065 kmavrommatis 76u IPv6 0x392c3fa3b6dcbbcf 0t0
> >
> >> TCP
> >
> >>>> localhost:49206->localhost:sunproxyadmin (CLOSE_WAIT)
> >
> >>>> java 92065 kmavrommatis 77u IPv6 0x392c3fa39fd12e0f 0t0
> >
> >> TCP
> >
> >>>> localhost:49207->localhost:sunproxyadmin (CLOSE_WAIT)
> >
> >>>> java 92065 kmavrommatis 78u IPv6 0x392c3fa39fdcdbcf 0t0
> >
> >> TCP
> >
> >>>> localhost:49208->localhost:sunproxyadmin (CLOSE_WAIT)
> >
> >>>> java 92065 kmavrommatis 79u IPv6 0x392c3fa3b62cde0f 0t0
> >
> >> TCP
> >
> >>>> localhost:49209->localhost:sunproxyadmin (CLOSE_WAIT)
> >
> >>>> java 92065 kmavrommatis 80u IPv6 0x392c3fa39fa2714f 0t0
> >
> >> TCP
> >
> >>>> localhost:49210->localhost:sunproxyadmin (CLOSE_WAIT)
> >
> >>>> java 92065 kmavrommatis 81u IPv6 0x392c3fa3b6c32acf 0t0
> >
> >> TCP
> >
> >>>> localhost:49211->localhost:sunproxyadmin (CLOSE_WAIT)
> >
> >>>> java 92065 kmavrommatis 82u IPv6 0x392c3fa3b6aa714f 0t0
> >
> >> TCP
> >
> >>>> localhost:49212->localhost:sunproxyadmin (CLOSE_WAIT)
> >
> >
> >
> >>>> process 92065 is:
> >
> >>>> /usr/bin/java -Djava.ext.dirs=../lib
> >
> >>>> -Djava.util.logging.config.file=../etc/logging.properties
> >
> >>>> -Dorg.apache.oodt.cas.filemgr.properties=../etc/filemgr.properties
> >
> >>>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager --portNum
> 9000
> >
> >
> >>>> *********************************************************
> >
> >>>> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL
> AND
> >
> >>>> MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE
> >>>> USE
> >
> >>>> OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
> >
> >>>> If the reader is not the intended recipient, or the employee or
> >
> >> agent
> >
> >>>> responsible to deliver it to the intended recipient, you are
> hereby
> >
> >>>> notified that any dissemination, distribution or copying of this
> >
> >>>> communication is strictly prohibited. If you have received this
> >
> >>>> communication in error, please reply to the sender to notify us of
> >
> >>>> the error and delete the original message. Thank You.
> >
> >
> >
> >
> >
> >>> --
> >
> >>> *Lewis*
> >
> >
> >
> >
> >
> >> --
> >
> >> *Lewis*
> >
> > *********************************************************
> > THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND
> > MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE USE
> > OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
> > If the reader is not the intended recipient, or the employee or agent
> > responsible to deliver it to the intended recipient, you are hereby
> > notified that any dissemination, distribution or copying of this
> > communication is strictly prohibited. If you have received this
> > communication in error, please reply to the sender to notify us of
> the
> > error and delete the original message. Thank You.
*********************************************************
THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS
CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED
INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL
OR INDIVIDUALS NAMED ABOVE.
If the reader is not the intended recipient, or the
employee or agent responsible to deliver it to the
intended recipient, you are hereby notified that any
dissemination, distribution or copying of this
communication is strictly prohibited. If you have
received this communication in error, please reply to the
sender to notify us of the error and delete the original
message. Thank You.
Re: UPDATE-PROBABLY SOLVED: File manager keeps connections to SOLR
in CLOSE_WAIT state for hours
Posted by "Ramirez, Paul M (398J)" <pa...@jpl.nasa.gov>.
Konstantinos,
Could you open a Jira issue for this and attach a patch? This would be helpful for the project.
https://issues.apache.org/jira/browse/OODT/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel
Thanks,
Paul Ramirez
> On Jul 10, 2014, at 5:23 AM, "Konstantinos Mavrommatis" <km...@celgene.com> wrote:
>
> Hi,
>
> Following the suggestion at
>
> http://blogs.nuxeo.com/development/2013/02/using-httpclient-properly-avoid-closewait-tcp-connections/
>
>
>
> I modified the code in
>
> src/main/java/org/apache/oodt/cas/filemgr/catalog/solr/SolrClient.java
>
>
>
> and added the line highlighted in red (method.setRequestHeader(“Connection”,”close”).
>
> This seems to have resolved the problem at least until now ingestion of few thousand files have not produced any stale connection.
>
> Please let me know your thoughts on this solution.
>
> Best,
>
> Konstantinos
>
>
>
> ======================================================
>
> private String doHttp(HttpMethod method) throws Exception {
>
>
>
> StringBuilder response = new StringBuilder();
>
> BufferedReader br = null;
>
> try {
>
>
>
> // send request
>
> HttpClient httpClient = new HttpClient();
>
> // 10 July 2014. attempting to avoid problems with CLOSE_WAIT connections Based on
>
> //http://blogs.nuxeo.com/development/2013/02/using-httpclient-properly-avoid-closewait-tcp-connections/
>
> method.setRequestHeader("Connection", "close");
>
> int statusCode = httpClient.executeMethod(method);
>
>
>
> // read response
>
> if (statusCode != HttpStatus.SC_OK) {
>
>
>
> // still consume the response
>
> method.getResponseBodyAsString();
>
> throw new CatalogException("HTTP method failed: " + method.getStatusLine());
>
>
>
> } else {
>
>
>
> // read the response body.
>
> br = new BufferedReader(new InputStreamReader(method.getResponseBodyAsStream()));
>
> String readLine;
>
> while(((readLine = br.readLine()) != null)) {
>
> response.append(readLine);
>
> }
>
>
>
> =======================================================
>
>
>
> Information about the system:
>
> OS is MacOS Mavericks,
>
> SOLR version : 4.6.1 , SOLR runs on a single server, no replication at all.
>
> Tomcat verision: 7.0.50
>
> OODT version is 0.6
>
> Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
>
>
>
>
>
>
>
>> -----Original Message-----
>
>> From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
>
>> Sent: Wednesday, July 09, 2014 4:57 AM
>
>> To: dev@oodt.apache.org
>
>> Subject: Re: File manager keeps connections to SOLR in CLOSE_WAIT state
>
>> for hours
>
>
>> It's just occoured to me that everything in
>
>> org.apache.oodt.cas.filemgr.catalog.solr.SolrCatalog (well,
>
>> specifically
>
>> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient) is done through
>
>> HTTPClient so all of the above may not be relevant.
>
>> Can you please scope all the same?
>
>> Thanks
>
>> Lewis
>
>
>
>> On Tue, Jul 8, 2014 at 10:54 PM, Lewis John Mcgibbney <
>
>> lewis.mcgibbney@gmail.com<ma...@gmail.com>> wrote:
>
>
>>> Hi Konstantinos,
>
>>> OK, I was ablel to scope this one and I have a few questions for you.
>
>>> 1) Which version of Solr are you using? Is it 3.5, 3.6, 4.0-ALPHA?
>
>> If
>
>>> so please scope this issue [0], the solution would be to upgrade if
>
>>> you are not too long ahead with ingestion as fixes in Solr are worth
>
>>> having based on recent release cycles.
>
>>> 2) How many cores do you have on Solr server? Also what kind of setUp
>
>>> do you have? Replication at all?
>
>>> In recent versions of Solr 4.X SolrJ clients should now call
>
>>> shutdown() on their SolrServer object to let it know they don't want
>
>>> to re-use any existing connections anymore, and when Solr internally
>
>>> uses SolrJ to talk to other nodes in SolrCloud it should be doing
>
>> this
>
>>> (as of 4.0-ALPHA)so this is why I ask.
>
>>> Lewis
>
>
>>> [0] https://issues.apache.org/jira/browse/SOLR-3280
>
>
>
>>> On Tue, Jul 8, 2014 at 7:14 AM, Konstantinos Mavrommatis <
>
>>> kmavrommatis@celgene.com<ma...@celgene.com>> wrote:
>
>
>>>> Hi,
>
>>>> I have setup OODT filemanager on port 9000, using SOLR as the
>
>>>> indexing service on port 8081. They are both setup on the same
>
>>>> computer, while crawler runs on a number of different compute nodes
>
>>>> spread across the local network and the cloud.
>
>
>>>> When the crawler runs and ingests files I notice that there are
>
>>>> several connections that open to solr and remain in CLOSE_WAIT state
>
>> for hours.
>
>>>> any idea why this happens? Moving forward I am planning to use
>
>>>> several hundreds of crawler instances, each running on different
>
>>>> computer, that will create thousands of such connections and will
>
>>>> probably create problems to the system.
>
>>>> Thanks in advance for any help
>
>>>> Kostas
>
>
>>>> $lsof -i :8081
>
>>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF
>
>> NODE
>
>>>> NAME
>
>>>> java 92065 kmavrommatis 75u IPv6 0x392c3fa3b63b29cf 0t0
>
>> TCP
>
>>>> localhost:49205->localhost:sunproxyadmin (CLOSE_WAIT)
>
>>>> java 92065 kmavrommatis 76u IPv6 0x392c3fa3b6dcbbcf 0t0
>
>> TCP
>
>>>> localhost:49206->localhost:sunproxyadmin (CLOSE_WAIT)
>
>>>> java 92065 kmavrommatis 77u IPv6 0x392c3fa39fd12e0f 0t0
>
>> TCP
>
>>>> localhost:49207->localhost:sunproxyadmin (CLOSE_WAIT)
>
>>>> java 92065 kmavrommatis 78u IPv6 0x392c3fa39fdcdbcf 0t0
>
>> TCP
>
>>>> localhost:49208->localhost:sunproxyadmin (CLOSE_WAIT)
>
>>>> java 92065 kmavrommatis 79u IPv6 0x392c3fa3b62cde0f 0t0
>
>> TCP
>
>>>> localhost:49209->localhost:sunproxyadmin (CLOSE_WAIT)
>
>>>> java 92065 kmavrommatis 80u IPv6 0x392c3fa39fa2714f 0t0
>
>> TCP
>
>>>> localhost:49210->localhost:sunproxyadmin (CLOSE_WAIT)
>
>>>> java 92065 kmavrommatis 81u IPv6 0x392c3fa3b6c32acf 0t0
>
>> TCP
>
>>>> localhost:49211->localhost:sunproxyadmin (CLOSE_WAIT)
>
>>>> java 92065 kmavrommatis 82u IPv6 0x392c3fa3b6aa714f 0t0
>
>> TCP
>
>>>> localhost:49212->localhost:sunproxyadmin (CLOSE_WAIT)
>
>
>
>>>> process 92065 is:
>
>>>> /usr/bin/java -Djava.ext.dirs=../lib
>
>>>> -Djava.util.logging.config.file=../etc/logging.properties
>
>>>> -Dorg.apache.oodt.cas.filemgr.properties=../etc/filemgr.properties
>
>>>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager --portNum 9000
>
>
>>>> *********************************************************
>
>>>> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND
>
>>>> MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE USE
>
>>>> OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
>
>>>> If the reader is not the intended recipient, or the employee or
>
>> agent
>
>>>> responsible to deliver it to the intended recipient, you are hereby
>
>>>> notified that any dissemination, distribution or copying of this
>
>>>> communication is strictly prohibited. If you have received this
>
>>>> communication in error, please reply to the sender to notify us of
>
>>>> the error and delete the original message. Thank You.
>
>
>
>
>
>>> --
>
>>> *Lewis*
>
>
>
>
>
>> --
>
>> *Lewis*
>
> *********************************************************
> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS
> CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED
> INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL
> OR INDIVIDUALS NAMED ABOVE.
> If the reader is not the intended recipient, or the
> employee or agent responsible to deliver it to the
> intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this
> communication is strictly prohibited. If you have
> received this communication in error, please reply to the
> sender to notify us of the error and delete the original
> message. Thank You.
Re: UPDATE-PROBABLY SOLVED: File manager keeps connections to SOLR in
CLOSE_WAIT state for hours
Posted by Lewis John Mcgibbney <le...@gmail.com>.
This is fantastic.
Are you able to submit a patch for this?
If not then I will do so.
Thanks again.
Lewis
On Thu, Jul 10, 2014 at 5:22 AM, Konstantinos Mavrommatis <
kmavrommatis@celgene.com> wrote:
> Hi,
>
> Following the suggestion at
>
>
> http://blogs.nuxeo.com/development/2013/02/using-httpclient-properly-avoid-closewait-tcp-connections/
>
>
>
> I modified the code in
>
> src/main/java/org/apache/oodt/cas/filemgr/catalog/solr/SolrClient.java
>
>
>
> and added the line highlighted in red
> (method.setRequestHeader(“Connection”,”close”).
>
> This seems to have resolved the problem at least until now ingestion of
> few thousand files have not produced any stale connection.
>
> Please let me know your thoughts on this solution.
>
> Best,
>
> Konstantinos
>
>
>
> ======================================================
>
> private String doHttp(HttpMethod method) throws Exception {
>
>
>
> StringBuilder response = new StringBuilder();
>
> BufferedReader br = null;
>
> try {
>
>
>
> // send request
>
> HttpClient httpClient = new HttpClient();
>
> // 10 July 2014. attempting to avoid problems with
> CLOSE_WAIT connections Based on
>
> //
> http://blogs.nuxeo.com/development/2013/02/using-httpclient-properly-avoid-closewait-tcp-connections/
>
> method.setRequestHeader("Connection", "close");
>
> int statusCode = httpClient.executeMethod(method);
>
>
>
> // read response
>
> if (statusCode != HttpStatus.SC_OK) {
>
>
>
> // still consume the response
>
> method.getResponseBodyAsString();
>
> throw new CatalogException("HTTP method failed: " +
> method.getStatusLine());
>
>
>
> } else {
>
>
>
> // read the response body.
>
> br = new BufferedReader(new
> InputStreamReader(method.getResponseBodyAsStream()));
>
> String readLine;
>
> while(((readLine = br.readLine()) != null)) {
>
> response.append(readLine);
>
> }
>
>
>
> =======================================================
>
>
>
> Information about the system:
>
> OS is MacOS Mavericks,
>
> SOLR version : 4.6.1 , SOLR runs on a single server, no replication at all.
>
> Tomcat verision: 7.0.50
>
> OODT version is 0.6
>
> Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
>
>
>
>
>
>
>
> > -----Original Message-----
>
> > From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
>
> > Sent: Wednesday, July 09, 2014 4:57 AM
>
> > To: dev@oodt.apache.org
>
> > Subject: Re: File manager keeps connections to SOLR in CLOSE_WAIT state
>
> > for hours
>
> >
>
> > It's just occoured to me that everything in
>
> > org.apache.oodt.cas.filemgr.catalog.solr.SolrCatalog (well,
>
> > specifically
>
> > org.apache.oodt.cas.filemgr.catalog.solr.SolrClient) is done through
>
> > HTTPClient so all of the above may not be relevant.
>
> > Can you please scope all the same?
>
> > Thanks
>
> > Lewis
>
> >
>
> >
>
> > On Tue, Jul 8, 2014 at 10:54 PM, Lewis John Mcgibbney <
>
> > lewis.mcgibbney@gmail.com<ma...@gmail.com>> wrote:
>
> >
>
> > > Hi Konstantinos,
>
> > > OK, I was ablel to scope this one and I have a few questions for you.
>
> > > 1) Which version of Solr are you using? Is it 3.5, 3.6, 4.0-ALPHA?
>
> > If
>
> > > so please scope this issue [0], the solution would be to upgrade if
>
> > > you are not too long ahead with ingestion as fixes in Solr are worth
>
> > > having based on recent release cycles.
>
> > > 2) How many cores do you have on Solr server? Also what kind of setUp
>
> > > do you have? Replication at all?
>
> > > In recent versions of Solr 4.X SolrJ clients should now call
>
> > > shutdown() on their SolrServer object to let it know they don't want
>
> > > to re-use any existing connections anymore, and when Solr internally
>
> > > uses SolrJ to talk to other nodes in SolrCloud it should be doing
>
> > this
>
> > > (as of 4.0-ALPHA)so this is why I ask.
>
> > > Lewis
>
> > >
>
> > > [0] https://issues.apache.org/jira/browse/SOLR-3280
>
> > >
>
> > >
>
> > > On Tue, Jul 8, 2014 at 7:14 AM, Konstantinos Mavrommatis <
>
> > > kmavrommatis@celgene.com<ma...@celgene.com>> wrote:
>
> > >
>
> > >> Hi,
>
> > >> I have setup OODT filemanager on port 9000, using SOLR as the
>
> > >> indexing service on port 8081. They are both setup on the same
>
> > >> computer, while crawler runs on a number of different compute nodes
>
> > >> spread across the local network and the cloud.
>
> > >>
>
> > >> When the crawler runs and ingests files I notice that there are
>
> > >> several connections that open to solr and remain in CLOSE_WAIT state
>
> > for hours.
>
> > >> any idea why this happens? Moving forward I am planning to use
>
> > >> several hundreds of crawler instances, each running on different
>
> > >> computer, that will create thousands of such connections and will
>
> > >> probably create problems to the system.
>
> > >> Thanks in advance for any help
>
> > >> Kostas
>
> > >>
>
> > >> $lsof -i :8081
>
> > >> COMMAND PID USER FD TYPE DEVICE SIZE/OFF
>
> > NODE
>
> > >> NAME
>
> > >> java 92065 kmavrommatis 75u IPv6 0x392c3fa3b63b29cf 0t0
>
> > TCP
>
> > >> localhost:49205->localhost:sunproxyadmin (CLOSE_WAIT)
>
> > >> java 92065 kmavrommatis 76u IPv6 0x392c3fa3b6dcbbcf 0t0
>
> > TCP
>
> > >> localhost:49206->localhost:sunproxyadmin (CLOSE_WAIT)
>
> > >> java 92065 kmavrommatis 77u IPv6 0x392c3fa39fd12e0f 0t0
>
> > TCP
>
> > >> localhost:49207->localhost:sunproxyadmin (CLOSE_WAIT)
>
> > >> java 92065 kmavrommatis 78u IPv6 0x392c3fa39fdcdbcf 0t0
>
> > TCP
>
> > >> localhost:49208->localhost:sunproxyadmin (CLOSE_WAIT)
>
> > >> java 92065 kmavrommatis 79u IPv6 0x392c3fa3b62cde0f 0t0
>
> > TCP
>
> > >> localhost:49209->localhost:sunproxyadmin (CLOSE_WAIT)
>
> > >> java 92065 kmavrommatis 80u IPv6 0x392c3fa39fa2714f 0t0
>
> > TCP
>
> > >> localhost:49210->localhost:sunproxyadmin (CLOSE_WAIT)
>
> > >> java 92065 kmavrommatis 81u IPv6 0x392c3fa3b6c32acf 0t0
>
> > TCP
>
> > >> localhost:49211->localhost:sunproxyadmin (CLOSE_WAIT)
>
> > >> java 92065 kmavrommatis 82u IPv6 0x392c3fa3b6aa714f 0t0
>
> > TCP
>
> > >> localhost:49212->localhost:sunproxyadmin (CLOSE_WAIT)
>
> > >>
>
> > >>
>
> > >> process 92065 is:
>
> > >> /usr/bin/java -Djava.ext.dirs=../lib
>
> > >> -Djava.util.logging.config.file=../etc/logging.properties
>
> > >> -Dorg.apache.oodt.cas.filemgr.properties=../etc/filemgr.properties
>
> > >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager --portNum 9000
>
> > >>
>
> > >> *********************************************************
>
> > >> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND
>
> > >> MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE USE
>
> > >> OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE.
>
> > >> If the reader is not the intended recipient, or the employee or
>
> > agent
>
> > >> responsible to deliver it to the intended recipient, you are hereby
>
> > >> notified that any dissemination, distribution or copying of this
>
> > >> communication is strictly prohibited. If you have received this
>
> > >> communication in error, please reply to the sender to notify us of
>
> > >> the error and delete the original message. Thank You.
>
> > >>
>
> > >
>
> > >
>
> > >
>
> > > --
>
> > > *Lewis*
>
> > >
>
> >
>
> >
>
> >
>
> > --
>
> > *Lewis*
>
> *********************************************************
> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS
> CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED
> INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL
> OR INDIVIDUALS NAMED ABOVE.
> If the reader is not the intended recipient, or the
> employee or agent responsible to deliver it to the
> intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this
> communication is strictly prohibited. If you have
> received this communication in error, please reply to the
> sender to notify us of the error and delete the original
> message. Thank You.
>
--
*Lewis*