You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Jason Rutherglen <ja...@gmail.com> on 2010/06/03 20:13:17 UTC
Monitoring low level IO
This is more of a unix related question than Lucene specific
however because Lucene is being used, I'm asking here as perhaps
other people have run into a similar issue.
On an Amazon EC2 merge, read, and write operations are possibly
blocking due to underlying IO. Is there a tool that you have
used to monitor this type of thing?
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Monitoring low level IO
Posted by Fe...@emc.com.
If you are running with a Sun JVM 1.6, one possible way is to monitor the low level JVM java IO calls using BTrace (See http://kenai.com/projects/btrace/pages/Home). This is kind of lightweight custom java profiler.
It's quite easy to setup but will require that you know what IO calls are involved to write a few java lines to track what you need to.
For example to track the reads calls start/end including the duration:
@OnMethod(clazz = "java.io.RandomAccessFile", method = "read")
public static void onRandomReadStart() {
print(timestamp("y/M/d HH:mm:ss.SSS"));
print(strcat(", ", name(currentThread())));
print(strcat(", ", "IO read start");
}
@OnMethod(clazz = "java.io.RandomAccessFile", method = "read", location=@Location(value=Kind.RETURN))
public static void onRandomReadEnd(@Duration long d) {
print(timestamp("y/M/d HH:mm:ss.SSS"));
print(strcat(", ", name(currentThread())));
print(strcat(", ", str(d));
print(strcat(", ", "IO read end");
}
-----Original Message-----
From: Jason Rutherglen [mailto:jason.rutherglen@gmail.com]
Sent: jeudi 3 juin 2010 20:13
To: java-user@lucene.apache.org
Subject: Monitoring low level IO
This is more of a unix related question than Lucene specific
however because Lucene is being used, I'm asking here as perhaps
other people have run into a similar issue.
On an Amazon EC2 merge, read, and write operations are possibly
blocking due to underlying IO. Is there a tool that you have
used to monitor this type of thing?
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Monitoring low level IO
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Ah, there is another one I came across several months back - http://wiki.sdn.sap.com/wiki/display/Java/JPicus.
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/
----- Original Message ----
> From: Otis Gospodnetic <ot...@yahoo.com>
> To: java-user@lucene.apache.org
> Sent: Fri, June 4, 2010 1:54:15 AM
> Subject: Re: Monitoring low level IO
>
> Other than iostat, vmstat and such?
Otis
----- Original
> Message ----
> From: Jason Rutherglen <
> ymailto="mailto:jason.rutherglen@gmail.com"
> href="mailto:jason.rutherglen@gmail.com">jason.rutherglen@gmail.com>
>
> To:
> href="mailto:java-user@lucene.apache.org">java-user@lucene.apache.org
>
> Sent: Thu, June 3, 2010 2:13:17 PM
> Subject: Monitoring low level
> IO
>
> This is more of a unix related question than Lucene
> specific
however because
> Lucene is being used, I'm asking here as
> perhaps
other people have run into a
> similar issue.
On an
> Amazon EC2 merge, read, and write operations are
> possibly
blocking
> due to underlying IO. Is there a tool that you have
used
> to monitor
> this type of
>
> thing?
---------------------------------------------------------------------
To
>
> unsubscribe, e-mail:
> href="mailto:
> ymailto="mailto:java-user-unsubscribe@lucene.apache.org"
> href="mailto:java-user-unsubscribe@lucene.apache.org">java-user-unsubscribe@lucene.apache.org">
> ymailto="mailto:java-user-unsubscribe@lucene.apache.org"
> href="mailto:java-user-unsubscribe@lucene.apache.org">java-user-unsubscribe@lucene.apache.org
For
>
> additional commands, e-mail:
> ymailto="mailto:
> ymailto="mailto:java-user-help@lucene.apache.org"
> href="mailto:java-user-help@lucene.apache.org">java-user-help@lucene.apache.org"
>
> href="mailto:
> href="mailto:java-user-help@lucene.apache.org">java-user-help@lucene.apache.org">
> ymailto="mailto:java-user-help@lucene.apache.org"
> href="mailto:java-user-help@lucene.apache.org">java-user-help@lucene.apache.org
---------------------------------------------------------------------
To
> unsubscribe, e-mail:
> href="mailto:java-user-unsubscribe@lucene.apache.org">java-user-unsubscribe@lucene.apache.org
For
> additional commands, e-mail:
> ymailto="mailto:java-user-help@lucene.apache.org"
> href="mailto:java-user-help@lucene.apache.org">java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Monitoring low level IO
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Other than iostat, vmstat and such?
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/
----- Original Message ----
> From: Jason Rutherglen <ja...@gmail.com>
> To: java-user@lucene.apache.org
> Sent: Thu, June 3, 2010 2:13:17 PM
> Subject: Monitoring low level IO
>
> This is more of a unix related question than Lucene specific
however because
> Lucene is being used, I'm asking here as perhaps
other people have run into a
> similar issue.
On an Amazon EC2 merge, read, and write operations are
> possibly
blocking due to underlying IO. Is there a tool that you have
used
> to monitor this type of
> thing?
---------------------------------------------------------------------
To
> unsubscribe, e-mail:
> href="mailto:java-user-unsubscribe@lucene.apache.org">java-user-unsubscribe@lucene.apache.org
For
> additional commands, e-mail:
> ymailto="mailto:java-user-help@lucene.apache.org"
> href="mailto:java-user-help@lucene.apache.org">java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: numDeletedDocs()
Posted by "Woolf, Ross" <Ro...@BMC.com>.
Oh, my mistake, thanks. I clicked on a link and I thought it took me to the indexWriter in javadocs. Once I retraced my steps it took me to the indexReader. This solves my problem :).
-----Original Message-----
From: Michael McCandless [mailto:lucene@mikemccandless.com]
Sent: Thursday, June 03, 2010 2:02 PM
To: java-user@lucene.apache.org
Subject: Re: numDeletedDocs()
Hmm... I don't think IndexWriter has ever had a numDeletedDocs() (w/ no params)?
(IndexReader does).
Mike
On Thu, Jun 3, 2010 at 3:50 PM, Woolf, Ross <Ro...@bmc.com> wrote:
> There seems to be a mismatch between the IndexWriter().numDeletedDocs() method as stated in the javadocs supplied in the 2.9.2 download and what is actual.
>
> JavaDocs for 2.9.2 as came with the 2.9.2 download
>
> numDeletedDocs
> public int numDeletedDocs()Returns the number of deleted documents. (No parameter required)
>
> ----------------------
> Source code for 2.9.2
> public int numDeletedDocs(SegmentInfo info) throws IOException { (Parameter required)
>
>
> Why is there no longer a no parameter numDeleteDocs as stated in the JavaDocs? I'm not sure how I use the "experimental" SegmentInfo just to get the delete count in my index? Any help appreciated.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: numDeletedDocs()
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Btw. folks, http://search-lucene.com/ has a really handy source code search with auto-completion for Lucene, Solr, etc. For example, I typed in: numDel - and immediately found those methods. Use it. :)
Otis----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/
----- Original Message ----
> From: Michael McCandless <lu...@mikemccandless.com>
> To: java-user@lucene.apache.org
> Sent: Thu, June 3, 2010 4:02:09 PM
> Subject: Re: numDeletedDocs()
>
> Hmm... I don't think IndexWriter has ever had a numDeletedDocs() (w/ no
> params)?
(IndexReader does).
Mike
On Thu, Jun 3, 2010 at
> 3:50 PM, Woolf, Ross <
> href="mailto:Ross_Woolf@bmc.com">Ross_Woolf@bmc.com> wrote:
> There
> seems to be a mismatch between the IndexWriter().numDeletedDocs() method as
> stated in the javadocs supplied in the 2.9.2 download and what is
> actual.
>
> JavaDocs for 2.9.2 as came with the 2.9.2
> download
>
> numDeletedDocs
> public int
> numDeletedDocs()Returns the number of deleted documents. (No parameter
> required)
>
> ----------------------
> Source code for
> 2.9.2
> public int numDeletedDocs(SegmentInfo info) throws IOException {
> (Parameter required)
>
>
> Why is there no longer a no
> parameter numDeleteDocs as stated in the JavaDocs? I'm not sure how I use the
> "experimental" SegmentInfo just to get the delete count in my index? Any help
> appreciated.
>
>
>
> ---------------------------------------------------------------------
> To
> unsubscribe, e-mail:
> href="mailto:java-user-unsubscribe@lucene.apache.org">java-user-unsubscribe@lucene.apache.org
>
> For additional commands, e-mail:
> ymailto="mailto:java-user-help@lucene.apache.org"
> href="mailto:java-user-help@lucene.apache.org">java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To
> unsubscribe, e-mail:
> href="mailto:java-user-unsubscribe@lucene.apache.org">java-user-unsubscribe@lucene.apache.org
For
> additional commands, e-mail:
> ymailto="mailto:java-user-help@lucene.apache.org"
> href="mailto:java-user-help@lucene.apache.org">java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: numDeletedDocs()
Posted by Michael McCandless <lu...@mikemccandless.com>.
Hmm... I don't think IndexWriter has ever had a numDeletedDocs() (w/ no params)?
(IndexReader does).
Mike
On Thu, Jun 3, 2010 at 3:50 PM, Woolf, Ross <Ro...@bmc.com> wrote:
> There seems to be a mismatch between the IndexWriter().numDeletedDocs() method as stated in the javadocs supplied in the 2.9.2 download and what is actual.
>
> JavaDocs for 2.9.2 as came with the 2.9.2 download
>
> numDeletedDocs
> public int numDeletedDocs()Returns the number of deleted documents. (No parameter required)
>
> ----------------------
> Source code for 2.9.2
> public int numDeletedDocs(SegmentInfo info) throws IOException { (Parameter required)
>
>
> Why is there no longer a no parameter numDeleteDocs as stated in the JavaDocs? I'm not sure how I use the "experimental" SegmentInfo just to get the delete count in my index? Any help appreciated.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
numDeletedDocs()
Posted by "Woolf, Ross" <Ro...@BMC.com>.
There seems to be a mismatch between the IndexWriter().numDeletedDocs() method as stated in the javadocs supplied in the 2.9.2 download and what is actual.
JavaDocs for 2.9.2 as came with the 2.9.2 download
numDeletedDocs
public int numDeletedDocs()Returns the number of deleted documents. (No parameter required)
----------------------
Source code for 2.9.2
public int numDeletedDocs(SegmentInfo info) throws IOException { (Parameter required)
Why is there no longer a no parameter numDeleteDocs as stated in the JavaDocs? I'm not sure how I use the "experimental" SegmentInfo just to get the delete count in my index? Any help appreciated.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org