You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by dr <bf...@126.com> on 2015/08/20 13:49:01 UTC
IO probleam in forceMerge(1)
Hi ,all
Currenty i met a probleam with forceMerge(1). During forceMerging some of my machine spent too much time(10-20 hours for a index size of 15GB). By using some command like top, iostat, jstack. I found that the avg of cpu, and disk is too low, nearly zero. And the size of the the index files increased very very slowly. From jstack, i found that the application is hanging on IO operations, like FileInputStream or DirectByteArray. But when i did some copy operation on the machine the speed may become normal.
Should this issue be related with the JVM, Operating System and hardware?
Did anybody met the same probleam, or have any idea about Java, Linux and IO.
Some basic info about my system and application.
| Lucece version | 4.10.3 |
| Jdk version | jdk 1.7.0_05 |
| OS | redhat 4.4.4 |
| cpu | IntelX5650 |
| Memory | 24GB |
Best Reagrds
Re:Re:Re: IO probleam in forceMerge(1)
Posted by dr <bf...@126.com>.
I used to say "when i did some copy operation on the machine the speed may become normal.", it seems not correct. But when forceMerge hangs on IO, if i use the command "echo 3 > /proc/sys/vm/drop_caches", the io become normal again.
At 2015-08-21 11:59:30, "dr" <bf...@126.com> wrote:
>
>Also i noticed that in the begin the speed was normal, and it slowed down later. The total space of the index is 15GB, and *.fdt takes 12GB. I allocated 8GB memory for forceMerge, is this enough for that? I readed the code and doc, it seems that *.fdt is readed and stored in block of 16KB, i guess it won't need too much memory to merge them.
>
>
>At 2015-08-20 22:00:27, "Michael McCandless" <lu...@mikemccandless.com> wrote:
>>I would first try upgrading you JVM: 1.7.0_05 is ancient.
>>
>>Mike McCandless
>>
>>http://blog.mikemccandless.com
>>
>>
>>On Thu, Aug 20, 2015 at 7:49 AM, dr <bf...@126.com> wrote:
>>> Hi ,all
>>> Currenty i met a probleam with forceMerge(1). During forceMerging some of my machine spent too much time(10-20 hours for a index size of 15GB). By using some command like top, iostat, jstack. I found that the avg of cpu, and disk is too low, nearly zero. And the size of the the index files increased very very slowly. From jstack, i found that the application is hanging on IO operations, like FileInputStream or DirectByteArray. But when i did some copy operation on the machine the speed may become normal.
>>>
>>> Should this issue be related with the JVM, Operating System and hardware?
>>> Did anybody met the same probleam, or have any idea about Java, Linux and IO.
>>> Some basic info about my system and application.
>>>
>>> | Lucece version | 4.10.3 |
>>> | Jdk version | jdk 1.7.0_05 |
>>> | OS | redhat 4.4.4 |
>>> | cpu | IntelX5650 |
>>> | Memory | 24GB |
>>>
>>>
>>> Best Reagrds
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>For additional commands, e-mail: java-user-help@lucene.apache.org
>>
Re:Re: IO probleam in forceMerge(1)
Posted by dr <bf...@126.com>.
Also i noticed that in the begin the speed was normal, and it slowed down later. The total space of the index is 15GB, and *.fdt takes 12GB. I allocated 8GB memory for forceMerge, is this enough for that? I readed the code and doc, it seems that *.fdt is readed and stored in block of 16KB, i guess it won't need too much memory to merge them.
At 2015-08-20 22:00:27, "Michael McCandless" <lu...@mikemccandless.com> wrote:
>I would first try upgrading you JVM: 1.7.0_05 is ancient.
>
>Mike McCandless
>
>http://blog.mikemccandless.com
>
>
>On Thu, Aug 20, 2015 at 7:49 AM, dr <bf...@126.com> wrote:
>> Hi ,all
>> Currenty i met a probleam with forceMerge(1). During forceMerging some of my machine spent too much time(10-20 hours for a index size of 15GB). By using some command like top, iostat, jstack. I found that the avg of cpu, and disk is too low, nearly zero. And the size of the the index files increased very very slowly. From jstack, i found that the application is hanging on IO operations, like FileInputStream or DirectByteArray. But when i did some copy operation on the machine the speed may become normal.
>>
>> Should this issue be related with the JVM, Operating System and hardware?
>> Did anybody met the same probleam, or have any idea about Java, Linux and IO.
>> Some basic info about my system and application.
>>
>> | Lucece version | 4.10.3 |
>> | Jdk version | jdk 1.7.0_05 |
>> | OS | redhat 4.4.4 |
>> | cpu | IntelX5650 |
>> | Memory | 24GB |
>>
>>
>> Best Reagrds
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>For additional commands, e-mail: java-user-help@lucene.apache.org
>
Re: IO probleam in forceMerge(1)
Posted by Michael McCandless <lu...@mikemccandless.com>.
I would first try upgrading you JVM: 1.7.0_05 is ancient.
Mike McCandless
http://blog.mikemccandless.com
On Thu, Aug 20, 2015 at 7:49 AM, dr <bf...@126.com> wrote:
> Hi ,all
> Currenty i met a probleam with forceMerge(1). During forceMerging some of my machine spent too much time(10-20 hours for a index size of 15GB). By using some command like top, iostat, jstack. I found that the avg of cpu, and disk is too low, nearly zero. And the size of the the index files increased very very slowly. From jstack, i found that the application is hanging on IO operations, like FileInputStream or DirectByteArray. But when i did some copy operation on the machine the speed may become normal.
>
> Should this issue be related with the JVM, Operating System and hardware?
> Did anybody met the same probleam, or have any idea about Java, Linux and IO.
> Some basic info about my system and application.
>
> | Lucece version | 4.10.3 |
> | Jdk version | jdk 1.7.0_05 |
> | OS | redhat 4.4.4 |
> | cpu | IntelX5650 |
> | Memory | 24GB |
>
>
> Best Reagrds
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org