You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Varun Dhussa <va...@mapmyindia.com> on 2009/02/18 11:36:07 UTC
Lucene search performance on Sun UltraSparc T2 (T5120) servers
Hi,
I have had a bad experience when migrating my application from Intel
Xeon based servers to Sun UltraSparc T2 T5120 servers. Lucene fuzzy
search just does not perform. A search which took approximately 500 ms
takes more than 6 seconds to execute.
The index has about 100,000,000 records. So, I tried to split it into 10
indices and used the ParallelSearcher on it, but still got similar results.
I am guessing that this is because the distance implementation used by
Lucene requires higher clock speed and can't be parallelized much.
Please advice
--
Varun Dhussa
Product Architect
CE InfoSystems (P) Ltd
http://www.mapmyindia.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene search performance on Sun UltraSparc T2 (T5120) servers
Posted by Glen Newton <gl...@gmail.com>.
I will look a little deeper into the information you supplied and
comment, but will suggest this on my initial cursory review:
1 - You have 32GB of memory. Using the 64bit VM, try using a 16GB or 24GB heap;
2 - Turn-on huge pages:
-XX:+UseLargePages
-XX:LargePageSizeInBytes=256m
3 - Turn on aggressive opts:
-XX:+AggressiveOpts
4 - Try using bound threads:
-XX:+UseBoundThreads
5 - Try using thread-local object allocation:
-XX:+UseTLAB
6 - Try Biased Locking:
-XX:+UseBiasedLocking
I consider #1-3 most likely to help you. I would first try them
separately then in combination. #4-6 I am not sure given your hardware
(of which I am very envious!), but would try nonetheless.
I am more used to working with quad dual core ia64 machines with 8GB
memory running Linux. And large indexes, with tests up to 1B documents
and ~700GB in size, in a single Lucene index.
Please let me know how these things help, if at all.
thanks,
Glen
2009/2/19 Varun Dhussa <va...@mapmyindia.com>:
> Hi,
>
> The details are as follows:
> Solaris version: Solaris 10 U5 and U6
> For the Java Setup, I have tried with:
> Sun JDK 1.5 (32 & 64)
> Sun JDK 1.6 (32 & 64)
> Heap Space: 2G from 32 bit and 4G for 64 bit (Set the same values for both
> XMS and XMX)
>
> Disk: Tried with ZFS (U6) and UFS (U5)
> I reduced the Inode density on the partition since page faults were
> happening initially. Also, the block size was 8K.
> The following reports are on Solaris 10 U5 with UFS (8K block size).
>
>
> The vmstat report is below:
>
> # vmstat 3
> kthr memory page disk faults cpu
> r b w swap free re mf pi po fr de sr s0 s2 -- -- in sy cs us sy
> id
> 0 0 0 43408368 29603112 13 107 4 7 6 0 0 2 -0 0 0 458 589 321 0 0
> 99
> 0 0 0 22517072 30241872 830 7709 0 0 0 0 0 0 0 0 0 2537 30699 3899 36 3
> 61
> 0 0 0 24666416 30483248 860 7942 0 0 0 0 0 0 0 0 0 2235 31886 3362 28 3
> 69
> 0 0 0 24184008 30450728 791 7813 0 0 0 0 0 0 0 0 0 2335 31426 3740 33 3
> 65
> 0 0 0 23472024 30442168 813 7822 0 0 0 0 0 0 0 0 0 2494 31312 3923 35 3
> 62
> 0 0 0 22751768 30297240 787 7654 0 0 0 0 0 0 0 0 0 2498 31820 3892 35 3
> 62
> 0 0 0 25385568 30527600 835 7963 0 0 0 0 0 0 0 0 0 2360 32144 3691 29 3
> 68
> 0 0 0 22741056 30281376 787 7643 0 0 0 0 0 0 0 0 0 2249 30008 3484 37 3
> 60
> 0 0 0 22753800 30373072 847 7923 0 0 0 0 0 0 0 0 0 2501 31375 3854 33 3
> 64
> 0 0 0 23725600 30431312 841 8049 0 0 0 0 0 0 0 0 0 2362 32514 3752 32 3
> 65
> 0 0 0 22743008 30256368 826 7719 0 0 0 0 0 0 0 0 0 2427 32101 3773 34 3
> 63
> 0 0 0 23950336 30405312 818 7925 0 0 0 0 0 0 0 0 0 2337 32095 3606 31 3
> 66
> 0 0 0 22752872 30319312 778 7486 0 0 0 0 0 0 0 0 0 2435 30277 3769 37 3
> 60
> 0 0 0 24657848 30401408 836 7797 0 0 0 0 0 0 0 0 0 2469 31879 3770 32 3
> 65
> 0 0 0 23460408 30331056 776 7560 0 0 0 0 0 0 0 0 0 2461 31344 3883 37 3
> 60
> 0 0 0 22748352 30302248 792 7534 0 0 0 0 0 0 0 0 0 2424 30683 3733 36 3
> 61
> 0 0 0 23953848 30389280 837 7868 0 0 0 0 0 0 0 0 0 2280 30710 3511 34 3
> 63
> 0 0 0 22736592 30249672 810 7716 0 0 0 0 0 0 0 0 0 2345 30980 3593 34 3
> 63
> kthr memory page disk faults cpu
> r b w swap free re mf pi po fr de sr s0 s2 -- -- in sy cs us sy
> id
> 0 0 0 23951544 30400016 834 7992 0 0 0 0 0 0 0 0 0 2431 32647 3776 31 3
> 66
> 0 0 0 21797992 30245536 812 7635 0 0 0 0 0 0 0 0 0 2402 30213 3693 39 3
> 59
> 0 0 0 22745264 30307728 781 7539 0 0 0 0 0 0 0 0 0 2437 30744 3860 38 3
> 59
> 0 0 0 22748536 30286456 851 7987 0 0 0 0 0 0 0 0 0 2433 32842 3678 29 3
> 68
> 0 0 0 22747680 30297544 791 7514 0 0 0 0 0 0 0 0 0 2369 30625 3655 38 3
> 59
> 0 0 0 23104848 30309488 793 7524 0 0 0 0 0 0 0 0 0 2566 31190 3942 36 3
> 61
> 0 0 0 23460368 30340688 810 7784 0 0 0 0 0 0 0 0 0 2302 31430 3688 33 3
> 64
> 0 0 0 22031344 30215504 792 7397 0 0 0 0 0 0 0 0 0 2477 30733 3850 39 3
> 58
> 0 0 0 22744168 30277312 834 7829 0 0 0 0 0 0 0 0 0 2370 29926 3592 35 3
> 63
> 0 0 0 24177728 30432600 806 7789 0 0 0 0 0 0 0 0 0 2379 31870 3731 32 3
> 65
> 0 0 0 23467896 30382728 802 7810 0 0 0 0 0 0 0 0 0 2527 31866 3980 33 3
> 64
> 0 0 0 22757912 30331080 823 7847 0 0 0 0 0 0 0 0 0 2496 31580 3871 33 3
> 64
> 0 0 0 22043312 30215904 819 7558 0 0 0 0 0 0 0 0 0 2253 30252 3455 36 3
> 61
> 0 0 0 22744952 30275008 793 7657 0 0 0 0 0 0 0 0 0 2499 31840 3937 36 3
> 61
> 0 0 0 23715200 30419008 843 7906 0 0 0 0 0 0 0 0 0 2404 31741 3707 33 3
> 64
> 0 0 0 21307184 30148472 788 7515 0 0 0 0 0 0 0 0 0 2430 31340 3819 39 3
> 58
> 0 0 0 23459544 30332160 817 7687 0 0 0 0 0 0 0 0 0 2353 31011 3526 34 3
> 63
> 0 0 0 23467304 30376392 819 7846 0 0 0 0 0 0 0 0 0 2522 31918 3933 32 3
> 65
> 0 0 0 23473464 30421992 830 7854 0 0 0 0 0 0 0 0 0 2407 31714 3827 33 3
> 65
> kthr memory page disk faults cpu
> r b w swap free re mf pi po fr de sr s0 s2 -- -- in sy cs us sy
> id
> 0 0 0 22508648 30240256 799 7447 0 0 0 0 0 0 0 0 0 2524 30483 3911 39 3
> 58
> 0 0 0 22737496 30276864 837 7731 0 0 0 0 0 0 0 0 0 2310 30289 3473 36 3
> 61
> 0 0 0 22515488 30312728 792 7646 0 0 0 0 0 0 0 0 0 2472 31610 3867 35 3
> 62
> 0 0 0 22756248 30251584 823 7669 0 0 0 0 0 0 0 0 0 2637 31413 4119 36 3
> 60
> 0 0 0 23940168 30379336 833 7973 0 0 0 0 0 0 0 0 0 2240 32325 3472 31 3
> 66
> 0 0 0 24184736 30455928 846 7838 0 0 0 0 0 0 0 0 0 2288 31508 3538 32 3
> 65
> 0 0 0 22751808 30309984 776 7614 0 0 0 0 0 0 0 0 0 2480 31486 3906 36 3
> 61
> 0 0 0 22989624 30309536 813 7626 0 0 0 0 0 0 0 0 0 2537 30347 3988 38 3
> 59
> 0 0 0 22739224 30276936 814 7692 0 0 0 0 0 0 0 0 0 2261 31528 3409 34 3
> 63
> 0 0 0 23236200 30350216 853 7968 0 0 0 0 0 0 0 0 0 2676 32636 4167 32 3
> 65
> 0 0 0 24194016 30506960 836 8005 0 0 0 0 0 0 0 0 0 2222 31503 3382 30 3
> 67
> 0 0 0 23480504 30424536 834 7848 0 0 0 0 0 0 0 0 0 2390 32039 3736 32 3
> 65
> 0 0 0 22739048 30233176 812 7742 0 0 0 0 0 0 0 0 0 2604 31693 4029 35 3
> 62
> 0 0 0 23461664 30358528 805 7700 0 0 0 0 0 0 0 0 0 2463 30936 3922 36 3
> 61
> 0 0 0 22031616 30257808 796 7533 0 0 0 0 0 0 0 0 0 2226 29829 3377 37 3
> 60
> 0 0 0 23458168 30327688 813 7773 0 0 0 0 0 0 0 0 0 2388 31519 3763 33 3
> 64
> 0 0 0 23465184 30371904 834 7796 0 0 0 0 0 0 0 0 0 2510 32101 3933 34 3
> 63
> 0 0 0 22513648 30312240 800 7596 0 0 0 0 0 0 0 0 0 2416 30538 3734 38 3
> 59
> 0 0 0 22272824 30228912 802 7640 0 0 0 0 0 0 0 0 0 2292 31395 3469 34 3
> 63
> kthr memory page disk faults cpu
> r b w swap free re mf pi po fr de sr s0 s2 -- -- in sy cs us sy
> id
> 0 0 0 22744032 30253480 830 7758 0 0 0 0 0 0 0 0 0 2568 31896 3935 35 3
> 62
> 0 0 0 23287696 30368432 808 7736 0 0 0 0 0 0 0 0 0 2439 30929 3844 35 3
> 62
> 0 0 0 22038504 30253112 794 7480 0 0 0 0 0 0 0 0 0 2213 29774 3433 37 3
> 60
> 0 0 0 24660144 30480240 846 8103 0 0 0 0 0 0 0 0 0 2548 32825 3944 30 3
> 66
> 0 0 0 24183648 30445120 823 7895 0 0 0 0 0 0 0 0 0 2441 32699 3822 31 3
> 66
> 0 0 0 23469136 30391744 830 7916 0 0 0 0 0 0 0 0 0 2441 31731 3813 32 3
> 65
> 0 0 0 22516312 30289688 796 7592 0 0 0 0 0 0 0 0 0 2318 30310 3616 36 3
> 61
> 0 0 0 23943448 30363248 821 7821 0 0 0 0 0 0 0 0 0 2355 31789 3641 31 3
> 66
> 0 0 0 22742768 30284896 795 7602 0 0 0 0 0 0 0 0 0 2387 30608 3675 37 3
> 60
> 0 0 0 23475456 30416136 852 7991 0 0 0 0 0 0 0 0 0 2489 32066 3871 31 3
> 66
> 0 0 0 22756848 30233968 798 7588 0 0 0 0 0 0 0 0 0 2464 31106 3836 37 3
> 60
>
> Cooltest report said virtually nothing.
>
> System Configuration
>
> Host name mmiAS02
> System name SUNW,SPARC-Enterprise-T5120
> Effective UID 0
> Cooltst version 3.0.1
> OS Solaris
> OS release 5.10
> OS version Generic_127127-11
> Distro Solaris
> BIOS/PROM OBP 4.28.0 2008/01/22 21:10
> Memory 32640 MB
> Chip UltraSPARC-T2
> MHz 1165
> Architecture SPARC
> # of Virtual CPUs 64
> P0: 1165 MHz UltraSPARC-T2
> P1: 1165 MHz UltraSPARC-T2
> P2: 1165 MHz UltraSPARC-T2
> P3: 1165 MHz UltraSPARC-T2
> P4: 1165 MHz UltraSPARC-T2
> P5: 1165 MHz UltraSPARC-T2
> P6: 1165 MHz UltraSPARC-T2
> P7: 1165 MHz UltraSPARC-T2
> P8: 1165 MHz UltraSPARC-T2
> P9: 1165 MHz UltraSPARC-T2
> P10: 1165 MHz UltraSPARC-T2
> P11: 1165 MHz UltraSPARC-T2
> P12: 1165 MHz UltraSPARC-T2
> P13: 1165 MHz UltraSPARC-T2
> P14: 1165 MHz UltraSPARC-T2
> P15: 1165 MHz UltraSPARC-T2
> P16: 1165 MHz UltraSPARC-T2
> P17: 1165 MHz UltraSPARC-T2
> P18: 1165 MHz UltraSPARC-T2
> P19: 1165 MHz UltraSPARC-T2
> P20: 1165 MHz UltraSPARC-T2
> P21: 1165 MHz UltraSPARC-T2
> P22: 1165 MHz UltraSPARC-T2
> P23: 1165 MHz UltraSPARC-T2
> P24: 1165 MHz UltraSPARC-T2
> P25: 1165 MHz UltraSPARC-T2
> P26: 1165 MHz UltraSPARC-T2
> P27: 1165 MHz UltraSPARC-T2
> P28: 1165 MHz UltraSPARC-T2
> P29: 1165 MHz UltraSPARC-T2
> P30: 1165 MHz UltraSPARC-T2
> P31: 1165 MHz UltraSPARC-T2
> P32: 1165 MHz UltraSPARC-T2
> P33: 1165 MHz UltraSPARC-T2
> P34: 1165 MHz UltraSPARC-T2
> P35: 1165 MHz UltraSPARC-T2
> P36: 1165 MHz UltraSPARC-T2
> P37: 1165 MHz UltraSPARC-T2
> P38: 1165 MHz UltraSPARC-T2
> P39: 1165 MHz UltraSPARC-T2
> P40: 1165 MHz UltraSPARC-T2
> P41: 1165 MHz UltraSPARC-T2
> P42: 1165 MHz UltraSPARC-T2
> P43: 1165 MHz UltraSPARC-T2
> P44: 1165 MHz UltraSPARC-T2
> P45: 1165 MHz UltraSPARC-T2
> P46: 1165 MHz UltraSPARC-T2
> P47: 1165 MHz UltraSPARC-T2
> P48: 1165 MHz UltraSPARC-T2
> P49: 1165 MHz UltraSPARC-T2
> P50: 1165 MHz UltraSPARC-T2
> P51: 1165 MHz UltraSPARC-T2
> P52: 1165 MHz UltraSPARC-T2
> P53: 1165 MHz UltraSPARC-T2
> P54: 1165 MHz UltraSPARC-T2
> P55: 1165 MHz UltraSPARC-T2
> P56: 1165 MHz UltraSPARC-T2
> P57: 1165 MHz UltraSPARC-T2
> P58: 1165 MHz UltraSPARC-T2
> P59: 1165 MHz UltraSPARC-T2
> P60: 1165 MHz UltraSPARC-T2
> P61: 1165 MHz UltraSPARC-T2
> P62: 1165 MHz UltraSPARC-T2
> P63: 1165 MHz UltraSPARC-T2
> OS release detail:
> Solaris 10 5/08 s10s_u5wos_10 SPARC Copyright 2008 Sun Microsystems, Inc.
> All Rights Reserved. Use is subject to license terms. Assembled 24 March
> 2008
>
> Workload Measurements
>
> Observed system for 10 min
> in intervals of 10 sec
> Cycles 44768051692942
> Instructions 3980806371547
> CPI 11.25 **
> FP instructions 5821938521
> Emulated FP instructions 0
> FP Percentage 0.1%
> The following applies to the measurement interval with the
> busiest single thread or process:
> Peak thread utilization at 2009-02-13 14:36:16
> Corresponding file name 1234515976
> CPU utilization 39.4%
> Command java
> PID/LWPID 16396/1
> Thread utilization 0.6%
> More detail on processes and threads is in data/process.out
>
> **Cycles per Instruction (CPI) is not comparable between UltraSPARC
> T1 and T2 processors and conventional processors. Conventional
> processors execute an idle loop when there is no work to do, so
> CPI may be artificially low, especially when the system is
> somewhat idle. The UltraSPARC T1 and T2 "park" idle threads,
> consuming no energy, when there is no work to do, so CPI may
> be artificially high, especially when the system is somewhat idle.
>
> Advice
>
> Floating Point GREEN
> Observed floating point content was not excessive for
> an UltraSPARC T1 processor. Floating point content is not
> a limitation for UltraSPARC T2.
>
> Parallelism GREEN
> The observed workload has sufficient threads of execution to
> efficiently utilize the multiple cores and threads of an
> UltraSPARC T1 or UltraSPARC T2 processor.
>
>
>
> Varun Dhussa
> Product Architect
> CE InfoSystems (P) Ltd
> http://www.mapmyindia.com
>
>
>
> Glen Newton wrote:
>>
>> Could you give some configuration details:
>> - Solaris version
>> - Java VM version, heap size, and any other flags
>> - disk setup
>>
>> You should also consider using huge pages (see
>>
>> http://zzzoot.blogspot.com/2009/02/java-mysql-increased-performance-with.html)
>>
>> I will also be posting performance gains using huge pages for Java
>> Lucene Linux large scale indexing in the next week or so...
>>
>> -glen
>>
>>
>>
>> 2009/2/18 Varun Dhussa <va...@mapmyindia.com>:
>>
>>>
>>> Hi,
>>>
>>> I have had a bad experience when migrating my application from Intel Xeon
>>> based servers to Sun UltraSparc T2 T5120 servers. Lucene fuzzy search
>>> just
>>> does not perform. A search which took approximately 500 ms takes more
>>> than 6
>>> seconds to execute.
>>>
>>> The index has about 100,000,000 records. So, I tried to split it into 10
>>> indices and used the ParallelSearcher on it, but still got similar
>>> results.
>>>
>>> I am guessing that this is because the distance implementation used by
>>> Lucene requires higher clock speed and can't be parallelized much.
>>>
>>> Please advice
>>>
>>> --
>>> Varun Dhussa
>>> Product Architect
>>> CE InfoSystems (P) Ltd
>>> http://www.mapmyindia.com
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
-
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene search performance on Sun UltraSparc T2 (T5120) servers
Posted by Varun Dhussa <va...@mapmyindia.com>.
Hi,
The details are as follows:
Solaris version: Solaris 10 U5 and U6
For the Java Setup, I have tried with:
Sun JDK 1.5 (32 & 64)
Sun JDK 1.6 (32 & 64)
Heap Space: 2G from 32 bit and 4G for 64 bit (Set the same values for
both XMS and XMX)
Disk: Tried with ZFS (U6) and UFS (U5)
I reduced the Inode density on the partition since page faults were
happening initially. Also, the block size was 8K.
The following reports are on Solaris 10 U5 with UFS (8K block size).
The vmstat report is below:
# vmstat 3
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s0 s2 -- -- in sy cs us
sy id
0 0 0 43408368 29603112 13 107 4 7 6 0 0 2 -0 0 0 458 589 321 0
0 99
0 0 0 22517072 30241872 830 7709 0 0 0 0 0 0 0 0 0 2537 30699 3899 36
3 61
0 0 0 24666416 30483248 860 7942 0 0 0 0 0 0 0 0 0 2235 31886 3362 28
3 69
0 0 0 24184008 30450728 791 7813 0 0 0 0 0 0 0 0 0 2335 31426 3740 33
3 65
0 0 0 23472024 30442168 813 7822 0 0 0 0 0 0 0 0 0 2494 31312 3923 35
3 62
0 0 0 22751768 30297240 787 7654 0 0 0 0 0 0 0 0 0 2498 31820 3892 35
3 62
0 0 0 25385568 30527600 835 7963 0 0 0 0 0 0 0 0 0 2360 32144 3691 29
3 68
0 0 0 22741056 30281376 787 7643 0 0 0 0 0 0 0 0 0 2249 30008 3484 37
3 60
0 0 0 22753800 30373072 847 7923 0 0 0 0 0 0 0 0 0 2501 31375 3854 33
3 64
0 0 0 23725600 30431312 841 8049 0 0 0 0 0 0 0 0 0 2362 32514 3752 32
3 65
0 0 0 22743008 30256368 826 7719 0 0 0 0 0 0 0 0 0 2427 32101 3773 34
3 63
0 0 0 23950336 30405312 818 7925 0 0 0 0 0 0 0 0 0 2337 32095 3606 31
3 66
0 0 0 22752872 30319312 778 7486 0 0 0 0 0 0 0 0 0 2435 30277 3769 37
3 60
0 0 0 24657848 30401408 836 7797 0 0 0 0 0 0 0 0 0 2469 31879 3770 32
3 65
0 0 0 23460408 30331056 776 7560 0 0 0 0 0 0 0 0 0 2461 31344 3883 37
3 60
0 0 0 22748352 30302248 792 7534 0 0 0 0 0 0 0 0 0 2424 30683 3733 36
3 61
0 0 0 23953848 30389280 837 7868 0 0 0 0 0 0 0 0 0 2280 30710 3511 34
3 63
0 0 0 22736592 30249672 810 7716 0 0 0 0 0 0 0 0 0 2345 30980 3593 34
3 63
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s0 s2 -- -- in sy cs us
sy id
0 0 0 23951544 30400016 834 7992 0 0 0 0 0 0 0 0 0 2431 32647 3776 31
3 66
0 0 0 21797992 30245536 812 7635 0 0 0 0 0 0 0 0 0 2402 30213 3693 39
3 59
0 0 0 22745264 30307728 781 7539 0 0 0 0 0 0 0 0 0 2437 30744 3860 38
3 59
0 0 0 22748536 30286456 851 7987 0 0 0 0 0 0 0 0 0 2433 32842 3678 29
3 68
0 0 0 22747680 30297544 791 7514 0 0 0 0 0 0 0 0 0 2369 30625 3655 38
3 59
0 0 0 23104848 30309488 793 7524 0 0 0 0 0 0 0 0 0 2566 31190 3942 36
3 61
0 0 0 23460368 30340688 810 7784 0 0 0 0 0 0 0 0 0 2302 31430 3688 33
3 64
0 0 0 22031344 30215504 792 7397 0 0 0 0 0 0 0 0 0 2477 30733 3850 39
3 58
0 0 0 22744168 30277312 834 7829 0 0 0 0 0 0 0 0 0 2370 29926 3592 35
3 63
0 0 0 24177728 30432600 806 7789 0 0 0 0 0 0 0 0 0 2379 31870 3731 32
3 65
0 0 0 23467896 30382728 802 7810 0 0 0 0 0 0 0 0 0 2527 31866 3980 33
3 64
0 0 0 22757912 30331080 823 7847 0 0 0 0 0 0 0 0 0 2496 31580 3871 33
3 64
0 0 0 22043312 30215904 819 7558 0 0 0 0 0 0 0 0 0 2253 30252 3455 36
3 61
0 0 0 22744952 30275008 793 7657 0 0 0 0 0 0 0 0 0 2499 31840 3937 36
3 61
0 0 0 23715200 30419008 843 7906 0 0 0 0 0 0 0 0 0 2404 31741 3707 33
3 64
0 0 0 21307184 30148472 788 7515 0 0 0 0 0 0 0 0 0 2430 31340 3819 39
3 58
0 0 0 23459544 30332160 817 7687 0 0 0 0 0 0 0 0 0 2353 31011 3526 34
3 63
0 0 0 23467304 30376392 819 7846 0 0 0 0 0 0 0 0 0 2522 31918 3933 32
3 65
0 0 0 23473464 30421992 830 7854 0 0 0 0 0 0 0 0 0 2407 31714 3827 33
3 65
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s0 s2 -- -- in sy cs us
sy id
0 0 0 22508648 30240256 799 7447 0 0 0 0 0 0 0 0 0 2524 30483 3911 39
3 58
0 0 0 22737496 30276864 837 7731 0 0 0 0 0 0 0 0 0 2310 30289 3473 36
3 61
0 0 0 22515488 30312728 792 7646 0 0 0 0 0 0 0 0 0 2472 31610 3867 35
3 62
0 0 0 22756248 30251584 823 7669 0 0 0 0 0 0 0 0 0 2637 31413 4119 36
3 60
0 0 0 23940168 30379336 833 7973 0 0 0 0 0 0 0 0 0 2240 32325 3472 31
3 66
0 0 0 24184736 30455928 846 7838 0 0 0 0 0 0 0 0 0 2288 31508 3538 32
3 65
0 0 0 22751808 30309984 776 7614 0 0 0 0 0 0 0 0 0 2480 31486 3906 36
3 61
0 0 0 22989624 30309536 813 7626 0 0 0 0 0 0 0 0 0 2537 30347 3988 38
3 59
0 0 0 22739224 30276936 814 7692 0 0 0 0 0 0 0 0 0 2261 31528 3409 34
3 63
0 0 0 23236200 30350216 853 7968 0 0 0 0 0 0 0 0 0 2676 32636 4167 32
3 65
0 0 0 24194016 30506960 836 8005 0 0 0 0 0 0 0 0 0 2222 31503 3382 30
3 67
0 0 0 23480504 30424536 834 7848 0 0 0 0 0 0 0 0 0 2390 32039 3736 32
3 65
0 0 0 22739048 30233176 812 7742 0 0 0 0 0 0 0 0 0 2604 31693 4029 35
3 62
0 0 0 23461664 30358528 805 7700 0 0 0 0 0 0 0 0 0 2463 30936 3922 36
3 61
0 0 0 22031616 30257808 796 7533 0 0 0 0 0 0 0 0 0 2226 29829 3377 37
3 60
0 0 0 23458168 30327688 813 7773 0 0 0 0 0 0 0 0 0 2388 31519 3763 33
3 64
0 0 0 23465184 30371904 834 7796 0 0 0 0 0 0 0 0 0 2510 32101 3933 34
3 63
0 0 0 22513648 30312240 800 7596 0 0 0 0 0 0 0 0 0 2416 30538 3734 38
3 59
0 0 0 22272824 30228912 802 7640 0 0 0 0 0 0 0 0 0 2292 31395 3469 34
3 63
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s0 s2 -- -- in sy cs us
sy id
0 0 0 22744032 30253480 830 7758 0 0 0 0 0 0 0 0 0 2568 31896 3935 35
3 62
0 0 0 23287696 30368432 808 7736 0 0 0 0 0 0 0 0 0 2439 30929 3844 35
3 62
0 0 0 22038504 30253112 794 7480 0 0 0 0 0 0 0 0 0 2213 29774 3433 37
3 60
0 0 0 24660144 30480240 846 8103 0 0 0 0 0 0 0 0 0 2548 32825 3944 30
3 66
0 0 0 24183648 30445120 823 7895 0 0 0 0 0 0 0 0 0 2441 32699 3822 31
3 66
0 0 0 23469136 30391744 830 7916 0 0 0 0 0 0 0 0 0 2441 31731 3813 32
3 65
0 0 0 22516312 30289688 796 7592 0 0 0 0 0 0 0 0 0 2318 30310 3616 36
3 61
0 0 0 23943448 30363248 821 7821 0 0 0 0 0 0 0 0 0 2355 31789 3641 31
3 66
0 0 0 22742768 30284896 795 7602 0 0 0 0 0 0 0 0 0 2387 30608 3675 37
3 60
0 0 0 23475456 30416136 852 7991 0 0 0 0 0 0 0 0 0 2489 32066 3871 31
3 66
0 0 0 22756848 30233968 798 7588 0 0 0 0 0 0 0 0 0 2464 31106 3836 37
3 60
Cooltest report said virtually nothing.
System Configuration
Host name mmiAS02
System name SUNW,SPARC-Enterprise-T5120
Effective UID 0
Cooltst version 3.0.1
OS Solaris
OS release 5.10
OS version Generic_127127-11
Distro Solaris
BIOS/PROM OBP 4.28.0 2008/01/22 21:10
Memory 32640 MB
Chip UltraSPARC-T2
MHz 1165
Architecture SPARC
# of Virtual CPUs 64
P0: 1165 MHz UltraSPARC-T2
P1: 1165 MHz UltraSPARC-T2
P2: 1165 MHz UltraSPARC-T2
P3: 1165 MHz UltraSPARC-T2
P4: 1165 MHz UltraSPARC-T2
P5: 1165 MHz UltraSPARC-T2
P6: 1165 MHz UltraSPARC-T2
P7: 1165 MHz UltraSPARC-T2
P8: 1165 MHz UltraSPARC-T2
P9: 1165 MHz UltraSPARC-T2
P10: 1165 MHz UltraSPARC-T2
P11: 1165 MHz UltraSPARC-T2
P12: 1165 MHz UltraSPARC-T2
P13: 1165 MHz UltraSPARC-T2
P14: 1165 MHz UltraSPARC-T2
P15: 1165 MHz UltraSPARC-T2
P16: 1165 MHz UltraSPARC-T2
P17: 1165 MHz UltraSPARC-T2
P18: 1165 MHz UltraSPARC-T2
P19: 1165 MHz UltraSPARC-T2
P20: 1165 MHz UltraSPARC-T2
P21: 1165 MHz UltraSPARC-T2
P22: 1165 MHz UltraSPARC-T2
P23: 1165 MHz UltraSPARC-T2
P24: 1165 MHz UltraSPARC-T2
P25: 1165 MHz UltraSPARC-T2
P26: 1165 MHz UltraSPARC-T2
P27: 1165 MHz UltraSPARC-T2
P28: 1165 MHz UltraSPARC-T2
P29: 1165 MHz UltraSPARC-T2
P30: 1165 MHz UltraSPARC-T2
P31: 1165 MHz UltraSPARC-T2
P32: 1165 MHz UltraSPARC-T2
P33: 1165 MHz UltraSPARC-T2
P34: 1165 MHz UltraSPARC-T2
P35: 1165 MHz UltraSPARC-T2
P36: 1165 MHz UltraSPARC-T2
P37: 1165 MHz UltraSPARC-T2
P38: 1165 MHz UltraSPARC-T2
P39: 1165 MHz UltraSPARC-T2
P40: 1165 MHz UltraSPARC-T2
P41: 1165 MHz UltraSPARC-T2
P42: 1165 MHz UltraSPARC-T2
P43: 1165 MHz UltraSPARC-T2
P44: 1165 MHz UltraSPARC-T2
P45: 1165 MHz UltraSPARC-T2
P46: 1165 MHz UltraSPARC-T2
P47: 1165 MHz UltraSPARC-T2
P48: 1165 MHz UltraSPARC-T2
P49: 1165 MHz UltraSPARC-T2
P50: 1165 MHz UltraSPARC-T2
P51: 1165 MHz UltraSPARC-T2
P52: 1165 MHz UltraSPARC-T2
P53: 1165 MHz UltraSPARC-T2
P54: 1165 MHz UltraSPARC-T2
P55: 1165 MHz UltraSPARC-T2
P56: 1165 MHz UltraSPARC-T2
P57: 1165 MHz UltraSPARC-T2
P58: 1165 MHz UltraSPARC-T2
P59: 1165 MHz UltraSPARC-T2
P60: 1165 MHz UltraSPARC-T2
P61: 1165 MHz UltraSPARC-T2
P62: 1165 MHz UltraSPARC-T2
P63: 1165 MHz UltraSPARC-T2
OS release detail:
Solaris 10 5/08 s10s_u5wos_10 SPARC Copyright 2008 Sun Microsystems,
Inc. All Rights Reserved. Use is subject to license terms. Assembled 24
March 2008
Workload Measurements
Observed system for 10 min
in intervals of 10 sec
Cycles 44768051692942
Instructions 3980806371547
CPI 11.25 **
FP instructions 5821938521
Emulated FP instructions 0
FP Percentage 0.1%
The following applies to the measurement interval with the
busiest single thread or process:
Peak thread utilization at 2009-02-13 14:36:16
Corresponding file name 1234515976
CPU utilization 39.4%
Command java
PID/LWPID 16396/1
Thread utilization 0.6%
More detail on processes and threads is in data/process.out
**Cycles per Instruction (CPI) is not comparable between UltraSPARC
T1 and T2 processors and conventional processors. Conventional
processors execute an idle loop when there is no work to do, so
CPI may be artificially low, especially when the system is
somewhat idle. The UltraSPARC T1 and T2 "park" idle threads,
consuming no energy, when there is no work to do, so CPI may
be artificially high, especially when the system is somewhat idle.
Advice
Floating Point GREEN
Observed floating point content was not excessive for
an UltraSPARC T1 processor. Floating point content is not
a limitation for UltraSPARC T2.
Parallelism GREEN
The observed workload has sufficient threads of execution to
efficiently utilize the multiple cores and threads of an
UltraSPARC T1 or UltraSPARC T2 processor.
Varun Dhussa
Product Architect
CE InfoSystems (P) Ltd
http://www.mapmyindia.com
Glen Newton wrote:
> Could you give some configuration details:
> - Solaris version
> - Java VM version, heap size, and any other flags
> - disk setup
>
> You should also consider using huge pages (see
> http://zzzoot.blogspot.com/2009/02/java-mysql-increased-performance-with.html)
>
> I will also be posting performance gains using huge pages for Java
> Lucene Linux large scale indexing in the next week or so...
>
> -glen
>
>
>
> 2009/2/18 Varun Dhussa <va...@mapmyindia.com>:
>
>> Hi,
>>
>> I have had a bad experience when migrating my application from Intel Xeon
>> based servers to Sun UltraSparc T2 T5120 servers. Lucene fuzzy search just
>> does not perform. A search which took approximately 500 ms takes more than 6
>> seconds to execute.
>>
>> The index has about 100,000,000 records. So, I tried to split it into 10
>> indices and used the ParallelSearcher on it, but still got similar results.
>>
>> I am guessing that this is because the distance implementation used by
>> Lucene requires higher clock speed and can't be parallelized much.
>>
>> Please advice
>>
>> --
>> Varun Dhussa
>> Product Architect
>> CE InfoSystems (P) Ltd
>> http://www.mapmyindia.com
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene search performance on Sun UltraSparc T2 (T5120) servers
Posted by Glen Newton <gl...@gmail.com>.
Could you give some configuration details:
- Solaris version
- Java VM version, heap size, and any other flags
- disk setup
You should also consider using huge pages (see
http://zzzoot.blogspot.com/2009/02/java-mysql-increased-performance-with.html)
I will also be posting performance gains using huge pages for Java
Lucene Linux large scale indexing in the next week or so...
-glen
2009/2/18 Varun Dhussa <va...@mapmyindia.com>:
> Hi,
>
> I have had a bad experience when migrating my application from Intel Xeon
> based servers to Sun UltraSparc T2 T5120 servers. Lucene fuzzy search just
> does not perform. A search which took approximately 500 ms takes more than 6
> seconds to execute.
>
> The index has about 100,000,000 records. So, I tried to split it into 10
> indices and used the ParallelSearcher on it, but still got similar results.
>
> I am guessing that this is because the distance implementation used by
> Lucene requires higher clock speed and can't be parallelized much.
>
> Please advice
>
> --
> Varun Dhussa
> Product Architect
> CE InfoSystems (P) Ltd
> http://www.mapmyindia.com
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
-
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Phrase indexing and searching with Lucene
Posted by Chris Hostetter <ho...@fucit.org>.
: Subject: Phrase indexing and searching with Lucene
: References: <49...@mapmyindia.com>
: <18...@web26005.mail.ukl.yahoo.com>
http://people.apache.org/~hossman/#threadhijack
Thread Hijacking on Mailing Lists
When starting a new discussion on a mailing list, please do not reply to
an existing message, instead start a fresh email. Even if you change the
subject line of your email, other mail headers still track which thread
you replied to and your question is "hidden" in that thread and gets less
attention. It makes following discussions in the mailing list archives
particularly difficult.
See Also: http://en.wikipedia.org/wiki/Thread_hijacking
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Phrase indexing and searching with Lucene
Posted by Erick Erickson <er...@gmail.com>.
It looks to me like what you're trying to do is akin to document similarity,
which
I haven't had to delve into. But it's been discussed on the user list a few
times,
so perhaps your best bet would be to search the mail archives for that
topic.
Best
Erick
On Thu, Feb 19, 2009 at 3:14 AM, Nada Mimouni <
mimouni@tk.informatik.tu-darmstadt.de> wrote:
>
> Hello,
>
> Thank you Erick for this detailed answer, that makes things clearer in my
> mind.
>
> >I'm still not clear why the built-in phrase query syntax won't work.
>
> I have programmed a set of java classes (I use Lucene classes) to index and
> search into a collection of documents for a set of queries.
> To test my system, I use a corpus which consists in a collection of queries
> (n queries) and documents (m documents).
> I started by creating one index for all queries and another one for all
> documents. Then I make the search to match between the queries index and
> documents index.
> I use a trec evaluation tool to generate a file that gives all hits
> (matches) between the queryID and documentID with different scores.
>
> In this first step, I just index terms, therefore the search process (as I
> have it now) looks only for term matches between the query terms and the
> documents terms.
> Now I want to get better results (better matching) by adding phrases to
> terms.
>
> I don't know exactly whether it makes a difference if I index phrases and
> terms (erick, erickson, thinks, small, thoughts, erick erickson, erickson
> thinks, small thoughts, erickson thoughts) and then search for both, or just
> keep the indexing process as it is (erick, erickson, thinks, small,
> thoughts) and then make a search for phrases (PhraseQuery : erick erickson,
> erickson thinks, small thoughts, erickson thoughts) and terms.
> Any idea?
>
>
> >Some examples of what you put in your index and what searches
> >you expect to return results for your example AND searches you do
> >NOT want to hit that document would be a great help.
>
> input:
>
> *Query*
> 898 Why is the sun bright?
>
> *Documents*
> 7568 Star, large celestial body composed of gravitationally contained hot
> gases emitting electromagnetic radiation, especially light, as a result of
> nuclear reactions inside the star. The sun is a star.
> 7567 The sun has a magnitude of -26.7, inasmuch as it is about 10 billion
> times as bright as Sirius in the earth's sky.
>
> output:
>
> qID dID score
> 898 7568 0,13 (not relevant)
> 898 7567 1 (relevant)
>
>
> In this example, Lucene matches document 7567 to be relevant to he query
> (since it contains all query terms), however bright here is relative to
> Sirius (what we need is to get "sun bright").
>
>
>
>
> Best
> Nada
>
>
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: Wed 2/18/2009 3:24 PM
> To: java-user@lucene.apache.org
> Subject: Re: Phrase indexing and searching with Lucene
>
> I'm still not clear why the built-in phrase query syntax won't work. If I
> index the following terms (erick, erickson, thinks, small, thoughts)
> in a single field, then searching for "erick erickson" (as a phrase query,
> i.e. with double quotes when sent through a query parser or constructing
> a PhraseQuery yourself) will generate a hit but "erick thinks" won't
> generate a hit (unless you specify slop).
>
> "thinks small thoughts" would also generate a hit
>
> If you're saying that you only want to match on *all* the tokens, i.e.
> the only way to get a hit on the above would be to search for
> "erick erickson thinks small thoughts", then you can create a
> field that's UN_ANALYZED. If you do this, though, beware
> that you have to do things like lower-case terms yourself when
> indexing.
>
> I have no idea what IndexTermGenerator is or what it does, but I'm
> assuming that it just generates single words.
>
> Some examples of what you put in your index and what searches
> you expect to return results for your example AND searches you do
> NOT want to hit that document would be a great help.
>
> As far as searching for both, constructing a BooleanQuery with regular
> TermQuerys and PhraseQuerys would work if you're constructing
> your queries programmatically, or just using a Lucene query
> like +termfield:word +phrasefield:"erick erickson thinks" would
> work. Or, if you just require that the phrase exists you could do
> it all in one field like
> +field:word +field:"erick erickson thinks"
>
>
>
> Best
> Erick
>
>
> On Wed, Feb 18, 2009 at 8:42 AM, Nada Mimouni <
> mimouni@tk.informatik.tu-darmstadt.de> wrote:
>
> >
> >
> > Thank you Erick.
> >
> > I need first to index phrases, the built-in phrase processing (with
> double
> > quotes) comes in the search step.
> > Is there any difference between :
> > 1) start by indexing phrases and then make a phrase search
> > 2) index terms and then search for phrases
> >
> >
> > To make things clearer:
> >
> > What I am doing now:
> > - In the indexing step: I am using "IndexTermGenerator" to generate
> term
> > based indexes, one index for all queries I have and another one for
> > documents (term means single word).
> > - In the search step : Lucene matches terms in queries index with terms
> in
> > documents index.
> >
> > What I need to do:
> > - Index phrases ("multi" words) in addition to terms (single words)
> > - Search for both : phrases and terms
> >
> >
> > Is there any idea on how to proceed?
> >
> > Regards
> > Nada
> >
> >
> > -----Original Message-----
> > From: Erick Erickson [mailto:erickerickson@gmail.com]
> > Sent: Wed 2/18/2009 2:10 PM
> > To: java-user@lucene.apache.org
> > Subject: Re: Phrase indexing and searching with Lucene
> >
> > Have you tried the built-in phrase processing with double quotes? e.g.
> > "this is a phrase"?
> >
> > See the Term section at
> > http://lucene.apache.org/java/2_4_0/queryparsersyntax.html
> >
> > Best
> > Erick
> >
> > On Wed, Feb 18, 2009 at 5:57 AM, Nada Mimouni <
> > mimouni@tk.informatik.tu-darmstadt.de> wrote:
> >
> > >
> > >
> > > Hello everybody,
> > >
> > > I use Lucene to index and search into text documents.
> > > At present, I just index and search for single words. I want to extend
> > this
> > > to phrases (or nGrams).
> > >
> > > Could anyone please give me details on how to index phrases and then
> make
> > a
> > > phrase search?
> > >
> > > Thank you very much in advance for your help.
> > >
> > > Nada Mimouni
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
RE: Phrase indexing and searching with Lucene
Posted by Nada Mimouni <mi...@tk.informatik.tu-darmstadt.de>.
Hello,
Thank you Erick for this detailed answer, that makes things clearer in my mind.
>I'm still not clear why the built-in phrase query syntax won't work.
I have programmed a set of java classes (I use Lucene classes) to index and search into a collection of documents for a set of queries.
To test my system, I use a corpus which consists in a collection of queries (n queries) and documents (m documents).
I started by creating one index for all queries and another one for all documents. Then I make the search to match between the queries index and documents index.
I use a trec evaluation tool to generate a file that gives all hits (matches) between the queryID and documentID with different scores.
In this first step, I just index terms, therefore the search process (as I have it now) looks only for term matches between the query terms and the documents terms.
Now I want to get better results (better matching) by adding phrases to terms.
I don't know exactly whether it makes a difference if I index phrases and terms (erick, erickson, thinks, small, thoughts, erick erickson, erickson thinks, small thoughts, erickson thoughts) and then search for both, or just keep the indexing process as it is (erick, erickson, thinks, small, thoughts) and then make a search for phrases (PhraseQuery : erick erickson, erickson thinks, small thoughts, erickson thoughts) and terms.
Any idea?
>Some examples of what you put in your index and what searches
>you expect to return results for your example AND searches you do
>NOT want to hit that document would be a great help.
input:
*Query*
898 Why is the sun bright?
*Documents*
7568 Star, large celestial body composed of gravitationally contained hot gases emitting electromagnetic radiation, especially light, as a result of nuclear reactions inside the star. The sun is a star.
7567 The sun has a magnitude of -26.7, inasmuch as it is about 10 billion times as bright as Sirius in the earth's sky.
output:
qID dID score
898 7568 0,13 (not relevant)
898 7567 1 (relevant)
In this example, Lucene matches document 7567 to be relevant to he query (since it contains all query terms), however bright here is relative to Sirius (what we need is to get "sun bright").
Best
Nada
-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com]
Sent: Wed 2/18/2009 3:24 PM
To: java-user@lucene.apache.org
Subject: Re: Phrase indexing and searching with Lucene
I'm still not clear why the built-in phrase query syntax won't work. If I
index the following terms (erick, erickson, thinks, small, thoughts)
in a single field, then searching for "erick erickson" (as a phrase query,
i.e. with double quotes when sent through a query parser or constructing
a PhraseQuery yourself) will generate a hit but "erick thinks" won't
generate a hit (unless you specify slop).
"thinks small thoughts" would also generate a hit
If you're saying that you only want to match on *all* the tokens, i.e.
the only way to get a hit on the above would be to search for
"erick erickson thinks small thoughts", then you can create a
field that's UN_ANALYZED. If you do this, though, beware
that you have to do things like lower-case terms yourself when
indexing.
I have no idea what IndexTermGenerator is or what it does, but I'm
assuming that it just generates single words.
Some examples of what you put in your index and what searches
you expect to return results for your example AND searches you do
NOT want to hit that document would be a great help.
As far as searching for both, constructing a BooleanQuery with regular
TermQuerys and PhraseQuerys would work if you're constructing
your queries programmatically, or just using a Lucene query
like +termfield:word +phrasefield:"erick erickson thinks" would
work. Or, if you just require that the phrase exists you could do
it all in one field like
+field:word +field:"erick erickson thinks"
Best
Erick
On Wed, Feb 18, 2009 at 8:42 AM, Nada Mimouni <
mimouni@tk.informatik.tu-darmstadt.de> wrote:
>
>
> Thank you Erick.
>
> I need first to index phrases, the built-in phrase processing (with double
> quotes) comes in the search step.
> Is there any difference between :
> 1) start by indexing phrases and then make a phrase search
> 2) index terms and then search for phrases
>
>
> To make things clearer:
>
> What I am doing now:
> - In the indexing step: I am using "IndexTermGenerator" to generate term
> based indexes, one index for all queries I have and another one for
> documents (term means single word).
> - In the search step : Lucene matches terms in queries index with terms in
> documents index.
>
> What I need to do:
> - Index phrases ("multi" words) in addition to terms (single words)
> - Search for both : phrases and terms
>
>
> Is there any idea on how to proceed?
>
> Regards
> Nada
>
>
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: Wed 2/18/2009 2:10 PM
> To: java-user@lucene.apache.org
> Subject: Re: Phrase indexing and searching with Lucene
>
> Have you tried the built-in phrase processing with double quotes? e.g.
> "this is a phrase"?
>
> See the Term section at
> http://lucene.apache.org/java/2_4_0/queryparsersyntax.html
>
> Best
> Erick
>
> On Wed, Feb 18, 2009 at 5:57 AM, Nada Mimouni <
> mimouni@tk.informatik.tu-darmstadt.de> wrote:
>
> >
> >
> > Hello everybody,
> >
> > I use Lucene to index and search into text documents.
> > At present, I just index and search for single words. I want to extend
> this
> > to phrases (or nGrams).
> >
> > Could anyone please give me details on how to index phrases and then make
> a
> > phrase search?
> >
> > Thank you very much in advance for your help.
> >
> > Nada Mimouni
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
Re: Phrase indexing and searching with Lucene
Posted by Erick Erickson <er...@gmail.com>.
I'm still not clear why the built-in phrase query syntax won't work. If I
index the following terms (erick, erickson, thinks, small, thoughts)
in a single field, then searching for "erick erickson" (as a phrase query,
i.e. with double quotes when sent through a query parser or constructing
a PhraseQuery yourself) will generate a hit but "erick thinks" won't
generate a hit (unless you specify slop).
"thinks small thoughts" would also generate a hit
If you're saying that you only want to match on *all* the tokens, i.e.
the only way to get a hit on the above would be to search for
"erick erickson thinks small thoughts", then you can create a
field that's UN_ANALYZED. If you do this, though, beware
that you have to do things like lower-case terms yourself when
indexing.
I have no idea what IndexTermGenerator is or what it does, but I'm
assuming that it just generates single words.
Some examples of what you put in your index and what searches
you expect to return results for your example AND searches you do
NOT want to hit that document would be a great help.
As far as searching for both, constructing a BooleanQuery with regular
TermQuerys and PhraseQuerys would work if you're constructing
your queries programmatically, or just using a Lucene query
like +termfield:word +phrasefield:"erick erickson thinks" would
work. Or, if you just require that the phrase exists you could do
it all in one field like
+field:word +field:"erick erickson thinks"
Best
Erick
On Wed, Feb 18, 2009 at 8:42 AM, Nada Mimouni <
mimouni@tk.informatik.tu-darmstadt.de> wrote:
>
>
> Thank you Erick.
>
> I need first to index phrases, the built-in phrase processing (with double
> quotes) comes in the search step.
> Is there any difference between :
> 1) start by indexing phrases and then make a phrase search
> 2) index terms and then search for phrases
>
>
> To make things clearer:
>
> What I am doing now:
> - In the indexing step: I am using "IndexTermGenerator" to generate term
> based indexes, one index for all queries I have and another one for
> documents (term means single word).
> - In the search step : Lucene matches terms in queries index with terms in
> documents index.
>
> What I need to do:
> - Index phrases ("multi" words) in addition to terms (single words)
> - Search for both : phrases and terms
>
>
> Is there any idea on how to proceed?
>
> Regards
> Nada
>
>
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: Wed 2/18/2009 2:10 PM
> To: java-user@lucene.apache.org
> Subject: Re: Phrase indexing and searching with Lucene
>
> Have you tried the built-in phrase processing with double quotes? e.g.
> "this is a phrase"?
>
> See the Term section at
> http://lucene.apache.org/java/2_4_0/queryparsersyntax.html
>
> Best
> Erick
>
> On Wed, Feb 18, 2009 at 5:57 AM, Nada Mimouni <
> mimouni@tk.informatik.tu-darmstadt.de> wrote:
>
> >
> >
> > Hello everybody,
> >
> > I use Lucene to index and search into text documents.
> > At present, I just index and search for single words. I want to extend
> this
> > to phrases (or nGrams).
> >
> > Could anyone please give me details on how to index phrases and then make
> a
> > phrase search?
> >
> > Thank you very much in advance for your help.
> >
> > Nada Mimouni
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
RE: Phrase indexing and searching with Lucene
Posted by Nada Mimouni <mi...@tk.informatik.tu-darmstadt.de>.
Thank you Erick.
I need first to index phrases, the built-in phrase processing (with double quotes) comes in the search step.
Is there any difference between :
1) start by indexing phrases and then make a phrase search
2) index terms and then search for phrases
To make things clearer:
What I am doing now:
- In the indexing step: I am using "IndexTermGenerator" to generate term based indexes, one index for all queries I have and another one for documents (term means single word).
- In the search step : Lucene matches terms in queries index with terms in documents index.
What I need to do:
- Index phrases ("multi" words) in addition to terms (single words)
- Search for both : phrases and terms
Is there any idea on how to proceed?
Regards
Nada
-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com]
Sent: Wed 2/18/2009 2:10 PM
To: java-user@lucene.apache.org
Subject: Re: Phrase indexing and searching with Lucene
Have you tried the built-in phrase processing with double quotes? e.g.
"this is a phrase"?
See the Term section at
http://lucene.apache.org/java/2_4_0/queryparsersyntax.html
Best
Erick
On Wed, Feb 18, 2009 at 5:57 AM, Nada Mimouni <
mimouni@tk.informatik.tu-darmstadt.de> wrote:
>
>
> Hello everybody,
>
> I use Lucene to index and search into text documents.
> At present, I just index and search for single words. I want to extend this
> to phrases (or nGrams).
>
> Could anyone please give me details on how to index phrases and then make a
> phrase search?
>
> Thank you very much in advance for your help.
>
> Nada Mimouni
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
Re: Phrase indexing and searching with Lucene
Posted by Erick Erickson <er...@gmail.com>.
Have you tried the built-in phrase processing with double quotes? e.g.
"this is a phrase"?
See the Term section at
http://lucene.apache.org/java/2_4_0/queryparsersyntax.html
Best
Erick
On Wed, Feb 18, 2009 at 5:57 AM, Nada Mimouni <
mimouni@tk.informatik.tu-darmstadt.de> wrote:
>
>
> Hello everybody,
>
> I use Lucene to index and search into text documents.
> At present, I just index and search for single words. I want to extend this
> to phrases (or nGrams).
>
> Could anyone please give me details on how to index phrases and then make a
> phrase search?
>
> Thank you very much in advance for your help.
>
> Nada Mimouni
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
Phrase indexing and searching with Lucene
Posted by Nada Mimouni <mi...@tk.informatik.tu-darmstadt.de>.
Hello everybody,
I use Lucene to index and search into text documents.
At present, I just index and search for single words. I want to extend this to phrases (or nGrams).
Could anyone please give me details on how to index phrases and then make a phrase search?
Thank you very much in advance for your help.
Nada Mimouni
Re: Lucene search performance on Sun UltraSparc T2 (T5120) servers
Posted by eks dev <ek...@yahoo.co.uk>.
Have you tried NGram SpellChecker + Query expansion? This is quite similar to your proposal, you have your priority queue in SpellChecker
----- Original Message ----
> From: mark harwood <ma...@yahoo.co.uk>
> To: java-user@lucene.apache.org
> Sent: Wednesday, 18 February, 2009 11:54:18
> Subject: Re: Lucene search performance on Sun UltraSparc T2 (T5120) servers
>
>
> I was having some thoughts recently about speeding up fuzzy search.
>
> The current system does edit-distance on all terms A-Z, single threaded. Prefix
> length can reduce the search space and there is a "minimum similarity" threshold
> but that's roughly where we are. Multithreading this to make use of multiple
> CPUs is one option to look at but I was mainly thinking about smarter ways to do
> the fuzzy scan:
>
> I had the notion that we could move to a solution where a priority queue keeps
> the "best matches so far" and as you progress through the termEnum you could
> bail out of edit distance calculations quickly using a rough(cheap) assessment
> of if the current term is likely to make the cut (i.e. beat the current lowest
> score in the priority queue). It would make sense to populate the priority queue
> ASAP with terms that are most likely to be the best matches and these will be
> the ones that share a reasonable length prefix.
> As an example - searching for Obama~
>
> 1) Create "best matches" priority queue
> 2) Scan all terms from oba to obz populating priority queue
> 3) Scan all terms from "a" to "oba" and "obz" to "z", exiting quickly if the
> term fails to meet lowest score in the priority queue.
>
> How we "exit quickly" and how we determine what prefix to use in 2) are to be
> determined but the principle seems reasonable
>
> Thoughts?
>
>
>
>
> ----- Original Message ----
> From: Varun Dhussa
> To: java-user@lucene.apache.org
> Sent: Wednesday, 18 February, 2009 10:36:07
> Subject: Lucene search performance on Sun UltraSparc T2 (T5120) servers
>
> Hi,
>
> I have had a bad experience when migrating my application from Intel Xeon based
> servers to Sun UltraSparc T2 T5120 servers. Lucene fuzzy search just does not
> perform. A search which took approximately 500 ms takes more than 6 seconds to
> execute.
>
> The index has about 100,000,000 records. So, I tried to split it into 10 indices
> and used the ParallelSearcher on it, but still got similar results.
>
> I am guessing that this is because the distance implementation used by Lucene
> requires higher clock speed and can't be parallelized much.
>
> Please advice
>
> -- Varun Dhussa
> Product Architect
> CE InfoSystems (P) Ltd
> http://www.mapmyindia.com
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene search performance on Sun UltraSparc T2 (T5120) servers
Posted by Michael Stoppelman <st...@gmail.com>.
Fuzzy search tends to be super heavy on CPU because of the Levenstein
distance algo. We use it for a small index 60MB for spell correcting and our
QPS suffers as a result.
There was recently a discussion of a new fuzzy algorithm:
https://issues.apache.org/jira/browse/LUCENE-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12661286#action_12661286
M
On Wed, Feb 18, 2009 at 5:59 AM, Varun Dhussa <va...@mapmyindia.com> wrote:
> The method suggested would make the speed faster, but I doubt whether it
> would be substantial on processors with slower clock speed. Keeping in mind
> that most processors are going multi-core, it would make sense to
> multi-thread the scan.
>
> Any remarks are welcome!
>
> Varun Dhussa
> Product Architect
> CE InfoSystems (P) Ltd
> http://www.mapmyindia.com
>
>
>
> mark harwood wrote:
>
>> I was having some thoughts recently about speeding up fuzzy search.
>>
>> The current system does edit-distance on all terms A-Z, single threaded.
>> Prefix length can reduce the search space and there is a "minimum
>> similarity" threshold but that's roughly where we are. Multithreading this
>> to make use of multiple CPUs is one option to look at but I was mainly
>> thinking about smarter ways to do the fuzzy scan:
>>
>> I had the notion that we could move to a solution where a priority queue
>> keeps the "best matches so far" and as you progress through the termEnum you
>> could bail out of edit distance calculations quickly using a rough(cheap)
>> assessment of if the current term is likely to make the cut (i.e. beat the
>> current lowest score in the priority queue). It would make sense to populate
>> the priority queue ASAP with terms that are most likely to be the best
>> matches and these will be the ones that share a reasonable length prefix.
>> As an example - searching for Obama~
>>
>> 1) Create "best matches" priority queue
>> 2) Scan all terms from oba to obz populating priority queue
>> 3) Scan all terms from "a" to "oba" and "obz" to "z", exiting quickly if
>> the term fails to meet lowest score in the priority queue.
>>
>> How we "exit quickly" and how we determine what prefix to use in 2) are to
>> be determined but the principle seems reasonable
>>
>> Thoughts?
>>
>>
>>
>>
>> ----- Original Message ----
>> From: Varun Dhussa <va...@mapmyindia.com>
>> To: java-user@lucene.apache.org
>> Sent: Wednesday, 18 February, 2009 10:36:07
>> Subject: Lucene search performance on Sun UltraSparc T2 (T5120) servers
>>
>> Hi,
>>
>> I have had a bad experience when migrating my application from Intel Xeon
>> based servers to Sun UltraSparc T2 T5120 servers. Lucene fuzzy search just
>> does not perform. A search which took approximately 500 ms takes more than 6
>> seconds to execute.
>>
>> The index has about 100,000,000 records. So, I tried to split it into 10
>> indices and used the ParallelSearcher on it, but still got similar results.
>>
>> I am guessing that this is because the distance implementation used by
>> Lucene requires higher clock speed and can't be parallelized much.
>>
>> Please advice
>>
>> -- Varun Dhussa
>> Product Architect
>> CE InfoSystems (P) Ltd
>> http://www.mapmyindia.com
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Lucene search performance on Sun UltraSparc T2 (T5120) servers
Posted by Varun Dhussa <va...@mapmyindia.com>.
The method suggested would make the speed faster, but I doubt whether it
would be substantial on processors with slower clock speed. Keeping in
mind that most processors are going multi-core, it would make sense to
multi-thread the scan.
Any remarks are welcome!
Varun Dhussa
Product Architect
CE InfoSystems (P) Ltd
http://www.mapmyindia.com
mark harwood wrote:
> I was having some thoughts recently about speeding up fuzzy search.
>
> The current system does edit-distance on all terms A-Z, single threaded. Prefix length can reduce the search space and there is a "minimum similarity" threshold but that's roughly where we are. Multithreading this to make use of multiple CPUs is one option to look at but I was mainly thinking about smarter ways to do the fuzzy scan:
>
> I had the notion that we could move to a solution where a priority queue keeps the "best matches so far" and as you progress through the termEnum you could bail out of edit distance calculations quickly using a rough(cheap) assessment of if the current term is likely to make the cut (i.e. beat the current lowest score in the priority queue). It would make sense to populate the priority queue ASAP with terms that are most likely to be the best matches and these will be the ones that share a reasonable length prefix.
> As an example - searching for Obama~
>
> 1) Create "best matches" priority queue
> 2) Scan all terms from oba to obz populating priority queue
> 3) Scan all terms from "a" to "oba" and "obz" to "z", exiting quickly if the term fails to meet lowest score in the priority queue.
>
> How we "exit quickly" and how we determine what prefix to use in 2) are to be determined but the principle seems reasonable
>
> Thoughts?
>
>
>
>
> ----- Original Message ----
> From: Varun Dhussa <va...@mapmyindia.com>
> To: java-user@lucene.apache.org
> Sent: Wednesday, 18 February, 2009 10:36:07
> Subject: Lucene search performance on Sun UltraSparc T2 (T5120) servers
>
> Hi,
>
> I have had a bad experience when migrating my application from Intel Xeon based servers to Sun UltraSparc T2 T5120 servers. Lucene fuzzy search just does not perform. A search which took approximately 500 ms takes more than 6 seconds to execute.
>
> The index has about 100,000,000 records. So, I tried to split it into 10 indices and used the ParallelSearcher on it, but still got similar results.
>
> I am guessing that this is because the distance implementation used by Lucene requires higher clock speed and can't be parallelized much.
>
> Please advice
>
> -- Varun Dhussa
> Product Architect
> CE InfoSystems (P) Ltd
> http://www.mapmyindia.com
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene search performance on Sun UltraSparc T2 (T5120) servers
Posted by mark harwood <ma...@yahoo.co.uk>.
I was having some thoughts recently about speeding up fuzzy search.
The current system does edit-distance on all terms A-Z, single threaded. Prefix length can reduce the search space and there is a "minimum similarity" threshold but that's roughly where we are. Multithreading this to make use of multiple CPUs is one option to look at but I was mainly thinking about smarter ways to do the fuzzy scan:
I had the notion that we could move to a solution where a priority queue keeps the "best matches so far" and as you progress through the termEnum you could bail out of edit distance calculations quickly using a rough(cheap) assessment of if the current term is likely to make the cut (i.e. beat the current lowest score in the priority queue). It would make sense to populate the priority queue ASAP with terms that are most likely to be the best matches and these will be the ones that share a reasonable length prefix.
As an example - searching for Obama~
1) Create "best matches" priority queue
2) Scan all terms from oba to obz populating priority queue
3) Scan all terms from "a" to "oba" and "obz" to "z", exiting quickly if the term fails to meet lowest score in the priority queue.
How we "exit quickly" and how we determine what prefix to use in 2) are to be determined but the principle seems reasonable
Thoughts?
----- Original Message ----
From: Varun Dhussa <va...@mapmyindia.com>
To: java-user@lucene.apache.org
Sent: Wednesday, 18 February, 2009 10:36:07
Subject: Lucene search performance on Sun UltraSparc T2 (T5120) servers
Hi,
I have had a bad experience when migrating my application from Intel Xeon based servers to Sun UltraSparc T2 T5120 servers. Lucene fuzzy search just does not perform. A search which took approximately 500 ms takes more than 6 seconds to execute.
The index has about 100,000,000 records. So, I tried to split it into 10 indices and used the ParallelSearcher on it, but still got similar results.
I am guessing that this is because the distance implementation used by Lucene requires higher clock speed and can't be parallelized much.
Please advice
-- Varun Dhussa
Product Architect
CE InfoSystems (P) Ltd
http://www.mapmyindia.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org