You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-commits@lucene.apache.org by gs...@apache.org on 2006/12/16 21:23:31 UTC
svn commit: r487874 - in /lucene/java/trunk: CHANGES.txt
docs/fileformats.html docs/fileformats.pdf
src/site/src/documentation/content/xdocs/fileformats.xml
Author: gsingers
Date: Sat Dec 16 12:23:30 2006
New Revision: 487874
URL: http://svn.apache.org/viewvc?view=rev&rev=487874
Log:
LUCENE-713. Updated fileformats.xml to include how position and offset info is stored in the TVF file.
Modified:
lucene/java/trunk/CHANGES.txt
lucene/java/trunk/docs/fileformats.html
lucene/java/trunk/docs/fileformats.pdf
lucene/java/trunk/src/site/src/documentation/content/xdocs/fileformats.xml
Modified: lucene/java/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/lucene/java/trunk/CHANGES.txt?view=diff&rev=487874&r1=487873&r2=487874
==============================================================================
--- lucene/java/trunk/CHANGES.txt (original)
+++ lucene/java/trunk/CHANGES.txt Sat Dec 16 12:23:30 2006
@@ -331,6 +331,10 @@
5. Added in Developer and System Requriements sections under Resources (Grant Ingersoll)
+ 6. LUCENE-713 Updated the Term Vector section of File Formats to include documentation on how Offset and Position info are stored in the TVF file. (Grant Ingersoll, Samir Abdou)
+
+ 7. Added in link to Clover Test Code Coverage Reports under the Develop section in Resources (Grant Ingersoll)
+
Build
1. Added in clover test code coverage per http://issues.apache.org/jira/browse/LUCENE-721 To enable clover code coverage, you must have clover.jar in the ANT classpath and specify -Drun.clover=true on the command line.(Michael Busch and Grant Ingersoll)
Modified: lucene/java/trunk/docs/fileformats.html
URL: http://svn.apache.org/viewvc/lucene/java/trunk/docs/fileformats.html?view=diff&rev=487874&r1=487873&r2=487874
==============================================================================
--- lucene/java/trunk/docs/fileformats.html (original)
+++ lucene/java/trunk/docs/fileformats.html Sat Dec 16 12:23:30 2006
@@ -2029,9 +2029,9 @@
<p>The Field or .tvf file.</p>
<p>This file contains, for each field that has a term vector stored, a list of
- the terms and their frequencies.</p>
+ the terms, their frequencies and, optionally, position and offest information.</p>
-<p>Field (.tvf) --> TVFVersion<NumTerms, NumDistinct, TermFreqs>
+<p>Field (.tvf) --> TVFVersion<NumTerms, Position/Offset, TermFreqs>
<sup>NumFields</sup>
</p>
@@ -2040,9 +2040,9 @@
<p>NumTerms --> VInt</p>
-<p>NumDistinct --> VInt -- Future Use</p>
+<p>Position/Offset --> Byte</p>
-<p>TermFreqs --> <TermText, TermFreq>
+<p>TermFreqs --> <TermText, TermFreq, Positions?, Offsets?>
<sup>NumTerms</sup>
</p>
@@ -2055,18 +2055,40 @@
<p>TermFreq --> VInt</p>
-<p>Term
- text prefixes are shared. The PrefixLength is the number of initial
- characters from the previous term which must be pre-pended to a
- term's suffix in order to form the term's text. Thus, if the
- previous term's text was "bone" and the term is "boy",
- the PrefixLength is two and the suffix is "y".
- </p>
+<p>Positions --> <VInt><sup>TermFreq</sup>
+</p>
+
+<p>Offsets --> <VInt, VInt><sup>TermFreq</sup>
+</p>
+
+<br>
+
+<p>Notes:</p>
+
+<ul>
+
+<li>Position/Offset byte stores whether this term vector has position or offset information stored.</li>
+
+<li>Term
+ text prefixes are shared. The PrefixLength is the number of initial
+ characters from the previous term which must be pre-pended to a
+ term's suffix in order to form the term's text. Thus, if the
+ previous term's text was "bone" and the term is "boy",
+ the PrefixLength is two and the suffix is "y".
+ </li>
+
+<li>Positions are stored as delta encoded VInts. This means we only store the difference of the current position from the last position</li>
+
+<li>Offsets are stored as delta encoded VInts. The first VInt is the startOffset, the second is the endOffset.</li>
+
+</ul>
+
+
</li>
</ol>
-<a name="N1073C"></a><a name="Deleted Documents"></a>
+<a name="N10757"></a><a name="Deleted Documents"></a>
<h3 class="boxed">Deleted Documents</h3>
<p>The .del file is
optional, and only exists when a segment contains deletions.
@@ -2138,7 +2160,7 @@
</div>
-<a name="N1077F"></a><a name="Limitations"></a>
+<a name="N1079A"></a><a name="Limitations"></a>
<h2 class="boxed">Limitations</h2>
<div class="section">
<p>There
Modified: lucene/java/trunk/docs/fileformats.pdf
URL: http://svn.apache.org/viewvc/lucene/java/trunk/docs/fileformats.pdf?view=diff&rev=487874&r1=487873&r2=487874
==============================================================================
--- lucene/java/trunk/docs/fileformats.pdf (original)
+++ lucene/java/trunk/docs/fileformats.pdf Sat Dec 16 12:23:30 2006
@@ -267,10 +267,10 @@
>>
endobj
52 0 obj
-<< /Length 627 /Filter [ /ASCII85Decode /FlateDecode ]
+<< /Length 626 /Filter [ /ASCII85Decode /FlateDecode ]
>>
stream
-Gaua<9l...@JEjrOXKVCmG>;YNb!8Q%"It"ZH^M`5Kt7)h#.CYc+nB~>
+Gaua<9lHOU&;KZOME1)G1A*"Qa/1:`X/JWjPY7JJ@_E^+4qa/uS,W7nZbe&k6;GIhB#JK21EOkIB$n4i![UA[gIDAV&Hsq;FR\P2),#iA&WkQ4IA;m@i=<0>UPOo(8\d=.\KE\6]nHh@mJ`TUZKTK?ah+H&3!ZUpC_.jXDAR+;%5Gp[N00TN'V:dIF,,@S1QNp4'W!gmNZ'#j1(j+h@OdLc@^/TH$?"R-BeT5JGgFTm$O86fBX%gp'8=@lb=1W_i(MhTGuajc/\/(V1b\^6>:gN+_,SnJVI'A+1hmOA011MHq4#)UX0V^tcJj[c/sF=$lt1"#4;;%gb4K9^1pj\NeY_S4`HO2B-><&r["XZpq9GdC6?`4Yn(2,VpA*H-Cj4F7r!tXeN?Y/0"#SI>']hkg131$WaGYCT;l:<V.V>oq4r'r_].R^HVe0RjcTGoEblt(`!;?7p0H[Xbd>cMkAr;bF^a'1e.Kl/mcG2X?)I;'b9:ptkLN'fdO9jA$*;*$TR%MD#ltL(,AkL-`E_D6\/4#1;!Hd,^6i6B&Qnm:Y=njp.+">E"Zr>M2#>c3TTrlu2:*s-7eZ-Z)Z(cBS\?ZE!JM48TlIk%M>uAqrQoIfWc?^iT~>
endstream
endobj
53 0 obj
@@ -559,10 +559,10 @@
>>
endobj
95 0 obj
-<< /Length 1543 /Filter [ /ASCII85Decode /FlateDecode ]
+<< /Length 1483 /Filter [ /ASCII85Decode /FlateDecode ]
>>
stream
-GatU4>>HJ&'RnB3Y]p2E`/Hj&Au8b6<-B4f'rkq=H!0iL8=[UrAt5Db!3&2sp0@+W0W(KM6/q5+cL!^6>V7D=/LNb1"E!i#eKbH?JEHI6rl_hI53"K9P_Z1Kk[F&FjRPMGY^g*g[U1RF,#k`s2j9mp($H`k8._8cSDnh0cn[uGcPBFFf72<'=0\sWF#9Z:NG#TNB03X#Q<385#?q*j]I>+n3B,cm;H?]WV<=jSc7Z>o+Z)jR\0)Dl3\&2Uq-)n7dAqM^i1Ri"\J+ZFY)[9Zg;lEZP<bhQ'OY:Em.W%ofR^%QYR:flFGeek'PuU%8$KH87:o%[Z)Ub(Ng=tsdA/c>4RPT\"rP?2&oeoc(29uR^%745ZIH9=1ks4;8nr-Z>\i=oqSe'!6t_RVM.t7F9]?OQBXdS7gH4>>s*W=]$Wd^9]Ed'Hnh@AS^"boEh9]9']LlaA:;I,_:o(/J_"W<0""EKh5qn7Kl0-7&/A_:"j)L@s:jSj,Dj85NM(#]5\LGJC>3%1_8rl8jcmIVb62-@Cma"&L]Y%J(M5&Kcpdt(qdXVMd)YVTob'[3?=gMn;0pV2S)&<mT#?IW*PE-C14Kq-)U\uUGZ7rQ"#P<B7*kMOfTrr.O!IpHr*5G*]?27qj+,QHr9CXiNDV*U5nfn4R,Q]gXh<0uG;i$E\\eb+Mg&[VNogV`,*kre[)Ee$s=+m)HJ$otQ=56i1oG!iM$iuc<VYVtZc%:t:@/`LQ*?'&(9P>7Fj\??<)#+S=_76VC;m*q0]ZqQ$+<sE31=mg5h#j\,P1,[C)b?i[b!spX#q4;`9Wd2LWZ"3EXRWfaOE.ZGEo;RsdHi"H[BH/4./4JlhE`F&3iIon1^UhB<7<ccpiJW$BFKLGptB^o%EVHC."9rbVp6+`d3G_]-P%[C-iO2+o7V5SGWqOANk&"GAqC:FX\L]:WqYFtjY)GVq4oZH9qr`e3/N8!R86;.q;agS,?/KAJ+9]me@+L=pXaE
BRM1DXVj5#!7#6L@h%+'27te;h7Q:H5f$1b(@$#h;X#^o><C5?`qgaD07kEq6,fYT!%g3[O,rjsrX?0_rONILi=<1)AR-pjnYGcjK%%<q=?.m#-h6MW#!r5oTX\nk9gLY$K`_,C\1$h,.c%5qZ?F+#5eP=Y!5S=<!bW6V:N;EE8\<:PO"qC3ec6cCZjWT5@aptG+L3]Vh&N?j#M.3"I]'.Y3;+uEVKT#^5]5RW1D6PXb/+H'GWMZ#fHMK<AUK0;)Z.[=OE]n;8+4O0c0217>c-,.fF9ERK/n+PQ!sDr,#=mEM(oi'kTj+W9_n=S.p3\Ft"1!94:_[CDo7dF[/B8`!3i`Q]\c=hUY.k:/.)JP4Vs*KY?A[I.RQ7-]\e%"b1Q*mDlpQKGaa3QmDcI(%A%I?FcXh"bIFI@@)8bO)<]+EP0H<S'lC2n]as1u#2[N[)d#:I>`o6q4360.2UR5a'F@Q2&%aTjam[AZdil24f'0FVK3ObcDCYAfn?8K;qS*I9E@G7-'adqQsMSZRrh@:IW,oDP<'Z[*>l&GEjW*U\Y~>
+Gau0D95iiK&AJ$C#e=aQ`/5t"H5)8gR^Ohl&StB6aJJ,e[]*AZ/:'b,NrOMt&">qCbU/)hfPF(fB0OTL^DZSpRJi8lIa!W@!rr7i`K'=f5nLhMT=06qWUI]R'sj44^Lqa4(;F!KMmQDS2KQH0%f?lr3)R;P\p!$,<jK...@itfKFmPHQKFN>\gn2IO@*F[L-u_\/+qk<\lM@Nb(Nl]O&.:U=P^"3[Y$1rf6pfK';c/sY/)KBK"b7qc)s]\L$`l8+Z=%QnpIlcOjTd5`Go"mCpm!63!25D]Ts>3#_V5\"cf3o615YB$:IqbXhc@NDkHX@(sBfDa2NG\i%h]>(!Us(>amdn;/uVh>IkC
Kj@XHoQ*`msgW\%4S]'`^9WJ(>TuL7B?A!$%DJC6b:)b]Q:;dCq@`_8d@;#cjB_?'A3fY$.ULcJ^7"Jm#G@]`b^B`@9?5VAc6@Su,M!3PX9#adfmeN"'>.qpADoS%^ChZlt0ue1A/B(X#a=&^"is:7Q+6sp#[gFZ*Je4TUS?X5C2&3(*bVLEb2i_m?$LL_s+FMZbMuk`5=T;kJPj;b3iV1n!VfXBQ[&PKjW[bmDY-QVK"kn_f8Jl:EU%K2^h9<8N...@Rk>";uFj+W>u5nML=mnl#lsEGK21_flWnrgWY:dUI:,I`sLg?8;M_F][r,bGuc#A>jQ)=]IDI1$ds<';92%9>G*\aXP1ks!1`d*^Xm@\~>
endstream
endobj
96 0 obj
@@ -574,10 +574,10 @@
>>
endobj
97 0 obj
-<< /Length 1591 /Filter [ /ASCII85Decode /FlateDecode ]
+<< /Length 1816 /Filter [ /ASCII85Decode /FlateDecode ]
>>
stream
-GatU4gMZ%0&:O:SkV5A?L=BKt;4uds6&MDGGh\"N&%"dY67#JZ;82DKG9kkh:ncYqMBK0Z,3Qs%B1?%,B0WJ1T4Z=$_=-D8U4c^an&&MaCVMMBa!&/s$#;?oD79B%m11_n@Q$>O-SsJIU@?(%rhO0?T=WT7\,NsCaXZIg^-!q9S:27FAZ@r8VGR$<_X;O'/Jd4R_T0#[NdYjED68:-Blt84Y?F4b4e^i(eeF9OnRBn=PYkmsQ%&bSb=Si'_thWapadmTZ@H!A8P<A,&W//7)#E9,ik-@pn9Q-YS5@[!18o3kn_FV/;AB'_V+P=81P-#h['b_=/Vek3V(R0a9?X%@40I!6o('T&7blMBhX,\%GnCR=g1,V,rYn0.m(NNkS@2Q6MjP?*!i02NJ8fSI=g^c9'MZ:ReKE%GpqAb(kL)[-)!1.!i^VG."RB36*gU.#T#IiaX,$3+r[VRhb)P$R]XQGc^LGWk:qmrU+XPm?W^NPbla9PJZXF-XO+n"g$",7k`R=mja?jQh%RM@:1kKZ?K+uLGPD`:]`2Q,IFlgr_-mhigRFt1<\Lo2(\(IAV&uEZ[fatWJ8p5S&(a+pgq:'O(6:3,A?Mu*FR;qjM(7A!&$C3YH=iI,`6H"bF9'1bj$!o-'/ok*aK6L2_f'0>.'7aklO?;cVKD?]cF=c67M)lgs\dCl`k_[^o)M1JfQ>2A8\#GDET63L**7U^;b:mb.UCMoM<MBUQJl<L=-*M&(^5IWr.Tp1%-*Ca?5k`Z(,m2N.SBQ!p,JBID7\7f?'WVmm!uLSu>+=%WIB.`S&PmTlnU@==3Nkkp&@I8AGLGcWG!k%.)Qiu`'&4hjN$p@Rk'JO&]!**W0EF)P8(6*_UXTt'b=kT/,\.)]9'[*5`<E.u([WhcJaoXVA9?,`Grn>2D<Ms"_V.jF))ANakF,Sm?Q0[;U@N]r`]'Dl1c!>GKsZ&ZpXsXtj&i!2Z#;X#a=bb]K0:Mh'N2td5@):
_:QIu=AQ:g-qVgn(<rXZ-fT`nt<rqFqk.'-W5[Z=5FRK9WC4MC.a6uq?U.Y20bu;U9q6J"HC9J]hA]4<@X&U7SRX"kZCt^%ka['icI048`pEnlZ#\D#b\>YG!%Y-40bTtH;e;^6eE7W#h5p86QV0R]^j"3_sU<:Nqn.pn)i"H&&qm?`=;DO:SNkNdDQVS'tf2S:Q\o;q1>0=.6&-e)S\0,'^RoosIXK`nFRn;mB1?N5E^L2)GM0s3b5g)cd$u?hel06^EQ:YpUK1+%.>C;'54$l3"+`r1RfVnO*Rrll@B^HKI%!:\KH8.=Sctk2[<l#o."CCS'@hRRYfU!k*Dd2nQ<GmIf=QN]G7brIl>'cH[[CAh,4@el96#2d]MRYR/LQu_15V1q6%[]VN*VC"l=2KdMBaXtT"hkH<d;27F0.f$FW>u8OQ.;^-LP9NG,*E**l0L2EOMJI^&$W[t&l3No*bP%)Ee/Bd0)s(<\hOtnL`,So$O@1KejKI"@@ft5hm5Wc_IpQb&$e<!l#,B)56s?R[llW?2PpBq?"T%%5cO)1B:5^YKa%@='oW368&PNNlOht]Sa@?;JFVnY$jDa'IfU%\>6P~>
+GatU4gMZ%0&:O:SkV5A?7\oI1U-QDoK+pbdD7o8RAfkQA[Y7>J<hatEo]&S:Z8n-uDi,Z.,1>IUc^Yl!^7>,6S:$`4CFaiU-]sC)bP-GTfCe4rbk3^]4iaS$[@7"JRJ-r&'Lne`=Rs4bZ2F(D;.s+_ef+SXcUY$;iOMPa(=/G(LIGTZ7sq]!9^L=c>Pqm046ejT_SfX?eu-+["mFg3O]Kib:Gei:-=DJ.S"(8,^:-:1b?mrW3Za16i8Uj`][\S'/5/)tF=RV_SqfBXQ92*gU4d,n[7N1D=^%k=<q;IiEAgg$E<0lg7<VQZH;&)a:*4Yd!!h%a9t/P\[BX<IO4+0#9k9WEmMbb,d64fjEmC9TR.224[QAWAUt)Qu!A*Cqkh<Pq\ZeJ*"-Z-6S5q)>J2@JTcV)j)^g).5eX:jU66"mi0-;Z),7)%'i'di:D8!=PHb!3>Ze5k(0gB[S?C8X.o9<T'C85Z"7+#/Hco[EQGfBCK9f@E'YUY_0kGc9W@!N+<o.Nr8(mg\sKU/u9dUI"ab4SSMp5/kVG","I%>Tb?MPY[)2>0'/8h[W5GkY&8`Z[gN[\JAQ];J6q?E,dBcm7*MM*fE/1qHZ^iUf'0I&.PsBE5[;\0g*6plN?^'`I\]Eo)XP-@j)Ta>Jjki@m4Qdf$lB/&8VM!:!NKAfV87#KT58=Cq`m`]W&Z/,+&7d$o&dDT>kBn"@?@q[G14ScTBbY-TIKM31#mcrF`Z\@H$!LV9?oneD>H`]!+-FjeR3L$gXNgXA\_po+t]1Y=$\m[iZDQIS@tErR.<467+Z]^+j0P>>?slFDa%+Cs;-Q/HUpH;44[BG__/Z)Rep!3Ve4/#[&-BHOJOd!-"g?..%0>'/u57E]u>Dlj[gU!hGlR;f\h=\l/s?5K(f)uM%UA;BE"!"gBFNhsYORb"C;Ipe=tLZ="&JiWIe<JbDPcp_e.W*$3')3&ELN^_6.Zh#"JDk1d(F<f^IE+$t3f0I&
@2`d7BTnFM<$#P@t62&^77qbG@7'ocX9O*?6H@=9P&[4QZ;TZFtObA>s"&r:-'PQi&isT,7\>j]HcLuA)=TrYQY`@s;6pD\kZ2iVY-aPp1J6YVp"3]fQ@IVeX'Gkj[>+<JG?)r?1&W_,WqKejpjh.JDT%4fT?V.:7km"mc0NrnB'A+Y77MIgg_RpQk,chb`(^$60VQkhEWQ-tO1HJEr+u0^+>*fM3=E&S-\T(^!?c;XPPGoNFNB20A*+80LRLH,tJTFO[PBY#3?r3FJmJ$lq$V%!B9Sgn5I#lo)0+J.H,O`k^!\q=i^[_/\,D$nXCDN.n*HuaZQT+j$[2F`:\Y7IU-NpjHCRV.`mOpaKGBFlbLgWa/eR*N#a'OEc**fZ3UMp>&G=)mU;.f>:f%!'Q+V5&b;nr6Jg5-L`cD73[/<\&o1iGX/]L<1ed[Uj6i>R^.ooX`I5#G[BHt3(na@mOZ^,2S'VYgQ.2KiMibHbTo%l$DfU3'glAsl2s`e9Fo7d5]7,X*+5'e39&c)8Pm#\J,'+]pV+gs`OM/S!Ln-:0lGQ,P9;gd-/fBgO1mLlQeZ;bB]4)m-#)Q4[t;JhM\T3#GJ+.KudD7mRX!gIL426*(X.<s91D#@S8<hsck>ib^OW!K$rNLg"WG[*Z9og?M9'=)DTtK3iDbE=Y$!e8gle>P@?rN,:sF=iHkO'k=#`Oc.)L2u,hReWo(nG`;$M!sED%_"GLpLY22pjEm3*e<->qhm;lQV;#:6Z63t_PB^33Li*$(e[Om_!CGD@T@nr@H:I.B(@mrqQIQO+emYCZ[(:%W(20]@WlL"<r<LA$+cH~>
endstream
endobj
98 0 obj
@@ -588,276 +588,291 @@
/Contents 97 0 R
>>
endobj
+99 0 obj
+<< /Length 433 /Filter [ /ASCII85Decode /FlateDecode ]
+ >>
+stream
+Gar'%_+qm%%#44r$6M3`R3XU`H=Vi++V#VE0#418ZF*T^N],YEgQQ6?P+jFk5K@s6ht)dhB&n_$f\5pb&rIC_34AiXo_+5"jNqXDOuG/BaDEDWEI3gb#^&_gILIicJn%J!aEI&53TejXG;iH)_H!$]XgaYN(`i*KRiq9$R_mOoU,JSqAV8X0>Qqr4IajtS`nOM(.h,84)mX*<"p]iZ5K.bsLiVQKJ7NlFG]F\]I\3eN=&;L_AB\KQE`hRPm$`oI;6TZ]aHMq?3$E.m?#I6I)7P0'9ID$Z&u?H+VoV<1roU4EqqniT$>mtJ7&]1nokV8^^W0;-2l":e:iRA0@#7P_9O\#PZ3mg/XbUM<DTcnU<ZkDGj;/WV';m\\PRV@f#h3qnhJCCbDZ`eh*9qZab,30$Xf=Ln\%@05Vq[tQUDNA0]niCe~>
+endstream
+endobj
100 0 obj
+<< /Type /Page
+/Parent 1 0 R
+/MediaBox [ 0 0 612 792 ]
+/Resources 3 0 R
+/Contents 99 0 R
+>>
+endobj
+102 0 obj
<<
/Title (\376\377\0\61\0\40\0\111\0\156\0\144\0\145\0\170\0\40\0\106\0\151\0\154\0\145\0\40\0\106\0\157\0\162\0\155\0\141\0\164\0\163)
- /Parent 99 0 R
- /Next 101 0 R
+ /Parent 101 0 R
+ /Next 103 0 R
/A 9 0 R
>> endobj
-101 0 obj
+103 0 obj
<<
/Title (\376\377\0\62\0\40\0\104\0\145\0\146\0\151\0\156\0\151\0\164\0\151\0\157\0\156\0\163)
- /Parent 99 0 R
- /First 102 0 R
- /Last 105 0 R
- /Prev 100 0 R
- /Next 106 0 R
+ /Parent 101 0 R
+ /First 104 0 R
+ /Last 107 0 R
+ /Prev 102 0 R
+ /Next 108 0 R
/Count -4
/A 11 0 R
>> endobj
-102 0 obj
+104 0 obj
<<
/Title (\376\377\0\62\0\56\0\61\0\40\0\111\0\156\0\166\0\145\0\162\0\164\0\145\0\144\0\40\0\111\0\156\0\144\0\145\0\170\0\151\0\156\0\147)
- /Parent 101 0 R
- /Next 103 0 R
+ /Parent 103 0 R
+ /Next 105 0 R
/A 13 0 R
>> endobj
-103 0 obj
+105 0 obj
<<
/Title (\376\377\0\62\0\56\0\62\0\40\0\124\0\171\0\160\0\145\0\163\0\40\0\157\0\146\0\40\0\106\0\151\0\145\0\154\0\144\0\163)
- /Parent 101 0 R
- /Prev 102 0 R
- /Next 104 0 R
+ /Parent 103 0 R
+ /Prev 104 0 R
+ /Next 106 0 R
/A 15 0 R
>> endobj
-104 0 obj
+106 0 obj
<<
/Title (\376\377\0\62\0\56\0\63\0\40\0\123\0\145\0\147\0\155\0\145\0\156\0\164\0\163)
- /Parent 101 0 R
- /Prev 103 0 R
- /Next 105 0 R
+ /Parent 103 0 R
+ /Prev 105 0 R
+ /Next 107 0 R
/A 17 0 R
>> endobj
-105 0 obj
+107 0 obj
<<
/Title (\376\377\0\62\0\56\0\64\0\40\0\104\0\157\0\143\0\165\0\155\0\145\0\156\0\164\0\40\0\116\0\165\0\155\0\142\0\145\0\162\0\163)
- /Parent 101 0 R
- /Prev 104 0 R
+ /Parent 103 0 R
+ /Prev 106 0 R
/A 19 0 R
>> endobj
-106 0 obj
+108 0 obj
<<
/Title (\376\377\0\63\0\40\0\117\0\166\0\145\0\162\0\166\0\151\0\145\0\167)
- /Parent 99 0 R
- /Prev 101 0 R
- /Next 107 0 R
+ /Parent 101 0 R
+ /Prev 103 0 R
+ /Next 109 0 R
/A 21 0 R
>> endobj
-107 0 obj
+109 0 obj
<<
/Title (\376\377\0\64\0\40\0\106\0\151\0\154\0\145\0\40\0\116\0\141\0\155\0\151\0\156\0\147)
- /Parent 99 0 R
- /Prev 106 0 R
- /Next 108 0 R
+ /Parent 101 0 R
+ /Prev 108 0 R
+ /Next 110 0 R
/A 23 0 R
>> endobj
-108 0 obj
+110 0 obj
<<
/Title (\376\377\0\65\0\40\0\120\0\162\0\151\0\155\0\151\0\164\0\151\0\166\0\145\0\40\0\124\0\171\0\160\0\145\0\163)
- /Parent 99 0 R
- /First 109 0 R
- /Last 114 0 R
- /Prev 107 0 R
- /Next 115 0 R
+ /Parent 101 0 R
+ /First 111 0 R
+ /Last 116 0 R
+ /Prev 109 0 R
+ /Next 117 0 R
/Count -6
/A 25 0 R
>> endobj
-109 0 obj
+111 0 obj
<<
/Title (\376\377\0\65\0\56\0\61\0\40\0\102\0\171\0\164\0\145)
- /Parent 108 0 R
- /Next 110 0 R
+ /Parent 110 0 R
+ /Next 112 0 R
/A 27 0 R
>> endobj
-110 0 obj
+112 0 obj
<<
/Title (\376\377\0\65\0\56\0\62\0\40\0\125\0\111\0\156\0\164\0\63\0\62)
- /Parent 108 0 R
- /Prev 109 0 R
- /Next 111 0 R
+ /Parent 110 0 R
+ /Prev 111 0 R
+ /Next 113 0 R
/A 29 0 R
>> endobj
-111 0 obj
+113 0 obj
<<
/Title (\376\377\0\65\0\56\0\63\0\40\0\125\0\151\0\156\0\164\0\66\0\64)
- /Parent 108 0 R
- /Prev 110 0 R
- /Next 112 0 R
+ /Parent 110 0 R
+ /Prev 112 0 R
+ /Next 114 0 R
/A 31 0 R
>> endobj
-112 0 obj
+114 0 obj
<<
/Title (\376\377\0\65\0\56\0\64\0\40\0\126\0\111\0\156\0\164)
- /Parent 108 0 R
- /Prev 111 0 R
- /Next 113 0 R
+ /Parent 110 0 R
+ /Prev 113 0 R
+ /Next 115 0 R
/A 33 0 R
>> endobj
-113 0 obj
+115 0 obj
<<
/Title (\376\377\0\65\0\56\0\65\0\40\0\103\0\150\0\141\0\162\0\163)
- /Parent 108 0 R
- /Prev 112 0 R
- /Next 114 0 R
+ /Parent 110 0 R
+ /Prev 114 0 R
+ /Next 116 0 R
/A 35 0 R
>> endobj
-114 0 obj
+116 0 obj
<<
/Title (\376\377\0\65\0\56\0\66\0\40\0\123\0\164\0\162\0\151\0\156\0\147)
- /Parent 108 0 R
- /Prev 113 0 R
+ /Parent 110 0 R
+ /Prev 115 0 R
/A 37 0 R
>> endobj
-115 0 obj
+117 0 obj
<<
/Title (\376\377\0\66\0\40\0\120\0\145\0\162\0\55\0\111\0\156\0\144\0\145\0\170\0\40\0\106\0\151\0\154\0\145\0\163)
- /Parent 99 0 R
- /First 116 0 R
- /Last 119 0 R
- /Prev 108 0 R
- /Next 120 0 R
+ /Parent 101 0 R
+ /First 118 0 R
+ /Last 121 0 R
+ /Prev 110 0 R
+ /Next 122 0 R
/Count -4
/A 39 0 R
>> endobj
-116 0 obj
+118 0 obj
<<
/Title (\376\377\0\66\0\56\0\61\0\40\0\123\0\145\0\147\0\155\0\145\0\156\0\164\0\163\0\40\0\106\0\151\0\154\0\145)
- /Parent 115 0 R
- /Next 117 0 R
+ /Parent 117 0 R
+ /Next 119 0 R
/A 41 0 R
>> endobj
-117 0 obj
+119 0 obj
<<
/Title (\376\377\0\66\0\56\0\62\0\40\0\114\0\157\0\143\0\153\0\40\0\106\0\151\0\154\0\145)
- /Parent 115 0 R
- /Prev 116 0 R
- /Next 118 0 R
+ /Parent 117 0 R
+ /Prev 118 0 R
+ /Next 120 0 R
/A 43 0 R
>> endobj
-118 0 obj
+120 0 obj
<<
/Title (\376\377\0\66\0\56\0\63\0\40\0\104\0\145\0\154\0\145\0\164\0\141\0\142\0\154\0\145\0\40\0\106\0\151\0\154\0\145)
- /Parent 115 0 R
- /Prev 117 0 R
- /Next 119 0 R
+ /Parent 117 0 R
+ /Prev 119 0 R
+ /Next 121 0 R
/A 45 0 R
>> endobj
-119 0 obj
+121 0 obj
<<
/Title (\376\377\0\66\0\56\0\64\0\40\0\103\0\157\0\155\0\160\0\157\0\165\0\156\0\144\0\40\0\106\0\151\0\154\0\145\0\163)
- /Parent 115 0 R
- /Prev 118 0 R
+ /Parent 117 0 R
+ /Prev 120 0 R
/A 47 0 R
>> endobj
-120 0 obj
+122 0 obj
<<
/Title (\376\377\0\67\0\40\0\120\0\145\0\162\0\55\0\123\0\145\0\147\0\155\0\145\0\156\0\164\0\40\0\106\0\151\0\154\0\145\0\163)
- /Parent 99 0 R
- /First 121 0 R
- /Last 127 0 R
- /Prev 115 0 R
- /Next 128 0 R
+ /Parent 101 0 R
+ /First 123 0 R
+ /Last 129 0 R
+ /Prev 117 0 R
+ /Next 130 0 R
/Count -7
/A 49 0 R
>> endobj
-121 0 obj
+123 0 obj
<<
/Title (\376\377\0\67\0\56\0\61\0\40\0\106\0\151\0\145\0\154\0\144\0\163)
- /Parent 120 0 R
- /Next 122 0 R
+ /Parent 122 0 R
+ /Next 124 0 R
/A 51 0 R
>> endobj
-122 0 obj
+124 0 obj
<<
/Title (\376\377\0\67\0\56\0\62\0\40\0\124\0\145\0\162\0\155\0\40\0\104\0\151\0\143\0\164\0\151\0\157\0\156\0\141\0\162\0\171)
- /Parent 120 0 R
- /Prev 121 0 R
- /Next 123 0 R
+ /Parent 122 0 R
+ /Prev 123 0 R
+ /Next 125 0 R
/A 56 0 R
>> endobj
-123 0 obj
+125 0 obj
<<
/Title (\376\377\0\67\0\56\0\63\0\40\0\106\0\162\0\145\0\161\0\165\0\145\0\156\0\143\0\151\0\145\0\163)
- /Parent 120 0 R
- /Prev 122 0 R
- /Next 124 0 R
+ /Parent 122 0 R
+ /Prev 124 0 R
+ /Next 126 0 R
/A 58 0 R
>> endobj
-124 0 obj
+126 0 obj
<<
/Title (\376\377\0\67\0\56\0\64\0\40\0\120\0\157\0\163\0\151\0\164\0\151\0\157\0\156\0\163)
- /Parent 120 0 R
- /Prev 123 0 R
- /Next 125 0 R
+ /Parent 122 0 R
+ /Prev 125 0 R
+ /Next 127 0 R
/A 60 0 R
>> endobj
-125 0 obj
+127 0 obj
<<
/Title (\376\377\0\67\0\56\0\65\0\40\0\116\0\157\0\162\0\155\0\141\0\154\0\151\0\172\0\141\0\164\0\151\0\157\0\156\0\40\0\106\0\141\0\143\0\164\0\157\0\162\0\163)
- /Parent 120 0 R
- /Prev 124 0 R
- /Next 126 0 R
+ /Parent 122 0 R
+ /Prev 126 0 R
+ /Next 128 0 R
/A 62 0 R
>> endobj
-126 0 obj
+128 0 obj
<<
/Title (\376\377\0\67\0\56\0\66\0\40\0\124\0\145\0\162\0\155\0\40\0\126\0\145\0\143\0\164\0\157\0\162\0\163)
- /Parent 120 0 R
- /Prev 125 0 R
- /Next 127 0 R
+ /Parent 122 0 R
+ /Prev 127 0 R
+ /Next 129 0 R
/A 64 0 R
>> endobj
-127 0 obj
+129 0 obj
<<
/Title (\376\377\0\67\0\56\0\67\0\40\0\104\0\145\0\154\0\145\0\164\0\145\0\144\0\40\0\104\0\157\0\143\0\165\0\155\0\145\0\156\0\164\0\163)
- /Parent 120 0 R
- /Prev 126 0 R
+ /Parent 122 0 R
+ /Prev 128 0 R
/A 66 0 R
>> endobj
-128 0 obj
+130 0 obj
<<
/Title (\376\377\0\70\0\40\0\114\0\151\0\155\0\151\0\164\0\141\0\164\0\151\0\157\0\156\0\163)
- /Parent 99 0 R
- /Prev 120 0 R
+ /Parent 101 0 R
+ /Prev 122 0 R
/A 68 0 R
>> endobj
-129 0 obj
+131 0 obj
<< /Type /Font
/Subtype /Type1
/Name /F3
/BaseFont /Helvetica-Bold
/Encoding /WinAnsiEncoding >>
endobj
-130 0 obj
+132 0 obj
<< /Type /Font
/Subtype /Type1
/Name /F5
/BaseFont /Times-Roman
/Encoding /WinAnsiEncoding >>
endobj
-131 0 obj
+133 0 obj
<< /Type /Font
/Subtype /Type1
/Name /F1
/BaseFont /Helvetica
/Encoding /WinAnsiEncoding >>
endobj
-132 0 obj
+134 0 obj
<< /Type /Font
/Subtype /Type1
/Name /F9
/BaseFont /Courier
/Encoding /WinAnsiEncoding >>
endobj
-133 0 obj
+135 0 obj
<< /Type /Font
/Subtype /Type1
/Name /F2
/BaseFont /Helvetica-Oblique
/Encoding /WinAnsiEncoding >>
endobj
-134 0 obj
+136 0 obj
<< /Type /Font
/Subtype /Type1
/Name /F7
@@ -866,19 +881,19 @@
endobj
1 0 obj
<< /Type /Pages
-/Count 17
-/Kids [6 0 R 53 0 R 70 0 R 72 0 R 74 0 R 76 0 R 78 0 R 80 0 R 82 0 R 84 0 R 86 0 R 88 0 R 90 0 R 92 0 R 94 0 R 96 0 R 98 0 R ] >>
+/Count 18
+/Kids [6 0 R 53 0 R 70 0 R 72 0 R 74 0 R 76 0 R 78 0 R 80 0 R 82 0 R 84 0 R 86 0 R 88 0 R 90 0 R 92 0 R 94 0 R 96 0 R 98 0 R 100 0 R ] >>
endobj
2 0 obj
<< /Type /Catalog
/Pages 1 0 R
- /Outlines 99 0 R
+ /Outlines 101 0 R
/PageMode /UseOutlines
>>
endobj
3 0 obj
<<
-/Font << /F3 129 0 R /F5 130 0 R /F1 131 0 R /F9 132 0 R /F2 133 0 R /F7 134 0 R >>
+/Font << /F3 131 0 R /F5 132 0 R /F1 133 0 R /F9 134 0 R /F2 135 0 R /F7 136 0 R >>
/ProcSet [ /PDF /ImageC /Text ] >>
endobj
9 0 obj
@@ -1046,163 +1061,165 @@
66 0 obj
<<
/S /GoTo
-/D [96 0 R /XYZ 85.0 257.8 null]
+/D [98 0 R /XYZ 85.0 659.0 null]
>>
endobj
68 0 obj
<<
/S /GoTo
-/D [98 0 R /XYZ 85.0 325.4 null]
+/D [98 0 R /XYZ 85.0 197.347 null]
>>
endobj
-99 0 obj
+101 0 obj
<<
- /First 100 0 R
- /Last 128 0 R
+ /First 102 0 R
+ /Last 130 0 R
>> endobj
xref
-0 135
+0 137
0000000000 65535 f
-0000043036 00000 n
-0000043207 00000 n
-0000043299 00000 n
+0000043842 00000 n
+0000044021 00000 n
+0000044114 00000 n
0000000015 00000 n
0000000071 00000 n
0000001279 00000 n
0000001399 00000 n
0000001571 00000 n
-0000043439 00000 n
+0000044254 00000 n
0000001706 00000 n
-0000043502 00000 n
+0000044317 00000 n
0000001841 00000 n
-0000043568 00000 n
+0000044383 00000 n
0000001978 00000 n
-0000043634 00000 n
+0000044449 00000 n
0000002115 00000 n
-0000043698 00000 n
+0000044513 00000 n
0000002252 00000 n
-0000043764 00000 n
+0000044579 00000 n
0000002389 00000 n
-0000043830 00000 n
+0000044645 00000 n
0000002526 00000 n
-0000043894 00000 n
+0000044709 00000 n
0000002663 00000 n
-0000043960 00000 n
+0000044775 00000 n
0000002800 00000 n
-0000044024 00000 n
+0000044839 00000 n
0000002937 00000 n
-0000044090 00000 n
+0000044905 00000 n
0000003074 00000 n
-0000044156 00000 n
+0000044971 00000 n
0000003211 00000 n
-0000044221 00000 n
+0000045036 00000 n
0000003348 00000 n
-0000044287 00000 n
+0000045102 00000 n
0000003485 00000 n
-0000044351 00000 n
+0000045166 00000 n
0000003621 00000 n
-0000044417 00000 n
+0000045232 00000 n
0000003758 00000 n
-0000044483 00000 n
+0000045298 00000 n
0000003894 00000 n
-0000044547 00000 n
+0000045362 00000 n
0000004031 00000 n
-0000044611 00000 n
+0000045426 00000 n
0000004168 00000 n
-0000044677 00000 n
+0000045492 00000 n
0000004304 00000 n
-0000044743 00000 n
+0000045558 00000 n
0000004441 00000 n
-0000044807 00000 n
+0000045622 00000 n
0000004577 00000 n
-0000005296 00000 n
-0000005419 00000 n
-0000005488 00000 n
-0000044873 00000 n
-0000005621 00000 n
-0000044937 00000 n
-0000005754 00000 n
-0000045001 00000 n
-0000005887 00000 n
-0000045065 00000 n
-0000006020 00000 n
-0000045131 00000 n
-0000006153 00000 n
-0000045195 00000 n
-0000006285 00000 n
-0000045259 00000 n
-0000006418 00000 n
-0000008552 00000 n
-0000008660 00000 n
-0000010890 00000 n
-0000010998 00000 n
-0000013346 00000 n
-0000013454 00000 n
-0000015660 00000 n
-0000015768 00000 n
-0000018211 00000 n
-0000018319 00000 n
-0000020364 00000 n
-0000020472 00000 n
-0000022596 00000 n
-0000022704 00000 n
-0000024343 00000 n
-0000024451 00000 n
-0000025797 00000 n
-0000025905 00000 n
-0000027743 00000 n
-0000027851 00000 n
-0000029537 00000 n
-0000029645 00000 n
-0000031424 00000 n
-0000031532 00000 n
-0000033335 00000 n
-0000033443 00000 n
-0000035079 00000 n
-0000035187 00000 n
-0000036871 00000 n
-0000045323 00000 n
-0000036979 00000 n
-0000037178 00000 n
-0000037395 00000 n
-0000037601 00000 n
-0000037809 00000 n
+0000005295 00000 n
+0000005418 00000 n
+0000005487 00000 n
+0000045688 00000 n
+0000005620 00000 n
+0000045752 00000 n
+0000005753 00000 n
+0000045816 00000 n
+0000005886 00000 n
+0000045880 00000 n
+0000006019 00000 n
+0000045946 00000 n
+0000006152 00000 n
+0000046010 00000 n
+0000006284 00000 n
+0000046074 00000 n
+0000006417 00000 n
+0000008551 00000 n
+0000008659 00000 n
+0000010889 00000 n
+0000010997 00000 n
+0000013345 00000 n
+0000013453 00000 n
+0000015659 00000 n
+0000015767 00000 n
+0000018210 00000 n
+0000018318 00000 n
+0000020363 00000 n
+0000020471 00000 n
+0000022595 00000 n
+0000022703 00000 n
+0000024342 00000 n
+0000024450 00000 n
+0000025796 00000 n
+0000025904 00000 n
+0000027742 00000 n
+0000027850 00000 n
+0000029536 00000 n
+0000029644 00000 n
+0000031423 00000 n
+0000031531 00000 n
+0000033334 00000 n
+0000033442 00000 n
+0000035018 00000 n
+0000035126 00000 n
+0000037035 00000 n
+0000037143 00000 n
+0000037668 00000 n
+0000046140 00000 n
+0000037777 00000 n
0000037977 00000 n
-0000038177 00000 n
-0000038334 00000 n
-0000038508 00000 n
-0000038748 00000 n
-0000038877 00000 n
-0000039031 00000 n
-0000039185 00000 n
-0000039329 00000 n
-0000039479 00000 n
-0000039620 00000 n
-0000039859 00000 n
-0000040041 00000 n
-0000040214 00000 n
-0000040417 00000 n
-0000040605 00000 n
-0000040856 00000 n
-0000040997 00000 n
-0000041206 00000 n
-0000041392 00000 n
-0000041566 00000 n
-0000041811 00000 n
-0000042002 00000 n
-0000042208 00000 n
-0000042368 00000 n
-0000042482 00000 n
-0000042593 00000 n
-0000042702 00000 n
-0000042809 00000 n
-0000042926 00000 n
+0000038195 00000 n
+0000038401 00000 n
+0000038609 00000 n
+0000038777 00000 n
+0000038977 00000 n
+0000039135 00000 n
+0000039310 00000 n
+0000039551 00000 n
+0000039680 00000 n
+0000039834 00000 n
+0000039988 00000 n
+0000040132 00000 n
+0000040282 00000 n
+0000040423 00000 n
+0000040663 00000 n
+0000040845 00000 n
+0000041018 00000 n
+0000041221 00000 n
+0000041409 00000 n
+0000041661 00000 n
+0000041802 00000 n
+0000042011 00000 n
+0000042197 00000 n
+0000042371 00000 n
+0000042616 00000 n
+0000042807 00000 n
+0000043013 00000 n
+0000043174 00000 n
+0000043288 00000 n
+0000043399 00000 n
+0000043508 00000 n
+0000043615 00000 n
+0000043732 00000 n
trailer
<<
-/Size 135
+/Size 137
/Root 2 0 R
/Info 4 0 R
>>
startxref
-45376
+46194
%%EOF
Modified: lucene/java/trunk/src/site/src/documentation/content/xdocs/fileformats.xml
URL: http://svn.apache.org/viewvc/lucene/java/trunk/src/site/src/documentation/content/xdocs/fileformats.xml?view=diff&rev=487874&r1=487873&r2=487874
==============================================================================
--- lucene/java/trunk/src/site/src/documentation/content/xdocs/fileformats.xml (original)
+++ lucene/java/trunk/src/site/src/documentation/content/xdocs/fileformats.xml Sat Dec 16 12:23:30 2006
@@ -1484,27 +1484,38 @@
<li>
<p>The Field or .tvf file.</p>
<p>This file contains, for each field that has a term vector stored, a list of
- the terms and their frequencies.</p>
- <p>Field (.tvf) --> TVFVersion<NumTerms, NumDistinct, TermFreqs>
+ the terms, their frequencies and, optionally, position and offest information.</p>
+ <p>Field (.tvf) --> TVFVersion<NumTerms, Position/Offset, TermFreqs>
<sup>NumFields</sup>
</p>
<p>TVFVersion --> Int</p>
<p>NumTerms --> VInt</p>
- <p>NumDistinct --> VInt -- Future Use</p>
- <p>TermFreqs --> <TermText, TermFreq>
+ <p>Position/Offset --> Byte</p>
+ <p>TermFreqs --> <TermText, TermFreq, Positions?, Offsets?>
<sup>NumTerms</sup>
</p>
<p>TermText --> <PrefixLength, Suffix></p>
<p>PrefixLength --> VInt</p>
<p>Suffix --> String</p>
<p>TermFreq --> VInt</p>
- <p>Term
- text prefixes are shared. The PrefixLength is the number of initial
- characters from the previous term which must be pre-pended to a
- term's suffix in order to form the term's text. Thus, if the
- previous term's text was "bone" and the term is "boy",
- the PrefixLength is two and the suffix is "y".
- </p>
+ <p>Positions --> <VInt><sup>TermFreq</sup></p>
+ <p>Offsets --> <VInt, VInt><sup>TermFreq</sup></p>
+ <br/>
+ <p>Notes:</p>
+ <ul>
+ <li>Position/Offset byte stores whether this term vector has position or offset information stored.</li>
+ <li>Term
+ text prefixes are shared. The PrefixLength is the number of initial
+ characters from the previous term which must be pre-pended to a
+ term's suffix in order to form the term's text. Thus, if the
+ previous term's text was "bone" and the term is "boy",
+ the PrefixLength is two and the suffix is "y".
+ </li>
+ <li>Positions are stored as delta encoded VInts. This means we only store the difference of the current position from the last position</li>
+ <li>Offsets are stored as delta encoded VInts. The first VInt is the startOffset, the second is the endOffset.</li>
+ </ul>
+
+
</li>
</ol>
</section>