You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hive.apache.org by pr...@apache.org on 2015/10/23 01:30:40 UTC
[1/2] hive git commit: HIVE-11807: Set ORC buffer size in relation to
set stripe size (Owen O'Malley reviewed by Gopal V)
Repository: hive
Updated Branches:
refs/heads/branch-1 30ca4fbc7 -> 51a0c03f4
http://git-wip-us.apache.org/repos/asf/hive/blob/51a0c03f/ql/src/test/resources/orc-file-dump-dictionary-threshold.out
----------------------------------------------------------------------
diff --git a/ql/src/test/resources/orc-file-dump-dictionary-threshold.out b/ql/src/test/resources/orc-file-dump-dictionary-threshold.out
index 13e316e..2b30be3 100644
--- a/ql/src/test/resources/orc-file-dump-dictionary-threshold.out
+++ b/ql/src/test/resources/orc-file-dump-dictionary-threshold.out
@@ -2,7 +2,7 @@ Structure for TestFileDump.testDump.orc
File Version: 0.12 with HIVE_8732
Rows: 21000
Compression: ZLIB
-Compression size: 10000
+Compression size: 4096
Type: struct<i:int,l:bigint,s:string>
Stripe Statistics:
@@ -39,15 +39,15 @@ File Statistics:
Column 3: count: 21000 hasNull: false min: Darkness,-230 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974-15004-15124-15252-15294-15356-15530-15610-16316-16936-17024-17122-17214-17310-17528-17682-17742-17870-17878-18010-18410-18524-18788-19204-19254-19518-19596-19786-19874-19904-20390-20752-20936 sum: 6910238
Stripes:
- Stripe: offset: 3 data: 151108 rows: 5000 tail: 68 index: 718
+ Stripe: offset: 3 data: 163602 rows: 5000 tail: 68 index: 720
Stream: column 0 section ROW_INDEX start: 3 length 17
- Stream: column 1 section ROW_INDEX start: 20 length 162
- Stream: column 2 section ROW_INDEX start: 182 length 174
- Stream: column 3 section ROW_INDEX start: 356 length 365
- Stream: column 1 section DATA start: 721 length 20029
- Stream: column 2 section DATA start: 20750 length 40035
- Stream: column 3 section DATA start: 60785 length 86757
- Stream: column 3 section LENGTH start: 147542 length 4287
+ Stream: column 1 section ROW_INDEX start: 20 length 166
+ Stream: column 2 section ROW_INDEX start: 186 length 171
+ Stream: column 3 section ROW_INDEX start: 357 length 366
+ Stream: column 1 section DATA start: 723 length 20035
+ Stream: column 2 section DATA start: 20758 length 40050
+ Stream: column 3 section DATA start: 60808 length 99226
+ Stream: column 3 section LENGTH start: 160034 length 4291
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DIRECT_V2
@@ -55,30 +55,30 @@ Stripes:
Row group indices for column 1:
Entry 0: count: 1000 hasNull: false min: -2132329551 max: 2145911404 sum: 61941331718 positions: 0,0,0
Entry 1: count: 1000 hasNull: false min: -2138433136 max: 2145210552 sum: 14574030042 positions: 0,2050,488
- Entry 2: count: 1000 hasNull: false min: -2147115959 max: 2137805337 sum: -2032493169 positions: 0,6150,464
- Entry 3: count: 1000 hasNull: false min: -2137828953 max: 2145877119 sum: -3167202608 positions: 10003,250,440
- Entry 4: count: 1000 hasNull: false min: -2146452517 max: 2142394906 sum: 88361503212 positions: 10003,4350,416
+ Entry 2: count: 1000 hasNull: false min: -2147115959 max: 2137805337 sum: -2032493169 positions: 4099,2054,464
+ Entry 3: count: 1000 hasNull: false min: -2137828953 max: 2145877119 sum: -3167202608 positions: 8198,2058,440
+ Entry 4: count: 1000 hasNull: false min: -2146452517 max: 2142394906 sum: 88361503212 positions: 12297,2062,416
Row group indices for column 2:
Entry 0: count: 1000 hasNull: false min: -9206837518492372266 max: 9169230975203934579 positions: 0,0,0
- Entry 1: count: 1000 hasNull: false min: -9188878639954124284 max: 9213664245516510068 positions: 0,4098,488
- Entry 2: count: 1000 hasNull: false min: -9211329013123260308 max: 9217851628057711416 positions: 10003,2294,464
- Entry 3: count: 1000 hasNull: false min: -9185745718227889962 max: 9181722705210917931 positions: 20006,490,440
- Entry 4: count: 1000 hasNull: false min: -9216505819108477308 max: 9196474183833079923 positions: 20006,8686,416
+ Entry 1: count: 1000 hasNull: false min: -9188878639954124284 max: 9213664245516510068 positions: 4099,2,488
+ Entry 2: count: 1000 hasNull: false min: -9211329013123260308 max: 9217851628057711416 positions: 12297,6,464
+ Entry 3: count: 1000 hasNull: false min: -9185745718227889962 max: 9181722705210917931 positions: 20495,10,440
+ Entry 4: count: 1000 hasNull: false min: -9216505819108477308 max: 9196474183833079923 positions: 28693,14,416
Row group indices for column 3:
Entry 0: count: 1000 hasNull: false min: Darkness,-230 max: worst-54-290-346-648-908-996 sum: 18442 positions: 0,0,0,0,0
- Entry 1: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966 sum: 46338 positions: 2777,8442,0,695,18
- Entry 2: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660 sum: 75448 positions: 13595,4780,0,1554,14
- Entry 3: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788 sum: 104868 positions: 31432,228,0,2372,90
- Entry 4: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744 sum: 136158 positions: 54111,5096,0,3354,108
- Stripe: offset: 151897 data: 336358 rows: 5000 tail: 69 index: 954
- Stream: column 0 section ROW_INDEX start: 151897 length 17
- Stream: column 1 section ROW_INDEX start: 151914 length 153
- Stream: column 2 section ROW_INDEX start: 152067 length 169
- Stream: column 3 section ROW_INDEX start: 152236 length 615
- Stream: column 1 section DATA start: 152851 length 20029
- Stream: column 2 section DATA start: 172880 length 40035
- Stream: column 3 section DATA start: 212915 length 270789
- Stream: column 3 section LENGTH start: 483704 length 5505
+ Entry 1: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966 sum: 46338 positions: 4767,2058,0,695,18
+ Entry 2: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660 sum: 75448 positions: 16464,3340,0,1554,14
+ Entry 3: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788 sum: 104868 positions: 36532,964,0,2372,90
+ Entry 4: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744 sum: 136158 positions: 63067,3432,0,3354,108
+ Stripe: offset: 164393 data: 368335 rows: 5000 tail: 69 index: 956
+ Stream: column 0 section ROW_INDEX start: 164393 length 17
+ Stream: column 1 section ROW_INDEX start: 164410 length 157
+ Stream: column 2 section ROW_INDEX start: 164567 length 166
+ Stream: column 3 section ROW_INDEX start: 164733 length 616
+ Stream: column 1 section DATA start: 165349 length 20035
+ Stream: column 2 section DATA start: 185384 length 40050
+ Stream: column 3 section DATA start: 225434 length 302715
+ Stream: column 3 section LENGTH start: 528149 length 5535
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DIRECT_V2
@@ -86,30 +86,30 @@ Stripes:
Row group indices for column 1:
Entry 0: count: 1000 hasNull: false min: -2146021688 max: 2146838901 sum: -50979197646 positions: 0,0,0
Entry 1: count: 1000 hasNull: false min: -2143569489 max: 2141223179 sum: 22810066834 positions: 0,2050,488
- Entry 2: count: 1000 hasNull: false min: -2140649392 max: 2146301701 sum: -31694882346 positions: 0,6150,464
- Entry 3: count: 1000 hasNull: false min: -2147390285 max: 2146299933 sum: 79371934221 positions: 10003,250,440
- Entry 4: count: 1000 hasNull: false min: -2145928262 max: 2147224606 sum: -34469378822 positions: 10003,4350,416
+ Entry 2: count: 1000 hasNull: false min: -2140649392 max: 2146301701 sum: -31694882346 positions: 4099,2054,464
+ Entry 3: count: 1000 hasNull: false min: -2147390285 max: 2146299933 sum: 79371934221 positions: 8198,2058,440
+ Entry 4: count: 1000 hasNull: false min: -2145928262 max: 2147224606 sum: -34469378822 positions: 12297,2062,416
Row group indices for column 2:
Entry 0: count: 1000 hasNull: false min: -9222178666167296739 max: 9191250610515369723 positions: 0,0,0
- Entry 1: count: 1000 hasNull: false min: -9220148577547102875 max: 9213945522531717278 positions: 0,4098,488
- Entry 2: count: 1000 hasNull: false min: -9220818777591257749 max: 9221301751385928177 positions: 10003,2294,464
- Entry 3: count: 1000 hasNull: false min: -9220031433030423388 max: 9207856144487414148 positions: 20006,490,440
- Entry 4: count: 1000 hasNull: false min: -9201438531577205959 max: 9212462124593119846 positions: 20006,8686,416
+ Entry 1: count: 1000 hasNull: false min: -9220148577547102875 max: 9213945522531717278 positions: 4099,2,488
+ Entry 2: count: 1000 hasNull: false min: -9220818777591257749 max: 9221301751385928177 positions: 12297,6,464
+ Entry 3: count: 1000 hasNull: false min: -9220031433030423388 max: 9207856144487414148 positions: 20495,10,440
+ Entry 4: count: 1000 hasNull: false min: -9201438531577205959 max: 9212462124593119846 positions: 28693,14,416
Row group indices for column 3:
Entry 0: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726 sum: 166320 positions: 0,0,0,0,0
- Entry 1: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994 sum: 193436 positions: 37112,6320,0,967,90
- Entry 2: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988 sum: 224740 positions: 80822,9756,0,1945,222
- Entry 3: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984 sum: 252094 positions: 137149,4496,0,3268,48
- Entry 4: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938 sum: 281404 positions: 197972,6590,0,4064,342
- Stripe: offset: 489278 data: 558031 rows: 5000 tail: 69 index: 1181
- Stream: column 0 section ROW_INDEX start: 489278 length 17
- Stream: column 1 section ROW_INDEX start: 489295 length 166
- Stream: column 2 section ROW_INDEX start: 489461 length 171
- Stream: column 3 section ROW_INDEX start: 489632 length 827
- Stream: column 1 section DATA start: 490459 length 20029
- Stream: column 2 section DATA start: 510488 length 40035
- Stream: column 3 section DATA start: 550523 length 492258
- Stream: column 3 section LENGTH start: 1042781 length 5709
+ Entry 1: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994 sum: 193436 positions: 43833,2480,0,967,90
+ Entry 2: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988 sum: 224740 positions: 94117,3404,0,1945,222
+ Entry 3: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984 sum: 252094 positions: 155111,2864,0,3268,48
+ Entry 4: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938 sum: 281404 positions: 224570,1006,0,4064,342
+ Stripe: offset: 533753 data: 606074 rows: 5000 tail: 69 index: 1427
+ Stream: column 0 section ROW_INDEX start: 533753 length 17
+ Stream: column 1 section ROW_INDEX start: 533770 length 167
+ Stream: column 2 section ROW_INDEX start: 533937 length 168
+ Stream: column 3 section ROW_INDEX start: 534105 length 1075
+ Stream: column 1 section DATA start: 535180 length 20035
+ Stream: column 2 section DATA start: 555215 length 40050
+ Stream: column 3 section DATA start: 595265 length 540210
+ Stream: column 3 section LENGTH start: 1135475 length 5779
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DIRECT_V2
@@ -117,30 +117,30 @@ Stripes:
Row group indices for column 1:
Entry 0: count: 1000 hasNull: false min: -2138229212 max: 2144818981 sum: -22823642812 positions: 0,0,0
Entry 1: count: 1000 hasNull: false min: -2145842720 max: 2144179881 sum: -12562754334 positions: 0,2050,488
- Entry 2: count: 1000 hasNull: false min: -2143045885 max: 2146718321 sum: 82993638644 positions: 0,6150,464
- Entry 3: count: 1000 hasNull: false min: -2144745617 max: 2146570474 sum: 25138722367 positions: 10003,250,440
- Entry 4: count: 1000 hasNull: false min: -2140127150 max: 2135081620 sum: 68346511655 positions: 10003,4350,416
+ Entry 2: count: 1000 hasNull: false min: -2143045885 max: 2146718321 sum: 82993638644 positions: 4099,2054,464
+ Entry 3: count: 1000 hasNull: false min: -2144745617 max: 2146570474 sum: 25138722367 positions: 8198,2058,440
+ Entry 4: count: 1000 hasNull: false min: -2140127150 max: 2135081620 sum: 68346511655 positions: 12297,2062,416
Row group indices for column 2:
Entry 0: count: 1000 hasNull: false min: -9204340807292138409 max: 9208698732685326961 positions: 0,0,0
- Entry 1: count: 1000 hasNull: false min: -9221963099397084326 max: 9222722740629726770 positions: 0,4098,488
- Entry 2: count: 1000 hasNull: false min: -9210480084701091299 max: 9207767402467343058 positions: 10003,2294,464
- Entry 3: count: 1000 hasNull: false min: -9195038026813631215 max: 9199201928563274421 positions: 20006,490,440
- Entry 4: count: 1000 hasNull: false min: -9215483580266514322 max: 9220102792864959501 positions: 20006,8686,416
+ Entry 1: count: 1000 hasNull: false min: -9221963099397084326 max: 9222722740629726770 positions: 4099,2,488
+ Entry 2: count: 1000 hasNull: false min: -9210480084701091299 max: 9207767402467343058 positions: 12297,6,464
+ Entry 3: count: 1000 hasNull: false min: -9195038026813631215 max: 9199201928563274421 positions: 20495,10,440
+ Entry 4: count: 1000 hasNull: false min: -9215483580266514322 max: 9220102792864959501 positions: 28693,14,416
Row group indices for column 3:
Entry 0: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876 sum: 313880 positions: 0,0,0,0,0
- Entry 1: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964 sum: 349542 positions: 80352,3880,0,1097,28
- Entry 2: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976 sum: 386538 posit
ions: 170641,3422,0,2077,162
- Entry 3: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802
-12976-13216-13246-13502-13766 sum: 421660 positions: 268420,9960,0,3369,16
- Entry 4: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298
-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974 sum: 453606 positions: 377916,1620,0,4041,470
- Stripe: offset: 1048559 data: 792850 rows: 5000 tail: 69 index: 1369
- Stream: column 0 section ROW_INDEX start: 1048559 length 17
- Stream: column 1 section ROW_INDEX start: 1048576 length 153
- Stream: column 2 section ROW_INDEX start: 1048729 length 172
- Stream: column 3 section ROW_INDEX start: 1048901 length 1027
- Stream: column 1 section DATA start: 1049928 length 20029
- Stream: column 2 section DATA start: 1069957 length 40035
- Stream: column 3 section DATA start: 1109992 length 727038
- Stream: column 3 section LENGTH start: 1837030 length 5748
+ Entry 1: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964 sum: 349542 positions: 87800,2584,0,1097,28
+ Entry 2: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976 sum: 386538 posit
ions: 185635,3966,0,2077,162
+ Entry 3: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802
-12976-13216-13246-13502-13766 sum: 421660 positions: 295550,1384,0,3369,16
+ Entry 4: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298
-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974 sum: 453606 positions: 412768,1156,0,4041,470
+ Stripe: offset: 1141323 data: 864001 rows: 5000 tail: 69 index: 1975
+ Stream: column 0 section ROW_INDEX start: 1141323 length 17
+ Stream: column 1 section ROW_INDEX start: 1141340 length 156
+ Stream: column 2 section ROW_INDEX start: 1141496 length 168
+ Stream: column 3 section ROW_INDEX start: 1141664 length 1634
+ Stream: column 1 section DATA start: 1143298 length 20035
+ Stream: column 2 section DATA start: 1163333 length 40050
+ Stream: column 3 section DATA start: 1203383 length 798014
+ Stream: column 3 section LENGTH start: 2001397 length 5902
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DIRECT_V2
@@ -148,30 +148,30 @@ Stripes:
Row group indices for column 1:
Entry 0: count: 1000 hasNull: false min: -2145319330 max: 2146998132 sum: -50856753363 positions: 0,0,0
Entry 1: count: 1000 hasNull: false min: -2134288866 max: 2147453086 sum: -17911019023 positions: 0,2050,488
- Entry 2: count: 1000 hasNull: false min: -2139010804 max: 2144727593 sum: -24993151857 positions: 0,6150,464
- Entry 3: count: 1000 hasNull: false min: -2145378214 max: 2144098933 sum: -18055164052 positions: 10003,250,440
- Entry 4: count: 1000 hasNull: false min: -2140494429 max: 2144595861 sum: -41863916235 positions: 10003,4350,416
+ Entry 2: count: 1000 hasNull: false min: -2139010804 max: 2144727593 sum: -24993151857 positions: 4099,2054,464
+ Entry 3: count: 1000 hasNull: false min: -2145378214 max: 2144098933 sum: -18055164052 positions: 8198,2058,440
+ Entry 4: count: 1000 hasNull: false min: -2140494429 max: 2144595861 sum: -41863916235 positions: 12297,2062,416
Row group indices for column 2:
Entry 0: count: 1000 hasNull: false min: -9172774601303513941 max: 9212917101275642143 positions: 0,0,0
- Entry 1: count: 1000 hasNull: false min: -9218164880949195469 max: 9222919052987871506 positions: 0,4098,488
- Entry 2: count: 1000 hasNull: false min: -9222731174895935707 max: 9214167447015056056 positions: 10003,2294,464
- Entry 3: count: 1000 hasNull: false min: -9196276654247395117 max: 9210639275226058005 positions: 20006,490,440
- Entry 4: count: 1000 hasNull: false min: -9197393848859294562 max: 9208134757538374043 positions: 20006,8686,416
+ Entry 1: count: 1000 hasNull: false min: -9218164880949195469 max: 9222919052987871506 positions: 4099,2,488
+ Entry 2: count: 1000 hasNull: false min: -9222731174895935707 max: 9214167447015056056 positions: 12297,6,464
+ Entry 3: count: 1000 hasNull: false min: -9196276654247395117 max: 9210639275226058005 positions: 20495,10,440
+ Entry 4: count: 1000 hasNull: false min: -9197393848859294562 max: 9208134757538374043 positions: 28693,14,416
Row group indices for column 3:
Entry 0: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188-14246-14340-14364-14394-14762-14850-14964-15048 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836
-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974-15004-15124-15252-15294-15356-15530-15610 sum: 492916 positions: 0,0,0,0,0
- Entry 1: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188-14246-14340-14364-14394-14762-14850-14964-15048-15494-15674-15726-16006 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008
-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974-15004-15124-15252-15294-15356-15530-15610-16316-16936 sum: 527290 positions: 126968,2916,0,1077,140
- Entry 2: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188-14246-14340-14364-14394-14762-14850-14964-15048-15494-15674-15726-16006-16056-16180-16304-16332-16452-16598-16730-16810-16994-17210 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9
650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974-15004-15124-15252-15294-15356-15530-15610-16316-16936-17024-17122-17214-17310-17528-17682-17742-17870-17878 sum: 568274 positions: 263111,206,0,1926,462
- Entry 3: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188-14246-14340-14364-14394-14762-14850-14964-15048-15494-15674-15726-16006-16056-16180-16304-16332-16452-16598-16730-16810-16994-17210-17268-17786-17962-18214 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-91
28-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974-15004-15124-15252-15294-15356-15530-15610-16316-16936-17024-17122-17214-17310-17528-17682-17742-17870-17878-18010-18410-18524-18788 sum: 594578 positions: 407371,8480,0,3444,250
- Entry 4: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188-14246-14340-14364-14394-14762-14850-14964-15048-15494-15674-15726-16006-16056-16180-16304-16332-16452-16598-16730-16810-16994-17210-17268-17786-17962-18214-18444-18446-18724-18912-18952-19164 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8
390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974-15004-15124-15252-15294-15356-15530-15610-16316-16936-17024-17122-17214-17310-17528-17682-17742-17870-17878-18010-18410-18524-18788-19204-19254-19518-19596-19786-19874-19904 sum: 631944 positions: 562094,3058,0,4643,292
- Stripe: offset: 1842847 data: 188033 rows: 1000 tail: 67 index: 841
- Stream: column 0 section ROW_INDEX start: 1842847 length 12
- Stream: column 1 section ROW_INDEX start: 1842859 length 38
- Stream: column 2 section ROW_INDEX start: 1842897 length 41
- Stream: column 3 section ROW_INDEX start: 1842938 length 750
- Stream: column 1 section DATA start: 1843688 length 4007
- Stream: column 2 section DATA start: 1847695 length 8007
- Stream: column 3 section DATA start: 1855702 length 174759
- Stream: column 3 section LENGTH start: 2030461 length 1260
+ Entry 1: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188-14246-14340-14364-14394-14762-14850-14964-15048-15494-15674-15726-16006 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008
-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974-15004-15124-15252-15294-15356-15530-15610-16316-16936 sum: 527290 positions: 139298,1396,0,1077,140
+ Entry 2: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188-14246-14340-14364-14394-14762-14850-14964-15048-15494-15674-15726-16006-16056-16180-16304-16332-16452-16598-16730-16810-16994-17210 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9
650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974-15004-15124-15252-15294-15356-15530-15610-16316-16936-17024-17122-17214-17310-17528-17682-17742-17870-17878 sum: 568274 positions: 286457,302,0,1926,462
+ Entry 3: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188-14246-14340-14364-14394-14762-14850-14964-15048-15494-15674-15726-16006-16056-16180-16304-16332-16452-16598-16730-16810-16994-17210-17268-17786-17962-18214 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-91
28-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974-15004-15124-15252-15294-15356-15530-15610-16316-16936-17024-17122-17214-17310-17528-17682-17742-17870-17878-18010-18410-18524-18788 sum: 594578 positions: 447943,3328,0,3444,250
+ Entry 4: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188-14246-14340-14364-14394-14762-14850-14964-15048-15494-15674-15726-16006-16056-16180-16304-16332-16452-16598-16730-16810-16994-17210-17268-17786-17962-18214-18444-18446-18724-18912-18952-19164 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8
390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974-15004-15124-15252-15294-15356-15530-15610-16316-16936-17024-17122-17214-17310-17528-17682-17742-17870-17878-18010-18410-18524-18788-19204-19254-19518-19596-19786-19874-19904 sum: 631944 positions: 616471,3986,3778,547,292
+ Stripe: offset: 2007368 data: 207295 rows: 1000 tail: 67 index: 841
+ Stream: column 0 section ROW_INDEX start: 2007368 length 12
+ Stream: column 1 section ROW_INDEX start: 2007380 length 38
+ Stream: column 2 section ROW_INDEX start: 2007418 length 41
+ Stream: column 3 section ROW_INDEX start: 2007459 length 750
+ Stream: column 1 section DATA start: 2008209 length 4007
+ Stream: column 2 section DATA start: 2012216 length 8010
+ Stream: column 3 section DATA start: 2020226 length 194018
+ Stream: column 3 section LENGTH start: 2214244 length 1260
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DIRECT_V2
@@ -183,6 +183,6 @@ Stripes:
Row group indices for column 3:
Entry 0: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188-14246-14340-14364-14394-14762-14850-14964-15048-15494-15674-15726-16006-16056-16180-16304-16332-16452-16598-16730-16810-16994-17210-17268-17786-17962-18214-18444-18446-18724-18912-18952-19164-19348-19400-19546-19776-19896-20084 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-
7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974-15004-15124-15252-15294-15356-15530-15610-16316-16936-17024-17122-17214-17310-17528-17682-17742-17870-17878-18010-18410-18524-18788-19204-19254-19518-19596-19786-19874-19904-20390-20752-20936 sum: 670762 positions: 0,0,0,0,0
-File length: 2033643 bytes
+File length: 2217685 bytes
Padding length: 0 bytes
Padding ratio: 0%
http://git-wip-us.apache.org/repos/asf/hive/blob/51a0c03f/ql/src/test/resources/orc-file-dump.json
----------------------------------------------------------------------
diff --git a/ql/src/test/resources/orc-file-dump.json b/ql/src/test/resources/orc-file-dump.json
index fe31d5e..99ea7da 100644
--- a/ql/src/test/resources/orc-file-dump.json
+++ b/ql/src/test/resources/orc-file-dump.json
@@ -4,7 +4,7 @@
"writerVersion": "HIVE_8732",
"numberOfRows": 21000,
"compression": "ZLIB",
- "compressionBufferSize": 10000,
+ "compressionBufferSize": 4096,
"schemaString": "struct<i:int,l:bigint,s:string>",
"schema": [
{
@@ -254,8 +254,8 @@
"stripeNumber": 1,
"stripeInformation": {
"offset": 3,
- "indexLength": 863,
- "dataLength": 63749,
+ "indexLength": 970,
+ "dataLength": 63770,
"footerLength": 90,
"rowCount": 5000
},
@@ -270,60 +270,60 @@
"columnId": 1,
"section": "ROW_INDEX",
"startOffset": 20,
- "length": 165
+ "length": 167
},
{
"columnId": 2,
"section": "ROW_INDEX",
- "startOffset": 185,
- "length": 174
+ "startOffset": 187,
+ "length": 171
},
{
"columnId": 3,
"section": "ROW_INDEX",
- "startOffset": 359,
+ "startOffset": 358,
"length": 103
},
{
"columnId": 3,
"section": "BLOOM_FILTER",
- "startOffset": 462,
- "length": 404
+ "startOffset": 461,
+ "length": 512
},
{
"columnId": 1,
"section": "DATA",
- "startOffset": 866,
- "length": 20029
+ "startOffset": 973,
+ "length": 20035
},
{
"columnId": 2,
"section": "DATA",
- "startOffset": 20895,
- "length": 40035
+ "startOffset": 21008,
+ "length": 40050
},
{
"columnId": 3,
"section": "PRESENT",
- "startOffset": 60930,
+ "startOffset": 61058,
"length": 17
},
{
"columnId": 3,
"section": "DATA",
- "startOffset": 60947,
+ "startOffset": 61075,
"length": 3510
},
{
"columnId": 3,
"section": "LENGTH",
- "startOffset": 64457,
+ "startOffset": 64585,
"length": 25
},
{
"columnId": 3,
"section": "DICTIONARY_DATA",
- "startOffset": 64482,
+ "startOffset": 64610,
"length": 133
}
],
@@ -494,77 +494,77 @@
{
"stripeNumber": 2,
"stripeInformation": {
- "offset": 64705,
- "indexLength": 854,
- "dataLength": 63742,
- "footerLength": 90,
+ "offset": 64833,
+ "indexLength": 961,
+ "dataLength": 63763,
+ "footerLength": 88,
"rowCount": 5000
},
"streams": [
{
"columnId": 0,
"section": "ROW_INDEX",
- "startOffset": 64705,
+ "startOffset": 64833,
"length": 17
},
{
"columnId": 1,
"section": "ROW_INDEX",
- "startOffset": 64722,
- "length": 164
+ "startOffset": 64850,
+ "length": 166
},
{
"columnId": 2,
"section": "ROW_INDEX",
- "startOffset": 64886,
- "length": 169
+ "startOffset": 65016,
+ "length": 166
},
{
"columnId": 3,
"section": "ROW_INDEX",
- "startOffset": 65055,
+ "startOffset": 65182,
"length": 100
},
{
"columnId": 3,
"section": "BLOOM_FILTER",
- "startOffset": 65155,
- "length": 404
+ "startOffset": 65282,
+ "length": 512
},
{
"columnId": 1,
"section": "DATA",
- "startOffset": 65559,
- "length": 20029
+ "startOffset": 65794,
+ "length": 20035
},
{
"columnId": 2,
"section": "DATA",
- "startOffset": 85588,
- "length": 40035
+ "startOffset": 85829,
+ "length": 40050
},
{
"columnId": 3,
"section": "PRESENT",
- "startOffset": 125623,
+ "startOffset": 125879,
"length": 17
},
{
"columnId": 3,
"section": "DATA",
- "startOffset": 125640,
+ "startOffset": 125896,
"length": 3503
},
{
"columnId": 3,
"section": "LENGTH",
- "startOffset": 129143,
+ "startOffset": 129399,
"length": 25
},
{
"columnId": 3,
"section": "DICTIONARY_DATA",
- "startOffset": 129168,
+ "startOffset": 129424,
"length": 133
}
],
@@ -735,77 +735,77 @@
{
"stripeNumber": 3,
"stripeInformation": {
- "offset": 129391,
- "indexLength": 853,
- "dataLength": 63749,
- "footerLength": 90,
+ "offset": 129645,
+ "indexLength": 962,
+ "dataLength": 63770,
+ "footerLength": 91,
"rowCount": 5000
},
"streams": [
{
"columnId": 0,
"section": "ROW_INDEX",
- "startOffset": 129391,
+ "startOffset": 129645,
"length": 17
},
{
"columnId": 1,
"section": "ROW_INDEX",
- "startOffset": 129408,
- "length": 160
+ "startOffset": 129662,
+ "length": 164
},
{
"columnId": 2,
"section": "ROW_INDEX",
- "startOffset": 129568,
- "length": 170
+ "startOffset": 129826,
+ "length": 167
},
{
"columnId": 3,
"section": "ROW_INDEX",
- "startOffset": 129738,
+ "startOffset": 129993,
"length": 102
},
{
"columnId": 3,
"section": "BLOOM_FILTER",
- "startOffset": 129840,
- "length": 404
+ "startOffset": 130095,
+ "length": 512
},
{
"columnId": 1,
"section": "DATA",
- "startOffset": 130244,
- "length": 20029
+ "startOffset": 130607,
+ "length": 20035
},
{
"columnId": 2,
"section": "DATA",
- "startOffset": 150273,
- "length": 40035
+ "startOffset": 150642,
+ "length": 40050
},
{
"columnId": 3,
"section": "PRESENT",
- "startOffset": 190308,
+ "startOffset": 190692,
"length": 17
},
{
"columnId": 3,
"section": "DATA",
- "startOffset": 190325,
+ "startOffset": 190709,
"length": 3510
},
{
"columnId": 3,
"section": "LENGTH",
- "startOffset": 193835,
+ "startOffset": 194219,
"length": 25
},
{
"columnId": 3,
"section": "DICTIONARY_DATA",
- "startOffset": 193860,
+ "startOffset": 194244,
"length": 133
}
],
@@ -976,77 +976,77 @@
{
"stripeNumber": 4,
"stripeInformation": {
- "offset": 194083,
- "indexLength": 866,
- "dataLength": 63735,
- "footerLength": 90,
+ "offset": 194468,
+ "indexLength": 973,
+ "dataLength": 63756,
+ "footerLength": 91,
"rowCount": 5000
},
"streams": [
{
"columnId": 0,
"section": "ROW_INDEX",
- "startOffset": 194083,
+ "startOffset": 194468,
"length": 17
},
{
"columnId": 1,
"section": "ROW_INDEX",
- "startOffset": 194100,
- "length": 164
+ "startOffset": 194485,
+ "length": 166
},
{
"columnId": 2,
"section": "ROW_INDEX",
- "startOffset": 194264,
- "length": 174
+ "startOffset": 194651,
+ "length": 171
},
{
"columnId": 3,
"section": "ROW_INDEX",
- "startOffset": 194438,
+ "startOffset": 194822,
"length": 107
},
{
"columnId": 3,
"section": "BLOOM_FILTER",
- "startOffset": 194545,
- "length": 404
+ "startOffset": 194929,
+ "length": 512
},
{
"columnId": 1,
"section": "DATA",
- "startOffset": 194949,
- "length": 20029
+ "startOffset": 195441,
+ "length": 20035
},
{
"columnId": 2,
"section": "DATA",
- "startOffset": 214978,
- "length": 40035
+ "startOffset": 215476,
+ "length": 40050
},
{
"columnId": 3,
"section": "PRESENT",
- "startOffset": 255013,
+ "startOffset": 255526,
"length": 17
},
{
"columnId": 3,
"section": "DATA",
- "startOffset": 255030,
+ "startOffset": 255543,
"length": 3496
},
{
"columnId": 3,
"section": "LENGTH",
- "startOffset": 258526,
+ "startOffset": 259039,
"length": 25
},
{
"columnId": 3,
"section": "DICTIONARY_DATA",
- "startOffset": 258551,
+ "startOffset": 259064,
"length": 133
}
],
@@ -1217,9 +1217,9 @@
{
"stripeNumber": 5,
"stripeInformation": {
- "offset": 258774,
+ "offset": 259288,
"indexLength": 433,
- "dataLength": 12940,
+ "dataLength": 12943,
"footerLength": 83,
"rowCount": 1000
},
@@ -1227,67 +1227,67 @@
{
"columnId": 0,
"section": "ROW_INDEX",
- "startOffset": 258774,
+ "startOffset": 259288,
"length": 12
},
{
"columnId": 1,
"section": "ROW_INDEX",
- "startOffset": 258786,
+ "startOffset": 259300,
"length": 38
},
{
"columnId": 2,
"section": "ROW_INDEX",
- "startOffset": 258824,
+ "startOffset": 259338,
"length": 41
},
{
"columnId": 3,
"section": "ROW_INDEX",
- "startOffset": 258865,
+ "startOffset": 259379,
"length": 41
},
{
"columnId": 3,
"section": "BLOOM_FILTER",
- "startOffset": 258906,
+ "startOffset": 259420,
"length": 301
},
{
"columnId": 1,
"section": "DATA",
- "startOffset": 259207,
+ "startOffset": 259721,
"length": 4007
},
{
"columnId": 2,
"section": "DATA",
- "startOffset": 263214,
- "length": 8007
+ "startOffset": 263728,
+ "length": 8010
},
{
"columnId": 3,
"section": "PRESENT",
- "startOffset": 271221,
+ "startOffset": 271738,
"length": 16
},
{
"columnId": 3,
"section": "DATA",
- "startOffset": 271237,
+ "startOffset": 271754,
"length": 752
},
{
"columnId": 3,
"section": "LENGTH",
- "startOffset": 271989,
+ "startOffset": 272506,
"length": 25
},
{
"columnId": 3,
"section": "DICTIONARY_DATA",
- "startOffset": 272014,
+ "startOffset": 272531,
"length": 133
}
],
@@ -1348,7 +1348,7 @@
}]
}
],
- "fileLength": 272779,
+ "fileLength": 273300,
"paddingLength": 0,
"paddingRatio": 0
}
http://git-wip-us.apache.org/repos/asf/hive/blob/51a0c03f/ql/src/test/resources/orc-file-dump.out
----------------------------------------------------------------------
diff --git a/ql/src/test/resources/orc-file-dump.out b/ql/src/test/resources/orc-file-dump.out
index 2f5962b..6457018 100644
--- a/ql/src/test/resources/orc-file-dump.out
+++ b/ql/src/test/resources/orc-file-dump.out
@@ -2,7 +2,7 @@ Structure for TestFileDump.testDump.orc
File Version: 0.12 with HIVE_8732
Rows: 21000
Compression: ZLIB
-Compression size: 10000
+Compression size: 4096
Type: struct<i:int,l:bigint,s:string>
Stripe Statistics:
@@ -39,16 +39,16 @@ File Statistics:
Column 3: count: 21000 hasNull: false min: Darkness, max: worst sum: 81761
Stripes:
- Stripe: offset: 3 data: 63765 rows: 5000 tail: 79 index: 441
+ Stripe: offset: 3 data: 63786 rows: 5000 tail: 79 index: 439
Stream: column 0 section ROW_INDEX start: 3 length 17
- Stream: column 1 section ROW_INDEX start: 20 length 164
- Stream: column 2 section ROW_INDEX start: 184 length 173
- Stream: column 3 section ROW_INDEX start: 357 length 87
- Stream: column 1 section DATA start: 444 length 20029
- Stream: column 2 section DATA start: 20473 length 40035
- Stream: column 3 section DATA start: 60508 length 3543
- Stream: column 3 section LENGTH start: 64051 length 25
- Stream: column 3 section DICTIONARY_DATA start: 64076 length 133
+ Stream: column 1 section ROW_INDEX start: 20 length 166
+ Stream: column 2 section ROW_INDEX start: 186 length 169
+ Stream: column 3 section ROW_INDEX start: 355 length 87
+ Stream: column 1 section DATA start: 442 length 20035
+ Stream: column 2 section DATA start: 20477 length 40050
+ Stream: column 3 section DATA start: 60527 length 3543
+ Stream: column 3 section LENGTH start: 64070 length 25
+ Stream: column 3 section DICTIONARY_DATA start: 64095 length 133
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DIRECT_V2
@@ -56,31 +56,31 @@ Stripes:
Row group indices for column 1:
Entry 0: count: 1000 hasNull: false min: -2145365268 max: 2135491313 sum: 7521792925 positions: 0,0,0
Entry 1: count: 1000 hasNull: false min: -2139452528 max: 2147223299 sum: -12923774313 positions: 0,2050,488
- Entry 2: count: 1000 hasNull: false min: -2142420586 max: 2143898386 sum: -25521983511 positions: 0,6150,464
- Entry 3: count: 1000 hasNull: false min: -2137233441 max: 2144267163 sum: 40993386199 positions: 10003,250,440
- Entry 4: count: 1000 hasNull: false min: -2146021688 max: 2146838901 sum: -9553628474 positions: 10003,4350,416
+ Entry 2: count: 1000 hasNull: false min: -2142420586 max: 2143898386 sum: -25521983511 positions: 4099,2054,464
+ Entry 3: count: 1000 hasNull: false min: -2137233441 max: 2144267163 sum: 40993386199 positions: 8198,2058,440
+ Entry 4: count: 1000 hasNull: false min: -2146021688 max: 2146838901 sum: -9553628474 positions: 12297,2062,416
Row group indices for column 2:
Entry 0: count: 1000 hasNull: false min: -9200577545527640566 max: 9175500305011173751 positions: 0,0,0
- Entry 1: count: 1000 hasNull: false min: -9203618157670445774 max: 9208123824411178101 positions: 0,4098,488
- Entry 2: count: 1000 hasNull: false min: -9218592812243954469 max: 9221351515892923972 positions: 10003,2294,464
- Entry 3: count: 1000 hasNull: false min: -9206585617947511272 max: 9167703224425685487 positions: 20006,490,440
- Entry 4: count: 1000 hasNull: false min: -9206645795733282496 max: 9221614132680747961 positions: 20006,8686,416
+ Entry 1: count: 1000 hasNull: false min: -9203618157670445774 max: 9208123824411178101 positions: 4099,2,488
+ Entry 2: count: 1000 hasNull: false min: -9218592812243954469 max: 9221351515892923972 positions: 12297,6,464
+ Entry 3: count: 1000 hasNull: false min: -9206585617947511272 max: 9167703224425685487 positions: 20495,10,440
+ Entry 4: count: 1000 hasNull: false min: -9206645795733282496 max: 9221614132680747961 positions: 28693,14,416
Row group indices for column 3:
Entry 0: count: 1000 hasNull: false min: Darkness, max: worst sum: 3862 positions: 0,0,0
Entry 1: count: 1000 hasNull: false min: Darkness, max: worst sum: 3884 positions: 0,659,149
Entry 2: count: 1000 hasNull: false min: Darkness, max: worst sum: 3893 positions: 0,1531,3
Entry 3: count: 1000 hasNull: false min: Darkness, max: worst sum: 3798 positions: 0,2281,32
Entry 4: count: 1000 hasNull: false min: Darkness, max: worst sum: 3843 positions: 0,3033,45
- Stripe: offset: 64288 data: 63754 rows: 5000 tail: 79 index: 433
- Stream: column 0 section ROW_INDEX start: 64288 length 17
- Stream: column 1 section ROW_INDEX start: 64305 length 162
- Stream: column 2 section ROW_INDEX start: 64467 length 171
- Stream: column 3 section ROW_INDEX start: 64638 length 83
- Stream: column 1 section DATA start: 64721 length 20029
- Stream: column 2 section DATA start: 84750 length 40035
- Stream: column 3 section DATA start: 124785 length 3532
- Stream: column 3 section LENGTH start: 128317 length 25
- Stream: column 3 section DICTIONARY_DATA start: 128342 length 133
+ Stripe: offset: 64307 data: 63775 rows: 5000 tail: 79 index: 432
+ Stream: column 0 section ROW_INDEX start: 64307 length 17
+ Stream: column 1 section ROW_INDEX start: 64324 length 164
+ Stream: column 2 section ROW_INDEX start: 64488 length 168
+ Stream: column 3 section ROW_INDEX start: 64656 length 83
+ Stream: column 1 section DATA start: 64739 length 20035
+ Stream: column 2 section DATA start: 84774 length 40050
+ Stream: column 3 section DATA start: 124824 length 3532
+ Stream: column 3 section LENGTH start: 128356 length 25
+ Stream: column 3 section DICTIONARY_DATA start: 128381 length 133
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DIRECT_V2
@@ -88,31 +88,31 @@ Stripes:
Row group indices for column 1:
Entry 0: count: 1000 hasNull: false min: -2143799121 max: 2145249879 sum: -6966266181 positions: 0,0,0
Entry 1: count: 1000 hasNull: false min: -2146733128 max: 2147001622 sum: -35930106333 positions: 0,2050,488
- Entry 2: count: 1000 hasNull: false min: -2144302712 max: 2146299933 sum: 6944230435 positions: 0,6150,464
- Entry 3: count: 1000 hasNull: false min: -2145172948 max: 2144335014 sum: -29624404959 positions: 10003,250,440
- Entry 4: count: 1000 hasNull: false min: -2146428427 max: 2144067253 sum: 65584220465 positions: 10003,4350,416
+ Entry 2: count: 1000 hasNull: false min: -2144302712 max: 2146299933 sum: 6944230435 positions: 4099,2054,464
+ Entry 3: count: 1000 hasNull: false min: -2145172948 max: 2144335014 sum: -29624404959 positions: 8198,2058,440
+ Entry 4: count: 1000 hasNull: false min: -2146428427 max: 2144067253 sum: 65584220465 positions: 12297,2062,416
Row group indices for column 2:
Entry 0: count: 1000 hasNull: false min: -9218450653857701562 max: 9189819526332228512 positions: 0,0,0
- Entry 1: count: 1000 hasNull: false min: -9220818777591257749 max: 9178821722829648113 positions: 0,4098,488
- Entry 2: count: 1000 hasNull: false min: -9220031433030423388 max: 9210838931786956852 positions: 10003,2294,464
- Entry 3: count: 1000 hasNull: false min: -9208195729739635607 max: 9222259462014003839 positions: 20006,490,440
- Entry 4: count: 1000 hasNull: false min: -9174271499932339698 max: 9212277876771676916 positions: 20006,8686,416
+ Entry 1: count: 1000 hasNull: false min: -9220818777591257749 max: 9178821722829648113 positions: 4099,2,488
+ Entry 2: count: 1000 hasNull: false min: -9220031433030423388 max: 9210838931786956852 positions: 12297,6,464
+ Entry 3: count: 1000 hasNull: false min: -9208195729739635607 max: 9222259462014003839 positions: 20495,10,440
+ Entry 4: count: 1000 hasNull: false min: -9174271499932339698 max: 9212277876771676916 positions: 28693,14,416
Row group indices for column 3:
Entry 0: count: 1000 hasNull: false min: Darkness, max: worst sum: 3923 positions: 0,0,0
Entry 1: count: 1000 hasNull: false min: Darkness, max: worst sum: 3869 positions: 0,761,12
Entry 2: count: 1000 hasNull: false min: Darkness, max: worst sum: 3817 positions: 0,1472,70
Entry 3: count: 1000 hasNull: false min: Darkness, max: worst sum: 3931 positions: 0,2250,43
Entry 4: count: 1000 hasNull: false min: Darkness, max: worst sum: 3964 positions: 0,2978,88
- Stripe: offset: 128554 data: 63766 rows: 5000 tail: 79 index: 437
- Stream: column 0 section ROW_INDEX start: 128554 length 17
- Stream: column 1 section ROW_INDEX start: 128571 length 159
- Stream: column 2 section ROW_INDEX start: 128730 length 171
- Stream: column 3 section ROW_INDEX start: 128901 length 90
- Stream: column 1 section DATA start: 128991 length 20029
- Stream: column 2 section DATA start: 149020 length 40035
- Stream: column 3 section DATA start: 189055 length 3544
- Stream: column 3 section LENGTH start: 192599 length 25
- Stream: column 3 section DICTIONARY_DATA start: 192624 length 133
+ Stripe: offset: 128593 data: 63787 rows: 5000 tail: 79 index: 438
+ Stream: column 0 section ROW_INDEX start: 128593 length 17
+ Stream: column 1 section ROW_INDEX start: 128610 length 163
+ Stream: column 2 section ROW_INDEX start: 128773 length 168
+ Stream: column 3 section ROW_INDEX start: 128941 length 90
+ Stream: column 1 section DATA start: 129031 length 20035
+ Stream: column 2 section DATA start: 149066 length 40050
+ Stream: column 3 section DATA start: 189116 length 3544
+ Stream: column 3 section LENGTH start: 192660 length 25
+ Stream: column 3 section DICTIONARY_DATA start: 192685 length 133
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DIRECT_V2
@@ -120,31 +120,31 @@ Stripes:
Row group indices for column 1:
Entry 0: count: 1000 hasNull: false min: -2146993718 max: 2144179881 sum: -7829543271 positions: 0,0,0
Entry 1: count: 1000 hasNull: false min: -2144095505 max: 2144883384 sum: 51623839692 positions: 0,2050,488
- Entry 2: count: 1000 hasNull: false min: -2144113995 max: 2143773575 sum: 56574412741 positions: 0,6150,464
- Entry 3: count: 1000 hasNull: false min: -2146954065 max: 2146794873 sum: 4336083432 positions: 10003,250,440
- Entry 4: count: 1000 hasNull: false min: -2135511523 max: 2147378179 sum: 27955949957 positions: 10003,4350,416
+ Entry 2: count: 1000 hasNull: false min: -2144113995 max: 2143773575 sum: 56574412741 positions: 4099,2054,464
+ Entry 3: count: 1000 hasNull: false min: -2146954065 max: 2146794873 sum: 4336083432 positions: 8198,2058,440
+ Entry 4: count: 1000 hasNull: false min: -2135511523 max: 2147378179 sum: 27955949957 positions: 12297,2062,416
Row group indices for column 2:
Entry 0: count: 1000 hasNull: false min: -9211978436552246208 max: 9179058898902097152 positions: 0,0,0
- Entry 1: count: 1000 hasNull: false min: -9195645160817780503 max: 9189147759444307708 positions: 0,4098,488
- Entry 2: count: 1000 hasNull: false min: -9202888157616520823 max: 9193561362676960747 positions: 10003,2294,464
- Entry 3: count: 1000 hasNull: false min: -9216318198067839390 max: 9221286760675829363 positions: 20006,490,440
- Entry 4: count: 1000 hasNull: false min: -9218342074710552826 max: 9222303228623055266 positions: 20006,8686,416
+ Entry 1: count: 1000 hasNull: false min: -9195645160817780503 max: 9189147759444307708 positions: 4099,2,488
+ Entry 2: count: 1000 hasNull: false min: -9202888157616520823 max: 9193561362676960747 positions: 12297,6,464
+ Entry 3: count: 1000 hasNull: false min: -9216318198067839390 max: 9221286760675829363 positions: 20495,10,440
+ Entry 4: count: 1000 hasNull: false min: -9218342074710552826 max: 9222303228623055266 positions: 28693,14,416
Row group indices for column 3:
Entry 0: count: 1000 hasNull: false min: Darkness, max: worst sum: 3817 positions: 0,0,0
Entry 1: count: 1000 hasNull: false min: Darkness, max: worst sum: 4008 positions: 0,634,174
Entry 2: count: 1000 hasNull: false min: Darkness, max: worst sum: 3999 positions: 0,1469,69
Entry 3: count: 1000 hasNull: false min: Darkness, max: worst sum: 3817 positions: 0,2133,194
Entry 4: count: 1000 hasNull: false min: Darkness, max: worst sum: 4000 positions: 0,3005,43
- Stripe: offset: 192836 data: 63796 rows: 5000 tail: 79 index: 440
- Stream: column 0 section ROW_INDEX start: 192836 length 17
- Stream: column 1 section ROW_INDEX start: 192853 length 162
- Stream: column 2 section ROW_INDEX start: 193015 length 170
- Stream: column 3 section ROW_INDEX start: 193185 length 91
- Stream: column 1 section DATA start: 193276 length 20029
- Stream: column 2 section DATA start: 213305 length 40035
- Stream: column 3 section DATA start: 253340 length 3574
- Stream: column 3 section LENGTH start: 256914 length 25
- Stream: column 3 section DICTIONARY_DATA start: 256939 length 133
+ Stripe: offset: 192897 data: 63817 rows: 5000 tail: 79 index: 440
+ Stream: column 0 section ROW_INDEX start: 192897 length 17
+ Stream: column 1 section ROW_INDEX start: 192914 length 165
+ Stream: column 2 section ROW_INDEX start: 193079 length 167
+ Stream: column 3 section ROW_INDEX start: 193246 length 91
+ Stream: column 1 section DATA start: 193337 length 20035
+ Stream: column 2 section DATA start: 213372 length 40050
+ Stream: column 3 section DATA start: 253422 length 3574
+ Stream: column 3 section LENGTH start: 256996 length 25
+ Stream: column 3 section DICTIONARY_DATA start: 257021 length 133
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DIRECT_V2
@@ -152,31 +152,31 @@ Stripes:
Row group indices for column 1:
Entry 0: count: 1000 hasNull: false min: -2141355639 max: 2145520931 sum: 2726719912 positions: 0,0,0
Entry 1: count: 1000 hasNull: false min: -2138324170 max: 2140167376 sum: -23606674002 positions: 0,2050,488
- Entry 2: count: 1000 hasNull: false min: -2146658006 max: 2144329742 sum: -41530109703 positions: 0,6150,464
- Entry 3: count: 1000 hasNull: false min: -2144207593 max: 2139456355 sum: 13559842458 positions: 10003,250,440
- Entry 4: count: 1000 hasNull: false min: -2145744719 max: 2145417153 sum: 57383770571 positions: 10003,4350,416
+ Entry 2: count: 1000 hasNull: false min: -2146658006 max: 2144329742 sum: -41530109703 positions: 4099,2054,464
+ Entry 3: count: 1000 hasNull: false min: -2144207593 max: 2139456355 sum: 13559842458 positions: 8198,2058,440
+ Entry 4: count: 1000 hasNull: false min: -2145744719 max: 2145417153 sum: 57383770571 positions: 12297,2062,416
Row group indices for column 2:
Entry 0: count: 1000 hasNull: false min: -9222731174895935707 max: 9214167447015056056 positions: 0,0,0
- Entry 1: count: 1000 hasNull: false min: -9222758097219661129 max: 9221043130193737406 positions: 0,4098,488
- Entry 2: count: 1000 hasNull: false min: -9174483776261243438 max: 9208134757538374043 positions: 10003,2294,464
- Entry 3: count: 1000 hasNull: false min: -9174329712613510612 max: 9197412874152820822 positions: 20006,490,440
- Entry 4: count: 1000 hasNull: false min: -9221162005892422758 max: 9220625004936875965 positions: 20006,8686,416
+ Entry 1: count: 1000 hasNull: false min: -9222758097219661129 max: 9221043130193737406 positions: 4099,2,488
+ Entry 2: count: 1000 hasNull: false min: -9174483776261243438 max: 9208134757538374043 positions: 12297,6,464
+ Entry 3: count: 1000 hasNull: false min: -9174329712613510612 max: 9197412874152820822 positions: 20495,10,440
+ Entry 4: count: 1000 hasNull: false min: -9221162005892422758 max: 9220625004936875965 positions: 28693,14,416
Row group indices for column 3:
Entry 0: count: 1000 hasNull: false min: Darkness, max: worst sum: 3901 positions: 0,0,0
Entry 1: count: 1000 hasNull: false min: Darkness, max: worst sum: 3900 positions: 0,431,431
Entry 2: count: 1000 hasNull: false min: Darkness, max: worst sum: 3909 positions: 0,1485,52
Entry 3: count: 1000 hasNull: false min: Darkness, max: worst sum: 3947 positions: 0,2196,104
Entry 4: count: 1000 hasNull: false min: Darkness, max: worst sum: 3813 positions: 0,2934,131
- Stripe: offset: 257151 data: 12940 rows: 1000 tail: 71 index: 131
- Stream: column 0 section ROW_INDEX start: 257151 length 12
- Stream: column 1 section ROW_INDEX start: 257163 length 38
- Stream: column 2 section ROW_INDEX start: 257201 length 41
- Stream: column 3 section ROW_INDEX start: 257242 length 40
- Stream: column 1 section DATA start: 257282 length 4007
- Stream: column 2 section DATA start: 261289 length 8007
- Stream: column 3 section DATA start: 269296 length 768
- Stream: column 3 section LENGTH start: 270064 length 25
- Stream: column 3 section DICTIONARY_DATA start: 270089 length 133
+ Stripe: offset: 257233 data: 12943 rows: 1000 tail: 71 index: 131
+ Stream: column 0 section ROW_INDEX start: 257233 length 12
+ Stream: column 1 section ROW_INDEX start: 257245 length 38
+ Stream: column 2 section ROW_INDEX start: 257283 length 41
+ Stream: column 3 section ROW_INDEX start: 257324 length 40
+ Stream: column 1 section DATA start: 257364 length 4007
+ Stream: column 2 section DATA start: 261371 length 8010
+ Stream: column 3 section DATA start: 269381 length 768
+ Stream: column 3 section LENGTH start: 270149 length 25
+ Stream: column 3 section DICTIONARY_DATA start: 270174 length 133
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DIRECT_V2
@@ -188,6 +188,6 @@ Stripes:
Row group indices for column 3:
Entry 0: count: 1000 hasNull: false min: Darkness, max: worst sum: 3866 positions: 0,0,0
-File length: 270838 bytes
+File length: 270923 bytes
Padding length: 0 bytes
Padding ratio: 0%
http://git-wip-us.apache.org/repos/asf/hive/blob/51a0c03f/ql/src/test/resources/orc-file-has-null.out
----------------------------------------------------------------------
diff --git a/ql/src/test/resources/orc-file-has-null.out b/ql/src/test/resources/orc-file-has-null.out
index fdc3862..44c60b7 100644
--- a/ql/src/test/resources/orc-file-has-null.out
+++ b/ql/src/test/resources/orc-file-has-null.out
@@ -2,7 +2,7 @@ Structure for TestOrcFile.testHasNull.orc
File Version: 0.12 with HIVE_8732
Rows: 20000
Compression: ZLIB
-Compression size: 10000
+Compression size: 4096
Type: struct<bytes1:binary,string1:string>
Stripe Statistics:
@@ -29,16 +29,16 @@ File Statistics:
Column 2: count: 7000 hasNull: true min: RG1 max: STRIPE-3 sum: 46000
Stripes:
- Stripe: offset: 3 data: 195 rows: 5000 tail: 64 index: 162
+ Stripe: offset: 3 data: 241 rows: 5000 tail: 67 index: 163
Stream: column 0 section ROW_INDEX start: 3 length 17
- Stream: column 1 section ROW_INDEX start: 20 length 63
- Stream: column 2 section ROW_INDEX start: 83 length 82
- Stream: column 1 section DATA start: 165 length 113
- Stream: column 1 section LENGTH start: 278 length 32
- Stream: column 2 section PRESENT start: 310 length 13
- Stream: column 2 section DATA start: 323 length 22
- Stream: column 2 section LENGTH start: 345 length 6
- Stream: column 2 section DICTIONARY_DATA start: 351 length 9
+ Stream: column 1 section ROW_INDEX start: 20 length 64
+ Stream: column 2 section ROW_INDEX start: 84 length 82
+ Stream: column 1 section DATA start: 166 length 159
+ Stream: column 1 section LENGTH start: 325 length 32
+ Stream: column 2 section PRESENT start: 357 length 13
+ Stream: column 2 section DATA start: 370 length 22
+ Stream: column 2 section LENGTH start: 392 length 6
+ Stream: column 2 section DICTIONARY_DATA start: 398 length 9
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DICTIONARY_V2[2]
@@ -48,16 +48,16 @@ Stripes:
Entry 2: count: 1000 hasNull: false min: RG3 max: RG3 sum: 3000 positions: 0,2,125,0,0,66,488
Entry 3: count: 0 hasNull: true positions: 0,4,125,0,0,136,488
Entry 4: count: 0 hasNull: true positions: 0,6,125,0,0,136,488
- Stripe: offset: 424 data: 156 rows: 5000 tail: 60 index: 119
- Stream: column 0 section ROW_INDEX start: 424 length 17
- Stream: column 1 section ROW_INDEX start: 441 length 63
- Stream: column 2 section ROW_INDEX start: 504 length 39
- Stream: column 1 section DATA start: 543 length 113
- Stream: column 1 section LENGTH start: 656 length 32
- Stream: column 2 section PRESENT start: 688 length 11
- Stream: column 2 section DATA start: 699 length 0
- Stream: column 2 section LENGTH start: 699 length 0
- Stream: column 2 section DICTIONARY_DATA start: 699 length 0
+ Stripe: offset: 474 data: 202 rows: 5000 tail: 64 index: 120
+ Stream: column 0 section ROW_INDEX start: 474 length 17
+ Stream: column 1 section ROW_INDEX start: 491 length 64
+ Stream: column 2 section ROW_INDEX start: 555 length 39
+ Stream: column 1 section DATA start: 594 length 159
+ Stream: column 1 section LENGTH start: 753 length 32
+ Stream: column 2 section PRESENT start: 785 length 11
+ Stream: column 2 section DATA start: 796 length 0
+ Stream: column 2 section LENGTH start: 796 length 0
+ Stream: column 2 section DICTIONARY_DATA start: 796 length 0
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DICTIONARY_V2[0]
@@ -67,15 +67,15 @@ Stripes:
Entry 2: count: 0 hasNull: true positions: 0,2,120,0,0,0,0
Entry 3: count: 0 hasNull: true positions: 0,4,115,0,0,0,0
Entry 4: count: 0 hasNull: true positions: 0,6,110,0,0,0,0
- Stripe: offset: 759 data: 186 rows: 5000 tail: 60 index: 148
- Stream: column 0 section ROW_INDEX start: 759 length 17
- Stream: column 1 section ROW_INDEX start: 776 length 63
- Stream: column 2 section ROW_INDEX start: 839 length 68
- Stream: column 1 section DATA start: 907 length 113
- Stream: column 1 section LENGTH start: 1020 length 32
- Stream: column 2 section DATA start: 1052 length 24
- Stream: column 2 section LENGTH start: 1076 length 6
- Stream: column 2 section DICTIONARY_DATA start: 1082 length 11
+ Stripe: offset: 860 data: 232 rows: 5000 tail: 63 index: 149
+ Stream: column 0 section ROW_INDEX start: 860 length 17
+ Stream: column 1 section ROW_INDEX start: 877 length 64
+ Stream: column 2 section ROW_INDEX start: 941 length 68
+ Stream: column 1 section DATA start: 1009 length 159
+ Stream: column 1 section LENGTH start: 1168 length 32
+ Stream: column 2 section DATA start: 1200 length 24
+ Stream: column 2 section LENGTH start: 1224 length 6
+ Stream: column 2 section DICTIONARY_DATA start: 1230 length 11
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DICTIONARY_V2[1]
@@ -85,16 +85,16 @@ Stripes:
Entry 2: count: 1000 hasNull: false min: STRIPE-3 max: STRIPE-3 sum: 8000 positions: 0,198,464
Entry 3: count: 1000 hasNull: false min: STRIPE-3 max: STRIPE-3 sum: 8000 positions: 0,330,440
Entry 4: count: 1000 hasNull: false min: STRIPE-3 max: STRIPE-3 sum: 8000 positions: 0,462,416
- Stripe: offset: 1153 data: 156 rows: 5000 tail: 60 index: 119
- Stream: column 0 section ROW_INDEX start: 1153 length 17
- Stream: column 1 section ROW_INDEX start: 1170 length 63
- Stream: column 2 section ROW_INDEX start: 1233 length 39
- Stream: column 1 section DATA start: 1272 length 113
- Stream: column 1 section LENGTH start: 1385 length 32
- Stream: column 2 section PRESENT start: 1417 length 11
- Stream: column 2 section DATA start: 1428 length 0
- Stream: column 2 section LENGTH start: 1428 length 0
- Stream: column 2 section DICTIONARY_DATA start: 1428 length 0
+ Stripe: offset: 1304 data: 202 rows: 5000 tail: 64 index: 120
+ Stream: column 0 section ROW_INDEX start: 1304 length 17
+ Stream: column 1 section ROW_INDEX start: 1321 length 64
+ Stream: column 2 section ROW_INDEX start: 1385 length 39
+ Stream: column 1 section DATA start: 1424 length 159
+ Stream: column 1 section LENGTH start: 1583 length 32
+ Stream: column 2 section PRESENT start: 1615 length 11
+ Stream: column 2 section DATA start: 1626 length 0
+ Stream: column 2 section LENGTH start: 1626 length 0
+ Stream: column 2 section DICTIONARY_DATA start: 1626 length 0
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DICTIONARY_V2[0]
@@ -105,6 +105,6 @@ Stripes:
Entry 3: count: 0 hasNull: true positions: 0,4,115,0,0,0,0
Entry 4: count: 0 hasNull: true positions: 0,6,110,0,0,0,0
-File length: 1736 bytes
+File length: 1940 bytes
Padding length: 0 bytes
Padding ratio: 0%
[2/2] hive git commit: HIVE-11807: Set ORC buffer size in relation to
set stripe size (Owen O'Malley reviewed by Gopal V)
Posted by pr...@apache.org.
HIVE-11807: Set ORC buffer size in relation to set stripe size (Owen O'Malley reviewed by Gopal V)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/51a0c03f
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/51a0c03f
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/51a0c03f
Branch: refs/heads/branch-1
Commit: 51a0c03f4fe1c90ca6ba9dbb7dad37a4a139c891
Parents: 30ca4fb
Author: Prasanth Jayachandran <j....@gmail.com>
Authored: Thu Oct 22 16:30:22 2015 -0700
Committer: Prasanth Jayachandran <j....@gmail.com>
Committed: Thu Oct 22 16:30:22 2015 -0700
----------------------------------------------------------------------
.../hadoop/hive/ql/io/orc/WriterImpl.java | 99 ++++----
.../hadoop/hive/ql/io/orc/TestOrcWideTable.java | 232 ++-----------------
.../resources/orc-file-dump-bloomfilter.out | 112 ++++-----
.../resources/orc-file-dump-bloomfilter2.out | 144 ++++++------
.../orc-file-dump-dictionary-threshold.out | 180 +++++++-------
ql/src/test/resources/orc-file-dump.json | 184 +++++++--------
ql/src/test/resources/orc-file-dump.out | 158 ++++++-------
ql/src/test/resources/orc-file-has-null.out | 80 +++----
8 files changed, 493 insertions(+), 696 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/51a0c03f/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java
----------------------------------------------------------------------
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java b/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java
index 58c7577..b5e6ad1 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java
@@ -22,7 +22,6 @@ import static com.google.common.base.Preconditions.checkArgument;
import java.io.IOException;
import java.io.OutputStream;
-import java.lang.management.ManagementFactory;
import java.nio.ByteBuffer;
import java.sql.Timestamp;
import java.util.ArrayList;
@@ -209,7 +208,8 @@ public class WriterImpl implements Writer, MemoryManager.Callback {
if (allColumns == null) {
allColumns = getColumnNamesFromInspector(inspector);
}
- this.bufferSize = getEstimatedBufferSize(allColumns, bufferSize);
+ this.bufferSize = getEstimatedBufferSize(defaultStripeSize,
+ countColumns(inspector), bufferSize);
if (version == OrcFile.Version.V_0_11) {
/* do not write bloom filters for ORC v11 */
this.bloomFilterColumns =
@@ -242,49 +242,58 @@ public class WriterImpl implements Writer, MemoryManager.Callback {
return joiner.join(fieldNames);
}
- @VisibleForTesting
- int getEstimatedBufferSize(int bs) {
- return getEstimatedBufferSize(conf.get(IOConstants.COLUMNS), bs);
- }
-
- int getEstimatedBufferSize(String colNames, int bs) {
- long availableMem = getMemoryAvailableForORC();
- if (colNames != null) {
- final int numCols = colNames.split(",").length;
- if (numCols > COLUMN_COUNT_THRESHOLD) {
- // In BufferedStream, there are 3 outstream buffers (compressed,
- // uncompressed and overflow) and list of previously compressed buffers.
- // Since overflow buffer is rarely used, lets consider only 2 allocation.
- // Also, initially, the list of compression buffers will be empty.
- final int outStreamBuffers = codec == null ? 1 : 2;
-
- // max possible streams per column is 5. For string columns, there is
- // ROW_INDEX, PRESENT, DATA, LENGTH, DICTIONARY_DATA streams.
- final int maxStreams = 5;
-
- // Lets assume 10% memory for holding dictionary in memory and other
- // object allocations
- final long miscAllocation = (long) (0.1f * availableMem);
-
- // compute the available memory
- final long remainingMem = availableMem - miscAllocation;
-
- int estBufferSize = (int) (remainingMem /
- (maxStreams * outStreamBuffers * numCols));
- estBufferSize = getClosestBufferSize(estBufferSize, bs);
- if (estBufferSize > bs) {
- estBufferSize = bs;
+ static int countColumns(ObjectInspector oi) {
+ switch (oi.getCategory()) {
+ case PRIMITIVE:
+ return 1;
+ case STRUCT: {
+ int result = 1;
+ for(StructField field:
+ ((StructObjectInspector) oi).getAllStructFieldRefs()) {
+ result += countColumns(field.getFieldObjectInspector());
}
-
- LOG.info("WIDE TABLE - Number of columns: " + numCols +
- " Chosen compression buffer size: " + estBufferSize);
- return estBufferSize;
+ return result;
+ }
+ case UNION: {
+ int result = 1;
+ for(ObjectInspector child:
+ ((UnionObjectInspector) oi).getObjectInspectors()) {
+ result += countColumns(child);
+ }
+ return result;
+ }
+ case LIST:
+ return 1 + countColumns(
+ ((ListObjectInspector)oi).getListElementObjectInspector());
+ case MAP: {
+ MapObjectInspector moi = (MapObjectInspector) oi;
+ return 1 + countColumns(moi.getMapKeyObjectInspector()) +
+ countColumns(moi.getMapValueObjectInspector());
}
+ default:
+ throw new IllegalArgumentException("Unknown category " +
+ oi.getCategory());
}
- return bs;
}
- private int getClosestBufferSize(int estBufferSize, int bs) {
+ @VisibleForTesting
+ static int getEstimatedBufferSize(long stripeSize, int numColumns, int bs) {
+ // The worst case is that there are 2 big streams per a column and
+ // we want to guarantee that each stream gets ~10 buffers.
+ // This keeps buffers small enough that we don't get really small stripe
+ // sizes.
+ int estBufferSize = (int) (stripeSize / (20 * numColumns));
+ estBufferSize = getClosestBufferSize(estBufferSize);
+ if (estBufferSize > bs) {
+ estBufferSize = bs;
+ } else {
+ LOG.info("WIDE TABLE - Number of columns: " + numColumns +
+ " Chosen compression buffer size: " + estBufferSize);
+ }
+ return estBufferSize;
+ }
+
+ private static int getClosestBufferSize(int estBufferSize) {
final int kb4 = 4 * 1024;
final int kb8 = 8 * 1024;
final int kb16 = 16 * 1024;
@@ -309,16 +318,6 @@ public class WriterImpl implements Writer, MemoryManager.Callback {
}
}
- // the assumption is only one ORC writer open at a time, which holds true for
- // most of the cases. HIVE-6455 forces single writer case.
- private long getMemoryAvailableForORC() {
- HiveConf.ConfVars poolVar = HiveConf.ConfVars.HIVE_ORC_FILE_MEMORY_POOL;
- double maxLoad = conf.getFloat(poolVar.varname, poolVar.defaultFloatVal);
- long totalMemoryPool = Math.round(ManagementFactory.getMemoryMXBean().
- getHeapMemoryUsage().getMax() * maxLoad);
- return totalMemoryPool;
- }
-
public static CompressionCodec createCodec(CompressionKind kind) {
switch (kind) {
case NONE:
http://git-wip-us.apache.org/repos/asf/hive/blob/51a0c03f/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcWideTable.java
----------------------------------------------------------------------
diff --git a/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcWideTable.java b/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcWideTable.java
index a3d3ec5..6b6cb2c 100644
--- a/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcWideTable.java
+++ b/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcWideTable.java
@@ -38,241 +38,39 @@ import org.junit.rules.TestName;
public class TestOrcWideTable {
- private static final int MEMORY_FOR_ORC = 512 * 1024 * 1024;
- Path workDir = new Path(System.getProperty("test.tmp.dir", "target" + File.separator + "test"
- + File.separator + "tmp"));
-
- Configuration conf;
- FileSystem fs;
- Path testFilePath;
- float memoryPercent;
-
- @Rule
- public TestName testCaseName = new TestName();
-
- @Before
- public void openFileSystem() throws Exception {
- conf = new Configuration();
- fs = FileSystem.getLocal(conf);
- testFilePath = new Path(workDir, "TestOrcFile." + testCaseName.getMethodName() + ".orc");
- fs.delete(testFilePath, false);
- // make sure constant memory is available for ORC always
- memoryPercent = (float) MEMORY_FOR_ORC / (float) ManagementFactory.getMemoryMXBean().
- getHeapMemoryUsage().getMax();
- conf.setFloat(HiveConf.ConfVars.HIVE_ORC_FILE_MEMORY_POOL.varname, memoryPercent);
- }
-
@Test
public void testBufferSizeFor1Col() throws IOException {
- ObjectInspector inspector;
- synchronized (TestOrcFile.class) {
- inspector = ObjectInspectorFactory.getReflectionObjectInspector(Long.class,
- ObjectInspectorFactory.ObjectInspectorOptions.JAVA);
- }
- int bufferSize = 128 * 1024;
- Writer writer = OrcFile.createWriter(
- testFilePath,
- OrcFile.writerOptions(conf).inspector(inspector).stripeSize(100000)
- .compress(CompressionKind.NONE).bufferSize(bufferSize));
- final int newBufferSize;
- if (writer instanceof WriterImpl) {
- WriterImpl orcWriter = (WriterImpl) writer;
- newBufferSize = orcWriter.getEstimatedBufferSize(bufferSize);
- assertEquals(bufferSize, newBufferSize);
- }
+ assertEquals(128 * 1024, WriterImpl.getEstimatedBufferSize(512 * 1024 * 1024,
+ 1, 128*1024));
}
@Test
- public void testBufferSizeFor1000Col() throws IOException {
- ObjectInspector inspector;
- synchronized (TestOrcFile.class) {
- inspector = ObjectInspectorFactory.getReflectionObjectInspector(Long.class,
- ObjectInspectorFactory.ObjectInspectorOptions.JAVA);
- }
- int bufferSize = 128 * 1024;
- String columns = getRandomColumnNames(1000);
- // just for testing. manually write the column names
- conf.set(IOConstants.COLUMNS, columns);
- Writer writer = OrcFile.createWriter(
- testFilePath,
- OrcFile.writerOptions(conf).inspector(inspector).stripeSize(100000)
- .compress(CompressionKind.NONE).bufferSize(bufferSize));
- final int newBufferSize;
- if (writer instanceof WriterImpl) {
- WriterImpl orcWriter = (WriterImpl) writer;
- newBufferSize = orcWriter.getEstimatedBufferSize(bufferSize);
- assertEquals(bufferSize, newBufferSize);
- }
+ public void testBufferSizeFor50Col() throws IOException {
+ assertEquals(256 * 1024, WriterImpl.getEstimatedBufferSize(256 * 1024 * 1024,
+ 50, 256*1024));
}
@Test
- public void testBufferSizeFor2000Col() throws IOException {
- ObjectInspector inspector;
- synchronized (TestOrcFile.class) {
- inspector = ObjectInspectorFactory.getReflectionObjectInspector(Long.class,
- ObjectInspectorFactory.ObjectInspectorOptions.JAVA);
- }
- int bufferSize = 256 * 1024;
- String columns = getRandomColumnNames(2000);
- // just for testing. manually write the column names
- conf.set(IOConstants.COLUMNS, columns);
- Writer writer = OrcFile.createWriter(
- testFilePath,
- OrcFile.writerOptions(conf).inspector(inspector).stripeSize(100000)
- .compress(CompressionKind.ZLIB).bufferSize(bufferSize));
- final int newBufferSize;
- if (writer instanceof WriterImpl) {
- WriterImpl orcWriter = (WriterImpl) writer;
- newBufferSize = orcWriter.getEstimatedBufferSize(bufferSize);
- assertEquals(32 * 1024, newBufferSize);
- }
+ public void testBufferSizeFor1000Col() throws IOException {
+ assertEquals(32 * 1024, WriterImpl.getEstimatedBufferSize(512 * 1024 * 1024,
+ 1000, 128*1024));
}
@Test
- public void testBufferSizeFor2000ColNoCompression() throws IOException {
- ObjectInspector inspector;
- synchronized (TestOrcFile.class) {
- inspector = ObjectInspectorFactory.getReflectionObjectInspector(Long.class,
- ObjectInspectorFactory.ObjectInspectorOptions.JAVA);
- }
- int bufferSize = 256 * 1024;
- String columns = getRandomColumnNames(2000);
- // just for testing. manually write the column names
- conf.set(IOConstants.COLUMNS, columns);
- Writer writer = OrcFile.createWriter(
- testFilePath,
- OrcFile.writerOptions(conf).inspector(inspector).stripeSize(100000)
- .compress(CompressionKind.NONE).bufferSize(bufferSize));
- final int newBufferSize;
- if (writer instanceof WriterImpl) {
- WriterImpl orcWriter = (WriterImpl) writer;
- newBufferSize = orcWriter.getEstimatedBufferSize(bufferSize);
- assertEquals(64 * 1024, newBufferSize);
- }
+ public void testBufferSizeFor2000Col() throws IOException {
+ assertEquals(16 * 1024, WriterImpl.getEstimatedBufferSize(512 * 1024 * 1024,
+ 2000, 256*1024));
}
@Test
public void testBufferSizeFor4000Col() throws IOException {
- ObjectInspector inspector;
- synchronized (TestOrcFile.class) {
- inspector = ObjectInspectorFactory.getReflectionObjectInspector(Long.class,
- ObjectInspectorFactory.ObjectInspectorOptions.JAVA);
- }
- int bufferSize = 256 * 1024;
- String columns = getRandomColumnNames(4000);
- // just for testing. manually write the column names
- conf.set(IOConstants.COLUMNS, columns);
- Writer writer = OrcFile.createWriter(
- testFilePath,
- OrcFile.writerOptions(conf).inspector(inspector).stripeSize(100000)
- .compress(CompressionKind.ZLIB).bufferSize(bufferSize));
- final int newBufferSize;
- if (writer instanceof WriterImpl) {
- WriterImpl orcWriter = (WriterImpl) writer;
- newBufferSize = orcWriter.getEstimatedBufferSize(bufferSize);
- assertEquals(16 * 1024, newBufferSize);
- }
- }
-
- @Test
- public void testBufferSizeFor4000ColNoCompression() throws IOException {
- ObjectInspector inspector;
- synchronized (TestOrcFile.class) {
- inspector = ObjectInspectorFactory.getReflectionObjectInspector(Long.class,
- ObjectInspectorFactory.ObjectInspectorOptions.JAVA);
- }
- int bufferSize = 256 * 1024;
- String columns = getRandomColumnNames(4000);
- // just for testing. manually write the column names
- conf.set(IOConstants.COLUMNS, columns);
- Writer writer = OrcFile.createWriter(
- testFilePath,
- OrcFile.writerOptions(conf).inspector(inspector).stripeSize(100000)
- .compress(CompressionKind.NONE).bufferSize(bufferSize));
- final int newBufferSize;
- if (writer instanceof WriterImpl) {
- WriterImpl orcWriter = (WriterImpl) writer;
- newBufferSize = orcWriter.getEstimatedBufferSize(bufferSize);
- assertEquals(32 * 1024, newBufferSize);
- }
+ assertEquals(8 * 1024, WriterImpl.getEstimatedBufferSize(512 * 1024 * 1024,
+ 4000, 256*1024));
}
@Test
public void testBufferSizeFor25000Col() throws IOException {
- ObjectInspector inspector;
- synchronized (TestOrcFile.class) {
- inspector = ObjectInspectorFactory.getReflectionObjectInspector(Long.class,
- ObjectInspectorFactory.ObjectInspectorOptions.JAVA);
- }
- int bufferSize = 256 * 1024;
- String columns = getRandomColumnNames(25000);
- // just for testing. manually write the column names
- conf.set(IOConstants.COLUMNS, columns);
- Writer writer = OrcFile.createWriter(
- testFilePath,
- OrcFile.writerOptions(conf).inspector(inspector).stripeSize(100000)
- .compress(CompressionKind.NONE).bufferSize(bufferSize));
- final int newBufferSize;
- if (writer instanceof WriterImpl) {
- WriterImpl orcWriter = (WriterImpl) writer;
- newBufferSize = orcWriter.getEstimatedBufferSize(bufferSize);
- // 4K is the minimum buffer size
- assertEquals(4 * 1024, newBufferSize);
- }
- }
-
- @Test
- public void testBufferSizeManualOverride1() throws IOException {
- ObjectInspector inspector;
- synchronized (TestOrcFile.class) {
- inspector = ObjectInspectorFactory.getReflectionObjectInspector(Long.class,
- ObjectInspectorFactory.ObjectInspectorOptions.JAVA);
- }
- int bufferSize = 1024;
- String columns = getRandomColumnNames(2000);
- // just for testing. manually write the column names
- conf.set(IOConstants.COLUMNS, columns);
- Writer writer = OrcFile.createWriter(
- testFilePath,
- OrcFile.writerOptions(conf).inspector(inspector).stripeSize(100000)
- .compress(CompressionKind.NONE).bufferSize(bufferSize));
- final int newBufferSize;
- if (writer instanceof WriterImpl) {
- WriterImpl orcWriter = (WriterImpl) writer;
- newBufferSize = orcWriter.getEstimatedBufferSize(bufferSize);
- assertEquals(bufferSize, newBufferSize);
- }
- }
-
- @Test
- public void testBufferSizeManualOverride2() throws IOException {
- ObjectInspector inspector;
- synchronized (TestOrcFile.class) {
- inspector = ObjectInspectorFactory.getReflectionObjectInspector(Long.class,
- ObjectInspectorFactory.ObjectInspectorOptions.JAVA);
- }
- int bufferSize = 2 * 1024;
- String columns = getRandomColumnNames(4000);
- // just for testing. manually write the column names
- conf.set(IOConstants.COLUMNS, columns);
- Writer writer = OrcFile.createWriter(
- testFilePath,
- OrcFile.writerOptions(conf).inspector(inspector).stripeSize(100000)
- .compress(CompressionKind.NONE).bufferSize(bufferSize));
- final int newBufferSize;
- if (writer instanceof WriterImpl) {
- WriterImpl orcWriter = (WriterImpl) writer;
- newBufferSize = orcWriter.getEstimatedBufferSize(bufferSize);
- assertEquals(bufferSize, newBufferSize);
- }
- }
-
- private String getRandomColumnNames(int n) {
- StringBuilder sb = new StringBuilder();
- for (int i = 0; i < n - 1; i++) {
- sb.append("col").append(i).append(",");
- }
- sb.append("col").append(n - 1);
- return sb.toString();
+ assertEquals(4 * 1024, WriterImpl.getEstimatedBufferSize(512 * 1024 * 1024,
+ 25000, 256*1024));
}
}
http://git-wip-us.apache.org/repos/asf/hive/blob/51a0c03f/ql/src/test/resources/orc-file-dump-bloomfilter.out
----------------------------------------------------------------------
diff --git a/ql/src/test/resources/orc-file-dump-bloomfilter.out b/ql/src/test/resources/orc-file-dump-bloomfilter.out
index add163c..3420135 100644
--- a/ql/src/test/resources/orc-file-dump-bloomfilter.out
+++ b/ql/src/test/resources/orc-file-dump-bloomfilter.out
@@ -2,7 +2,7 @@ Structure for TestFileDump.testDump.orc
File Version: 0.12 with HIVE_8732
Rows: 21000
Compression: ZLIB
-Compression size: 10000
+Compression size: 4096
Type: struct<i:int,l:bigint,s:string>
Stripe Statistics:
@@ -39,17 +39,17 @@ File Statistics:
Column 3: count: 21000 hasNull: false min: Darkness, max: worst sum: 81761
Stripes:
- Stripe: offset: 3 data: 63765 rows: 5000 tail: 86 index: 845
+ Stripe: offset: 3 data: 63786 rows: 5000 tail: 86 index: 951
Stream: column 0 section ROW_INDEX start: 3 length 17
- Stream: column 1 section ROW_INDEX start: 20 length 164
- Stream: column 2 section ROW_INDEX start: 184 length 173
- Stream: column 3 section ROW_INDEX start: 357 length 87
- Stream: column 3 section BLOOM_FILTER start: 444 length 404
- Stream: column 1 section DATA start: 848 length 20029
- Stream: column 2 section DATA start: 20877 length 40035
- Stream: column 3 section DATA start: 60912 length 3543
- Stream: column 3 section LENGTH start: 64455 length 25
- Stream: column 3 section DICTIONARY_DATA start: 64480 length 133
+ Stream: column 1 section ROW_INDEX start: 20 length 166
+ Stream: column 2 section ROW_INDEX start: 186 length 169
+ Stream: column 3 section ROW_INDEX start: 355 length 87
+ Stream: column 3 section BLOOM_FILTER start: 442 length 512
+ Stream: column 1 section DATA start: 954 length 20035
+ Stream: column 2 section DATA start: 20989 length 40050
+ Stream: column 3 section DATA start: 61039 length 3543
+ Stream: column 3 section LENGTH start: 64582 length 25
+ Stream: column 3 section DICTIONARY_DATA start: 64607 length 133
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DIRECT_V2
@@ -67,17 +67,17 @@ Stripes:
Entry 3: numHashFunctions: 4 bitCount: 6272 popCount: 138 loadFactor: 0.022 expectedFpp: 2.343647E-7
Entry 4: numHashFunctions: 4 bitCount: 6272 popCount: 138 loadFactor: 0.022 expectedFpp: 2.343647E-7
Stripe level merge: numHashFunctions: 4 bitCount: 6272 popCount: 138 loadFactor: 0.022 expectedFpp: 2.343647E-7
- Stripe: offset: 64699 data: 63754 rows: 5000 tail: 86 index: 837
- Stream: column 0 section ROW_INDEX start: 64699 length 17
- Stream: column 1 section ROW_INDEX start: 64716 length 162
- Stream: column 2 section ROW_INDEX start: 64878 length 171
- Stream: column 3 section ROW_INDEX start: 65049 length 83
- Stream: column 3 section BLOOM_FILTER start: 65132 length 404
- Stream: column 1 section DATA start: 65536 length 20029
- Stream: column 2 section DATA start: 85565 length 40035
- Stream: column 3 section DATA start: 125600 length 3532
- Stream: column 3 section LENGTH start: 129132 length 25
- Stream: column 3 section DICTIONARY_DATA start: 129157 length 133
+ Stripe: offset: 64826 data: 63775 rows: 5000 tail: 86 index: 944
+ Stream: column 0 section ROW_INDEX start: 64826 length 17
+ Stream: column 1 section ROW_INDEX start: 64843 length 164
+ Stream: column 2 section ROW_INDEX start: 65007 length 168
+ Stream: column 3 section ROW_INDEX start: 65175 length 83
+ Stream: column 3 section BLOOM_FILTER start: 65258 length 512
+ Stream: column 1 section DATA start: 65770 length 20035
+ Stream: column 2 section DATA start: 85805 length 40050
+ Stream: column 3 section DATA start: 125855 length 3532
+ Stream: column 3 section LENGTH start: 129387 length 25
+ Stream: column 3 section DICTIONARY_DATA start: 129412 length 133
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DIRECT_V2
@@ -95,17 +95,17 @@ Stripes:
Entry 3: numHashFunctions: 4 bitCount: 6272 popCount: 138 loadFactor: 0.022 expectedFpp: 2.343647E-7
Entry 4: numHashFunctions: 4 bitCount: 6272 popCount: 138 loadFactor: 0.022 expectedFpp: 2.343647E-7
Stripe level merge: numHashFunctions: 4 bitCount: 6272 popCount: 138 loadFactor: 0.022 expectedFpp: 2.343647E-7
- Stripe: offset: 129376 data: 63766 rows: 5000 tail: 86 index: 841
- Stream: column 0 section ROW_INDEX start: 129376 length 17
- Stream: column 1 section ROW_INDEX start: 129393 length 159
- Stream: column 2 section ROW_INDEX start: 129552 length 171
- Stream: column 3 section ROW_INDEX start: 129723 length 90
- Stream: column 3 section BLOOM_FILTER start: 129813 length 404
- Stream: column 1 section DATA start: 130217 length 20029
- Stream: column 2 section DATA start: 150246 length 40035
- Stream: column 3 section DATA start: 190281 length 3544
- Stream: column 3 section LENGTH start: 193825 length 25
- Stream: column 3 section DICTIONARY_DATA start: 193850 length 133
+ Stripe: offset: 129631 data: 63787 rows: 5000 tail: 86 index: 950
+ Stream: column 0 section ROW_INDEX start: 129631 length 17
+ Stream: column 1 section ROW_INDEX start: 129648 length 163
+ Stream: column 2 section ROW_INDEX start: 129811 length 168
+ Stream: column 3 section ROW_INDEX start: 129979 length 90
+ Stream: column 3 section BLOOM_FILTER start: 130069 length 512
+ Stream: column 1 section DATA start: 130581 length 20035
+ Stream: column 2 section DATA start: 150616 length 40050
+ Stream: column 3 section DATA start: 190666 length 3544
+ Stream: column 3 section LENGTH start: 194210 length 25
+ Stream: column 3 section DICTIONARY_DATA start: 194235 length 133
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DIRECT_V2
@@ -123,17 +123,17 @@ Stripes:
Entry 3: numHashFunctions: 4 bitCount: 6272 popCount: 138 loadFactor: 0.022 expectedFpp: 2.343647E-7
Entry 4: numHashFunctions: 4 bitCount: 6272 popCount: 138 loadFactor: 0.022 expectedFpp: 2.343647E-7
Stripe level merge: numHashFunctions: 4 bitCount: 6272 popCount: 138 loadFactor: 0.022 expectedFpp: 2.343647E-7
- Stripe: offset: 194069 data: 63796 rows: 5000 tail: 86 index: 844
- Stream: column 0 section ROW_INDEX start: 194069 length 17
- Stream: column 1 section ROW_INDEX start: 194086 length 162
- Stream: column 2 section ROW_INDEX start: 194248 length 170
- Stream: column 3 section ROW_INDEX start: 194418 length 91
- Stream: column 3 section BLOOM_FILTER start: 194509 length 404
- Stream: column 1 section DATA start: 194913 length 20029
- Stream: column 2 section DATA start: 214942 length 40035
- Stream: column 3 section DATA start: 254977 length 3574
- Stream: column 3 section LENGTH start: 258551 length 25
- Stream: column 3 section DICTIONARY_DATA start: 258576 length 133
+ Stripe: offset: 194454 data: 63817 rows: 5000 tail: 86 index: 952
+ Stream: column 0 section ROW_INDEX start: 194454 length 17
+ Stream: column 1 section ROW_INDEX start: 194471 length 165
+ Stream: column 2 section ROW_INDEX start: 194636 length 167
+ Stream: column 3 section ROW_INDEX start: 194803 length 91
+ Stream: column 3 section BLOOM_FILTER start: 194894 length 512
+ Stream: column 1 section DATA start: 195406 length 20035
+ Stream: column 2 section DATA start: 215441 length 40050
+ Stream: column 3 section DATA start: 255491 length 3574
+ Stream: column 3 section LENGTH start: 259065 length 25
+ Stream: column 3 section DICTIONARY_DATA start: 259090 length 133
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DIRECT_V2
@@ -151,17 +151,17 @@ Stripes:
Entry 3: numHashFunctions: 4 bitCount: 6272 popCount: 138 loadFactor: 0.022 expectedFpp: 2.343647E-7
Entry 4: numHashFunctions: 4 bitCount: 6272 popCount: 138 loadFactor: 0.022 expectedFpp: 2.343647E-7
Stripe level merge: numHashFunctions: 4 bitCount: 6272 popCount: 138 loadFactor: 0.022 expectedFpp: 2.343647E-7
- Stripe: offset: 258795 data: 12940 rows: 1000 tail: 78 index: 432
- Stream: column 0 section ROW_INDEX start: 258795 length 12
- Stream: column 1 section ROW_INDEX start: 258807 length 38
- Stream: column 2 section ROW_INDEX start: 258845 length 41
- Stream: column 3 section ROW_INDEX start: 258886 length 40
- Stream: column 3 section BLOOM_FILTER start: 258926 length 301
- Stream: column 1 section DATA start: 259227 length 4007
- Stream: column 2 section DATA start: 263234 length 8007
- Stream: column 3 section DATA start: 271241 length 768
- Stream: column 3 section LENGTH start: 272009 length 25
- Stream: column 3 section DICTIONARY_DATA start: 272034 length 133
+ Stripe: offset: 259309 data: 12943 rows: 1000 tail: 78 index: 432
+ Stream: column 0 section ROW_INDEX start: 259309 length 12
+ Stream: column 1 section ROW_INDEX start: 259321 length 38
+ Stream: column 2 section ROW_INDEX start: 259359 length 41
+ Stream: column 3 section ROW_INDEX start: 259400 length 40
+ Stream: column 3 section BLOOM_FILTER start: 259440 length 301
+ Stream: column 1 section DATA start: 259741 length 4007
+ Stream: column 2 section DATA start: 263748 length 8010
+ Stream: column 3 section DATA start: 271758 length 768
+ Stream: column 3 section LENGTH start: 272526 length 25
+ Stream: column 3 section DICTIONARY_DATA start: 272551 length 133
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DIRECT_V2
@@ -172,6 +172,6 @@ Stripes:
Entry 0: numHashFunctions: 4 bitCount: 6272 popCount: 138 loadFactor: 0.022 expectedFpp: 2.343647E-7
Stripe level merge: numHashFunctions: 4 bitCount: 6272 popCount: 138 loadFactor: 0.022 expectedFpp: 2.343647E-7
-File length: 272790 bytes
+File length: 273307 bytes
Padding length: 0 bytes
Padding ratio: 0%
http://git-wip-us.apache.org/repos/asf/hive/blob/51a0c03f/ql/src/test/resources/orc-file-dump-bloomfilter2.out
----------------------------------------------------------------------
diff --git a/ql/src/test/resources/orc-file-dump-bloomfilter2.out b/ql/src/test/resources/orc-file-dump-bloomfilter2.out
index 06b65ce..462d41f 100644
--- a/ql/src/test/resources/orc-file-dump-bloomfilter2.out
+++ b/ql/src/test/resources/orc-file-dump-bloomfilter2.out
@@ -2,7 +2,7 @@ Structure for TestFileDump.testDump.orc
File Version: 0.12 with HIVE_8732
Rows: 21000
Compression: ZLIB
-Compression size: 10000
+Compression size: 4096
Type: struct<i:int,l:bigint,s:string>
Stripe Statistics:
@@ -39,27 +39,27 @@ File Statistics:
Column 3: count: 21000 hasNull: false min: Darkness, max: worst sum: 81761
Stripes:
- Stripe: offset: 3 data: 63765 rows: 5000 tail: 85 index: 6935
+ Stripe: offset: 3 data: 63786 rows: 5000 tail: 85 index: 6974
Stream: column 0 section ROW_INDEX start: 3 length 17
- Stream: column 1 section ROW_INDEX start: 20 length 164
- Stream: column 2 section ROW_INDEX start: 184 length 173
- Stream: column 2 section BLOOM_FILTER start: 357 length 6494
- Stream: column 3 section ROW_INDEX start: 6851 length 87
- Stream: column 1 section DATA start: 6938 length 20029
- Stream: column 2 section DATA start: 26967 length 40035
- Stream: column 3 section DATA start: 67002 length 3543
- Stream: column 3 section LENGTH start: 70545 length 25
- Stream: column 3 section DICTIONARY_DATA start: 70570 length 133
+ Stream: column 1 section ROW_INDEX start: 20 length 166
+ Stream: column 2 section ROW_INDEX start: 186 length 169
+ Stream: column 2 section BLOOM_FILTER start: 355 length 6535
+ Stream: column 3 section ROW_INDEX start: 6890 length 87
+ Stream: column 1 section DATA start: 6977 length 20035
+ Stream: column 2 section DATA start: 27012 length 40050
+ Stream: column 3 section DATA start: 67062 length 3543
+ Stream: column 3 section LENGTH start: 70605 length 25
+ Stream: column 3 section DICTIONARY_DATA start: 70630 length 133
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DIRECT_V2
Encoding column 3: DICTIONARY_V2[35]
Row group indices for column 2:
Entry 0: count: 1000 hasNull: false min: -9200577545527640566 max: 9175500305011173751 positions: 0,0,0
- Entry 1: count: 1000 hasNull: false min: -9203618157670445774 max: 9208123824411178101 positions: 0,4098,488
- Entry 2: count: 1000 hasNull: false min: -9218592812243954469 max: 9221351515892923972 positions: 10003,2294,464
- Entry 3: count: 1000 hasNull: false min: -9206585617947511272 max: 9167703224425685487 positions: 20006,490,440
- Entry 4: count: 1000 hasNull: false min: -9206645795733282496 max: 9221614132680747961 positions: 20006,8686,416
+ Entry 1: count: 1000 hasNull: false min: -9203618157670445774 max: 9208123824411178101 positions: 4099,2,488
+ Entry 2: count: 1000 hasNull: false min: -9218592812243954469 max: 9221351515892923972 positions: 12297,6,464
+ Entry 3: count: 1000 hasNull: false min: -9206585617947511272 max: 9167703224425685487 positions: 20495,10,440
+ Entry 4: count: 1000 hasNull: false min: -9206645795733282496 max: 9221614132680747961 positions: 28693,14,416
Bloom filters for column 2:
Entry 0: numHashFunctions: 7 bitCount: 9600 popCount: 4931 loadFactor: 0.5136 expectedFpp: 0.009432924
Entry 1: numHashFunctions: 7 bitCount: 9600 popCount: 4956 loadFactor: 0.5163 expectedFpp: 0.009772834
@@ -67,27 +67,27 @@ Stripes:
Entry 3: numHashFunctions: 7 bitCount: 9600 popCount: 4971 loadFactor: 0.5178 expectedFpp: 0.009981772
Entry 4: numHashFunctions: 7 bitCount: 9600 popCount: 4949 loadFactor: 0.5155 expectedFpp: 0.009676614
Stripe level merge: numHashFunctions: 7 bitCount: 9600 popCount: 9347 loadFactor: 0.9736 expectedFpp: 0.829482
- Stripe: offset: 70788 data: 63754 rows: 5000 tail: 85 index: 6917
- Stream: column 0 section ROW_INDEX start: 70788 length 17
- Stream: column 1 section ROW_INDEX start: 70805 length 162
- Stream: column 2 section ROW_INDEX start: 70967 length 171
- Stream: column 2 section BLOOM_FILTER start: 71138 length 6484
- Stream: column 3 section ROW_INDEX start: 77622 length 83
- Stream: column 1 section DATA start: 77705 length 20029
- Stream: column 2 section DATA start: 97734 length 40035
- Stream: column 3 section DATA start: 137769 length 3532
- Stream: column 3 section LENGTH start: 141301 length 25
- Stream: column 3 section DICTIONARY_DATA start: 141326 length 133
+ Stripe: offset: 70848 data: 63775 rows: 5000 tail: 85 index: 6965
+ Stream: column 0 section ROW_INDEX start: 70848 length 17
+ Stream: column 1 section ROW_INDEX start: 70865 length 164
+ Stream: column 2 section ROW_INDEX start: 71029 length 168
+ Stream: column 2 section BLOOM_FILTER start: 71197 length 6533
+ Stream: column 3 section ROW_INDEX start: 77730 length 83
+ Stream: column 1 section DATA start: 77813 length 20035
+ Stream: column 2 section DATA start: 97848 length 40050
+ Stream: column 3 section DATA start: 137898 length 3532
+ Stream: column 3 section LENGTH start: 141430 length 25
+ Stream: column 3 section DICTIONARY_DATA start: 141455 length 133
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DIRECT_V2
Encoding column 3: DICTIONARY_V2[35]
Row group indices for column 2:
Entry 0: count: 1000 hasNull: false min: -9218450653857701562 max: 9189819526332228512 positions: 0,0,0
- Entry 1: count: 1000 hasNull: false min: -9220818777591257749 max: 9178821722829648113 positions: 0,4098,488
- Entry 2: count: 1000 hasNull: false min: -9220031433030423388 max: 9210838931786956852 positions: 10003,2294,464
- Entry 3: count: 1000 hasNull: false min: -9208195729739635607 max: 9222259462014003839 positions: 20006,490,440
- Entry 4: count: 1000 hasNull: false min: -9174271499932339698 max: 9212277876771676916 positions: 20006,8686,416
+ Entry 1: count: 1000 hasNull: false min: -9220818777591257749 max: 9178821722829648113 positions: 4099,2,488
+ Entry 2: count: 1000 hasNull: false min: -9220031433030423388 max: 9210838931786956852 positions: 12297,6,464
+ Entry 3: count: 1000 hasNull: false min: -9208195729739635607 max: 9222259462014003839 positions: 20495,10,440
+ Entry 4: count: 1000 hasNull: false min: -9174271499932339698 max: 9212277876771676916 positions: 28693,14,416
Bloom filters for column 2:
Entry 0: numHashFunctions: 7 bitCount: 9600 popCount: 4971 loadFactor: 0.5178 expectedFpp: 0.009981772
Entry 1: numHashFunctions: 7 bitCount: 9600 popCount: 4988 loadFactor: 0.5196 expectedFpp: 0.010223193
@@ -95,27 +95,27 @@ Stripes:
Entry 3: numHashFunctions: 7 bitCount: 9600 popCount: 4962 loadFactor: 0.5169 expectedFpp: 0.009855959
Entry 4: numHashFunctions: 7 bitCount: 9600 popCount: 4966 loadFactor: 0.5173 expectedFpp: 0.009911705
Stripe level merge: numHashFunctions: 7 bitCount: 9600 popCount: 9344 loadFactor: 0.9733 expectedFpp: 0.8276205
- Stripe: offset: 141544 data: 63766 rows: 5000 tail: 85 index: 6924
- Stream: column 0 section ROW_INDEX start: 141544 length 17
- Stream: column 1 section ROW_INDEX start: 141561 length 159
- Stream: column 2 section ROW_INDEX start: 141720 length 171
- Stream: column 2 section BLOOM_FILTER start: 141891 length 6487
- Stream: column 3 section ROW_INDEX start: 148378 length 90
- Stream: column 1 section DATA start: 148468 length 20029
- Stream: column 2 section DATA start: 168497 length 40035
- Stream: column 3 section DATA start: 208532 length 3544
- Stream: column 3 section LENGTH start: 212076 length 25
- Stream: column 3 section DICTIONARY_DATA start: 212101 length 133
+ Stripe: offset: 141673 data: 63787 rows: 5000 tail: 85 index: 6971
+ Stream: column 0 section ROW_INDEX start: 141673 length 17
+ Stream: column 1 section ROW_INDEX start: 141690 length 163
+ Stream: column 2 section ROW_INDEX start: 141853 length 168
+ Stream: column 2 section BLOOM_FILTER start: 142021 length 6533
+ Stream: column 3 section ROW_INDEX start: 148554 length 90
+ Stream: column 1 section DATA start: 148644 length 20035
+ Stream: column 2 section DATA start: 168679 length 40050
+ Stream: column 3 section DATA start: 208729 length 3544
+ Stream: column 3 section LENGTH start: 212273 length 25
+ Stream: column 3 section DICTIONARY_DATA start: 212298 length 133
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DIRECT_V2
Encoding column 3: DICTIONARY_V2[35]
Row group indices for column 2:
Entry 0: count: 1000 hasNull: false min: -9211978436552246208 max: 9179058898902097152 positions: 0,0,0
- Entry 1: count: 1000 hasNull: false min: -9195645160817780503 max: 9189147759444307708 positions: 0,4098,488
- Entry 2: count: 1000 hasNull: false min: -9202888157616520823 max: 9193561362676960747 positions: 10003,2294,464
- Entry 3: count: 1000 hasNull: false min: -9216318198067839390 max: 9221286760675829363 positions: 20006,490,440
- Entry 4: count: 1000 hasNull: false min: -9218342074710552826 max: 9222303228623055266 positions: 20006,8686,416
+ Entry 1: count: 1000 hasNull: false min: -9195645160817780503 max: 9189147759444307708 positions: 4099,2,488
+ Entry 2: count: 1000 hasNull: false min: -9202888157616520823 max: 9193561362676960747 positions: 12297,6,464
+ Entry 3: count: 1000 hasNull: false min: -9216318198067839390 max: 9221286760675829363 positions: 20495,10,440
+ Entry 4: count: 1000 hasNull: false min: -9218342074710552826 max: 9222303228623055266 positions: 28693,14,416
Bloom filters for column 2:
Entry 0: numHashFunctions: 7 bitCount: 9600 popCount: 4967 loadFactor: 0.5174 expectedFpp: 0.009925688
Entry 1: numHashFunctions: 7 bitCount: 9600 popCount: 5002 loadFactor: 0.521 expectedFpp: 0.01042575
@@ -123,27 +123,27 @@ Stripes:
Entry 3: numHashFunctions: 7 bitCount: 9600 popCount: 4943 loadFactor: 0.5149 expectedFpp: 0.009594797
Entry 4: numHashFunctions: 7 bitCount: 9600 popCount: 4930 loadFactor: 0.5135 expectedFpp: 0.009419539
Stripe level merge: numHashFunctions: 7 bitCount: 9600 popCount: 9333 loadFactor: 0.9722 expectedFpp: 0.82082444
- Stripe: offset: 212319 data: 63796 rows: 5000 tail: 85 index: 6925
- Stream: column 0 section ROW_INDEX start: 212319 length 17
- Stream: column 1 section ROW_INDEX start: 212336 length 162
- Stream: column 2 section ROW_INDEX start: 212498 length 170
- Stream: column 2 section BLOOM_FILTER start: 212668 length 6485
- Stream: column 3 section ROW_INDEX start: 219153 length 91
- Stream: column 1 section DATA start: 219244 length 20029
- Stream: column 2 section DATA start: 239273 length 40035
- Stream: column 3 section DATA start: 279308 length 3574
- Stream: column 3 section LENGTH start: 282882 length 25
- Stream: column 3 section DICTIONARY_DATA start: 282907 length 133
+ Stripe: offset: 212516 data: 63817 rows: 5000 tail: 85 index: 6964
+ Stream: column 0 section ROW_INDEX start: 212516 length 17
+ Stream: column 1 section ROW_INDEX start: 212533 length 165
+ Stream: column 2 section ROW_INDEX start: 212698 length 167
+ Stream: column 2 section BLOOM_FILTER start: 212865 length 6524
+ Stream: column 3 section ROW_INDEX start: 219389 length 91
+ Stream: column 1 section DATA start: 219480 length 20035
+ Stream: column 2 section DATA start: 239515 length 40050
+ Stream: column 3 section DATA start: 279565 length 3574
+ Stream: column 3 section LENGTH start: 283139 length 25
+ Stream: column 3 section DICTIONARY_DATA start: 283164 length 133
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DIRECT_V2
Encoding column 3: DICTIONARY_V2[35]
Row group indices for column 2:
Entry 0: count: 1000 hasNull: false min: -9222731174895935707 max: 9214167447015056056 positions: 0,0,0
- Entry 1: count: 1000 hasNull: false min: -9222758097219661129 max: 9221043130193737406 positions: 0,4098,488
- Entry 2: count: 1000 hasNull: false min: -9174483776261243438 max: 9208134757538374043 positions: 10003,2294,464
- Entry 3: count: 1000 hasNull: false min: -9174329712613510612 max: 9197412874152820822 positions: 20006,490,440
- Entry 4: count: 1000 hasNull: false min: -9221162005892422758 max: 9220625004936875965 positions: 20006,8686,416
+ Entry 1: count: 1000 hasNull: false min: -9222758097219661129 max: 9221043130193737406 positions: 4099,2,488
+ Entry 2: count: 1000 hasNull: false min: -9174483776261243438 max: 9208134757538374043 positions: 12297,6,464
+ Entry 3: count: 1000 hasNull: false min: -9174329712613510612 max: 9197412874152820822 positions: 20495,10,440
+ Entry 4: count: 1000 hasNull: false min: -9221162005892422758 max: 9220625004936875965 positions: 28693,14,416
Bloom filters for column 2:
Entry 0: numHashFunctions: 7 bitCount: 9600 popCount: 4951 loadFactor: 0.5157 expectedFpp: 0.009704026
Entry 1: numHashFunctions: 7 bitCount: 9600 popCount: 4969 loadFactor: 0.5176 expectedFpp: 0.009953696
@@ -151,17 +151,17 @@ Stripes:
Entry 3: numHashFunctions: 7 bitCount: 9600 popCount: 4941 loadFactor: 0.5147 expectedFpp: 0.009567649
Entry 4: numHashFunctions: 7 bitCount: 9600 popCount: 4993 loadFactor: 0.5201 expectedFpp: 0.010295142
Stripe level merge: numHashFunctions: 7 bitCount: 9600 popCount: 9353 loadFactor: 0.9743 expectedFpp: 0.8332165
- Stripe: offset: 283125 data: 12940 rows: 1000 tail: 78 index: 1468
- Stream: column 0 section ROW_INDEX start: 283125 length 12
- Stream: column 1 section ROW_INDEX start: 283137 length 38
- Stream: column 2 section ROW_INDEX start: 283175 length 41
- Stream: column 2 section BLOOM_FILTER start: 283216 length 1337
- Stream: column 3 section ROW_INDEX start: 284553 length 40
- Stream: column 1 section DATA start: 284593 length 4007
- Stream: column 2 section DATA start: 288600 length 8007
- Stream: column 3 section DATA start: 296607 length 768
- Stream: column 3 section LENGTH start: 297375 length 25
- Stream: column 3 section DICTIONARY_DATA start: 297400 length 133
+ Stripe: offset: 283382 data: 12943 rows: 1000 tail: 78 index: 1468
+ Stream: column 0 section ROW_INDEX start: 283382 length 12
+ Stream: column 1 section ROW_INDEX start: 283394 length 38
+ Stream: column 2 section ROW_INDEX start: 283432 length 41
+ Stream: column 2 section BLOOM_FILTER start: 283473 length 1337
+ Stream: column 3 section ROW_INDEX start: 284810 length 40
+ Stream: column 1 section DATA start: 284850 length 4007
+ Stream: column 2 section DATA start: 288857 length 8010
+ Stream: column 3 section DATA start: 296867 length 768
+ Stream: column 3 section LENGTH start: 297635 length 25
+ Stream: column 3 section DICTIONARY_DATA start: 297660 length 133
Encoding column 0: DIRECT
Encoding column 1: DIRECT_V2
Encoding column 2: DIRECT_V2
@@ -172,6 +172,6 @@ Stripes:
Entry 0: numHashFunctions: 7 bitCount: 9600 popCount: 4948 loadFactor: 0.5154 expectedFpp: 0.00966294
Stripe level merge: numHashFunctions: 7 bitCount: 9600 popCount: 4948 loadFactor: 0.5154 expectedFpp: 0.00966294
-File length: 298155 bytes
+File length: 298416 bytes
Padding length: 0 bytes
Padding ratio: 0%