You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by "anujphadke (Code Review)" <ge...@cloudera.org> on 2016/06/24 07:22:22 UTC
[Impala-CR](cdh5-trunk) IMPALA-3766: Applying LZ4 compression on buffers before spilling
anujphadke has uploaded a new change for review.
http://gerrit.cloudera.org:8080/3478
Change subject: IMPALA-3766: Applying LZ4 compression on buffers before spilling
......................................................................
IMPALA-3766: Applying LZ4 compression on buffers before spilling
Evalution patch to compress buffers before spilling it to disk
Change-Id: I4d49bd8d6d7643c84cefd1274c18b52907ca1488
(cherry picked from commit 08568166be91665f3b924675cd3fb8044690fcb2)
---
M be/src/runtime/buffered-block-mgr.cc
M be/src/runtime/buffered-block-mgr.h
2 files changed, 32 insertions(+), 7 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/78/3478/1
--
To view, visit http://gerrit.cloudera.org:8080/3478
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I4d49bd8d6d7643c84cefd1274c18b52907ca1488
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: anujphadke <ap...@cloudera.com>
[Impala-CR](cdh5-trunk) IMPALA-3766: Applying LZ4 compression on buffers before spilling
Posted by "anujphadke (Code Review)" <ge...@cloudera.org>.
anujphadke has posted comments on this change.
Change subject: IMPALA-3766: Applying LZ4 compression on buffers before spilling
......................................................................
Patch Set 2:
Yes /tmp is mounted on ssd
anuj@anuj-OptiPlex-9020:~/Impala$ df -T /tmp
Filesystem Type 1K-blocks Used Available Use% Mounted on
/dev/disk/by-uuid/a617ba28-de21-4166-af72-c55f6236c3f5 ext4 951445500 663818924 239272888 74% /
anuj@anuj-OptiPlex-9020:~/Impala$ sudo hdparm -I /dev/disk/by-uuid/a617ba28-de21-4166-af72-c55f6236c3f5
/dev/disk/by-uuid/a617ba28-de21-4166-af72-c55f6236c3f5:
ATA device, with non-removable media
Model Number: Samsung SSD 850 PRO 1TB
Serial Number: S252NSAG410674W
Firmware Revision: EXM02B6Q
Transport: Serial, ATA8-AST, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
--
To view, visit http://gerrit.cloudera.org:8080/3478
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d49bd8d6d7643c84cefd1274c18b52907ca1488
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: anujphadke <ap...@cloudera.com>
Gerrit-Reviewer: Mostafa Mokhtar <mm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: anujphadke <ap...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-trunk) IMPALA-3766: Applying LZ4 compression on buffers before spilling
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change.
Change subject: IMPALA-3766: Applying LZ4 compression on buffers before spilling
......................................................................
Patch Set 2:
Didn't realise the ball was in my court on this one. It looks like this is going to be beneficial in some circumstances but not others. We may need to think through if/when to enable this.
It would be interesting if we could benchmark this in a way that we were actually reading and writing to a spinning disk. That's when we'd expect to see a benefit. I think there are two things going on with the benchmark:
* < 1gb of spilled data is almost certainly going to fit in the OS buffer cache, so we're not going to see much disk I/O on the critical path.
* Disks and SSDs have different characteristics and trade-offs for spilling.
To elaborate on the second point: on spinning disks, I/O is slow, so we will probably see some perf benefit. On SSDs, I suspect compression will be slower than the I/O. However, space on SSDs is scarce so it's probably worth compressing the data just to save space even if we take a performance hit.
--
To view, visit http://gerrit.cloudera.org:8080/3478
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d49bd8d6d7643c84cefd1274c18b52907ca1488
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: anujphadke <ap...@cloudera.com>
Gerrit-Reviewer: Mostafa Mokhtar <mm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: anujphadke <ap...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-trunk) IMPALA-3766: Applying LZ4 compression on buffers before spilling
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change.
Change subject: IMPALA-3766: Applying LZ4 compression on buffers before spilling
......................................................................
Patch Set 1:
It would be good to get some numbers on how this compares to the uncompressed version in terms of bytes written to disk and end-to-end performance, I think we should get an idea of the gains before doing the detailed review.
* Total uncompressed bytes versus compressed bytes written - you could add an extra profile counter for this since we already have bytes_written_counter_.
* End-to-end query time on a release build.
* Disk space saved. This is a little tricky since we allocate 8mb file ranges, but don't write the whole 8mb. The files will be sparse so we will save disk space, but you need to do something like this to see it: http://extrabright.com/blog/2010/03/30/how-to-know-if-a-file-on-linux-is-sparse/
--
To view, visit http://gerrit.cloudera.org:8080/3478
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d49bd8d6d7643c84cefd1274c18b52907ca1488
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: anujphadke <ap...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: anujphadke <ap...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-trunk) IMPALA-3766: Applying LZ4 compression on buffers before spilling
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change.
Change subject: IMPALA-3766: Applying LZ4 compression on buffers before spilling
......................................................................
Patch Set 2:
This is interesting, I would have expected spilling was limited by I/O speed, but based on this it's probably more limited by compute. Do you know what kind of device your /tmp is mounted on? Is it an SSD.
--
To view, visit http://gerrit.cloudera.org:8080/3478
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d49bd8d6d7643c84cefd1274c18b52907ca1488
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: anujphadke <ap...@cloudera.com>
Gerrit-Reviewer: Mostafa Mokhtar <mm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: anujphadke <ap...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-trunk) IMPALA-3766: Applying LZ4 compression on buffers before spilling
Posted by "anujphadke (Code Review)" <ge...@cloudera.org>.
anujphadke has posted comments on this change.
Change subject: IMPALA-3766: Applying LZ4 compression on buffers before spilling
......................................................................
Patch Set 2:
Spilling (with compression)
Fetched 6001215 row(s) in 1747.67s
BytesWritten: 944.32 MB (990188382)
CompressedBytesWritten: 437.09 MB (458321721)
Spilling (without compression)
Total time - 21min 991ms (roughly 1680 secs)
--
To view, visit http://gerrit.cloudera.org:8080/3478
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d49bd8d6d7643c84cefd1274c18b52907ca1488
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: anujphadke <ap...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: anujphadke <ap...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-trunk) IMPALA-3766: Applying LZ4 compression on buffers before spilling
Posted by "anujphadke (Code Review)" <ge...@cloudera.org>.
anujphadke has posted comments on this change.
Change subject: IMPALA-3766: Applying LZ4 compression on buffers before spilling
......................................................................
Patch Set 1:
Will add a better commit message.
This is a evaluation patch.
--
To view, visit http://gerrit.cloudera.org:8080/3478
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d49bd8d6d7643c84cefd1274c18b52907ca1488
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: anujphadke <ap...@cloudera.com>
Gerrit-Reviewer: anujphadke <ap...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-trunk) IMPALA-3766: Applying LZ4 compression on buffers before spilling
Posted by "anujphadke (Code Review)" <ge...@cloudera.org>.
anujphadke has posted comments on this change.
Change subject: IMPALA-3766: Applying LZ4 compression on buffers before spilling
......................................................................
Patch Set 2:
Before -
[localhost:21000] > profile;
Query Runtime Profile:
Query (id=a348adf5718dd795:e38675e8e7ea1a94):
Summary:
Session ID: 624487d98d041692:7ce44c89f323fc8b
Session Type: BEESWAX
Start Time: 2016-07-07 15:44:59.063728000
End Time: 2016-07-07 16:50:19.658784000
Query Type: QUERY
Query State: FINISHED
Query Status: OK
Impala Version: impalad version 2.7.0-cdh5-INTERNAL DEBUG (build d43a6771b7a892df2bab8d62c25ec8154ac9fab4)
User: anuj
Connected User: anuj
Delegated User:
Network Address: ::ffff:127.0.0.1:59531
Default Db: tpch
Sql Statement: select * from lineitem order by l_quantity
Coordinator: anuj-OptiPlex-9020:22000
Query Options (non default): MEM_LIMIT=1073741824
Plan:
----------------
Estimated Per-Host Requirements: Memory=864.00MB VCores=1
02:MERGING-EXCHANGE [UNPARTITIONED]
| order by: l_quantity ASC
| hosts=1 per-host-mem=unavailable
| tuple-ids=1 row-size=263B cardinality=6001215
|
01:SORT
| order by: l_quantity ASC
| hosts=1 per-host-mem=160.00MB
| tuple-ids=1 row-size=263B cardinality=6001215
|
00:SCAN HDFS [tpch.lineitem, RANDOM]
partitions=1/1 files=1 size=718.94MB
table stats: 6001215 rows total
column stats: all
hosts=1 per-host-mem=704.00MB
tuple-ids=0 row-size=263B cardinality=6001215
----------------
Estimated Per-Host Mem: 905969664
Estimated Per-Host VCores: 1
Request Pool: default-pool
Admission result: Admitted immediately
ExecSummary:
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
---------------------------------------------------------------------------------------------------------------
02:MERGING-EXCHANGE 1 1s020ms 1s020ms 6.00M 6.00M 0 -1.00 B UNPARTITIONED
01:SORT 1 37s063ms 37s063ms 6.00M 6.00M 816.20 MB 160.00 MB
00:SCAN HDFS 1 344.851ms 344.851ms 6.00M 6.00M 201.39 MB 704.00 MB tpch.lineitem
Planner Timeline: 105.297ms
- Analysis finished: 11.531ms (11.531ms)
- Equivalence classes computed: 31.690ms (20.159ms)
- Single node plan created: 45.135ms (13.444ms)
- Runtime filters computed: 46.123ms (988.831us)
- Distributed plan created: 51.654ms (5.530ms)
- Planning finished: 105.297ms (53.643ms)
Query Timeline: 1h5m
- Start execution: 40.528us (40.528us)
- Planning finished: 5s618ms (5s618ms)
- Submit for admission: 5s620ms (2.164ms)
- Completed admission: 5s620ms (146.718us)
- Ready to start 1 remote fragments: 5s621ms (714.555us)
- All 1 remote fragments started: 5s622ms (1.010ms)
- Rows available: 31s511ms (25s888ms)
- First row fetched: 31s680ms (169.455ms)
- Unregister query: 1h5m (1h4m)
ImpalaServer:
- ClientFetchWaitTimer: 1h4m
- RowMaterializationTimer: 34s415ms
Execution Profile a348adf5718dd795:e38675e8e7ea1a94:(Total: 27s454ms, non-child: 0.000ns, % non-child: 0.00%)
Number of filters: 0
Filter routing table:
ID Src. Node Tgt. Node(s) Targets Target type Partition filter Pending (Expected) First arrived Completed
-------------------------------------------------------------------------------------------------------------------
Fragment start latencies: Count: 1, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 0, 90th %-ile: 0, 95th %-ile: 0, 99.9th %-ile: 0
Per Node Peak Memory Usage: anuj-OptiPlex-9020:22000(978.74 MB)
- FiltersReceived: 0 (0)
- FinalizationTimer: 0.000ns
Coordinator Fragment F01:(Total: 26s923ms, non-child: 14.099ms, % non-child: 0.05%)
MemoryUsage(1m4s): 12.26 MB, 20.12 MB, 22.13 MB, 19.77 MB, 21.58 MB, 20.77 MB, 20.43 MB, 21.78 MB, 19.75 MB, 22.03 MB, 20.41 MB, 20.87 MB, 21.66 MB, 19.83 MB, 22.03 MB, 20.11 MB, 21.33 MB, 21.15 MB, 20.08 MB, 22.14 MB, 19.90 MB, 21.69 MB, 20.84 MB, 20.45 MB, 22.01 MB, 19.73 MB, 22.11 MB, 20.45 MB, 20.92 MB, 21.70 MB, 19.84 MB, 22.17 MB, 20.05 MB, 21.29 MB, 21.27 MB, 20.08 MB, 21.99 MB, 19.89 MB, 21.69 MB, 20.88 MB, 20.44 MB, 22.03 MB, 19.77 MB, 22.00 MB, 20.53 MB, 20.71 MB, 21.70 MB, 19.89 MB, 22.15 MB, 20.20 MB, 21.22 MB, 21.39 MB, 19.93 MB, 22.16 MB, 19.93 MB, 21.56 MB, 20.96 MB, 20.34 MB, 22.05 MB, 19.71 MB, 15.62 MB
- AverageThreadTokens: 0.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 22.55 MB (23643536)
- PerHostPeakMemUsage: 0
- PrepareTime: 43.673us
- RowsProduced: 6.00M (6001215)
- TotalCpuTime: 1h5m
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 0.000ns
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 219 (219)
- BlocksRecycled: 32 (32)
- BufferedPins: 74 (74)
- BytesWritten: 944.31 MB (990183996)
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 819.20 MB (858993472)
- PeakMemoryUsage: 816.00 MB (855638016)
- TotalBufferWaitTime: 0.000ns
- TotalEncryptionTime: 0.000ns
- TotalIntegrityCheckTime: 0.000ns
- TotalReadBlockTime: 485.948ms
EXCHANGE_NODE (id=2):(Total: 26s909ms, non-child: 1s020ms, % non-child: 3.79%)
BytesReceived(1m4s): 8.15 MB, 23.92 MB, 36.90 MB, 48.93 MB, 62.21 MB, 74.37 MB, 87.28 MB, 100.01 MB, 112.26 MB, 125.53 MB, 137.54 MB, 150.62 MB, 163.13 MB, 175.50 MB, 188.63 MB, 200.69 MB, 213.86 MB, 226.13 MB, 238.83 MB, 251.80 MB, 263.83 MB, 277.07 MB, 289.19 MB, 302.03 MB, 314.76 MB, 326.91 MB, 340.17 MB, 352.13 MB, 365.15 MB, 377.61 MB, 389.96 MB, 403.11 MB, 415.06 MB, 428.29 MB, 440.57 MB, 453.15 MB, 466.12 MB, 478.23 MB, 491.48 MB, 503.60 MB, 516.43 MB, 529.18 MB, 541.29 MB, 554.54 MB, 566.55 MB, 579.55 MB, 592.15 MB, 604.48 MB, 617.63 MB, 629.62 MB, 642.76 MB, 655.07 MB, 667.58 MB, 680.65 MB, 692.70 MB, 705.94 MB, 718.11 MB, 730.91 MB, 743.72 MB, 755.81 MB, 766.16 MB
- BytesReceived: 766.85 MB (804096787)
- ConvertRowBatchTime: 0.000ns
- DeserializeRowBatchTimer: 3s974ms
- FirstBatchArrivalWaitTime: 25s888ms
- MergeGetNext: 997.745ms
- MergeGetNextBatch: 64.131ms
- PeakMemoryUsage: 0
- RowsReturned: 6.00M (6001215)
- RowsReturnedRate: 223.02 K/sec
- SendersBlockedTimer: 1h3m
- SendersBlockedTotalTimer(*): 1h3m
Averaged Fragment F00:(Total: 1h4m, non-child: 1h3m, % non-child: 98.77%)
split sizes: min: 718.94 MB, max: 718.94 MB, avg: 718.94 MB, stddev: 0
completion times: min:1h4m max:1h4m mean: 1h4m stddev:0.000ns
execution rates: min:190.64 KB/sec max:190.64 KB/sec mean:190.64 KB/sec stddev:0.00 /sec
num instances: 1
- AverageThreadTokens: 1.03
- BloomFilterBytes: 0
- PeakMemoryUsage: 969.24 MB (1016316584)
- PerHostPeakMemUsage: 978.74 MB (1026286248)
- PrepareTime: 143.403ms
- RowsProduced: 6.00M (6001215)
- TotalCpuTime: 2m42s
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1h3m
- TotalStorageWaitTime: 1s124ms
CodeGen:(Total: 212.028ms, non-child: 212.028ms, % non-child: 100.00%)
- CodegenTime: 1.478ms
- CompileTime: 6.577ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.12 MB (2224192)
- NumFunctions: 29 (29)
- NumInstructions: 529 (529)
- OptimizationTime: 74.084ms
- PrepareTime: 126.830ms
DataStreamSender (dst_id=2):(Total: 9s711ms, non-child: 9s711ms, % non-child: 100.00%)
- BytesSent: 766.85 MB (804096787)
- NetworkThroughput(*): 1.04 GB/sec
- OverallThroughput: 78.97 MB/sec
- PeakMemoryUsage: 680.00 B (680)
- RowsReturned: 6.00M (6001215)
- SerializeBatchTime: 8s992ms
- TransmitDataRPCTime: 717.249ms
- UncompressedRowBatchSize: 1.51 GB (1625021116)
SORT_NODE (id=1):(Total: 37s408ms, non-child: 37s063ms, % non-child: 99.08%)
- InMemorySortTime: 6s488ms
- InitialRunsCreated: 2 (2)
- MergeGetNext: 11s580ms
- MergeGetNextBatch: 4s490ms
- PeakMemoryUsage: 816.20 MB (855842816)
- RowsReturned: 6.00M (6001215)
- RowsReturnedRate: 160.42 K/sec
- SortDataSize: 1.49 GB (1600992812)
- SpilledRuns: 3 (3)
- TotalMergesPerformed: 1 (1)
HDFS_SCAN_NODE (id=0):(Total: 344.851ms, non-child: 344.851ms, % non-child: 100.00%)
- AverageHdfsReadThreadConcurrency: 0.10
- AverageScannerThreadConcurrency: 5.76
- BytesRead: 718.94 MB (753867192)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 718.94 MB (753867192)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 718.94 MB (753867192)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumDisksAccessed: 2 (2)
- NumScannerThreadsStarted: 6 (6)
- PeakMemoryUsage: 201.39 MB (211169280)
- PerReadThreadRawHdfsThroughput: 192.58 MB/sec
- RemoteScanRanges: 0 (0)
- RowsRead: 6.00M (6001215)
- RowsReturned: 6.00M (6001215)
- RowsReturnedRate: 17.40 M/sec
- ScanRangesComplete: 6 (6)
- ScannerThreadsInvoluntaryContextSwitches: 371 (371)
- ScannerThreadsTotalWallClockTime: 1m57s
- DelimiterParseTime: 3s697ms
- MaterializeTupleTime(*): 6s048ms
- ScannerThreadsSysTime: 35.892ms
- ScannerThreadsUserTime: 9s863ms
- ScannerThreadsVoluntaryContextSwitches: 5.90K (5903)
- TotalRawHdfsReadTime(*): 3s733ms
- TotalReadThroughput: 35.07 MB/sec
Fragment F00:
Instance a348adf5718dd795:e38675e8e7ea1a96 (host=anuj-OptiPlex-9020:22000):(Total: 1h4m, non-child: 1h3m, % non-child: 98.77%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:6/718.94 MB
MemoryUsage(1m4s): 270.90 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.58 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.51 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 28.77 MB
ThreadUsage(1m4s): 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 1.03
- BloomFilterBytes: 0
- PeakMemoryUsage: 969.24 MB (1016316584)
- PerHostPeakMemUsage: 978.74 MB (1026286248)
- PrepareTime: 143.403ms
- RowsProduced: 6.00M (6001215)
- TotalCpuTime: 2m42s
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1h3m
- TotalStorageWaitTime: 1s124ms
CodeGen:(Total: 212.028ms, non-child: 212.028ms, % non-child: 100.00%)
- CodegenTime: 1.478ms
- CompileTime: 6.577ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.12 MB (2224192)
- NumFunctions: 29 (29)
- NumInstructions: 529 (529)
- OptimizationTime: 74.084ms
- PrepareTime: 126.830ms
DataStreamSender (dst_id=2):(Total: 9s711ms, non-child: 9s711ms, % non-child: 100.00%)
- BytesSent: 766.85 MB (804096787)
- NetworkThroughput(*): 1.04 GB/sec
- OverallThroughput: 78.97 MB/sec
- PeakMemoryUsage: 680.00 B (680)
- RowsReturned: 6.00M (6001215)
- SerializeBatchTime: 8s992ms
- TransmitDataRPCTime: 717.249ms
- UncompressedRowBatchSize: 1.51 GB (1625021116)
SORT_NODE (id=1):(Total: 37s408ms, non-child: 37s063ms, % non-child: 99.08%)
ExecOption: Codegen Enabled
- InMemorySortTime: 6s488ms
- InitialRunsCreated: 2 (2)
- MergeGetNext: 11s580ms
- MergeGetNextBatch: 4s490ms
- PeakMemoryUsage: 816.20 MB (855842816)
- RowsReturned: 6.00M (6001215)
- RowsReturnedRate: 160.42 K/sec
- SortDataSize: 1.49 GB (1600992812)
- SpilledRuns: 3 (3)
- TotalMergesPerformed: 1 (1)
HDFS_SCAN_NODE (id=0):(Total: 344.851ms, non-child: 344.851ms, % non-child: 100.00%)
ExecOption: Expr Evaluation Codegen Disabled, Codegen enabled: 0 out of 6
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:6/718.94 MB
Hdfs Read Thread Concurrency Bucket: 0:92.68% 1:4.878% 2:2.439% 3:0% 4:0%
File Formats: TEXT/NONE:6
BytesRead(500.000ms): 0, 192.00 MB, 192.00 MB, 192.00 MB, 200.00 MB, 240.00 MB, 248.00 MB, 288.00 MB, 296.00 MB, 336.00 MB, 344.00 MB, 384.00 MB, 392.00 MB, 432.00 MB, 448.00 MB, 478.94 MB, 486.94 MB, 486.94 MB, 486.94 MB, 486.94 MB, 486.94 MB, 486.94 MB, 486.94 MB, 486.94 MB, 486.94 MB, 486.94 MB, 486.94 MB, 486.94 MB, 486.94 MB, 494.94 MB, 542.94 MB, 550.94 MB, 590.94 MB, 630.94 MB, 638.94 MB, 670.94 MB, 702.94 MB, 710.94 MB, 718.94 MB, 718.94 MB, 718.94 MB
- AverageHdfsReadThreadConcurrency: 0.10
- AverageScannerThreadConcurrency: 5.76
- BytesRead: 718.94 MB (753867192)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 718.94 MB (753867192)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 718.94 MB (753867192)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumDisksAccessed: 2 (2)
- NumScannerThreadsStarted: 6 (6)
- PeakMemoryUsage: 201.39 MB (211169280)
- PerReadThreadRawHdfsThroughput: 192.58 MB/sec
- RemoteScanRanges: 0 (0)
- RowsRead: 6.00M (6001215)
- RowsReturned: 6.00M (6001215)
- RowsReturnedRate: 17.40 M/sec
- ScanRangesComplete: 6 (6)
- ScannerThreadsInvoluntaryContextSwitches: 371 (371)
- ScannerThreadsTotalWallClockTime: 1m57s
- DelimiterParseTime: 3s697ms
- MaterializeTupleTime(*): 6s048ms
- ScannerThreadsSysTime: 35.892ms
- ScannerThreadsUserTime: 9s863ms
- ScannerThreadsVoluntaryContextSwitches: 5.90K (5903)
- TotalRawHdfsReadTime(*): 3s733ms
- TotalReadThroughput: 35.07 MB/sec
After -
[localhost:21000] > profile;
Query Runtime Profile:
Query (id=394ee7185b41be23:949a047c1e09ec82):
Summary:
Session ID: 84ece9f0c0c9e06:f6c333f0e5737cb5
Session Type: BEESWAX
Start Time: 2016-07-07 12:43:01.225510000
End Time: 2016-07-07 13:48:28.978336000
Query Type: QUERY
Query State: FINISHED
Query Status: OK
Impala Version: impalad version 2.7.0-cdh5-INTERNAL DEBUG (build 34e49668e40386c7e50f87f01e4c5ad926983717)
User: anuj
Connected User: anuj
Delegated User:
Network Address: ::ffff:127.0.0.1:57170
Default Db: tpch
Sql Statement: select * from lineitem order by l_quantity
Coordinator: anuj-OptiPlex-9020:22000
Query Options (non default): MEM_LIMIT=1073741824
Plan:
----------------
Estimated Per-Host Requirements: Memory=864.00MB VCores=1
02:MERGING-EXCHANGE [UNPARTITIONED]
| order by: l_quantity ASC
| hosts=1 per-host-mem=unavailable
| tuple-ids=1 row-size=263B cardinality=6001215
|
01:SORT
| order by: l_quantity ASC
| hosts=1 per-host-mem=160.00MB
| tuple-ids=1 row-size=263B cardinality=6001215
|
00:SCAN HDFS [tpch.lineitem, RANDOM]
partitions=1/1 files=1 size=718.94MB
table stats: 6001215 rows total
column stats: all
hosts=1 per-host-mem=704.00MB
tuple-ids=0 row-size=263B cardinality=6001215
----------------
Estimated Per-Host Mem: 905969664
Estimated Per-Host VCores: 1
Request Pool: default-pool
Admission result: Admitted immediately
ExecSummary:
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
---------------------------------------------------------------------------------------------------------------
02:MERGING-EXCHANGE 1 1s021ms 1s021ms 6.00M 6.00M 0 -1.00 B UNPARTITIONED
01:SORT 1 41s010ms 41s010ms 6.00M 6.00M 816.20 MB 160.00 MB
00:SCAN HDFS 1 412.149ms 412.149ms 6.00M 6.00M 201.76 MB 704.00 MB tpch.lineitem
Planner Timeline: 116.792ms
- Analysis finished: 11.939ms (11.939ms)
- Equivalence classes computed: 32.413ms (20.473ms)
- Single node plan created: 46.386ms (13.973ms)
- Runtime filters computed: 47.696ms (1.310ms)
- Distributed plan created: 54.638ms (6.942ms)
- Planning finished: 116.792ms (62.153ms)
Query Timeline: 1h5m
- Start execution: 43.421us (43.421us)
- Planning finished: 4s826ms (4s826ms)
- Submit for admission: 4s826ms (261.600us)
- Completed admission: 4s826ms (149.430us)
- Ready to start 1 remote fragments: 4s828ms (1.831ms)
- All 1 remote fragments started: 4s829ms (1.125ms)
- Rows available: 33s993ms (29s163ms)
- First row fetched: 34s394ms (401.447ms)
- Unregister query: 1h5m (1h4m)
ImpalaServer:
- ClientFetchWaitTimer: 1h4m
- RowMaterializationTimer: 34s291ms
Execution Profile 394ee7185b41be23:949a047c1e09ec82:(Total: 30s738ms, non-child: 0.000ns, % non-child: 0.00%)
Number of filters: 0
Filter routing table:
ID Src. Node Tgt. Node(s) Targets Target type Partition filter Pending (Expected) First arrived Completed
-------------------------------------------------------------------------------------------------------------------
Fragment start latencies: Count: 1, 25th %-ile: 1ms, 50th %-ile: 1ms, 75th %-ile: 1ms, 90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th %-ile: 1ms
Per Node Peak Memory Usage: anuj-OptiPlex-9020:22000(970.91 MB)
- FiltersReceived: 0 (0)
- FinalizationTimer: 0.000ns
Coordinator Fragment F01:(Total: 30s200ms, non-child: 15.938ms, % non-child: 0.05%)
MemoryUsage(1m4s): 10.99 MB, 19.34 MB, 22.10 MB, 19.26 MB, 21.25 MB, 20.63 MB, 19.67 MB, 21.83 MB, 19.01 MB, 21.66 MB, 20.05 MB, 20.10 MB, 21.68 MB, 19.01 MB, 22.03 MB, 19.70 MB, 20.70 MB, 21.08 MB, 19.28 MB, 22.10 MB, 19.27 MB, 21.21 MB, 20.80 MB, 19.59 MB, 21.98 MB, 19.02 MB, 21.66 MB, 20.18 MB, 20.09 MB, 21.72 MB, 18.95 MB, 21.94 MB, 19.71 MB, 20.50 MB, 21.30 MB, 19.15 MB, 22.05 MB, 19.39 MB, 21.13 MB, 20.86 MB, 19.51 MB, 21.99 MB, 19.07 MB, 21.40 MB, 20.29 MB, 19.82 MB, 21.82 MB, 19.00 MB, 21.91 MB, 19.93 MB, 20.42 MB, 21.44 MB, 19.04 MB, 22.03 MB, 19.51 MB, 20.87 MB, 21.00 MB, 19.45 MB, 22.05 MB, 19.22 MB, 17.13 MB
- AverageThreadTokens: 0.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 22.55 MB (23644896)
- PerHostPeakMemUsage: 0
- PrepareTime: 224.969us
- RowsProduced: 6.00M (6001215)
- TotalCpuTime: 1h5m
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 0.000ns
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 219 (219)
- BlocksRecycled: 32 (32)
- BufferedPins: 74 (74)
- BytesWritten: 944.32 MB (990186348)
- CompressedBytesWritten: 437.30 MB (458537208)
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 819.20 MB (858993472)
- PeakMemoryUsage: 816.00 MB (855638016)
- TotalBufferWaitTime: 0.000ns
- TotalEncryptionTime: 0.000ns
- TotalIntegrityCheckTime: 0.000ns
- TotalReadBlockTime: 226.005ms
EXCHANGE_NODE (id=2):(Total: 30s184ms, non-child: 1s021ms, % non-child: 3.39%)
BytesReceived(1m4s): 7.20 MB, 23.35 MB, 36.77 MB, 48.83 MB, 62.53 MB, 74.82 MB, 87.88 MB, 101.03 MB, 113.27 MB, 126.98 MB, 139.11 MB, 152.48 MB, 165.37 MB, 177.76 MB, 191.43 MB, 203.47 MB, 217.02 MB, 229.56 MB, 242.35 MB, 255.81 MB, 267.86 MB, 281.57 MB, 293.87 MB, 306.84 MB, 320.09 MB, 332.25 MB, 346.00 MB, 358.15 MB, 371.47 MB, 384.39 MB, 396.76 MB, 410.51 MB, 422.55 MB, 436.12 MB, 448.79 MB, 461.47 MB, 475.04 MB, 487.12 MB, 500.78 MB, 513.14 MB, 526.06 MB, 539.35 MB, 551.49 MB, 565.21 MB, 577.44 MB, 590.70 MB, 603.79 MB, 616.08 MB, 629.79 MB, 641.85 MB, 655.32 MB, 668.05 MB, 680.60 MB, 694.24 MB, 706.31 MB, 719.94 MB, 732.40 MB, 745.26 MB, 758.61 MB, 770.73 MB, 782.37 MB
- BytesReceived: 783.74 MB (821814808)
- ConvertRowBatchTime: 0.000ns
- DeserializeRowBatchTimer: 3s948ms
- FirstBatchArrivalWaitTime: 29s163ms
- MergeGetNext: 997.881ms
- MergeGetNextBatch: 65.561ms
- PeakMemoryUsage: 0
- RowsReturned: 6.00M (6001215)
- RowsReturnedRate: 198.81 K/sec
- SendersBlockedTimer: 1h3m
- SendersBlockedTotalTimer(*): 1h3m
Averaged Fragment F00:(Total: 1h4m, non-child: 1h3m, % non-child: 98.68%)
split sizes: min: 718.94 MB, max: 718.94 MB, avg: 718.94 MB, stddev: 0
completion times: min:1h4m max:1h4m mean: 1h4m stddev:0.000ns
execution rates: min:190.29 KB/sec max:190.29 KB/sec mean:190.29 KB/sec stddev:0.00 /sec
num instances: 1
- AverageThreadTokens: 1.03
- BloomFilterBytes: 0
- PeakMemoryUsage: 969.61 MB (1016709800)
- PerHostPeakMemUsage: 970.91 MB (1018077864)
- PrepareTime: 167.610ms
- RowsProduced: 6.00M (6001215)
- TotalCpuTime: 3m1s
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1h3m
- TotalStorageWaitTime: 1s303ms
CodeGen:(Total: 202.085ms, non-child: 202.085ms, % non-child: 100.00%)
- CodegenTime: 274.215us
- CompileTime: 3.591ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.12 MB (2224196)
- NumFunctions: 29 (29)
- NumInstructions: 529 (529)
- OptimizationTime: 54.022ms
- PrepareTime: 143.628ms
DataStreamSender (dst_id=2):(Total: 9s619ms, non-child: 9s619ms, % non-child: 100.00%)
- BytesSent: 783.74 MB (821814808)
- NetworkThroughput(*): 1.03 GB/sec
- OverallThroughput: 81.48 MB/sec
- PeakMemoryUsage: 680.00 B (680)
- RowsReturned: 6.00M (6001215)
- SerializeBatchTime: 8s876ms
- TransmitDataRPCTime: 741.034ms
- UncompressedRowBatchSize: 1.51 GB (1625021116)
SORT_NODE (id=1):(Total: 41s422ms, non-child: 41s010ms, % non-child: 99.01%)
- InMemorySortTime: 6s580ms
- InitialRunsCreated: 2 (2)
- MergeGetNext: 12s295ms
- MergeGetNextBatch: 5s318ms
- PeakMemoryUsage: 816.20 MB (855842816)
- RowsReturned: 6.00M (6001215)
- RowsReturnedRate: 144.88 K/sec
- SortDataSize: 1.49 GB (1600992812)
- SpilledRuns: 3 (3)
- TotalMergesPerformed: 1 (1)
HDFS_SCAN_NODE (id=0):(Total: 412.149ms, non-child: 412.149ms, % non-child: 100.00%)
- AverageHdfsReadThreadConcurrency: 0.24
- AverageScannerThreadConcurrency: 5.74
- BytesRead: 718.94 MB (753867192)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 718.94 MB (753867192)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 718.94 MB (753867192)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumDisksAccessed: 2 (2)
- NumScannerThreadsStarted: 6 (6)
- PeakMemoryUsage: 201.76 MB (211562496)
- PerReadThreadRawHdfsThroughput: 195.65 MB/sec
- RemoteScanRanges: 0 (0)
- RowsRead: 6.00M (6001215)
- RowsReturned: 6.00M (6001215)
- RowsReturnedRate: 14.56 M/sec
- ScanRangesComplete: 6 (6)
- ScannerThreadsInvoluntaryContextSwitches: 320 (320)
- ScannerThreadsTotalWallClockTime: 2m12s
- DelimiterParseTime: 3s826ms
- MaterializeTupleTime(*): 6s619ms
- ScannerThreadsSysTime: 20.468ms
- ScannerThreadsUserTime: 10s538ms
- ScannerThreadsVoluntaryContextSwitches: 6.01K (6006)
- TotalRawHdfsReadTime(*): 3s674ms
- TotalReadThroughput: 30.59 MB/sec
Fragment F00:
Instance 394ee7185b41be23:949a047c1e09ec84 (host=anuj-OptiPlex-9020:22000):(Total: 1h4m, non-child: 1h3m, % non-child: 98.68%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:6/718.94 MB
MemoryUsage(1m4s): 297.48 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 30.89 MB
ThreadUsage(1m4s): 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 1.03
- BloomFilterBytes: 0
- PeakMemoryUsage: 969.61 MB (1016709800)
- PerHostPeakMemUsage: 970.91 MB (1018077864)
- PrepareTime: 167.610ms
- RowsProduced: 6.00M (6001215)
- TotalCpuTime: 3m1s
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1h3m
- TotalStorageWaitTime: 1s303ms
CodeGen:(Total: 202.085ms, non-child: 202.085ms, % non-child: 100.00%)
- CodegenTime: 274.215us
- CompileTime: 3.591ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.12 MB (2224196)
- NumFunctions: 29 (29)
- NumInstructions: 529 (529)
- OptimizationTime: 54.022ms
- PrepareTime: 143.628ms
DataStreamSender (dst_id=2):(Total: 9s619ms, non-child: 9s619ms, % non-child: 100.00%)
- BytesSent: 783.74 MB (821814808)
- NetworkThroughput(*): 1.03 GB/sec
- OverallThroughput: 81.48 MB/sec
- PeakMemoryUsage: 680.00 B (680)
- RowsReturned: 6.00M (6001215)
- SerializeBatchTime: 8s876ms
- TransmitDataRPCTime: 741.034ms
- UncompressedRowBatchSize: 1.51 GB (1625021116)
SORT_NODE (id=1):(Total: 41s422ms, non-child: 41s010ms, % non-child: 99.01%)
ExecOption: Codegen Enabled
- InMemorySortTime: 6s580ms
- InitialRunsCreated: 2 (2)
- MergeGetNext: 12s295ms
- MergeGetNextBatch: 5s318ms
- PeakMemoryUsage: 816.20 MB (855842816)
- RowsReturned: 6.00M (6001215)
- RowsReturnedRate: 144.88 K/sec
- SortDataSize: 1.49 GB (1600992812)
- SpilledRuns: 3 (3)
- TotalMergesPerformed: 1 (1)
HDFS_SCAN_NODE (id=0):(Total: 412.149ms, non-child: 412.149ms, % non-child: 100.00%)
ExecOption: Expr Evaluation Codegen Disabled, Codegen enabled: 0 out of 6
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:6/718.94 MB
Hdfs Read Thread Concurrency Bucket: 0:91.3% 1:4.348% 2:0% 3:2.174% 4:2.174%
File Formats: TEXT/NONE:6
BytesRead(500.000ms): 0, 96.00 MB, 192.00 MB, 192.00 MB, 192.00 MB, 200.00 MB, 240.00 MB, 240.00 MB, 288.00 MB, 288.00 MB, 304.00 MB, 336.00 MB, 336.00 MB, 384.00 MB, 384.00 MB, 432.00 MB, 432.00 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 486.94 MB, 534.94 MB, 550.94 MB, 582.94 MB, 622.94 MB, 622.94 MB, 662.94 MB, 662.94 MB, 702.94 MB, 702.94 MB, 718.94 MB, 718.94 MB, 718.94 MB, 718.94 MB
- AverageHdfsReadThreadConcurrency: 0.24
- AverageScannerThreadConcurrency: 5.74
- BytesRead: 718.94 MB (753867192)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 718.94 MB (753867192)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 718.94 MB (753867192)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumDisksAccessed: 2 (2)
- NumScannerThreadsStarted: 6 (6)
- PeakMemoryUsage: 201.76 MB (211562496)
- PerReadThreadRawHdfsThroughput: 195.65 MB/sec
- RemoteScanRanges: 0 (0)
- RowsRead: 6.00M (6001215)
- RowsReturned: 6.00M (6001215)
- RowsReturnedRate: 14.56 M/sec
- ScanRangesComplete: 6 (6)
- ScannerThreadsInvoluntaryContextSwitches: 320 (320)
- ScannerThreadsTotalWallClockTime: 2m12s
- DelimiterParseTime: 3s826ms
- MaterializeTupleTime(*): 6s619ms
- ScannerThreadsSysTime: 20.468ms
- ScannerThreadsUserTime: 10s538ms
- ScannerThreadsVoluntaryContextSwitches: 6.01K (6006)
- TotalRawHdfsReadTime(*): 3s674ms
- TotalReadThroughput: 30.59 MB/sec
[localhost:21000] >
--
To view, visit http://gerrit.cloudera.org:8080/3478
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d49bd8d6d7643c84cefd1274c18b52907ca1488
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: anujphadke <ap...@cloudera.com>
Gerrit-Reviewer: Mostafa Mokhtar <mm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: anujphadke <ap...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-trunk) IMPALA-3766: Applying LZ4 compression on buffers before spilling
Posted by "anujphadke (Code Review)" <ge...@cloudera.org>.
anujphadke has uploaded a new patch set (#2).
Change subject: IMPALA-3766: Applying LZ4 compression on buffers before spilling
......................................................................
IMPALA-3766: Applying LZ4 compression on buffers before spilling
Evalution patch to compress buffers before spilling it to disk
Change-Id: I4d49bd8d6d7643c84cefd1274c18b52907ca1488
(cherry picked from commit 08568166be91665f3b924675cd3fb8044690fcb2)
---
M be/src/exec/hdfs-table-sink.h
M be/src/runtime/buffered-block-mgr.cc
M be/src/runtime/buffered-block-mgr.h
3 files changed, 40 insertions(+), 7 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/78/3478/2
--
To view, visit http://gerrit.cloudera.org:8080/3478
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4d49bd8d6d7643c84cefd1274c18b52907ca1488
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: anujphadke <ap...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: anujphadke <ap...@cloudera.com>
[Impala-CR](cdh5-trunk) IMPALA-3766: Applying LZ4 compression on buffers before spilling
Posted by "Jim Apple (Code Review)" <ge...@cloudera.org>.
Jim Apple has abandoned this change.
Change subject: IMPALA-3766: Applying LZ4 compression on buffers before spilling
......................................................................
Abandoned
--
To view, visit http://gerrit.cloudera.org:8080/3478
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: abandon
Gerrit-Change-Id: I4d49bd8d6d7643c84cefd1274c18b52907ca1488
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: anujphadke <ap...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@cloudera.com>
Gerrit-Reviewer: Mostafa Mokhtar <mm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: anujphadke <ap...@cloudera.com>
[Impala-CR](cdh5-trunk) IMPALA-3766: Applying LZ4 compression on buffers before spilling
Posted by "Mostafa Mokhtar (Code Review)" <ge...@cloudera.org>.
Mostafa Mokhtar has posted comments on this change.
Change subject: IMPALA-3766: Applying LZ4 compression on buffers before spilling
......................................................................
Patch Set 2:
Can you please attach the before and after profiles to IMPALA-3766.
--
To view, visit http://gerrit.cloudera.org:8080/3478
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d49bd8d6d7643c84cefd1274c18b52907ca1488
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: anujphadke <ap...@cloudera.com>
Gerrit-Reviewer: Mostafa Mokhtar <mm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: anujphadke <ap...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-trunk) IMPALA-3766: Applying LZ4 compression on buffers before spilling
Posted by "Jim Apple (Code Review)" <ge...@cloudera.org>.
Jim Apple has posted comments on this change.
Change subject: IMPALA-3766: Applying LZ4 compression on buffers before spilling
......................................................................
Patch Set 2:
Please update to using the new gerrit project, "Impala-ASF".
Instructions are here:
https://cwiki.apache.org/confluence/display/IMPALA/How+to+switch+to+Apache-hosted+git
Pushes to this project will be disabled on October 1.
--
To view, visit http://gerrit.cloudera.org:8080/3478
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d49bd8d6d7643c84cefd1274c18b52907ca1488
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: anujphadke <ap...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@cloudera.com>
Gerrit-Reviewer: Mostafa Mokhtar <mm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: anujphadke <ap...@cloudera.com>
Gerrit-HasComments: No