You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "abstractdog (via GitHub)" <gi...@apache.org> on 2026/09/16 13:50:02 UTC
[PR] Parquet io cache [hive]
abstractdog opened a new pull request, #6793:
URL: https://github.com/apache/hive/pull/6793
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducible example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4156039257
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ // Fallback path: no cache key, so we read status + footer straight from HDFS. Time both
+ // under HDFS_TIME_NS to match how ORC accounts for cache-miss footer reads.
+ final FileSystem fs = path.getFileSystem(jobConf);
+ long hdfsStart = counters.startTimeCounter();
+ final FileStatus stat = fs.getFileStatus(path);
+ counters.recordHdfsTime(hdfsStart);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ hdfsStart = counters.startTimeCounter();
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ counters.recordHdfsTime(hdfsStart);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ long startTime = counters.startTimeCounter();
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Exception e) {
+ // A shutdown-triggered InterruptedException must not be silently reported as an ordinary
+ // consumer error: preserve the interrupt on the thread so any caller sees it.
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ }
+ consumer.setError(e);
+ } finally {
+ counters.incrWallClockCounter(LlapIOCounters.TOTAL_IO_TIME_NS, startTime);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ // The LLAP IO summary keys off TABLE / FILE / STRIPES; set them the same way ORC does so a
+ // native Parquet fragment shows up in the summary with the same fields populated.
+ if (cacheTag != null) {
+ counters.setDesc(QueryFragmentCounters.Desc.TABLE, cacheTag.getTableName());
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.FILE, path
+ + (fileKey == null ? "" : " (" + fileKey + ")"));
+
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = selectRowGroups(blocks, fileSchema);
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ // STRIPES is the ORC term; for Parquet the analog is row groups. Reuse the same descriptor so
+ // the summary layout stays common and we don't have to teach the reporter about a new field.
+ counters.setDesc(QueryFragmentCounters.Desc.STRIPES, "0," + selected.size());
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /**
+ * The row groups this split has to read: the ones whose first data page falls inside the split, as
+ * Parquet's own input format assigns them, minus any the pushed-down predicate rules out on its
+ * statistics. Returned in file order, since the fetch pipeline reads them in that order.
+ */
+ private List<BlockMetaData> selectRowGroups(List<BlockMetaData> blocks, MessageType fileSchema)
+ throws IOException {
+ long splitStart = split.getStart();
+ long splitEnd = splitStart + split.getLength();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().getFirst().getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate == null) {
+ return selected;
+ }
+ return RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Whether the query needs row-lineage virtual columns ({@code ROW__LINEAGE__ID} /
+ * {@code LAST__UPDATED__SEQUENCE__NUMBER}) that the fallback reader augments the requested schema
+ * with via {@link RowLineageUtils#getRequestedSchemaWithRowLineageColumns}. The native pipeline
+ * doesn't propagate that augmentation through {@code Includes} yet, so we defer to the fallback
+ * reader instead of quietly emitting nulls in the lineage slots.
+ */
+ public boolean needsRowLineage() {
+ VectorizedRowBatchCtx rbCtx = Utilities.getVectorizedRowBatchCtx(jobConf);
+ if (rbCtx == null) {
+ return false;
+ }
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ return RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema, VirtualColumn.ROW_LINEAGE_ID)
+ || RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema,
+ VirtualColumn.LAST_UPDATED_SEQUENCE_NUMBER);
+ }
+
+ /**
+ * Leaf-column positions of the requested fields; {@code BlockMetaData.getColumns()} is a flat list
+ * of leaves in file-schema order, so a top-level primitive at ordinal {@code k} in the file schema
+ * may sit at a very different leaf index (e.g. a nested group of two leaves before it shifts it
+ * from {@code 1} to {@code 2}). We only reach this method when every requested field is a
+ * top-level primitive ({@link #projectsNestedTypes()} would have forced a fallback otherwise), so
+ * each requested field maps to exactly one leaf whose path is a single segment.
+ */
+ static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
+ List<ColumnDescriptor> allLeaves = fileSchema.getColumns();
+ List<Integer> selected = new ArrayList<>();
+ for (Type field : requestedSchema.getFields()) {
+ if (!fileSchema.containsField(field.getName())) {
+ continue;
+ }
+ for (int i = 0; i < allLeaves.size(); ++i) {
+ String[] path = allLeaves.get(i).getPath();
+ // Single-segment path == top-level primitive; skip leaves buried inside a group.
+ if (path.length == 1 && path[0].equals(field.getName())) {
+ selected.add(i);
+ break;
+ }
+ }
+ }
+ return selected.stream().mapToInt(Integer::intValue).toArray();
+ }
+
+ private static long bytes(int[] projected, BlockMetaData block) {
+ long total = 0;
+ for (int leaf : projected) {
+ total += block.getColumns().get(leaf).getTotalSize();
+ }
+ return total;
+ }
+
+ private static long bytes(Fetch fetch) {
+ long total = 0;
+ for (ColumnChunkMetaData chunk : fetch.batch.chunks()) {
+ total += chunk.getTotalSize();
+ }
+ return total;
+ }
+
+ /** One row group on its way in: buffers planned, missing ranges requested, not yet decoded. */
+ private static final class Fetch {
+ private final ParquetEncodedColumnBatch batch = new ParquetEncodedColumnBatch();
+ private final List<ColumnPlan> columns = new ArrayList<>();
+ private final List<Part> misses = new ArrayList<>();
+ private final List<Run> runs = new ArrayList<>();
+ }
+
+ /** The buffers covering one column chunk, in file order, and which stretches of them are new. */
+ private record ColumnPlan(List<Part> parts, List<MissRun> missRuns) {
+ ColumnPlan() {
+ this(new ArrayList<>(), new ArrayList<>());
+ }
+ }
+
+ /** One cache buffer's worth of a column chunk: a hit handed back by the cache, or a miss to fill. */
+ private static final class Part {
+ private MemoryBuffer buffer;
+ private final DiskRange range;
+ /** We hold one ref to release; until then a miss is a raw allocation to free. */
+ private boolean owned;
+
+ Part(MemoryBuffer buffer, DiskRange range, boolean owned) {
+ this.buffer = buffer;
+ this.range = range;
+ this.owned = owned;
+ }
+ }
+
+ /** {@code count} consecutive parts covering one contiguous missing sub-range; cached as a unit. */
+ private record MissRun(int firstPart, int count) {
+ }
+
+ /** One vectored range and the cache buffers it fills, in file order. */
+ private record Run(FileRange range, List<Part> parts) {
Review Comment:
ReadRange ?
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ // Fallback path: no cache key, so we read status + footer straight from HDFS. Time both
+ // under HDFS_TIME_NS to match how ORC accounts for cache-miss footer reads.
+ final FileSystem fs = path.getFileSystem(jobConf);
+ long hdfsStart = counters.startTimeCounter();
+ final FileStatus stat = fs.getFileStatus(path);
+ counters.recordHdfsTime(hdfsStart);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ hdfsStart = counters.startTimeCounter();
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ counters.recordHdfsTime(hdfsStart);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ long startTime = counters.startTimeCounter();
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Exception e) {
+ // A shutdown-triggered InterruptedException must not be silently reported as an ordinary
+ // consumer error: preserve the interrupt on the thread so any caller sees it.
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ }
+ consumer.setError(e);
+ } finally {
+ counters.incrWallClockCounter(LlapIOCounters.TOTAL_IO_TIME_NS, startTime);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ // The LLAP IO summary keys off TABLE / FILE / STRIPES; set them the same way ORC does so a
+ // native Parquet fragment shows up in the summary with the same fields populated.
+ if (cacheTag != null) {
+ counters.setDesc(QueryFragmentCounters.Desc.TABLE, cacheTag.getTableName());
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.FILE, path
+ + (fileKey == null ? "" : " (" + fileKey + ")"));
+
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = selectRowGroups(blocks, fileSchema);
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ // STRIPES is the ORC term; for Parquet the analog is row groups. Reuse the same descriptor so
+ // the summary layout stays common and we don't have to teach the reporter about a new field.
+ counters.setDesc(QueryFragmentCounters.Desc.STRIPES, "0," + selected.size());
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /**
+ * The row groups this split has to read: the ones whose first data page falls inside the split, as
+ * Parquet's own input format assigns them, minus any the pushed-down predicate rules out on its
+ * statistics. Returned in file order, since the fetch pipeline reads them in that order.
+ */
+ private List<BlockMetaData> selectRowGroups(List<BlockMetaData> blocks, MessageType fileSchema)
+ throws IOException {
+ long splitStart = split.getStart();
+ long splitEnd = splitStart + split.getLength();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().getFirst().getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate == null) {
+ return selected;
+ }
+ return RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Whether the query needs row-lineage virtual columns ({@code ROW__LINEAGE__ID} /
+ * {@code LAST__UPDATED__SEQUENCE__NUMBER}) that the fallback reader augments the requested schema
+ * with via {@link RowLineageUtils#getRequestedSchemaWithRowLineageColumns}. The native pipeline
+ * doesn't propagate that augmentation through {@code Includes} yet, so we defer to the fallback
+ * reader instead of quietly emitting nulls in the lineage slots.
+ */
+ public boolean needsRowLineage() {
+ VectorizedRowBatchCtx rbCtx = Utilities.getVectorizedRowBatchCtx(jobConf);
+ if (rbCtx == null) {
+ return false;
+ }
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ return RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema, VirtualColumn.ROW_LINEAGE_ID)
+ || RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema,
+ VirtualColumn.LAST_UPDATED_SEQUENCE_NUMBER);
+ }
+
+ /**
+ * Leaf-column positions of the requested fields; {@code BlockMetaData.getColumns()} is a flat list
+ * of leaves in file-schema order, so a top-level primitive at ordinal {@code k} in the file schema
+ * may sit at a very different leaf index (e.g. a nested group of two leaves before it shifts it
+ * from {@code 1} to {@code 2}). We only reach this method when every requested field is a
+ * top-level primitive ({@link #projectsNestedTypes()} would have forced a fallback otherwise), so
+ * each requested field maps to exactly one leaf whose path is a single segment.
+ */
+ static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
+ List<ColumnDescriptor> allLeaves = fileSchema.getColumns();
+ List<Integer> selected = new ArrayList<>();
+ for (Type field : requestedSchema.getFields()) {
+ if (!fileSchema.containsField(field.getName())) {
+ continue;
+ }
+ for (int i = 0; i < allLeaves.size(); ++i) {
+ String[] path = allLeaves.get(i).getPath();
+ // Single-segment path == top-level primitive; skip leaves buried inside a group.
+ if (path.length == 1 && path[0].equals(field.getName())) {
+ selected.add(i);
+ break;
+ }
+ }
+ }
+ return selected.stream().mapToInt(Integer::intValue).toArray();
+ }
+
+ private static long bytes(int[] projected, BlockMetaData block) {
+ long total = 0;
+ for (int leaf : projected) {
+ total += block.getColumns().get(leaf).getTotalSize();
+ }
+ return total;
+ }
+
+ private static long bytes(Fetch fetch) {
+ long total = 0;
+ for (ColumnChunkMetaData chunk : fetch.batch.chunks()) {
+ total += chunk.getTotalSize();
+ }
+ return total;
+ }
+
+ /** One row group on its way in: buffers planned, missing ranges requested, not yet decoded. */
+ private static final class Fetch {
+ private final ParquetEncodedColumnBatch batch = new ParquetEncodedColumnBatch();
+ private final List<ColumnPlan> columns = new ArrayList<>();
+ private final List<Part> misses = new ArrayList<>();
+ private final List<Run> runs = new ArrayList<>();
+ }
+
+ /** The buffers covering one column chunk, in file order, and which stretches of them are new. */
+ private record ColumnPlan(List<Part> parts, List<MissRun> missRuns) {
+ ColumnPlan() {
+ this(new ArrayList<>(), new ArrayList<>());
+ }
+ }
+
+ /** One cache buffer's worth of a column chunk: a hit handed back by the cache, or a miss to fill. */
+ private static final class Part {
+ private MemoryBuffer buffer;
+ private final DiskRange range;
+ /** We hold one ref to release; until then a miss is a raw allocation to free. */
+ private boolean owned;
+
+ Part(MemoryBuffer buffer, DiskRange range, boolean owned) {
+ this.buffer = buffer;
+ this.range = range;
+ this.owned = owned;
+ }
+ }
+
+ /** {@code count} consecutive parts covering one contiguous missing sub-range; cached as a unit. */
+ private record MissRun(int firstPart, int count) {
Review Comment:
Gap / planGap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4156642554
##########
ql/src/java/org/apache/hadoop/hive/llap/ParquetCacheLayout.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.llap;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.DataCache.DiskRangeListFactory;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+
+/**
+ * How Parquet column chunks are laid out in the LLAP cache and fetched from the file. Everything
+ * that caches a chunk goes through this, so a reader finding only part of one cached re-caches
+ * the gap on the boundaries the first reader used. A reader builds one from the daemon's allocator
+ * and configuration and uses it for every chunk, which is what keeps its layout consistent.
+ */
+public final class ParquetCacheLayout {
+
+ /**
+ * An object store fetches separate ranges concurrently but never splits one, so cutting buys
+ * parallelism while a large range amortises the request. Measured against S3 the knee is here:
+ * below 4 Mb throughput drops sharply, above 8 Mb it is flat. Trino also splits at 8 Mb.
+ */
+ private static final int MIN_RANGE_CAP_BYTES = 8 << 20;
+
+ public static final DiskRangeListFactory CACHE_CHUNK_FACTORY = CacheChunk::new;
+
+ private final int maxBuffer;
+ private final int minBuffer;
+ private final int maxRange;
+
+ /** The layout for a daemon: its allocator's maximum and the grain it caches non-ORC data at. */
+ public ParquetCacheLayout(Allocator allocator, Configuration conf) {
+ this(allocator.getMaxAllocation(), HiveConf.getSizeVar(conf, HiveConf.ConfVars.LLAP_IO_ENCODE_ALLOC_SIZE));
+ }
+
+ /**
+ * @param maxAlloc largest buffer the allocator hands out; rounded down to a power of two
+ * @param splitFloor size below which a remainder is left as one buffer rather than split further
+ */
+ public ParquetCacheLayout(int maxAlloc, long splitFloor) {
+ this.maxBuffer = Integer.highestOneBit(maxAlloc);
Review Comment:
i think we can drop the defensive caps
````
this.maxBuffer = maxAlloc;
this.minBuffer = (int) splitFloor;
this.maxRange = Math.max(MAX_RANGE_FLOOR_BYTES, maxAlloc);
````
##########
ql/src/java/org/apache/hadoop/hive/llap/ParquetCacheLayout.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.llap;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.DataCache.DiskRangeListFactory;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+
+/**
+ * How Parquet column chunks are laid out in the LLAP cache and fetched from the file. Everything
+ * that caches a chunk goes through this, so a reader finding only part of one cached re-caches
+ * the gap on the boundaries the first reader used. A reader builds one from the daemon's allocator
+ * and configuration and uses it for every chunk, which is what keeps its layout consistent.
+ */
+public final class ParquetCacheLayout {
+
+ /**
+ * An object store fetches separate ranges concurrently but never splits one, so cutting buys
+ * parallelism while a large range amortises the request. Measured against S3 the knee is here:
+ * below 4 Mb throughput drops sharply, above 8 Mb it is flat. Trino also splits at 8 Mb.
+ */
+ private static final int MIN_RANGE_CAP_BYTES = 8 << 20;
Review Comment:
MAX_RANGE_FLOOR_BYTES ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "aturoczy (via GitHub)" <gi...@apache.org>.
aturoczy commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4155871405
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ParquetEncodedDataConsumer.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.decode;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.type.DataTypePhysicalVariation;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.impl.ColumnVectorBatch;
+import org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl;
+import org.apache.hadoop.hive.llap.io.decode.ColumnVectorProducer.Includes;
+import org.apache.hadoop.hive.llap.io.encoded.ParquetEncodedColumnBatch;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonIOMetrics;
+import org.apache.hadoop.hive.ql.exec.vector.ColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.orc.encoded.Consumer;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetRowGroupDecoder;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedColumnReader;
+import org.apache.hadoop.hive.serde2.ColumnProjectionUtils;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
+import org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.parquet.HadoopReadOptions;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.compression.CompressionCodecFactory;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.metadata.FileMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+
+import com.google.common.base.Strings;
+
+/**
+ * The Parquet counterpart of {@link OrcEncodedDataConsumer}: it turns one row group's worth of
+ * cached column-chunk buffers ({@link ParquetEncodedColumnBatch}) into {@link ColumnVectorBatch}es
+ * and hands them downstream.
+ *
+ * <p>Rather than reimplementing Parquet value decoding, this consumer builds a
+ * {@link ParquetCachedPageReadStore} directly over the cached buffers (pages are parsed in place and
+ * decompressed lazily, no ParquetFileReader and no whole-chunk copy) and drives the very same
+ * {@link VectorizedColumnReader}s the non-cached vectorized reader uses (via {@link ParquetRowGroupDecoder}).
+ */
+public class ParquetEncodedDataConsumer
+ extends EncodedDataConsumer<Object, ParquetEncodedColumnBatch> {
+
+ private final Configuration jobConf;
+ private final boolean useDecimal64ColumnVectors;
+ private final CompressionCodecFactory codecFactory;
+ private final ParquetMetadataConverter converter;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private Path path;
+ private Map<String, Object> initialDefaults;
+
+ // Derived lazily (once) from the footer + job conf, then reused across row groups.
+ private List<TypeInfo> columnTypesList;
+ private List<Integer> colsToInclude;
+ private boolean readAllColumns;
+ private boolean skipTimestampConversion;
+ private boolean skipProlepticConversion;
+ private boolean legacyConversionEnabled;
+ private ZoneId writerTimezone;
+ private boolean schemaInitialized = false;
+
+ public ParquetEncodedDataConsumer(Consumer<ColumnVectorBatch> consumer, Includes includes,
+ QueryFragmentCounters counters, LlapDaemonIOMetrics ioMetrics, Configuration jobConf) {
+ super(consumer, includes.getPhysicalColumnIds().size(), ioMetrics, counters);
+ this.jobConf = jobConf;
+ this.useDecimal64ColumnVectors = HiveConf.getVar(jobConf,
+ ConfVars.HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_ENABLED)
+ .equalsIgnoreCase(HiveConf.HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_DECIMAL_64);
+ ParquetReadOptions options = HadoopReadOptions.builder(jobConf).build();
+ this.codecFactory = options.getCodecFactory();
+ this.converter = new ParquetMetadataConverter(options);
+ }
+
+ public void setFileMetadata(ParquetMetadata footer, MessageType requestedSchema, Path path) {
+ this.footer = footer;
+ this.requestedSchema = requestedSchema;
+ this.path = path;
+ }
+
+ public void setInitialDefaults(Map<String, Object> initialDefaults) {
+ this.initialDefaults = initialDefaults;
+ }
+
+ /**
+ * Derives the Hive type / projection lists and the conversion flags the same way
+ * {@link org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase} does from the job conf,
+ * so the shared {@link ParquetRowGroupDecoder} produces identical readers.
+ */
+ private void initSchema() {
+ FileMetaData fileMetaData = footer.getFileMetaData();
+ Map<String, String> kvMeta = fileMetaData.getKeyValueMetaData();
+
+ this.columnTypesList =
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES));
+ this.colsToInclude = ColumnProjectionUtils.getReadColumnIDs(jobConf);
+ this.readAllColumns = ColumnProjectionUtils.isReadAllColumns(jobConf);
+
+ this.writerTimezone = DataWritableReadSupport.getWriterTimeZoneId(kvMeta);
+ if (HiveConf.getBoolVar(jobConf, ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION)) {
+ this.skipTimestampConversion =
+ !Strings.nullToEmpty(fileMetaData.getCreatedBy()).startsWith("parquet-mr");
+ } else {
+ this.skipTimestampConversion = false;
+ }
+ Boolean proleptic = DataWritableReadSupport.getWriterDateProleptic(kvMeta);
+ if (proleptic == null) {
+ proleptic = HiveConf.getBoolVar(jobConf, ConfVars.HIVE_PARQUET_DATE_PROLEPTIC_GREGORIAN_DEFAULT);
+ }
+ this.skipProlepticConversion = proleptic;
+ this.legacyConversionEnabled = DataWritableReadSupport.getZoneConversionLegacy(kvMeta, jobConf);
+
+ this.schemaInitialized = true;
+ }
+
+ @Override
+ protected void decodeBatch(ParquetEncodedColumnBatch batch,
+ Consumer<ColumnVectorBatch> downstreamConsumer) throws InterruptedException {
+ if (!schemaInitialized) {
+ initSchema();
+ }
+
+ long startTime = counters.startTimeCounter();
+ // NOSONAR - S2093 does not apply: codecFactory is a per-consumer field with a per-batch
+ // release() (not close()); try-with-resources would tie it to this scope, which is wrong.
+ try { // NOSONAR
+ PageReadStore pages = new ParquetCachedPageReadStore(footer, batch, codecFactory, converter);
+
+ VectorizedColumnReader[] columnReaders =
+ new ParquetRowGroupDecoder(footer.getFileMetaData().getSchema(), initialDefaults).buildColumnReaders(
+ pages, requestedSchema, columnTypesList, colsToInclude, readAllColumns,
+ new ParquetRowGroupDecoder.TimestampConversionOptions(skipTimestampConversion,
+ writerTimezone, skipProlepticConversion, legacyConversionEnabled));
+
+ long rowsLeft = pages.getRowCount();
+ int batches = 0;
+ while (rowsLeft > 0) {
+ int batchSize = (int) Math.min(VectorizedRowBatch.DEFAULT_SIZE, rowsLeft);
+
+ ColumnVectorBatch cvb = takeBatch(batchSize);
+
+ // columnReaders[i] is requestedSchema field i, i.e. the i-th projected column.
+ for (int i = 0; i < columnReaders.length; ++i) {
+ if (columnReaders[i] == null) {
+ continue;
+ }
+ TypeInfo columnType = columnTypesList.get(readAllColumns ? i : colsToInclude.get(i));
+ ColumnVector cv = prepareColumnVector(cvb, i, columnType, batchSize);
+ columnReaders[i].readBatch(batchSize, cv, columnType);
+ }
+
+ downstreamConsumer.consumeData(cvb);
+ counters.incrCounter(LlapIOCounters.ROWS_EMITTED, batchSize);
+ rowsLeft -= batchSize;
+ ++batches;
+ }
+ counters.incrWallClockCounter(LlapIOCounters.DECODE_TIME_NS, startTime);
+ counters.incrCounter(LlapIOCounters.NUM_VECTOR_BATCHES, batches);
+ counters.incrCounter(LlapIOCounters.NUM_DECODED_BATCHES);
+ } catch (IOException | RuntimeException e) {
+ // parquet reports decode failures as runtime ParquetDecodingException.
+ LlapIoImpl.LOG.error("Parquet decodeBatch failed for rowGroup " + batch.rowGroupIx() + " of " + path, e);
+ downstreamConsumer.setError(e);
Review Comment:
Just because of my C# love, it is not possible to have something like this in java?
`xxxxx.("Parquet decode failed for row group %d of %s", batch.rowGroupIx(), path)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4216539052
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ // Fallback path: no cache key, so we read status + footer straight from HDFS. Time both
+ // under HDFS_TIME_NS to match how ORC accounts for cache-miss footer reads.
+ final FileSystem fs = path.getFileSystem(jobConf);
+ long hdfsStart = counters.startTimeCounter();
+ final FileStatus stat = fs.getFileStatus(path);
+ counters.recordHdfsTime(hdfsStart);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ hdfsStart = counters.startTimeCounter();
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ counters.recordHdfsTime(hdfsStart);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ long startTime = counters.startTimeCounter();
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Exception e) {
+ // A shutdown-triggered InterruptedException must not be silently reported as an ordinary
+ // consumer error: preserve the interrupt on the thread so any caller sees it.
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ }
+ consumer.setError(e);
+ } finally {
+ counters.incrWallClockCounter(LlapIOCounters.TOTAL_IO_TIME_NS, startTime);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ // The LLAP IO summary keys off TABLE / FILE / STRIPES; set them the same way ORC does so a
+ // native Parquet fragment shows up in the summary with the same fields populated.
+ if (cacheTag != null) {
+ counters.setDesc(QueryFragmentCounters.Desc.TABLE, cacheTag.getTableName());
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.FILE, path
+ + (fileKey == null ? "" : " (" + fileKey + ")"));
+
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = selectRowGroups(blocks, fileSchema);
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ // STRIPES is the ORC term; for Parquet the analog is row groups. Reuse the same descriptor so
+ // the summary layout stays common and we don't have to teach the reporter about a new field.
+ counters.setDesc(QueryFragmentCounters.Desc.STRIPES, "0," + selected.size());
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /**
+ * The row groups this split has to read: the ones whose first data page falls inside the split, as
+ * Parquet's own input format assigns them, minus any the pushed-down predicate rules out on its
+ * statistics. Returned in file order, since the fetch pipeline reads them in that order.
+ */
+ private List<BlockMetaData> selectRowGroups(List<BlockMetaData> blocks, MessageType fileSchema)
+ throws IOException {
+ long splitStart = split.getStart();
+ long splitEnd = splitStart + split.getLength();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().getFirst().getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate == null) {
+ return selected;
+ }
+ return RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Whether the query needs row-lineage virtual columns ({@code ROW__LINEAGE__ID} /
+ * {@code LAST__UPDATED__SEQUENCE__NUMBER}) that the fallback reader augments the requested schema
+ * with via {@link RowLineageUtils#getRequestedSchemaWithRowLineageColumns}. The native pipeline
+ * doesn't propagate that augmentation through {@code Includes} yet, so we defer to the fallback
+ * reader instead of quietly emitting nulls in the lineage slots.
+ */
+ public boolean needsRowLineage() {
+ VectorizedRowBatchCtx rbCtx = Utilities.getVectorizedRowBatchCtx(jobConf);
+ if (rbCtx == null) {
+ return false;
+ }
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ return RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema, VirtualColumn.ROW_LINEAGE_ID)
+ || RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema,
+ VirtualColumn.LAST_UPDATED_SEQUENCE_NUMBER);
+ }
+
+ /**
+ * Leaf-column positions of the requested fields; {@code BlockMetaData.getColumns()} is a flat list
+ * of leaves in file-schema order, so a top-level primitive at ordinal {@code k} in the file schema
+ * may sit at a very different leaf index (e.g. a nested group of two leaves before it shifts it
+ * from {@code 1} to {@code 2}). We only reach this method when every requested field is a
+ * top-level primitive ({@link #projectsNestedTypes()} would have forced a fallback otherwise), so
+ * each requested field maps to exactly one leaf whose path is a single segment.
+ */
+ static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
Review Comment:
ack, also `BlockMetaData` doesn't seem to be present in this scope, fixing the comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4216481779
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
Review Comment:
ack, created https://issues.apache.org/jira/browse/HIVE-30143
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #6793:
URL: https://github.com/apache/hive/pull/6793#issuecomment-5926567264
@deniskuzZ: this is now ready for final reviews, 10GB results are as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4044637203
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java:
##########
@@ -350,6 +353,11 @@ public void start() {
private boolean checkOrcSchemaEvolution() {
SchemaEvolution evolution = rp.getSchemaEvolution();
+ if (evolution == null) {
+ // No ORC-style schema evolution to validate (e.g. native parquet path);
Review Comment:
you might be looking at an old commit, the current state of PR already has an improved comment:
https://github.com/apache/hive/pull/6793/changes#diff-f00087a16c2c9fed9e0899d7a3718edad0215e33cc41fdd38895945ba45f2517R357-R361
```
// Only the ORC pipeline hangs an ORC SchemaEvolution off the ReadPipeline; other formats
// (native Parquet today) handle column resolution themselves, so there is nothing to
// validate here.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6793:
URL: https://github.com/apache/hive/pull/6793#issuecomment-5888751847
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6793) **Quality Gate passed**
Issues
 [69 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6793&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6793&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6793&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6793&metric=new_coverage&view=list)
 [0.2% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6793&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6793)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4157054796
##########
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetRowGroupDecoder.java:
##########
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.ql.io.parquet.vector;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.parquet.ParquetRuntimeException;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.InvalidSchemaException;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Reusable, stateless helper that builds the {@link VectorizedColumnReader} array for one Parquet
+ * row group, given an already-read {@link PageReadStore}. The logic here was lifted verbatim from
+ * {@link VectorizedParquetRecordReader} so that both the row-by-split reader and the LLAP
+ * cache-backed consumer ({@code ParquetEncodedDataConsumer}) can share a single, behavior-preserving
+ * implementation of the Hive-type-driven Parquet column-reader construction.
+ *
+ * <p>The instance only carries the {@code fileSchema} (used for the schema-evolution check). All
+ * per-row-group state lives in the supplied {@link PageReadStore}.
+ */
+public class ParquetRowGroupDecoder {
+
+ private static final int MAP_DEFINITION_LEVEL_MAX = 3;
+
+ /**
+ * Bag of writer-timezone / proleptic / legacy conversion flags forwarded to every primitive
+ * column reader. Grouped so the reader-construction entry point stays under the parameter cap.
+ */
+ public record TimestampConversionOptions(
+ boolean skipTimestampConversion,
+ ZoneId writerTimezone,
+ boolean skipProlepticConversion,
+ boolean legacyConversionEnabled) {
+ }
+
+ private final MessageType fileSchema;
+ private final Map<String, Object> initialDefaults;
+
+ public ParquetRowGroupDecoder(MessageType fileSchema, Map<String, Object> initialDefaults) {
+ this.fileSchema = fileSchema;
+ this.initialDefaults = initialDefaults;
+ }
+
+ /**
+ * Builds the per-(requested-)column {@link VectorizedColumnReader} array for a row group.
+ *
+ * @param pages the row group's page store (from {@code reader.readRowGroup(..)}
+ * / {@code readNextRowGroup()})
+ * @param requestedSchema the projected Parquet schema being read
+ * @param columnTypesList the Hive type infos for ALL table columns (indexed by table col id)
+ * @param colsToInclude the table column ids being read, in requested-schema field order;
+ * may be empty (e.g. {@code count(*)}), in which case all readers are null
+ * @param readAllColumns whether projection is "read all columns"
+ * @param options writer-timezone / proleptic / legacy conversion flags, see
+ * {@link TimestampConversionOptions}
+ * @return one reader per requested-schema field (null entries where no reader is needed)
+ */
+ public VectorizedColumnReader[] buildColumnReaders(
+ PageReadStore pages,
+ MessageType requestedSchema,
+ List<TypeInfo> columnTypesList,
+ List<Integer> colsToInclude,
+ boolean readAllColumns,
+ TimestampConversionOptions options) throws IOException {
+ List<Type> types = requestedSchema.getFields();
+ VectorizedColumnReader[] columnReaders = new VectorizedColumnReader[types.size()];
+
+ if (!readAllColumns) {
+ // certain queries like select count(*) from table do not have
+ // any projected columns and still have isReadAllColumns as false
+ // in such cases columnReaders are not needed
+ // However, if colsToInclude is not empty we should initialize each columnReader
+ if (!colsToInclude.isEmpty()) {
+ for (int i = 0; i < types.size(); ++i) {
Review Comment:
buildColumnReaders has two loops that differ only in colsToInclude.get(i) versus i; one loop with int tableCol = readAllColumns ? i : colsToInclude.get(i) and the count(*) guard in front.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4155244490
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ParquetColumnVectorProducer.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.decode;
+
+import java.io.IOException;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.impl.ColumnVectorBatch;
+import org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl;
+import org.apache.hadoop.hive.llap.io.encoded.ParquetEncodedDataReader;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonCacheMetrics;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonIOMetrics;
+import org.apache.hadoop.hive.ql.io.orc.encoded.Consumer;
+import org.apache.hadoop.hive.ql.io.sarg.SearchArgument;
+import org.apache.hadoop.hive.ql.plan.PartitionDesc;
+import org.apache.hadoop.hive.serde2.Deserializer;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.InputFormat;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.Reporter;
+
+public class ParquetColumnVectorProducer implements ColumnVectorProducer {
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration conf;
+ private final LlapDaemonCacheMetrics cacheMetrics;
+ private final LlapDaemonIOMetrics ioMetrics;
+
+ public ParquetColumnVectorProducer(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration conf, LlapDaemonCacheMetrics cacheMetrics, LlapDaemonIOMetrics ioMetrics) {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.conf = conf;
+ this.cacheMetrics = cacheMetrics;
+ this.ioMetrics = ioMetrics;
+ }
+
+ @Override
+ public ReadPipeline createReadPipeline(Consumer<ColumnVectorBatch> consumer, FileSplit split,
+ Includes includes, SearchArgument sarg, QueryFragmentCounters counters,
+ SchemaEvolutionFactory sef, InputFormat<?, ?> sourceInputFormat, Deserializer sourceSerDe,
+ Reporter reporter, JobConf job, Map<Path, PartitionDesc> parts) {
+ try {
+ cacheMetrics.incrCacheReadRequests();
+ ParquetEncodedDataConsumer edc =
+ new ParquetEncodedDataConsumer(consumer, includes, counters, ioMetrics, job);
+ ParquetEncodedDataReader reader = new ParquetEncodedDataReader(
+ lowLevelCache, bufferManager, conf, job, split, edc, counters);
+ reader.loadFooter();
+ // Null makes LlapInputFormat fall back to the normal VectorizedParquetRecordReader.
+ if (reader.projectsNestedTypes()) {
+ LlapIoImpl.LOG.info("Parquet native cache: falling back to normal reader for {} due to "
+ + "nested types in the projection", split.getPath());
+ return null;
+ }
+ /*
+ * Fallback path still caches (footer via LlapProxy, data via LlapCacheAwareFs); what's lost
+ * for this split is vectored reads, row-group lookahead, and its LLAP IO summary counters.
+ * Guard fires per split only for Iceberg-Parquet files that physically carry _row_id /
+ * _last_updated_sequence_number while the query projects those virtual columns.
+ *
+ * Table shape Fires?
+ * ------------------------------------------------------------ ------
+ * Non-Iceberg Parquet never
+ * Iceberg-Parquet, row lineage disabled never
Review Comment:
do we need n`eedsRowLineage()` and the comment block in this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4156085108
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ // Fallback path: no cache key, so we read status + footer straight from HDFS. Time both
+ // under HDFS_TIME_NS to match how ORC accounts for cache-miss footer reads.
+ final FileSystem fs = path.getFileSystem(jobConf);
+ long hdfsStart = counters.startTimeCounter();
+ final FileStatus stat = fs.getFileStatus(path);
+ counters.recordHdfsTime(hdfsStart);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ hdfsStart = counters.startTimeCounter();
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ counters.recordHdfsTime(hdfsStart);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ long startTime = counters.startTimeCounter();
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Exception e) {
+ // A shutdown-triggered InterruptedException must not be silently reported as an ordinary
+ // consumer error: preserve the interrupt on the thread so any caller sees it.
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ }
+ consumer.setError(e);
+ } finally {
+ counters.incrWallClockCounter(LlapIOCounters.TOTAL_IO_TIME_NS, startTime);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ // The LLAP IO summary keys off TABLE / FILE / STRIPES; set them the same way ORC does so a
+ // native Parquet fragment shows up in the summary with the same fields populated.
+ if (cacheTag != null) {
+ counters.setDesc(QueryFragmentCounters.Desc.TABLE, cacheTag.getTableName());
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.FILE, path
+ + (fileKey == null ? "" : " (" + fileKey + ")"));
+
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = selectRowGroups(blocks, fileSchema);
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ // STRIPES is the ORC term; for Parquet the analog is row groups. Reuse the same descriptor so
+ // the summary layout stays common and we don't have to teach the reporter about a new field.
+ counters.setDesc(QueryFragmentCounters.Desc.STRIPES, "0," + selected.size());
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /**
+ * The row groups this split has to read: the ones whose first data page falls inside the split, as
+ * Parquet's own input format assigns them, minus any the pushed-down predicate rules out on its
+ * statistics. Returned in file order, since the fetch pipeline reads them in that order.
+ */
+ private List<BlockMetaData> selectRowGroups(List<BlockMetaData> blocks, MessageType fileSchema)
+ throws IOException {
+ long splitStart = split.getStart();
+ long splitEnd = splitStart + split.getLength();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().getFirst().getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate == null) {
+ return selected;
+ }
+ return RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Whether the query needs row-lineage virtual columns ({@code ROW__LINEAGE__ID} /
+ * {@code LAST__UPDATED__SEQUENCE__NUMBER}) that the fallback reader augments the requested schema
+ * with via {@link RowLineageUtils#getRequestedSchemaWithRowLineageColumns}. The native pipeline
+ * doesn't propagate that augmentation through {@code Includes} yet, so we defer to the fallback
+ * reader instead of quietly emitting nulls in the lineage slots.
+ */
+ public boolean needsRowLineage() {
+ VectorizedRowBatchCtx rbCtx = Utilities.getVectorizedRowBatchCtx(jobConf);
+ if (rbCtx == null) {
+ return false;
+ }
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ return RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema, VirtualColumn.ROW_LINEAGE_ID)
+ || RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema,
+ VirtualColumn.LAST_UPDATED_SEQUENCE_NUMBER);
+ }
+
+ /**
+ * Leaf-column positions of the requested fields; {@code BlockMetaData.getColumns()} is a flat list
+ * of leaves in file-schema order, so a top-level primitive at ordinal {@code k} in the file schema
+ * may sit at a very different leaf index (e.g. a nested group of two leaves before it shifts it
+ * from {@code 1} to {@code 2}). We only reach this method when every requested field is a
+ * top-level primitive ({@link #projectsNestedTypes()} would have forced a fallback otherwise), so
+ * each requested field maps to exactly one leaf whose path is a single segment.
+ */
+ static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
+ List<ColumnDescriptor> allLeaves = fileSchema.getColumns();
+ List<Integer> selected = new ArrayList<>();
+ for (Type field : requestedSchema.getFields()) {
+ if (!fileSchema.containsField(field.getName())) {
+ continue;
+ }
+ for (int i = 0; i < allLeaves.size(); ++i) {
+ String[] path = allLeaves.get(i).getPath();
+ // Single-segment path == top-level primitive; skip leaves buried inside a group.
+ if (path.length == 1 && path[0].equals(field.getName())) {
+ selected.add(i);
+ break;
+ }
+ }
+ }
+ return selected.stream().mapToInt(Integer::intValue).toArray();
+ }
+
+ private static long bytes(int[] projected, BlockMetaData block) {
+ long total = 0;
+ for (int leaf : projected) {
+ total += block.getColumns().get(leaf).getTotalSize();
+ }
+ return total;
+ }
+
+ private static long bytes(Fetch fetch) {
+ long total = 0;
+ for (ColumnChunkMetaData chunk : fetch.batch.chunks()) {
+ total += chunk.getTotalSize();
+ }
+ return total;
+ }
+
+ /** One row group on its way in: buffers planned, missing ranges requested, not yet decoded. */
+ private static final class Fetch {
+ private final ParquetEncodedColumnBatch batch = new ParquetEncodedColumnBatch();
+ private final List<ColumnPlan> columns = new ArrayList<>();
+ private final List<Part> misses = new ArrayList<>();
+ private final List<Run> runs = new ArrayList<>();
+ }
+
+ /** The buffers covering one column chunk, in file order, and which stretches of them are new. */
+ private record ColumnPlan(List<Part> parts, List<MissRun> missRuns) {
+ ColumnPlan() {
+ this(new ArrayList<>(), new ArrayList<>());
+ }
+ }
+
+ /** One cache buffer's worth of a column chunk: a hit handed back by the cache, or a miss to fill. */
+ private static final class Part {
+ private MemoryBuffer buffer;
+ private final DiskRange range;
+ /** We hold one ref to release; until then a miss is a raw allocation to free. */
+ private boolean owned;
+
+ Part(MemoryBuffer buffer, DiskRange range, boolean owned) {
+ this.buffer = buffer;
+ this.range = range;
+ this.owned = owned;
+ }
+ }
+
+ /** {@code count} consecutive parts covering one contiguous missing sub-range; cached as a unit. */
+ private record MissRun(int firstPart, int count) {
+ }
+
+ /** One vectored range and the cache buffers it fills, in file order. */
+ private record Run(FileRange range, List<Part> parts) {
+ }
+
+ private Fetch startFetch(FSDataInputStream fileStream, ParquetRangeBuffers buffers, Allocator allocator,
+ int[] projected, BlockMetaData block, int rg)
+ throws IOException {
+ Fetch fetch = new Fetch();
+ ColumnChunkMetaData[] chunks = new ColumnChunkMetaData[projected.length];
+ for (int pc = 0; pc < projected.length; ++pc) {
+ chunks[pc] = block.getColumns().get(projected[pc]);
+ }
+ fetch.batch.init(fileKey, rg, chunks);
+ try {
+ for (ColumnChunkMetaData chunk : chunks) {
+ ColumnPlan column = new ColumnPlan();
+ fetch.columns.add(column);
+ planColumnChunk(allocator, chunk.getStartingPos(),
+ chunk.getStartingPos() + chunk.getTotalSize(), column, fetch.misses);
+ }
+ // The vectored dispatch is where non-async FS impls actually read; count it under HDFS_TIME.
+ long hdfsStart = counters.startTimeCounter();
+ requestMisses(fileStream, buffers, fetch, layout.maxRangeBytes());
+ counters.recordHdfsTime(hdfsStart);
+ } catch (Exception e) {
+ abandon(allocator, fetch);
+ throw e;
+ }
+ return fetch;
+ }
+
+ private void finishFetch(Allocator allocator, ParquetRangeBuffers buffers, Fetch fetch)
+ throws IOException, InterruptedException {
+ try {
+ // Awaiting the vectored futures is where the miss bytes actually arrive from HDFS.
+ long hdfsStart = counters.startTimeCounter();
+ receiveMisses(buffers, fetch);
+ counters.recordHdfsTime(hdfsStart);
+ for (ColumnPlan column : fetch.columns) {
+ putColumn(allocator, column);
+ }
+ for (int pc = 0; pc < fetch.columns.size(); ++pc) {
+ assemble(fetch.batch, pc, fetch.columns.get(pc).parts);
+ }
+ // consumeData returns the batch on success; after a throw it is still ours.
+ consumer.consumeData(fetch.batch);
+ } catch (Exception e) {
+ abandon(allocator, fetch);
+ throw e;
+ }
+ }
+
+ /** Drops a fetch that will not be decoded: outstanding requests are cancelled, buffers released. */
+ private void abandon(Allocator allocator, Fetch fetch) {
+ for (Run run : fetch.runs) {
+ run.range.getData().cancel(true);
+ }
+ for (ColumnPlan column : fetch.columns) {
+ for (Part part : column.parts) {
+ if (part.owned) {
+ bufferManager.decRefBuffer(part.buffer);
+ } else {
+ allocator.deallocate(part.buffer);
+ }
+ }
+ }
+ }
+
+ /** Package-private so a test can observe the read pattern. */
+ FSDataInputStream openFile(FileSystem fs) throws IOException {
+ return fs.open(path);
+ }
+
+ /** Lets the allocator abandon a wait for memory once the fragment is cancelled. */
+ private void allocateMultiple(Allocator allocator, MemoryBuffer[] dest, int size) {
+ if (allocator instanceof StoppableAllocator stoppable) {
+ stoppable.allocateMultiple(dest, size, DATA_BUFFER_FACTORY, isStopped);
+ } else {
+ allocator.allocateMultiple(dest, size, DATA_BUFFER_FACTORY);
+ }
+ }
+
+ /**
+ * Works out which buffers cover column chunk {@code [start, end)} without touching the file: cache
+ * hits as returned by getFileData, and freshly allocated power-of-two buffers for everything
+ * missing. Misses are appended to {@code allMisses} so the whole row group can be read at once.
+ */
+ private void planColumnChunk(Allocator allocator, long start, long end,
+ ColumnPlan column, List<Part> allMisses) throws IOException {
+ DiskRangeList head = new DiskRangeList(start, end);
+ if (fileKey != null) {
+ head = lowLevelCache.getFileData(fileKey, head, 0, ParquetCacheLayout.CACHE_CHUNK_FACTORY,
+ counters, new BooleanRef());
+ }
+ DiskRangeList current = head;
+ try {
+ for (; current != null; current = current.next) {
+ if (current.hasData()) {
+ column.parts.add(new Part(((CacheChunk) current).getBuffer(), current, true));
+ } else {
+ planMissRun(allocator, current, column, allMisses);
+ }
+ }
+ } catch (Exception e) {
+ // Hits past the failure point are still locked by getFileData; the caller only knows about
+ // the parts already recorded.
+ for (current = current.next; current != null; current = current.next) {
+ if (current.hasData()) {
+ bufferManager.decRefBuffer(((CacheChunk) current).getBuffer());
+ }
+ }
+ throw e;
+ }
+ }
+
+ /**
+ * Records one missing sub-range as a run of freshly allocated power-of-two buffers, and appends
+ * each part to {@code allMisses} so the caller can request them all in one vectored call.
+ */
+ private void planMissRun(Allocator allocator, DiskRangeList missing, ColumnPlan column,
+ List<Part> allMisses) throws IOException {
+ LlapHiveUtils.throwIfCacheOnlyRead(cacheOnly);
+ int[] sizes = layout.bufferSizes(missing.getEnd() - missing.getOffset());
+ column.missRuns.add(new MissRun(column.parts.size(), sizes.length));
+ long partFrom = missing.getOffset();
+ for (int size : sizes) {
+ MemoryBuffer[] one = new MemoryBuffer[1];
+ allocateMultiple(allocator, one, size);
+ // The cache accounts and serves the bytes up to the buffer's limit.
+ ByteBuffer raw = one[0].getByteBufferRaw();
+ raw.limit(raw.position() + size);
+ Part part = new Part(one[0], new DiskRange(partFrom, partFrom + size), false);
+ column.parts.add(part);
+ allMisses.add(part);
+ partFrom += size;
+ }
+ }
+
+ /**
+ * Requests every missing buffer of a row group in one vectored call, one range per run of
+ * adjacent buffers. Column chunks sit back to back in the file, so a projection that keeps
+ * neighbouring columns reads them together; runs are capped so the row group arrives as
+ * several concurrent requests rather than one. A filesystem without a vectored implementation
+ * reads the ranges in turn.
+ */
+ private static void requestMisses(FSDataInputStream fileStream, ParquetRangeBuffers buffers, Fetch fetch,
+ int maxRange) throws IOException {
+ List<Part> misses = fetch.misses;
+ if (misses.isEmpty()) {
+ return;
+ }
+ misses.sort(Comparator.comparingLong(part -> part.range.getOffset()));
+ List<Run> runs = new ArrayList<>();
+ int i = 0;
+ while (i < misses.size()) {
+ long from = misses.get(i).range.getOffset();
+ long to = misses.get(i).range.getEnd();
+ int j = i + 1;
+ while (j < misses.size()) {
+ DiskRange next = misses.get(j).range;
+ if (next.getOffset() != to || next.getEnd() - from > maxRange) {
+ break;
+ }
+ to = next.getEnd();
+ ++j;
+ }
+ runs.add(new Run(FileRange.createFileRange(from, (int) (to - from)), misses.subList(i, j)));
+ i = j;
+ }
+ List<FileRange> ranges = runs.stream().map(run -> run.range).collect(Collectors.toList());
+ fileStream.readVectored(ranges, buffers::allocate, buffers::release);
+ // Only requests the stream accepted are the fetch's to wait for or cancel.
+ fetch.runs.addAll(runs);
+ }
+
+ /** Waits for the requested runs and copies each into the cache buffers it covers. */
+ private static void receiveMisses(ParquetRangeBuffers buffers, Fetch fetch) throws IOException {
+ for (Run run : fetch.runs) {
+ ByteBuffer data = FutureIO.awaitFuture(run.range.getData());
+ for (Part part : run.parts) {
+ int length = part.range.getLength();
+ ByteBuffer src = data.duplicate();
+ src.position(data.position() + (int) (part.range.getOffset() - run.range.getOffset()));
+ src.limit(src.position() + length);
+ part.buffer.getByteBufferRaw().duplicate().put(src);
+ }
+ buffers.release(data);
+ }
+ }
+
+ /** Hands one column's freshly read buffers to the cache; every part then carries a ref we own. */
+ private void putColumn(Allocator allocator, ColumnPlan column) {
+ for (MissRun run : column.missRuns) {
+ List<Part> parts = column.parts.subList(run.firstPart, run.firstPart + run.count);
+ if (fileKey == null) {
+ for (Part part : parts) {
+ bufferManager.incRefBuffer(part.buffer);
+ part.owned = true;
+ }
+ // No cache key => no putFileData; each part is already on the ref-counted side
+ // (incRefBuffer + owned=true), so cleanup will decRef rather than deallocate. Skip the
+ // cache-put loop.
+ continue;
+ }
+ // One range at a time so a mid-run throw from putFileData (e.g. the length-mismatch guard
+ // in LowLevelCacheImpl) leaves ownership clean: parts already handled are marked owned so
+ // cleanup calls decRefBuffer on cache-owned memory, while the current and later ones stay
+ // raw allocations that cleanup can safely deallocate.
+ for (Part part : parts) {
+ MemoryBuffer fresh = part.buffer;
+ MemoryBuffer[] pair = new MemoryBuffer[] {fresh};
+ DiskRange[] range = new DiskRange[] {part.range};
+ lowLevelCache.putFileData(fileKey, range, pair, 0, Priority.NORMAL, counters, cacheTag);
+ if (pair[0] != fresh) {
+ // The cache kept its own buffer (locked for us) and unlocked ours without freeing it.
+ allocator.deallocate(fresh);
+ part.buffer = pair[0];
+ }
+ part.owned = true;
+ }
+ }
+ }
+
+ private static void assemble(ParquetEncodedColumnBatch batch, int pc, List<Part> parts) {
Review Comment:
setBatchColumn ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "aturoczy (via GitHub)" <gi...@apache.org>.
aturoczy commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4155832942
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
Review Comment:
yes, maybe the "read" is more expressive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4217291552
##########
ql/src/java/org/apache/hadoop/hive/llap/ParquetRangeBuffers.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.llap;
+
+import java.nio.ByteBuffer;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.StreamCapabilities;
+import org.apache.hadoop.io.ElasticByteBufferPool;
+
+/**
+ * Heap buffers for one stream's vectored reads. A stream returning slices of a larger read says
+ * so ({@link StreamCapabilities#VECTOREDIO_BUFFERS_SLICED}); those alias each other and are left
+ * to the collector. Other streams' buffers are exclusive and pooled for reuse across the file's
+ * row groups. The pool is per stream, not per JVM: {@link ElasticByteBufferPool} never evicts
+ * and has one lock, so a shared one would pin every executor's peak for the daemon's life.
+ */
+public final class ParquetRangeBuffers {
+ private final ElasticByteBufferPool pool;
+
+ private ParquetRangeBuffers(boolean pooled) {
+ this.pool = pooled ? new ElasticByteBufferPool() : null;
+ }
+
+ public static ParquetRangeBuffers forStream(FSDataInputStream stream) {
Review Comment:
ack
also, this is moved to a generic class named `VectoredReadBuffers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #6793:
URL: https://github.com/apache/hive/pull/6793#issuecomment-6360859747
> +1 Ship it!
>
> Re-checked on [198d593](https://github.com/apache/hive/commit/198d5937e4c32a01b220eeef8d334eda78b9508e). All open items are closed:
thanks! I'm fixing a possible flakiness in q test: changing a platform-dependent snappy to pure java lz4 compression to remove bytes number flakiness, then it should be ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4030031689
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -1890,7 +1889,8 @@ private static boolean hasOrcTimeInSchema(Properties tableProps, Schema tableSch
/**
* Vectorized reads of parquet files from columns with list or map type is only supported if the nested types are of
* primitive type category
- * check {@link VectorizedParquetRecordReader#checkListColumnSupport} for details on nested types under lists
+ * check {@link org.apache.hadoop.hive.ql.io.parquet.vector.ParquetRowGroupDecoder#checkListColumnSupport} for
Review Comment:
do we need full name with package here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4184498250
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ParquetCachedPageReadStore.java:
##########
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.decode;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.llap.io.encoded.ParquetEncodedColumnBatch;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.DataPage;
+import org.apache.parquet.column.page.DataPageV1;
+import org.apache.parquet.column.page.DataPageV2;
+import org.apache.parquet.column.page.DictionaryPage;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.column.page.PageReader;
+import org.apache.parquet.compression.CompressionCodecFactory;
+import org.apache.parquet.compression.CompressionCodecFactory.BytesInputDecompressor;
+import org.apache.parquet.format.DataPageHeader;
+import org.apache.parquet.format.DataPageHeaderV2;
+import org.apache.parquet.format.DictionaryPageHeader;
+import org.apache.parquet.format.PageHeader;
+import org.apache.parquet.format.Util;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.schema.PrimitiveType;
+
+/**
+ * {@link PageReadStore} over one row group's cached column-chunk buffers, so that the vectorized
+ * column readers decode straight out of the LLAP cache: no ParquetFileReader, and no copy of the
+ * chunk bytes.
+ *
+ * <p><b>Why this is a copy.</b> Parquet does the same thing over a file, but none of it is
+ * reachable from here: {@code ParquetFileReader.Chunk.readAllPages} is private, the store it fills
+ * ({@code org.apache.parquet.hadoop.ColumnChunkPageReadStore}) is package-private and so is its
+ * {@code addColumn}, and the {@link PageReader} it puts in ({@code ColumnChunkPageReader}) is a
+ * package-private {@code final} class. Only the glue below is re-implemented; page header parsing,
+ * encoding and statistics conversion, the page value types and decompression are all parquet's own
+ * public API.
+ *
+ * <p><b>On a parquet upgrade.</b> Exactly two members track upstream code, each with a counterpart
+ * to diff against the new release:
+ * <ul>
+ * <li>{@link #readAllPages} mirrors {@code ParquetFileReader.Chunk.readAllPages}: the page-header
+ * walk, the page-type switch and the mapping of header fields onto {@link DataPageV1},
+ * {@link DataPageV2} and {@link DictionaryPage}. A new page type, a new header field or a
+ * changed page constructor lands here.</li>
+ * <li>{@link CachedChunkPageReader} mirrors {@code ColumnChunkPageReader}: page queueing and
+ * lazy decompression. New {@link PageReader} behaviour lands here.</li>
+ * </ul>
+ * Also check whether {@link PageReadStore} gained or changed a {@code default} method, since this
+ * class implements none of them ({@code getRowIndexOffset}, {@code getRowIndexes} and
+ * {@code close} are all inherited). {@code TestParquetCachedPageReadStore} compares this store page
+ * for page against the stock one and pins the parquet version, so the upgrade fails the build
+ * rather than passing with a divergence.
+ *
+ * <p>Deliberately unsupported, because LLAP's Parquet path does not use them: the offset index (so
+ * {@code getFirstRowIndex} and {@code getIndexRowCount} keep their no-offset-index defaults),
+ * column encryption, and page checksum verification.
+ *
+ * <p>How it works:
+ * <ol>
+ * <li><b>Input</b>: a {@link ParquetEncodedColumnBatch} holding the already-cached
+ * {@link MemoryBuffer}s for one row group's projected column chunks, plus the file footer.</li>
+ * <li><b>Stitching and parsing</b>: per chunk, {@link #chunkBuffers} turns the cache buffers into a
+ * list of {@link ByteBuffer} slices covering exactly the chunk's byte region; {@link #readAllPages}
+ * then walks that byte range with {@link Util#readPageHeader}, building {@link DataPageV1} /
+ * {@link DataPageV2} / {@link DictionaryPage} objects whose payloads are
+ * {@link ByteBufferInputStream} views over the cache buffers - the page bytes are never copied.</li>
+ * <li><b>Lazy decompression</b>: each {@link CachedChunkPageReader} keeps the raw bytes and its
+ * {@link BytesInputDecompressor}, and decompresses only when the vectorized column reader
+ * actually calls {@link PageReader#readPage()}, so pages that get pruned never pay the codec cost.</li>
+ * <li><b>Output</b>: the resulting {@code ColumnPath -> PageReader} map backs
+ * {@link #getPageReader(ColumnDescriptor)}, so parquet's VectorizedColumnReader never notices
+ * that it is not reading from a file.</li>
+ * </ol>
+ */
+class ParquetCachedPageReadStore implements PageReadStore {
+
+ private final Map<ColumnPath, PageReader> readers = new HashMap<>();
+ private final long rowCount;
+
+ ParquetCachedPageReadStore(ParquetMetadata footer, ParquetEncodedColumnBatch batch,
+ CompressionCodecFactory codecFactory, ParquetMetadataConverter converter) throws IOException {
+ BlockMetaData block = footer.getBlocks().get(batch.rowGroupIx());
+ this.rowCount = block.getRowCount();
+ String createdBy = footer.getFileMetaData().getCreatedBy();
+ ColumnChunkMetaData[] chunks = batch.chunks();
+ // pc is the projection index: chunks[pc] and the batch's three buffer arrays are all indexed by
+ // it, while the footer's block lists columns in file order. Keying on the path keeps the two
+ // apart, so a projection that drops columns from the middle of the schema still lines up.
+ for (int pc = 0; pc < chunks.length; ++pc) {
+ ColumnChunkMetaData chunk = chunks[pc];
+ readers.put(chunk.getPath(), readAllPages(chunk, chunkBuffers(batch, pc), createdBy,
+ codecFactory.getDecompressor(chunk.getCodec()), converter));
+ }
+ }
+
+ /**
+ * Slices of the cached buffers covering exactly the chunk's byte region, in file order.
+ *
+ * <p>This is the only place in the class that touches bytes, and the only place the cache appears
+ * at all: the {@link MemoryBuffer}s are already-populated LLAP cache memory, so there is no read
+ * here and nothing to decide - by the time this runs, a cache hit and a miss that
+ * {@code ParquetEncodedDataReader} had to fetch look exactly the same.
+ *
+ * <p>The buffers are cache ranges aligned to absolute file offsets, not to this chunk, so the
+ * first and last one usually overhang the chunk (the first may even start before it). That is what
+ * the trimming below is for.
+ *
+ * <p>The trimmed slices have to tile the chunk exactly, which is checked as they are built: a
+ * buffer that does not continue where the previous one ended, or one that reaches past the chunk,
+ * means the batch is not carrying the buffers the chunk was planned with. Without the check the
+ * page walk would read the wrong bytes as page bytes, and fail much later with a decoding error
+ * about a corrupt header.
+ */
+ private static List<ByteBuffer> chunkBuffers(ParquetEncodedColumnBatch batch, int pc)
+ throws IOException {
+ ColumnChunkMetaData chunk = batch.chunks()[pc];
+ long chunkStart = chunk.getStartingPos();
+ long chunkEnd = chunkStart + chunk.getTotalSize();
+ MemoryBuffer[] columnBuffers = batch.columnBuffers()[pc];
+ long[] bufferOffsets = batch.bufferOffsets()[pc];
+ int[] bufferLengths = batch.bufferLengths()[pc];
+ List<ByteBuffer> slices = new ArrayList<>(columnBuffers.length);
+ long covered = chunkStart;
+ for (int i = 0; i < columnBuffers.length; ++i) {
+ long bufferStart = bufferOffsets[i];
+ long bufferEnd = bufferStart + bufferLengths[i];
+ long sliceStart = Math.max(chunkStart, bufferStart);
+ long sliceEnd = Math.min(chunkEnd, bufferEnd);
+ if (sliceStart != covered || sliceEnd <= sliceStart) {
+ throw new IOException("Cached buffer " + (i + 1) + " of " + columnBuffers.length + " for column chunk "
+ + chunk.getPath() + " covers [" + sliceStart + ", " + sliceEnd + ") of the chunk at ["
+ + chunkStart + ", " + chunkEnd + "), which does not continue at " + covered);
+ }
+ covered = sliceEnd;
+ // A dup, because the cache buffer is shared: moving position/limit must not be visible to
+ // another reader of the same buffer. slice() then keeps a view of the trimmed region, so the
+ // chunk bytes are still the cache's bytes - nothing is copied here or below.
+ ByteBuffer bb = columnBuffers[i].getByteBufferDup();
+ bb.position(bb.position() + (int) (sliceStart - bufferStart));
+ bb.limit(bb.position() + (int) (sliceEnd - sliceStart));
+ slices.add(bb.slice());
+ }
+ if (covered != chunkEnd) {
+ throw new IOException("Cached buffers for column chunk " + chunk.getPath() + " cover ["
+ + chunkStart + ", " + covered + "), but the chunk ends at " + chunkEnd);
+ }
+ return slices;
+ }
+
+ private static PageReader readAllPages(ColumnChunkMetaData chunk, List<ByteBuffer> buffers,
+ String createdBy, BytesInputDecompressor decompressor, ParquetMetadataConverter converter)
+ throws IOException {
+ // Reads across the buffer boundaries as if the chunk were contiguous, which it is not: a page
+ // header or a payload can straddle two cache buffers. sliceBuffers below hands out views into
+ // these same buffers, so every page this method builds points at cache memory for as long as it
+ // lives - the batch has to keep its refs until the consumer is done decoding.
+ ByteBufferInputStream stream = ByteBufferInputStream.wrap(buffers);
+ PrimitiveType type = chunk.getPrimitiveType();
+ List<DataPage> pages = new ArrayList<>();
+ DictionaryPage dictionaryPage = null;
+ long valuesRead = 0;
+ // Only data pages count towards valuesRead, so the dictionary page and any page type skipped
+ // below do not end the walk early; the footer's value count is the only terminator.
+ while (valuesRead < chunk.getValueCount()) {
+ PageHeader header = Util.readPageHeader(stream);
+ int uncompressedSize = header.getUncompressed_page_size();
+ int compressedSize = header.getCompressed_page_size();
+ switch (header.getType()) {
+ case DICTIONARY_PAGE:
+ if (dictionaryPage != null) {
+ throw new ParquetDecodingException("more than one dictionary page in column " + chunk.getPath());
+ }
+ DictionaryPageHeader dictHeader = header.getDictionary_page_header();
+ dictionaryPage = new DictionaryPage(BytesInput.from(stream.sliceBuffers(compressedSize)),
+ uncompressedSize, dictHeader.getNum_values(), converter.getEncoding(dictHeader.getEncoding()));
+ break;
+ case DATA_PAGE:
+ DataPageHeader v1 = header.getData_page_header();
+ pages.add(new DataPageV1(BytesInput.from(stream.sliceBuffers(compressedSize)), v1.getNum_values(),
+ uncompressedSize, converter.fromParquetStatistics(createdBy, v1.getStatistics(), type),
+ converter.getEncoding(v1.getRepetition_level_encoding()),
+ converter.getEncoding(v1.getDefinition_level_encoding()),
+ converter.getEncoding(v1.getEncoding())));
+ valuesRead += v1.getNum_values();
+ break;
+ case DATA_PAGE_V2:
+ DataPageHeaderV2 v2 = header.getData_page_header_v2();
+ // In V2 the level bytes sit inside the page but are never compressed, so only what is left
+ // after them is codec output. The three slices have to be taken in this order.
+ int dataSize = compressedSize
+ - v2.getRepetition_levels_byte_length() - v2.getDefinition_levels_byte_length();
+ BytesInput repetitionLevels = BytesInput.from(stream.sliceBuffers(v2.getRepetition_levels_byte_length()));
+ BytesInput definitionLevels = BytesInput.from(stream.sliceBuffers(v2.getDefinition_levels_byte_length()));
+ BytesInput values = BytesInput.from(stream.sliceBuffers(dataSize));
+ pages.add(new DataPageV2(v2.getNum_rows(), v2.getNum_nulls(), v2.getNum_values(),
+ repetitionLevels, definitionLevels, converter.getEncoding(v2.getEncoding()), values,
+ uncompressedSize, converter.fromParquetStatistics(createdBy, v2.getStatistics(), type),
+ v2.isIs_compressed()));
+ valuesRead += v2.getNum_values();
+ break;
+ default:
+ // A page type this parquet release knows and we do not (an index page, say). Skipping it
+ // by its compressed size keeps the walk aligned on the next header, as parquet does.
+ stream.skipFully(compressedSize);
+ }
+ }
+ if (valuesRead != chunk.getValueCount()) {
+ throw new IOException("Expected " + chunk.getValueCount() + " values in column chunk " + chunk.getPath()
+ + " at offset " + chunk.getStartingPos() + " but got " + valuesRead + " over " + pages.size() + " pages");
+ }
+ return new CachedChunkPageReader(decompressor, pages, dictionaryPage);
+ }
+
+ @Override
+ public long getRowCount() {
+ return rowCount;
+ }
+
+ @Override
+ public PageReader getPageReader(ColumnDescriptor descriptor) {
+ return readers.get(ColumnPath.get(descriptor.getPath()));
+ }
+
+ private static final class CachedChunkPageReader implements PageReader {
+ private final BytesInputDecompressor decompressor;
+ private final Queue<DataPage> compressedPages;
+ private final DictionaryPage compressedDictionaryPage;
+ private final long valueCount;
+
+ CachedChunkPageReader(BytesInputDecompressor decompressor, List<DataPage> compressedPages,
+ DictionaryPage compressedDictionaryPage) {
+ this.decompressor = decompressor;
+ this.compressedPages = new ArrayDeque<>(compressedPages);
+ this.compressedDictionaryPage = compressedDictionaryPage;
+ long count = 0;
+ for (DataPage p : compressedPages) {
+ count += p.getValueCount();
+ }
+ this.valueCount = count;
+ }
+
+ @Override
+ public long getTotalValueCount() {
+ return valueCount;
+ }
+
+ /**
+ * Decompresses one page, on demand. This is where the cached bytes stop being the cache's: the
+ * decompressed copy is an ordinary heap/direct buffer that the column reader owns and drops after
+ * the batch. Pages the reader never asks for are never decompressed, so a pruned page costs
+ * nothing beyond the header parse.
+ */
+ @Override
+ public DataPage readPage() {
+ DataPage compressedPage = compressedPages.poll();
+ if (compressedPage == null) {
+ return null;
+ }
+ return compressedPage.accept(new DataPage.Visitor<DataPage>() {
+ @Override
+ public DataPage visit(DataPageV1 page) {
+ try {
+ return new DataPageV1(decompressor.decompress(page.getBytes(), page.getUncompressedSize()),
+ page.getValueCount(), page.getUncompressedSize(), page.getStatistics(),
+ page.getRlEncoding(), page.getDlEncoding(), page.getValueEncoding());
+ } catch (IOException e) {
+ throw new ParquetDecodingException("could not decompress page", e);
+ }
+ }
+
+ @Override
+ public DataPage visit(DataPageV2 page) {
+ if (!page.isCompressed()) {
+ // Handed straight through, still a view over the cache buffers. Safe for the same reason
+ // the whole class is: the batch holds the refs until the consumer finishes the row group.
+ return page;
+ }
+ // getUncompressedSize() covers the levels, which were not compressed, so the codec's own
+ // output size is what is left after them.
+ int uncompressedSize = Math.toIntExact(page.getUncompressedSize()
+ - page.getDefinitionLevels().size() - page.getRepetitionLevels().size());
+ BytesInput data;
+ try {
+ data = decompressor.decompress(page.getData(), uncompressedSize);
Review Comment:
ack, reused at 3 places
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4184232313
##########
llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIo.java:
##########
@@ -74,11 +74,15 @@ InputFormat<NullWritable, T> getInputFormat(
* @param fileKey fileId of the Parquet file (either the Long fileId of HDFS or the SyntheticFileId).
* Optional, if it is not provided, it will be generated, see:
* org.apache.hadoop.hive.ql.io.HdfsUtils#getFileId()
+ * @param cacheHit optional out-parameter; when supplied, its {@code value} field is written
+ * with {@code true} on a cache hit and {@code false} on a miss, so the caller
+ * (which lives in llap-server and holds {@code QueryFragmentCounters}) can bump
+ * {@code METADATA_CACHE_HIT} / {@code METADATA_CACHE_MISS} for the LLAP IO summary.
* @return
* @throws IOException
*/
- MemoryBufferOrBuffers getParquetFooterBuffersFromCache(Path path, JobConf conf, @Nullable Object fileKey)
- throws IOException;
+ MemoryBufferOrBuffers getParquetFooterBuffersFromCache(Path path, JobConf conf, @Nullable Object fileKey,
Review Comment:
this makes this patch even larger, but as it's already huge, I think I can take care of it here for better code quality
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4155530935
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ParquetEncodedDataConsumer.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.decode;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.type.DataTypePhysicalVariation;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.impl.ColumnVectorBatch;
+import org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl;
+import org.apache.hadoop.hive.llap.io.decode.ColumnVectorProducer.Includes;
+import org.apache.hadoop.hive.llap.io.encoded.ParquetEncodedColumnBatch;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonIOMetrics;
+import org.apache.hadoop.hive.ql.exec.vector.ColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.orc.encoded.Consumer;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetRowGroupDecoder;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedColumnReader;
+import org.apache.hadoop.hive.serde2.ColumnProjectionUtils;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
+import org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.parquet.HadoopReadOptions;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.compression.CompressionCodecFactory;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.metadata.FileMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+
+import com.google.common.base.Strings;
+
+/**
+ * The Parquet counterpart of {@link OrcEncodedDataConsumer}: it turns one row group's worth of
+ * cached column-chunk buffers ({@link ParquetEncodedColumnBatch}) into {@link ColumnVectorBatch}es
+ * and hands them downstream.
+ *
+ * <p>Rather than reimplementing Parquet value decoding, this consumer builds a
+ * {@link ParquetCachedPageReadStore} directly over the cached buffers (pages are parsed in place and
+ * decompressed lazily, no ParquetFileReader and no whole-chunk copy) and drives the very same
+ * {@link VectorizedColumnReader}s the non-cached vectorized reader uses (via {@link ParquetRowGroupDecoder}).
+ */
+public class ParquetEncodedDataConsumer
+ extends EncodedDataConsumer<Object, ParquetEncodedColumnBatch> {
+
+ private final Configuration jobConf;
+ private final boolean useDecimal64ColumnVectors;
+ private final CompressionCodecFactory codecFactory;
+ private final ParquetMetadataConverter converter;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private Path path;
+ private Map<String, Object> initialDefaults;
+
+ // Derived lazily (once) from the footer + job conf, then reused across row groups.
+ private List<TypeInfo> columnTypesList;
+ private List<Integer> colsToInclude;
+ private boolean readAllColumns;
+ private boolean skipTimestampConversion;
+ private boolean skipProlepticConversion;
+ private boolean legacyConversionEnabled;
+ private ZoneId writerTimezone;
+ private boolean schemaInitialized = false;
+
+ public ParquetEncodedDataConsumer(Consumer<ColumnVectorBatch> consumer, Includes includes,
+ QueryFragmentCounters counters, LlapDaemonIOMetrics ioMetrics, Configuration jobConf) {
+ super(consumer, includes.getPhysicalColumnIds().size(), ioMetrics, counters);
+ this.jobConf = jobConf;
+ this.useDecimal64ColumnVectors = HiveConf.getVar(jobConf,
+ ConfVars.HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_ENABLED)
+ .equalsIgnoreCase(HiveConf.HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_DECIMAL_64);
+ ParquetReadOptions options = HadoopReadOptions.builder(jobConf).build();
+ this.codecFactory = options.getCodecFactory();
+ this.converter = new ParquetMetadataConverter(options);
+ }
+
+ public void setFileMetadata(ParquetMetadata footer, MessageType requestedSchema, Path path) {
+ this.footer = footer;
+ this.requestedSchema = requestedSchema;
+ this.path = path;
+ }
+
+ public void setInitialDefaults(Map<String, Object> initialDefaults) {
+ this.initialDefaults = initialDefaults;
+ }
+
+ /**
+ * Derives the Hive type / projection lists and the conversion flags the same way
+ * {@link org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase} does from the job conf,
+ * so the shared {@link ParquetRowGroupDecoder} produces identical readers.
+ */
+ private void initSchema() {
+ FileMetaData fileMetaData = footer.getFileMetaData();
+ Map<String, String> kvMeta = fileMetaData.getKeyValueMetaData();
+
+ this.columnTypesList =
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES));
+ this.colsToInclude = ColumnProjectionUtils.getReadColumnIDs(jobConf);
+ this.readAllColumns = ColumnProjectionUtils.isReadAllColumns(jobConf);
+
+ this.writerTimezone = DataWritableReadSupport.getWriterTimeZoneId(kvMeta);
+ if (HiveConf.getBoolVar(jobConf, ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION)) {
+ this.skipTimestampConversion =
+ !Strings.nullToEmpty(fileMetaData.getCreatedBy()).startsWith("parquet-mr");
+ } else {
+ this.skipTimestampConversion = false;
+ }
+ Boolean proleptic = DataWritableReadSupport.getWriterDateProleptic(kvMeta);
+ if (proleptic == null) {
+ proleptic = HiveConf.getBoolVar(jobConf, ConfVars.HIVE_PARQUET_DATE_PROLEPTIC_GREGORIAN_DEFAULT);
+ }
+ this.skipProlepticConversion = proleptic;
+ this.legacyConversionEnabled = DataWritableReadSupport.getZoneConversionLegacy(kvMeta, jobConf);
+
+ this.schemaInitialized = true;
+ }
+
+ @Override
+ protected void decodeBatch(ParquetEncodedColumnBatch batch,
+ Consumer<ColumnVectorBatch> downstreamConsumer) throws InterruptedException {
+ if (!schemaInitialized) {
+ initSchema();
+ }
+
+ long startTime = counters.startTimeCounter();
+ // NOSONAR - S2093 does not apply: codecFactory is a per-consumer field with a per-batch
+ // release() (not close()); try-with-resources would tie it to this scope, which is wrong.
+ try { // NOSONAR
+ PageReadStore pages = new ParquetCachedPageReadStore(footer, batch, codecFactory, converter);
+
+ VectorizedColumnReader[] columnReaders =
+ new ParquetRowGroupDecoder(footer.getFileMetaData().getSchema(), initialDefaults).buildColumnReaders(
+ pages, requestedSchema, columnTypesList, colsToInclude, readAllColumns,
+ new ParquetRowGroupDecoder.TimestampConversionOptions(skipTimestampConversion,
+ writerTimezone, skipProlepticConversion, legacyConversionEnabled));
+
+ long rowsLeft = pages.getRowCount();
+ int batches = 0;
+ while (rowsLeft > 0) {
+ int batchSize = (int) Math.min(VectorizedRowBatch.DEFAULT_SIZE, rowsLeft);
+
+ ColumnVectorBatch cvb = takeBatch(batchSize);
+
+ // columnReaders[i] is requestedSchema field i, i.e. the i-th projected column.
+ for (int i = 0; i < columnReaders.length; ++i) {
+ if (columnReaders[i] == null) {
+ continue;
+ }
+ TypeInfo columnType = columnTypesList.get(readAllColumns ? i : colsToInclude.get(i));
+ ColumnVector cv = prepareColumnVector(cvb, i, columnType, batchSize);
+ columnReaders[i].readBatch(batchSize, cv, columnType);
+ }
+
+ downstreamConsumer.consumeData(cvb);
+ counters.incrCounter(LlapIOCounters.ROWS_EMITTED, batchSize);
+ rowsLeft -= batchSize;
+ ++batches;
+ }
+ counters.incrWallClockCounter(LlapIOCounters.DECODE_TIME_NS, startTime);
+ counters.incrCounter(LlapIOCounters.NUM_VECTOR_BATCHES, batches);
+ counters.incrCounter(LlapIOCounters.NUM_DECODED_BATCHES);
+ } catch (IOException | RuntimeException e) {
+ // parquet reports decode failures as runtime ParquetDecodingException.
+ LlapIoImpl.LOG.error("Parquet decodeBatch failed for rowGroup " + batch.rowGroupIx() + " of " + path, e);
+ downstreamConsumer.setError(e);
+ } finally {
+ // Returns the pooled Hadoop decompressors after each row group; getDecompressor re-creates
+ // them. codecFactory is a per-consumer field with a per-batch release() (not close()) —
+ // try-with-resources would tie it to the outer decodeBatch scope, which is wrong here.
+ codecFactory.release(); // NOSONAR - see comment above (S2093 does not apply)
+ }
+ }
+
+ private ColumnVector prepareColumnVector(ColumnVectorBatch cvb, int idx, TypeInfo columnType,
Review Comment:
once per split, where initSchema runs:
````
// non-null here: the native path only serves vectorized plans (F6), and the Vectorizer always fills the array.
this.variations = Utilities.getVectorizedRowBatchCtx(jobConf).getRowdataTypePhysicalVariations();
````
prepareColumnVector already has tableColId (readAllColumns ? i : colsToInclude.get(i), line 177):
````
ColumnVector cv = VectorizedBatchUtil.createColumnVector(columnType, variations[tableColId]);
````
so we could delete physicalVariation, useDecimal64ColumnVectors and the consumer's read of HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_ENABLED.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4156049773
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ // Fallback path: no cache key, so we read status + footer straight from HDFS. Time both
+ // under HDFS_TIME_NS to match how ORC accounts for cache-miss footer reads.
+ final FileSystem fs = path.getFileSystem(jobConf);
+ long hdfsStart = counters.startTimeCounter();
+ final FileStatus stat = fs.getFileStatus(path);
+ counters.recordHdfsTime(hdfsStart);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ hdfsStart = counters.startTimeCounter();
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ counters.recordHdfsTime(hdfsStart);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ long startTime = counters.startTimeCounter();
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Exception e) {
+ // A shutdown-triggered InterruptedException must not be silently reported as an ordinary
+ // consumer error: preserve the interrupt on the thread so any caller sees it.
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ }
+ consumer.setError(e);
+ } finally {
+ counters.incrWallClockCounter(LlapIOCounters.TOTAL_IO_TIME_NS, startTime);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ // The LLAP IO summary keys off TABLE / FILE / STRIPES; set them the same way ORC does so a
+ // native Parquet fragment shows up in the summary with the same fields populated.
+ if (cacheTag != null) {
+ counters.setDesc(QueryFragmentCounters.Desc.TABLE, cacheTag.getTableName());
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.FILE, path
+ + (fileKey == null ? "" : " (" + fileKey + ")"));
+
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = selectRowGroups(blocks, fileSchema);
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ // STRIPES is the ORC term; for Parquet the analog is row groups. Reuse the same descriptor so
+ // the summary layout stays common and we don't have to teach the reporter about a new field.
+ counters.setDesc(QueryFragmentCounters.Desc.STRIPES, "0," + selected.size());
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /**
+ * The row groups this split has to read: the ones whose first data page falls inside the split, as
+ * Parquet's own input format assigns them, minus any the pushed-down predicate rules out on its
+ * statistics. Returned in file order, since the fetch pipeline reads them in that order.
+ */
+ private List<BlockMetaData> selectRowGroups(List<BlockMetaData> blocks, MessageType fileSchema)
+ throws IOException {
+ long splitStart = split.getStart();
+ long splitEnd = splitStart + split.getLength();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().getFirst().getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate == null) {
+ return selected;
+ }
+ return RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Whether the query needs row-lineage virtual columns ({@code ROW__LINEAGE__ID} /
+ * {@code LAST__UPDATED__SEQUENCE__NUMBER}) that the fallback reader augments the requested schema
+ * with via {@link RowLineageUtils#getRequestedSchemaWithRowLineageColumns}. The native pipeline
+ * doesn't propagate that augmentation through {@code Includes} yet, so we defer to the fallback
+ * reader instead of quietly emitting nulls in the lineage slots.
+ */
+ public boolean needsRowLineage() {
+ VectorizedRowBatchCtx rbCtx = Utilities.getVectorizedRowBatchCtx(jobConf);
+ if (rbCtx == null) {
+ return false;
+ }
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ return RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema, VirtualColumn.ROW_LINEAGE_ID)
+ || RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema,
+ VirtualColumn.LAST_UPDATED_SEQUENCE_NUMBER);
+ }
+
+ /**
+ * Leaf-column positions of the requested fields; {@code BlockMetaData.getColumns()} is a flat list
+ * of leaves in file-schema order, so a top-level primitive at ordinal {@code k} in the file schema
+ * may sit at a very different leaf index (e.g. a nested group of two leaves before it shifts it
+ * from {@code 1} to {@code 2}). We only reach this method when every requested field is a
+ * top-level primitive ({@link #projectsNestedTypes()} would have forced a fallback otherwise), so
+ * each requested field maps to exactly one leaf whose path is a single segment.
+ */
+ static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
+ List<ColumnDescriptor> allLeaves = fileSchema.getColumns();
+ List<Integer> selected = new ArrayList<>();
+ for (Type field : requestedSchema.getFields()) {
+ if (!fileSchema.containsField(field.getName())) {
+ continue;
+ }
+ for (int i = 0; i < allLeaves.size(); ++i) {
+ String[] path = allLeaves.get(i).getPath();
+ // Single-segment path == top-level primitive; skip leaves buried inside a group.
+ if (path.length == 1 && path[0].equals(field.getName())) {
+ selected.add(i);
+ break;
+ }
+ }
+ }
+ return selected.stream().mapToInt(Integer::intValue).toArray();
+ }
+
+ private static long bytes(int[] projected, BlockMetaData block) {
+ long total = 0;
+ for (int leaf : projected) {
+ total += block.getColumns().get(leaf).getTotalSize();
+ }
+ return total;
+ }
+
+ private static long bytes(Fetch fetch) {
+ long total = 0;
+ for (ColumnChunkMetaData chunk : fetch.batch.chunks()) {
+ total += chunk.getTotalSize();
+ }
+ return total;
+ }
+
+ /** One row group on its way in: buffers planned, missing ranges requested, not yet decoded. */
+ private static final class Fetch {
+ private final ParquetEncodedColumnBatch batch = new ParquetEncodedColumnBatch();
+ private final List<ColumnPlan> columns = new ArrayList<>();
+ private final List<Part> misses = new ArrayList<>();
+ private final List<Run> runs = new ArrayList<>();
+ }
+
+ /** The buffers covering one column chunk, in file order, and which stretches of them are new. */
+ private record ColumnPlan(List<Part> parts, List<MissRun> missRuns) {
+ ColumnPlan() {
+ this(new ArrayList<>(), new ArrayList<>());
+ }
+ }
+
+ /** One cache buffer's worth of a column chunk: a hit handed back by the cache, or a miss to fill. */
+ private static final class Part {
+ private MemoryBuffer buffer;
+ private final DiskRange range;
+ /** We hold one ref to release; until then a miss is a raw allocation to free. */
+ private boolean owned;
+
+ Part(MemoryBuffer buffer, DiskRange range, boolean owned) {
+ this.buffer = buffer;
+ this.range = range;
+ this.owned = owned;
+ }
+ }
+
+ /** {@code count} consecutive parts covering one contiguous missing sub-range; cached as a unit. */
+ private record MissRun(int firstPart, int count) {
+ }
+
+ /** One vectored range and the cache buffers it fills, in file order. */
+ private record Run(FileRange range, List<Part> parts) {
+ }
+
+ private Fetch startFetch(FSDataInputStream fileStream, ParquetRangeBuffers buffers, Allocator allocator,
+ int[] projected, BlockMetaData block, int rg)
+ throws IOException {
+ Fetch fetch = new Fetch();
+ ColumnChunkMetaData[] chunks = new ColumnChunkMetaData[projected.length];
+ for (int pc = 0; pc < projected.length; ++pc) {
+ chunks[pc] = block.getColumns().get(projected[pc]);
+ }
+ fetch.batch.init(fileKey, rg, chunks);
+ try {
+ for (ColumnChunkMetaData chunk : chunks) {
+ ColumnPlan column = new ColumnPlan();
+ fetch.columns.add(column);
+ planColumnChunk(allocator, chunk.getStartingPos(),
+ chunk.getStartingPos() + chunk.getTotalSize(), column, fetch.misses);
+ }
+ // The vectored dispatch is where non-async FS impls actually read; count it under HDFS_TIME.
+ long hdfsStart = counters.startTimeCounter();
+ requestMisses(fileStream, buffers, fetch, layout.maxRangeBytes());
+ counters.recordHdfsTime(hdfsStart);
+ } catch (Exception e) {
+ abandon(allocator, fetch);
+ throw e;
+ }
+ return fetch;
+ }
+
+ private void finishFetch(Allocator allocator, ParquetRangeBuffers buffers, Fetch fetch)
+ throws IOException, InterruptedException {
+ try {
+ // Awaiting the vectored futures is where the miss bytes actually arrive from HDFS.
+ long hdfsStart = counters.startTimeCounter();
+ receiveMisses(buffers, fetch);
+ counters.recordHdfsTime(hdfsStart);
+ for (ColumnPlan column : fetch.columns) {
+ putColumn(allocator, column);
+ }
+ for (int pc = 0; pc < fetch.columns.size(); ++pc) {
+ assemble(fetch.batch, pc, fetch.columns.get(pc).parts);
+ }
+ // consumeData returns the batch on success; after a throw it is still ours.
+ consumer.consumeData(fetch.batch);
+ } catch (Exception e) {
+ abandon(allocator, fetch);
+ throw e;
+ }
+ }
+
+ /** Drops a fetch that will not be decoded: outstanding requests are cancelled, buffers released. */
+ private void abandon(Allocator allocator, Fetch fetch) {
+ for (Run run : fetch.runs) {
+ run.range.getData().cancel(true);
+ }
+ for (ColumnPlan column : fetch.columns) {
+ for (Part part : column.parts) {
+ if (part.owned) {
+ bufferManager.decRefBuffer(part.buffer);
+ } else {
+ allocator.deallocate(part.buffer);
+ }
+ }
+ }
+ }
+
+ /** Package-private so a test can observe the read pattern. */
+ FSDataInputStream openFile(FileSystem fs) throws IOException {
+ return fs.open(path);
+ }
+
+ /** Lets the allocator abandon a wait for memory once the fragment is cancelled. */
+ private void allocateMultiple(Allocator allocator, MemoryBuffer[] dest, int size) {
+ if (allocator instanceof StoppableAllocator stoppable) {
+ stoppable.allocateMultiple(dest, size, DATA_BUFFER_FACTORY, isStopped);
+ } else {
+ allocator.allocateMultiple(dest, size, DATA_BUFFER_FACTORY);
+ }
+ }
+
+ /**
+ * Works out which buffers cover column chunk {@code [start, end)} without touching the file: cache
+ * hits as returned by getFileData, and freshly allocated power-of-two buffers for everything
+ * missing. Misses are appended to {@code allMisses} so the whole row group can be read at once.
+ */
+ private void planColumnChunk(Allocator allocator, long start, long end,
Review Comment:
planChunk? turns cache hits and gaps into parts without reading
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4157036857
##########
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetRowGroupDecoder.java:
##########
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.ql.io.parquet.vector;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.parquet.ParquetRuntimeException;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.InvalidSchemaException;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Reusable, stateless helper that builds the {@link VectorizedColumnReader} array for one Parquet
+ * row group, given an already-read {@link PageReadStore}. The logic here was lifted verbatim from
+ * {@link VectorizedParquetRecordReader} so that both the row-by-split reader and the LLAP
+ * cache-backed consumer ({@code ParquetEncodedDataConsumer}) can share a single, behavior-preserving
+ * implementation of the Hive-type-driven Parquet column-reader construction.
+ *
+ * <p>The instance only carries the {@code fileSchema} (used for the schema-evolution check). All
+ * per-row-group state lives in the supplied {@link PageReadStore}.
+ */
+public class ParquetRowGroupDecoder {
+
+ private static final int MAP_DEFINITION_LEVEL_MAX = 3;
+
+ /**
+ * Bag of writer-timezone / proleptic / legacy conversion flags forwarded to every primitive
+ * column reader. Grouped so the reader-construction entry point stays under the parameter cap.
+ */
+ public record TimestampConversionOptions(
+ boolean skipTimestampConversion,
+ ZoneId writerTimezone,
+ boolean skipProlepticConversion,
+ boolean legacyConversionEnabled) {
+ }
+
+ private final MessageType fileSchema;
+ private final Map<String, Object> initialDefaults;
+
+ public ParquetRowGroupDecoder(MessageType fileSchema, Map<String, Object> initialDefaults) {
+ this.fileSchema = fileSchema;
+ this.initialDefaults = initialDefaults;
+ }
+
+ /**
+ * Builds the per-(requested-)column {@link VectorizedColumnReader} array for a row group.
+ *
+ * @param pages the row group's page store (from {@code reader.readRowGroup(..)}
+ * / {@code readNextRowGroup()})
+ * @param requestedSchema the projected Parquet schema being read
+ * @param columnTypesList the Hive type infos for ALL table columns (indexed by table col id)
+ * @param colsToInclude the table column ids being read, in requested-schema field order;
+ * may be empty (e.g. {@code count(*)}), in which case all readers are null
+ * @param readAllColumns whether projection is "read all columns"
+ * @param options writer-timezone / proleptic / legacy conversion flags, see
+ * {@link TimestampConversionOptions}
+ * @return one reader per requested-schema field (null entries where no reader is needed)
+ */
+ public VectorizedColumnReader[] buildColumnReaders(
+ PageReadStore pages,
+ MessageType requestedSchema,
+ List<TypeInfo> columnTypesList,
+ List<Integer> colsToInclude,
+ boolean readAllColumns,
+ TimestampConversionOptions options) throws IOException {
+ List<Type> types = requestedSchema.getFields();
+ VectorizedColumnReader[] columnReaders = new VectorizedColumnReader[types.size()];
+
+ if (!readAllColumns) {
+ // certain queries like select count(*) from table do not have
+ // any projected columns and still have isReadAllColumns as false
+ // in such cases columnReaders are not needed
+ // However, if colsToInclude is not empty we should initialize each columnReader
+ if (!colsToInclude.isEmpty()) {
+ for (int i = 0; i < types.size(); ++i) {
+ columnReaders[i] = buildVectorizedParquetReader(
+ columnTypesList.get(colsToInclude.get(i)), types.get(i), pages,
+ requestedSchema.getColumns(), options, 0, 0);
+ }
+ }
+ } else {
+ for (int i = 0; i < types.size(); ++i) {
+ columnReaders[i] = buildVectorizedParquetReader(columnTypesList.get(i),
+ types.get(i), pages, requestedSchema.getColumns(), options, 0, 0);
+ }
+ }
+ return columnReaders;
+ }
+
+ private static List<ColumnDescriptor> getAllColumnDescriptorByType(
+ int depth,
+ Type type,
+ List<ColumnDescriptor> columns) throws ParquetRuntimeException {
+ List<ColumnDescriptor> res = new ArrayList<>();
+ for (ColumnDescriptor descriptor : columns) {
+ if (depth >= descriptor.getPath().length) {
+ throw new InvalidSchemaException("Corrupted Parquet schema");
+ }
+ if (type.getName().equals(descriptor.getPath()[depth])) {
+ res.add(descriptor);
+ }
+ }
+ return res;
+ }
+
+ // Nested types are unsupported here on purpose; callers detect that in advance
+ // (ParquetEncodedDataReader.projectsNestedTypes) and route the query to the non-native reader.
+ private static PrimitiveType getElementType(Type type) {
+ if (type.isPrimitive()) {
+ return type.asPrimitiveType();
+ }
+ if (type.asGroupType().getFields().size() > 1) {
+ throw new UnsupportedOperationException(
+ "Current Parquet Vectorization reader doesn't support nested type");
+ }
+
+ Type childType = type.asGroupType().getFields().getFirst();
+
+ // Parquet file generated using thrift may have child type as PrimitiveType
+ if (childType.isPrimitive()) {
+ return childType.asPrimitiveType();
+ } else {
+ return childType.asGroupType().getFields().getFirst().asPrimitiveType();
+ }
+ }
+
+ // Build VectorizedParquetColumnReader via Hive typeInfo and Parquet schema
+ private VectorizedColumnReader buildVectorizedParquetReader(
+ TypeInfo typeInfo,
+ Type type,
+ PageReadStore pages,
+ List<ColumnDescriptor> columnDescriptors,
+ TimestampConversionOptions options,
+ int depth, int currentDefLevel) throws IOException {
+ int typeDefLevel = currentDefLevel;
+ if (type.isRepetition(Type.Repetition.OPTIONAL) || type.isRepetition(Type.Repetition.REPEATED)) {
+ typeDefLevel++;
+ }
+ List<ColumnDescriptor> descriptors =
+ getAllColumnDescriptorByType(depth, type, columnDescriptors);
+ // Support for schema evolution: if the column from the current
+ // query schema is not present in the file schema, return a dummy
+ // reader that produces nulls. This allows queries to proceed even
+ // when new columns have been added after the file was written.
+ if (!fileSchema.getColumns().contains(descriptors.getFirst())) {
+ return new VectorizedDummyColumnReader(Optional.ofNullable(initialDefaults)
+ .map(defaults -> defaults.getOrDefault(descriptors.getFirst().getPath()[0], null)).orElse(null));
+ }
+ switch (typeInfo.getCategory()) {
Review Comment:
1. The three columnDescriptors == null || isEmpty() checks (PRIMITIVE, LIST, MAP) are unreachable. They test the input list, but descriptors.getFirst() on line 175 already ran before the switch; an empty input makes that throw NoSuchElementException first. Replace all three with one check before line 175, if (descriptors.isEmpty()) throw new InvalidSchemaException(…), which also gives the schema-evolution lookup a real precondition and the right exception type.
2. STRUCT's if (r != null) … else throw IllegalStateException is dead: buildVectorizedParquetReader never returns null, every path returns a reader or throws.
3. new VectorizedListColumnReader(descriptor, pages.getPageReader(descriptor), options.skipTimestampConversion(), options.writerTimezone(), options.skipProlepticConversion(), options.legacyConversionEnabled(), elementType, typeInfo) appears three times (LIST, MAP key, MAP value). A private listReader(ColumnDescriptor, Type elementType, TypeInfo) collapses them.
4. Optional.ofNullable(initialDefaults).map(…).orElse(null) is a null check wearing an Optional: initialDefaults == null ? null : initialDefaults.get(name).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4217414628
##########
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetRowGroupDecoder.java:
##########
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.ql.io.parquet.vector;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.parquet.ParquetRuntimeException;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.InvalidSchemaException;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Reusable, stateless helper that builds the {@link VectorizedColumnReader} array for one Parquet
+ * row group, given an already-read {@link PageReadStore}. The logic here was lifted verbatim from
+ * {@link VectorizedParquetRecordReader} so that both the row-by-split reader and the LLAP
+ * cache-backed consumer ({@code ParquetEncodedDataConsumer}) can share a single, behavior-preserving
+ * implementation of the Hive-type-driven Parquet column-reader construction.
+ *
+ * <p>The instance only carries the {@code fileSchema} (used for the schema-evolution check). All
+ * per-row-group state lives in the supplied {@link PageReadStore}.
+ */
+public class ParquetRowGroupDecoder {
+
+ private static final int MAP_DEFINITION_LEVEL_MAX = 3;
Review Comment:
ack, added also a comment to explain how different layouts can exist
the layout could be one of the following (not added to code, only here):
```
// canonical
optional group m (MAP) { // 1 field
repeated group key_value { // 2 fields <- the walk stops here
required binary key (STRING);
optional int32 value;
}
}
// names may differ in existing data; key and value are taken by position
optional group m (MAP) { // 1 field
repeated group map { // 2 fields <- stops here
required binary str (STRING);
required int32 num;
}
}
// MAP_KEY_VALUE written in place of MAP, with no MAP-annotated group above it
optional group m (MAP_KEY_VALUE) { // 1 field
repeated group map { // 2 fields <- stops here
required binary key (STRING);
optional int32 value;
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4217414628
##########
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetRowGroupDecoder.java:
##########
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.ql.io.parquet.vector;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.parquet.ParquetRuntimeException;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.InvalidSchemaException;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Reusable, stateless helper that builds the {@link VectorizedColumnReader} array for one Parquet
+ * row group, given an already-read {@link PageReadStore}. The logic here was lifted verbatim from
+ * {@link VectorizedParquetRecordReader} so that both the row-by-split reader and the LLAP
+ * cache-backed consumer ({@code ParquetEncodedDataConsumer}) can share a single, behavior-preserving
+ * implementation of the Hive-type-driven Parquet column-reader construction.
+ *
+ * <p>The instance only carries the {@code fileSchema} (used for the schema-evolution check). All
+ * per-row-group state lives in the supplied {@link PageReadStore}.
+ */
+public class ParquetRowGroupDecoder {
+
+ private static final int MAP_DEFINITION_LEVEL_MAX = 3;
Review Comment:
ack, added also a comment to explain different layouts
the layout could be one of the following (not added to code, only here):
```
// canonical
optional group m (MAP) { // 1 field
repeated group key_value { // 2 fields <- the walk stops here
required binary key (STRING);
optional int32 value;
}
}
// names may differ in existing data; key and value are taken by position
optional group m (MAP) { // 1 field
repeated group map { // 2 fields <- stops here
required binary str (STRING);
required int32 num;
}
}
// MAP_KEY_VALUE written in place of MAP, with no MAP-annotated group above it
optional group m (MAP_KEY_VALUE) { // 1 field
repeated group map { // 2 fields <- stops here
required binary key (STRING);
optional int32 value;
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4156001830
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ // Fallback path: no cache key, so we read status + footer straight from HDFS. Time both
+ // under HDFS_TIME_NS to match how ORC accounts for cache-miss footer reads.
+ final FileSystem fs = path.getFileSystem(jobConf);
+ long hdfsStart = counters.startTimeCounter();
+ final FileStatus stat = fs.getFileStatus(path);
+ counters.recordHdfsTime(hdfsStart);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ hdfsStart = counters.startTimeCounter();
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ counters.recordHdfsTime(hdfsStart);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ long startTime = counters.startTimeCounter();
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Exception e) {
+ // A shutdown-triggered InterruptedException must not be silently reported as an ordinary
+ // consumer error: preserve the interrupt on the thread so any caller sees it.
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ }
+ consumer.setError(e);
+ } finally {
+ counters.incrWallClockCounter(LlapIOCounters.TOTAL_IO_TIME_NS, startTime);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ // The LLAP IO summary keys off TABLE / FILE / STRIPES; set them the same way ORC does so a
+ // native Parquet fragment shows up in the summary with the same fields populated.
+ if (cacheTag != null) {
+ counters.setDesc(QueryFragmentCounters.Desc.TABLE, cacheTag.getTableName());
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.FILE, path
+ + (fileKey == null ? "" : " (" + fileKey + ")"));
+
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = selectRowGroups(blocks, fileSchema);
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ // STRIPES is the ORC term; for Parquet the analog is row groups. Reuse the same descriptor so
+ // the summary layout stays common and we don't have to teach the reporter about a new field.
+ counters.setDesc(QueryFragmentCounters.Desc.STRIPES, "0," + selected.size());
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /**
+ * The row groups this split has to read: the ones whose first data page falls inside the split, as
+ * Parquet's own input format assigns them, minus any the pushed-down predicate rules out on its
+ * statistics. Returned in file order, since the fetch pipeline reads them in that order.
+ */
+ private List<BlockMetaData> selectRowGroups(List<BlockMetaData> blocks, MessageType fileSchema)
+ throws IOException {
+ long splitStart = split.getStart();
+ long splitEnd = splitStart + split.getLength();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().getFirst().getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate == null) {
+ return selected;
+ }
+ return RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Whether the query needs row-lineage virtual columns ({@code ROW__LINEAGE__ID} /
+ * {@code LAST__UPDATED__SEQUENCE__NUMBER}) that the fallback reader augments the requested schema
+ * with via {@link RowLineageUtils#getRequestedSchemaWithRowLineageColumns}. The native pipeline
+ * doesn't propagate that augmentation through {@code Includes} yet, so we defer to the fallback
+ * reader instead of quietly emitting nulls in the lineage slots.
+ */
+ public boolean needsRowLineage() {
+ VectorizedRowBatchCtx rbCtx = Utilities.getVectorizedRowBatchCtx(jobConf);
+ if (rbCtx == null) {
+ return false;
+ }
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ return RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema, VirtualColumn.ROW_LINEAGE_ID)
+ || RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema,
+ VirtualColumn.LAST_UPDATED_SEQUENCE_NUMBER);
+ }
+
+ /**
+ * Leaf-column positions of the requested fields; {@code BlockMetaData.getColumns()} is a flat list
+ * of leaves in file-schema order, so a top-level primitive at ordinal {@code k} in the file schema
+ * may sit at a very different leaf index (e.g. a nested group of two leaves before it shifts it
+ * from {@code 1} to {@code 2}). We only reach this method when every requested field is a
+ * top-level primitive ({@link #projectsNestedTypes()} would have forced a fallback otherwise), so
+ * each requested field maps to exactly one leaf whose path is a single segment.
+ */
+ static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
+ List<ColumnDescriptor> allLeaves = fileSchema.getColumns();
+ List<Integer> selected = new ArrayList<>();
+ for (Type field : requestedSchema.getFields()) {
+ if (!fileSchema.containsField(field.getName())) {
+ continue;
+ }
+ for (int i = 0; i < allLeaves.size(); ++i) {
+ String[] path = allLeaves.get(i).getPath();
+ // Single-segment path == top-level primitive; skip leaves buried inside a group.
+ if (path.length == 1 && path[0].equals(field.getName())) {
+ selected.add(i);
+ break;
+ }
+ }
+ }
+ return selected.stream().mapToInt(Integer::intValue).toArray();
+ }
+
+ private static long bytes(int[] projected, BlockMetaData block) {
Review Comment:
private static long projectedChunkBytes(int[] leafIndexes, BlockMetaData block)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4156097197
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ // Fallback path: no cache key, so we read status + footer straight from HDFS. Time both
+ // under HDFS_TIME_NS to match how ORC accounts for cache-miss footer reads.
+ final FileSystem fs = path.getFileSystem(jobConf);
+ long hdfsStart = counters.startTimeCounter();
+ final FileStatus stat = fs.getFileStatus(path);
+ counters.recordHdfsTime(hdfsStart);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ hdfsStart = counters.startTimeCounter();
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ counters.recordHdfsTime(hdfsStart);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ long startTime = counters.startTimeCounter();
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Exception e) {
+ // A shutdown-triggered InterruptedException must not be silently reported as an ordinary
+ // consumer error: preserve the interrupt on the thread so any caller sees it.
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ }
+ consumer.setError(e);
+ } finally {
+ counters.incrWallClockCounter(LlapIOCounters.TOTAL_IO_TIME_NS, startTime);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ // The LLAP IO summary keys off TABLE / FILE / STRIPES; set them the same way ORC does so a
+ // native Parquet fragment shows up in the summary with the same fields populated.
+ if (cacheTag != null) {
+ counters.setDesc(QueryFragmentCounters.Desc.TABLE, cacheTag.getTableName());
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.FILE, path
+ + (fileKey == null ? "" : " (" + fileKey + ")"));
+
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = selectRowGroups(blocks, fileSchema);
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ // STRIPES is the ORC term; for Parquet the analog is row groups. Reuse the same descriptor so
+ // the summary layout stays common and we don't have to teach the reporter about a new field.
+ counters.setDesc(QueryFragmentCounters.Desc.STRIPES, "0," + selected.size());
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /**
+ * The row groups this split has to read: the ones whose first data page falls inside the split, as
+ * Parquet's own input format assigns them, minus any the pushed-down predicate rules out on its
+ * statistics. Returned in file order, since the fetch pipeline reads them in that order.
+ */
+ private List<BlockMetaData> selectRowGroups(List<BlockMetaData> blocks, MessageType fileSchema)
+ throws IOException {
+ long splitStart = split.getStart();
+ long splitEnd = splitStart + split.getLength();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().getFirst().getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate == null) {
+ return selected;
+ }
+ return RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Whether the query needs row-lineage virtual columns ({@code ROW__LINEAGE__ID} /
+ * {@code LAST__UPDATED__SEQUENCE__NUMBER}) that the fallback reader augments the requested schema
+ * with via {@link RowLineageUtils#getRequestedSchemaWithRowLineageColumns}. The native pipeline
+ * doesn't propagate that augmentation through {@code Includes} yet, so we defer to the fallback
+ * reader instead of quietly emitting nulls in the lineage slots.
+ */
+ public boolean needsRowLineage() {
+ VectorizedRowBatchCtx rbCtx = Utilities.getVectorizedRowBatchCtx(jobConf);
+ if (rbCtx == null) {
+ return false;
+ }
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ return RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema, VirtualColumn.ROW_LINEAGE_ID)
+ || RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema,
+ VirtualColumn.LAST_UPDATED_SEQUENCE_NUMBER);
+ }
+
+ /**
+ * Leaf-column positions of the requested fields; {@code BlockMetaData.getColumns()} is a flat list
+ * of leaves in file-schema order, so a top-level primitive at ordinal {@code k} in the file schema
+ * may sit at a very different leaf index (e.g. a nested group of two leaves before it shifts it
+ * from {@code 1} to {@code 2}). We only reach this method when every requested field is a
+ * top-level primitive ({@link #projectsNestedTypes()} would have forced a fallback otherwise), so
+ * each requested field maps to exactly one leaf whose path is a single segment.
+ */
+ static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
+ List<ColumnDescriptor> allLeaves = fileSchema.getColumns();
+ List<Integer> selected = new ArrayList<>();
+ for (Type field : requestedSchema.getFields()) {
+ if (!fileSchema.containsField(field.getName())) {
+ continue;
+ }
+ for (int i = 0; i < allLeaves.size(); ++i) {
+ String[] path = allLeaves.get(i).getPath();
+ // Single-segment path == top-level primitive; skip leaves buried inside a group.
+ if (path.length == 1 && path[0].equals(field.getName())) {
+ selected.add(i);
+ break;
+ }
+ }
+ }
+ return selected.stream().mapToInt(Integer::intValue).toArray();
+ }
+
+ private static long bytes(int[] projected, BlockMetaData block) {
+ long total = 0;
+ for (int leaf : projected) {
+ total += block.getColumns().get(leaf).getTotalSize();
+ }
+ return total;
+ }
+
+ private static long bytes(Fetch fetch) {
+ long total = 0;
+ for (ColumnChunkMetaData chunk : fetch.batch.chunks()) {
+ total += chunk.getTotalSize();
+ }
+ return total;
+ }
+
+ /** One row group on its way in: buffers planned, missing ranges requested, not yet decoded. */
+ private static final class Fetch {
+ private final ParquetEncodedColumnBatch batch = new ParquetEncodedColumnBatch();
+ private final List<ColumnPlan> columns = new ArrayList<>();
+ private final List<Part> misses = new ArrayList<>();
+ private final List<Run> runs = new ArrayList<>();
+ }
+
+ /** The buffers covering one column chunk, in file order, and which stretches of them are new. */
+ private record ColumnPlan(List<Part> parts, List<MissRun> missRuns) {
Review Comment:
ChunkPlan with fields parts, gaps ? - It is per column chunk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4156953624
##########
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetRowGroupDecoder.java:
##########
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.ql.io.parquet.vector;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.parquet.ParquetRuntimeException;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.InvalidSchemaException;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Reusable, stateless helper that builds the {@link VectorizedColumnReader} array for one Parquet
+ * row group, given an already-read {@link PageReadStore}. The logic here was lifted verbatim from
+ * {@link VectorizedParquetRecordReader} so that both the row-by-split reader and the LLAP
+ * cache-backed consumer ({@code ParquetEncodedDataConsumer}) can share a single, behavior-preserving
+ * implementation of the Hive-type-driven Parquet column-reader construction.
+ *
+ * <p>The instance only carries the {@code fileSchema} (used for the schema-evolution check). All
+ * per-row-group state lives in the supplied {@link PageReadStore}.
+ */
+public class ParquetRowGroupDecoder {
+
+ private static final int MAP_DEFINITION_LEVEL_MAX = 3;
+
+ /**
+ * Bag of writer-timezone / proleptic / legacy conversion flags forwarded to every primitive
+ * column reader. Grouped so the reader-construction entry point stays under the parameter cap.
+ */
+ public record TimestampConversionOptions(
+ boolean skipTimestampConversion,
+ ZoneId writerTimezone,
+ boolean skipProlepticConversion,
+ boolean legacyConversionEnabled) {
+ }
+
+ private final MessageType fileSchema;
+ private final Map<String, Object> initialDefaults;
+
+ public ParquetRowGroupDecoder(MessageType fileSchema, Map<String, Object> initialDefaults) {
+ this.fileSchema = fileSchema;
+ this.initialDefaults = initialDefaults;
+ }
+
+ /**
+ * Builds the per-(requested-)column {@link VectorizedColumnReader} array for a row group.
+ *
+ * @param pages the row group's page store (from {@code reader.readRowGroup(..)}
+ * / {@code readNextRowGroup()})
+ * @param requestedSchema the projected Parquet schema being read
+ * @param columnTypesList the Hive type infos for ALL table columns (indexed by table col id)
+ * @param colsToInclude the table column ids being read, in requested-schema field order;
+ * may be empty (e.g. {@code count(*)}), in which case all readers are null
+ * @param readAllColumns whether projection is "read all columns"
+ * @param options writer-timezone / proleptic / legacy conversion flags, see
+ * {@link TimestampConversionOptions}
+ * @return one reader per requested-schema field (null entries where no reader is needed)
+ */
+ public VectorizedColumnReader[] buildColumnReaders(
+ PageReadStore pages,
+ MessageType requestedSchema,
+ List<TypeInfo> columnTypesList,
+ List<Integer> colsToInclude,
+ boolean readAllColumns,
+ TimestampConversionOptions options) throws IOException {
+ List<Type> types = requestedSchema.getFields();
+ VectorizedColumnReader[] columnReaders = new VectorizedColumnReader[types.size()];
+
+ if (!readAllColumns) {
+ // certain queries like select count(*) from table do not have
+ // any projected columns and still have isReadAllColumns as false
+ // in such cases columnReaders are not needed
+ // However, if colsToInclude is not empty we should initialize each columnReader
+ if (!colsToInclude.isEmpty()) {
+ for (int i = 0; i < types.size(); ++i) {
+ columnReaders[i] = buildVectorizedParquetReader(
+ columnTypesList.get(colsToInclude.get(i)), types.get(i), pages,
+ requestedSchema.getColumns(), options, 0, 0);
+ }
+ }
+ } else {
+ for (int i = 0; i < types.size(); ++i) {
+ columnReaders[i] = buildVectorizedParquetReader(columnTypesList.get(i),
+ types.get(i), pages, requestedSchema.getColumns(), options, 0, 0);
+ }
+ }
+ return columnReaders;
+ }
+
+ private static List<ColumnDescriptor> getAllColumnDescriptorByType(
+ int depth,
+ Type type,
+ List<ColumnDescriptor> columns) throws ParquetRuntimeException {
+ List<ColumnDescriptor> res = new ArrayList<>();
+ for (ColumnDescriptor descriptor : columns) {
+ if (depth >= descriptor.getPath().length) {
+ throw new InvalidSchemaException("Corrupted Parquet schema");
+ }
+ if (type.getName().equals(descriptor.getPath()[depth])) {
+ res.add(descriptor);
+ }
+ }
+ return res;
+ }
+
+ // Nested types are unsupported here on purpose; callers detect that in advance
+ // (ParquetEncodedDataReader.projectsNestedTypes) and route the query to the non-native reader.
+ private static PrimitiveType getElementType(Type type) {
+ if (type.isPrimitive()) {
+ return type.asPrimitiveType();
+ }
+ if (type.asGroupType().getFields().size() > 1) {
+ throw new UnsupportedOperationException(
+ "Current Parquet Vectorization reader doesn't support nested type");
+ }
+
+ Type childType = type.asGroupType().getFields().getFirst();
+
+ // Parquet file generated using thrift may have child type as PrimitiveType
+ if (childType.isPrimitive()) {
+ return childType.asPrimitiveType();
+ } else {
+ return childType.asGroupType().getFields().getFirst().asPrimitiveType();
+ }
+ }
+
+ // Build VectorizedParquetColumnReader via Hive typeInfo and Parquet schema
+ private VectorizedColumnReader buildVectorizedParquetReader(
+ TypeInfo typeInfo,
+ Type type,
+ PageReadStore pages,
+ List<ColumnDescriptor> columnDescriptors,
+ TimestampConversionOptions options,
+ int depth, int currentDefLevel) throws IOException {
+ int typeDefLevel = currentDefLevel;
+ if (type.isRepetition(Type.Repetition.OPTIONAL) || type.isRepetition(Type.Repetition.REPEATED)) {
+ typeDefLevel++;
+ }
+ List<ColumnDescriptor> descriptors =
+ getAllColumnDescriptorByType(depth, type, columnDescriptors);
+ // Support for schema evolution: if the column from the current
+ // query schema is not present in the file schema, return a dummy
+ // reader that produces nulls. This allows queries to proceed even
+ // when new columns have been added after the file was written.
+ if (!fileSchema.getColumns().contains(descriptors.getFirst())) {
+ return new VectorizedDummyColumnReader(Optional.ofNullable(initialDefaults)
+ .map(defaults -> defaults.getOrDefault(descriptors.getFirst().getPath()[0], null)).orElse(null));
+ }
+ switch (typeInfo.getCategory()) {
+ case PRIMITIVE:
Review Comment:
is formatting good?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6793:
URL: https://github.com/apache/hive/pull/6793#issuecomment-5934795250
Thanks @abstractdog for the PR! added some comments. in future we should probably try to create smaller changes, wa really painful to review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "aturoczy (via GitHub)" <gi...@apache.org>.
aturoczy commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4027531809
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -4958,6 +4958,9 @@ public static enum ConfVars {
LLAP_IO_ENCODE_ENABLED("hive.llap.io.encode.enabled", true,
"Whether LLAP should try to re-encode and cache data for non-ORC formats. This is used\n" +
"on LLAP Server side to determine if the infrastructure for that is initialized."),
+ LLAP_IO_PARQUET_NATIVE_ENABLED("hive.llap.io.parquet.native.enabled", false,
Review Comment:
Wouldn't be this enabled by default?
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java:
##########
@@ -350,6 +353,11 @@ public void start() {
private boolean checkOrcSchemaEvolution() {
SchemaEvolution evolution = rp.getSchemaEvolution();
+ if (evolution == null) {
+ // No ORC-style schema evolution to validate (e.g. native parquet path);
Review Comment:
no orc-style?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4156091885
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ // Fallback path: no cache key, so we read status + footer straight from HDFS. Time both
+ // under HDFS_TIME_NS to match how ORC accounts for cache-miss footer reads.
+ final FileSystem fs = path.getFileSystem(jobConf);
+ long hdfsStart = counters.startTimeCounter();
+ final FileStatus stat = fs.getFileStatus(path);
+ counters.recordHdfsTime(hdfsStart);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ hdfsStart = counters.startTimeCounter();
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ counters.recordHdfsTime(hdfsStart);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ long startTime = counters.startTimeCounter();
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Exception e) {
+ // A shutdown-triggered InterruptedException must not be silently reported as an ordinary
+ // consumer error: preserve the interrupt on the thread so any caller sees it.
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ }
+ consumer.setError(e);
+ } finally {
+ counters.incrWallClockCounter(LlapIOCounters.TOTAL_IO_TIME_NS, startTime);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ // The LLAP IO summary keys off TABLE / FILE / STRIPES; set them the same way ORC does so a
+ // native Parquet fragment shows up in the summary with the same fields populated.
+ if (cacheTag != null) {
+ counters.setDesc(QueryFragmentCounters.Desc.TABLE, cacheTag.getTableName());
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.FILE, path
+ + (fileKey == null ? "" : " (" + fileKey + ")"));
+
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = selectRowGroups(blocks, fileSchema);
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ // STRIPES is the ORC term; for Parquet the analog is row groups. Reuse the same descriptor so
+ // the summary layout stays common and we don't have to teach the reporter about a new field.
+ counters.setDesc(QueryFragmentCounters.Desc.STRIPES, "0," + selected.size());
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /**
+ * The row groups this split has to read: the ones whose first data page falls inside the split, as
+ * Parquet's own input format assigns them, minus any the pushed-down predicate rules out on its
+ * statistics. Returned in file order, since the fetch pipeline reads them in that order.
+ */
+ private List<BlockMetaData> selectRowGroups(List<BlockMetaData> blocks, MessageType fileSchema)
+ throws IOException {
+ long splitStart = split.getStart();
+ long splitEnd = splitStart + split.getLength();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().getFirst().getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate == null) {
+ return selected;
+ }
+ return RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Whether the query needs row-lineage virtual columns ({@code ROW__LINEAGE__ID} /
+ * {@code LAST__UPDATED__SEQUENCE__NUMBER}) that the fallback reader augments the requested schema
+ * with via {@link RowLineageUtils#getRequestedSchemaWithRowLineageColumns}. The native pipeline
+ * doesn't propagate that augmentation through {@code Includes} yet, so we defer to the fallback
+ * reader instead of quietly emitting nulls in the lineage slots.
+ */
+ public boolean needsRowLineage() {
+ VectorizedRowBatchCtx rbCtx = Utilities.getVectorizedRowBatchCtx(jobConf);
+ if (rbCtx == null) {
+ return false;
+ }
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ return RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema, VirtualColumn.ROW_LINEAGE_ID)
+ || RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema,
+ VirtualColumn.LAST_UPDATED_SEQUENCE_NUMBER);
+ }
+
+ /**
+ * Leaf-column positions of the requested fields; {@code BlockMetaData.getColumns()} is a flat list
+ * of leaves in file-schema order, so a top-level primitive at ordinal {@code k} in the file schema
+ * may sit at a very different leaf index (e.g. a nested group of two leaves before it shifts it
+ * from {@code 1} to {@code 2}). We only reach this method when every requested field is a
+ * top-level primitive ({@link #projectsNestedTypes()} would have forced a fallback otherwise), so
+ * each requested field maps to exactly one leaf whose path is a single segment.
+ */
+ static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
+ List<ColumnDescriptor> allLeaves = fileSchema.getColumns();
+ List<Integer> selected = new ArrayList<>();
+ for (Type field : requestedSchema.getFields()) {
+ if (!fileSchema.containsField(field.getName())) {
+ continue;
+ }
+ for (int i = 0; i < allLeaves.size(); ++i) {
+ String[] path = allLeaves.get(i).getPath();
+ // Single-segment path == top-level primitive; skip leaves buried inside a group.
+ if (path.length == 1 && path[0].equals(field.getName())) {
+ selected.add(i);
+ break;
+ }
+ }
+ }
+ return selected.stream().mapToInt(Integer::intValue).toArray();
+ }
+
+ private static long bytes(int[] projected, BlockMetaData block) {
+ long total = 0;
+ for (int leaf : projected) {
+ total += block.getColumns().get(leaf).getTotalSize();
+ }
+ return total;
+ }
+
+ private static long bytes(Fetch fetch) {
+ long total = 0;
+ for (ColumnChunkMetaData chunk : fetch.batch.chunks()) {
+ total += chunk.getTotalSize();
+ }
+ return total;
+ }
+
+ /** One row group on its way in: buffers planned, missing ranges requested, not yet decoded. */
+ private static final class Fetch {
Review Comment:
RowGroupFetch ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4155079140
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java:
##########
@@ -171,7 +178,8 @@ private LlapIoImpl(Configuration conf) throws IOException {
MetadataCache metadataCache = null;
SerDeLowLevelCacheImpl serdeCache = null; // TODO: extract interface when needed
- BufferUsageManager bufferManagerOrc = null, bufferManagerGeneric = null;
+ BufferUsageManager bufferManagerData = null;
Review Comment:
I think `bufferManagerData` is pure redundancy with the `bufferManager` field and should be removed in favor of passing bufferManager directly, while bufferManagerGeneric is legitimately distinct and should stay
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4156647117
##########
ql/src/java/org/apache/hadoop/hive/llap/ParquetCacheLayout.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.llap;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.DataCache.DiskRangeListFactory;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+
+/**
+ * How Parquet column chunks are laid out in the LLAP cache and fetched from the file. Everything
+ * that caches a chunk goes through this, so a reader finding only part of one cached re-caches
+ * the gap on the boundaries the first reader used. A reader builds one from the daemon's allocator
+ * and configuration and uses it for every chunk, which is what keeps its layout consistent.
+ */
+public final class ParquetCacheLayout {
+
+ /**
+ * An object store fetches separate ranges concurrently but never splits one, so cutting buys
+ * parallelism while a large range amortises the request. Measured against S3 the knee is here:
+ * below 4 Mb throughput drops sharply, above 8 Mb it is flat. Trino also splits at 8 Mb.
+ */
+ private static final int MIN_RANGE_CAP_BYTES = 8 << 20;
+
+ public static final DiskRangeListFactory CACHE_CHUNK_FACTORY = CacheChunk::new;
+
+ private final int maxBuffer;
+ private final int minBuffer;
+ private final int maxRange;
Review Comment:
rangeCap ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4030353671
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,560 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ColumnVectorProducer.Includes;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final Includes includes;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split, Includes includes,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.includes = includes;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ final FileSystem fs = path.getFileSystem(jobConf);
+ final FileStatus stat = fs.getFileStatus(path);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Throwable t) {
+ consumer.setError(t);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final int maxAlloc = allocator.getMaxAllocation();
+ final long splitStart = split.getStart(), splitEnd = splitStart + split.getLength();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().get(0).getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate != null) {
+ selected = RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, maxAlloc, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, maxAlloc, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /** File-schema positions of the requested fields; column chunks follow the schema order. */
+ private static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
+ List<Integer> leaves = new ArrayList<>();
+ for (Type field : requestedSchema.getFields()) {
+ if (fileSchema.containsField(field.getName())) {
+ leaves.add(fileSchema.getFieldIndex(field.getName()));
+ }
+ }
+ return leaves.stream().mapToInt(Integer::intValue).toArray();
Review Comment:
`getFieldIndex` is an index among top-level fields, but `BlockMetaData.getColumns()` is flattened leaf-column order. For a file schema such as `struct nested {a,b}, x` and a projection of `x`, this selects `nested.b` instead of `x` (or makes the page reader missing), because the native fallback only rejects nested fields in the requested projection. Map each requested primitive to its matching leaf descriptor in `fileSchema.getColumns()` instead of using the top-level field index, and add a regression case with nested columns preceding a projected primitive.
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,560 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ColumnVectorProducer.Includes;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final Includes includes;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split, Includes includes,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.includes = includes;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ final FileSystem fs = path.getFileSystem(jobConf);
+ final FileStatus stat = fs.getFileStatus(path);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Throwable t) {
+ consumer.setError(t);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final int maxAlloc = allocator.getMaxAllocation();
+ final long splitStart = split.getStart(), splitEnd = splitStart + split.getLength();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().get(0).getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate != null) {
+ selected = RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, maxAlloc, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, maxAlloc, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /** File-schema positions of the requested fields; column chunks follow the schema order. */
+ private static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
+ List<Integer> leaves = new ArrayList<>();
+ for (Type field : requestedSchema.getFields()) {
+ if (fileSchema.containsField(field.getName())) {
+ leaves.add(fileSchema.getFieldIndex(field.getName()));
+ }
+ }
+ return leaves.stream().mapToInt(Integer::intValue).toArray();
+ }
+
+ private static long bytes(int[] projected, BlockMetaData block) {
+ long total = 0;
+ for (int leaf : projected) {
+ total += block.getColumns().get(leaf).getTotalSize();
+ }
+ return total;
+ }
+
+ private static long bytes(Fetch fetch) {
+ long total = 0;
+ for (ColumnChunkMetaData chunk : fetch.batch.chunks) {
+ total += chunk.getTotalSize();
+ }
+ return total;
+ }
+
+ /** One row group on its way in: buffers planned, missing ranges requested, not yet decoded. */
+ private static final class Fetch {
+ private final ParquetEncodedColumnBatch batch = new ParquetEncodedColumnBatch();
+ private final List<ColumnPlan> columns = new ArrayList<>();
+ private final List<Part> misses = new ArrayList<>();
+ private final List<Run> runs = new ArrayList<>();
+ }
+
+ /** The buffers covering one column chunk, in file order, and which stretches of them are new. */
+ private record ColumnPlan(List<Part> parts, List<MissRun> missRuns) {
+ ColumnPlan() {
+ this(new ArrayList<>(), new ArrayList<>());
+ }
+ }
+
+ /** One cache buffer's worth of a column chunk: a hit handed back by the cache, or a miss to fill. */
+ private static final class Part {
+ private MemoryBuffer buffer;
+ private final DiskRange range;
+ private final boolean miss;
+ /** We hold one ref to release; until then a miss is a raw allocation to free. */
+ private boolean owned;
+
+ Part(MemoryBuffer buffer, DiskRange range, boolean miss) {
+ this.buffer = buffer;
+ this.range = range;
+ this.miss = miss;
+ this.owned = !miss;
+ }
+ }
+
+ /** {@code count} consecutive parts covering one contiguous missing sub-range; cached as a unit. */
+ private record MissRun(int firstPart, int count) {
+ }
+
+ /** One vectored range and the cache buffers it fills, in file order. */
+ private record Run(FileRange range, List<Part> parts) {
+ }
+
+ private Fetch startFetch(FSDataInputStream fileStream, ParquetRangeBuffers buffers, Allocator allocator,
+ int maxAlloc, int[] projected, BlockMetaData block, int rg)
+ throws IOException {
+ Fetch fetch = new Fetch();
+ ColumnChunkMetaData[] chunks = new ColumnChunkMetaData[projected.length];
+ for (int pc = 0; pc < projected.length; ++pc) {
+ chunks[pc] = block.getColumns().get(projected[pc]);
+ }
+ fetch.batch.init(fileKey, rg, chunks);
+ try {
+ for (ColumnChunkMetaData chunk : chunks) {
+ ColumnPlan column = new ColumnPlan();
+ fetch.columns.add(column);
+ planColumnChunk(allocator, maxAlloc, chunk.getStartingPos(),
+ chunk.getStartingPos() + chunk.getTotalSize(), column, fetch.misses);
+ }
+ requestMisses(fileStream, buffers, fetch, layout.maxRangeBytes());
+ } catch (Throwable t) {
+ abandon(allocator, fetch);
+ throw t;
+ }
+ return fetch;
+ }
+
+ private void finishFetch(Allocator allocator, ParquetRangeBuffers buffers, Fetch fetch)
+ throws IOException, InterruptedException {
+ try {
+ receiveMisses(buffers, fetch);
+ for (ColumnPlan column : fetch.columns) {
+ putColumn(allocator, column);
+ }
+ for (int pc = 0; pc < fetch.columns.size(); ++pc) {
+ assemble(fetch.batch, pc, fetch.columns.get(pc).parts);
+ }
+ // consumeData returns the batch on success; after a throw it is still ours.
+ consumer.consumeData(fetch.batch);
+ } catch (Throwable t) {
+ abandon(allocator, fetch);
+ throw t;
+ }
+ }
+
+ /** Drops a fetch that will not be decoded: outstanding requests are cancelled, buffers released. */
+ private void abandon(Allocator allocator, Fetch fetch) {
+ for (Run run : fetch.runs) {
+ run.range.getData().cancel(true);
+ }
+ for (ColumnPlan column : fetch.columns) {
+ for (Part part : column.parts) {
+ if (part.owned) {
+ bufferManager.decRefBuffer(part.buffer);
+ } else {
+ allocator.deallocate(part.buffer);
+ }
+ }
+ }
+ }
+
+ /** Package-private so a test can observe the read pattern. */
+ FSDataInputStream openFile(FileSystem fs) throws IOException {
+ return fs.open(path);
+ }
+
+ /** Lets the allocator abandon a wait for memory once the fragment is cancelled. */
+ private void allocateMultiple(Allocator allocator, MemoryBuffer[] dest, int size) {
+ if (allocator instanceof StoppableAllocator) {
+ ((StoppableAllocator) allocator).allocateMultiple(dest, size, DATA_BUFFER_FACTORY, isStopped);
+ } else {
+ allocator.allocateMultiple(dest, size, DATA_BUFFER_FACTORY);
+ }
+ }
+
+ /**
+ * Works out which buffers cover column chunk {@code [start, end)} without touching the file: cache
+ * hits as returned by getFileData, and freshly allocated power-of-two buffers for everything
+ * missing. Misses are appended to {@code allMisses} so the whole row group can be read at once.
+ */
+ private void planColumnChunk(Allocator allocator, int maxAlloc, long start, long end,
+ ColumnPlan column, List<Part> allMisses) throws IOException {
+ DiskRangeList head = new DiskRangeList(start, end);
+ if (fileKey != null) {
+ head = lowLevelCache.getFileData(fileKey, head, 0, ParquetCacheLayout.CACHE_CHUNK_FACTORY,
+ counters, new BooleanRef());
+ }
+ DiskRangeList current = head;
+ try {
+ for (; current != null; current = current.next) {
+ if (current.hasData()) {
+ column.parts.add(new Part(((CacheChunk) current).getBuffer(), current, false));
+ continue;
+ }
+ LlapHiveUtils.throwIfCacheOnlyRead(cacheOnly);
+ int[] sizes = layout.bufferSizes(current.getEnd() - current.getOffset());
+ column.missRuns.add(new MissRun(column.parts.size(), sizes.length));
+ long partFrom = current.getOffset();
+ for (int size : sizes) {
+ MemoryBuffer[] one = new MemoryBuffer[1];
+ allocateMultiple(allocator, one, size);
+ // The cache accounts and serves the bytes up to the buffer's limit.
+ ByteBuffer raw = one[0].getByteBufferRaw();
+ raw.limit(raw.position() + size);
+ Part part = new Part(one[0], new DiskRange(partFrom, partFrom + size), true);
+ column.parts.add(part);
+ allMisses.add(part);
+ partFrom += size;
+ }
+ }
+ } catch (Throwable t) {
+ // Hits past the failure point are still locked by getFileData; the caller only knows about
+ // the parts already recorded.
+ for (current = current.next; current != null; current = current.next) {
+ if (current.hasData()) {
+ bufferManager.decRefBuffer(((CacheChunk) current).getBuffer());
+ }
+ }
+ throw t;
+ }
+ }
+
+ /**
+ * Requests every missing buffer of a row group in one vectored call, one range per run of
+ * adjacent buffers. Column chunks sit back to back in the file, so a projection that keeps
+ * neighbouring columns reads them together; runs are capped so the row group arrives as
+ * several concurrent requests rather than one. A filesystem without a vectored implementation
+ * reads the ranges in turn.
+ */
+ private static void requestMisses(FSDataInputStream fileStream, ParquetRangeBuffers buffers, Fetch fetch,
+ int maxRange) throws IOException {
+ List<Part> misses = fetch.misses;
+ if (misses.isEmpty()) {
+ return;
+ }
+ misses.sort(Comparator.comparingLong(part -> part.range.getOffset()));
+ List<Run> runs = new ArrayList<>();
+ for (int i = 0; i < misses.size(); ) {
+ long from = misses.get(i).range.getOffset();
+ long to = misses.get(i).range.getEnd();
+ int j = i + 1;
+ for (; j < misses.size(); ++j) {
+ DiskRange next = misses.get(j).range;
+ if (next.getOffset() != to || next.getEnd() - from > maxRange) {
+ break;
+ }
+ to = next.getEnd();
+ }
+ runs.add(new Run(FileRange.createFileRange(from, (int) (to - from)), misses.subList(i, j)));
+ i = j;
+ }
+ List<FileRange> ranges = new ArrayList<>(runs.size());
+ for (Run run : runs) {
+ ranges.add(run.range);
+ }
+ fileStream.readVectored(ranges, buffers::allocate, buffers::release);
+ // Only requests the stream accepted are the fetch's to wait for or cancel.
+ fetch.runs.addAll(runs);
+ }
+
+ /** Waits for the requested runs and copies each into the cache buffers it covers. */
+ private static void receiveMisses(ParquetRangeBuffers buffers, Fetch fetch) throws IOException {
+ for (Run run : fetch.runs) {
+ ByteBuffer data = FutureIO.awaitFuture(run.range.getData());
+ for (Part part : run.parts) {
+ int length = part.range.getLength();
+ ByteBuffer src = data.duplicate();
+ src.position(data.position() + (int) (part.range.getOffset() - run.range.getOffset()));
+ src.limit(src.position() + length);
+ part.buffer.getByteBufferRaw().duplicate().put(src);
+ }
+ buffers.release(data);
+ }
+ }
+
+ /** Hands one column's freshly read buffers to the cache; every part then carries a ref we own. */
+ private void putColumn(Allocator allocator, ColumnPlan column) {
+ for (MissRun run : column.missRuns) {
+ List<Part> parts = column.parts.subList(run.firstPart, run.firstPart + run.count);
+ if (fileKey == null) {
+ for (Part part : parts) {
+ bufferManager.incRefBuffer(part.buffer);
+ }
+ } else {
+ MemoryBuffer[] fresh = new MemoryBuffer[run.count];
+ MemoryBuffer[] cached = new MemoryBuffer[run.count];
+ DiskRange[] ranges = new DiskRange[run.count];
+ for (int i = 0; i < run.count; ++i) {
+ fresh[i] = cached[i] = parts.get(i).buffer;
+ ranges[i] = parts.get(i).range;
+ }
+ lowLevelCache.putFileData(fileKey, ranges, cached, 0, Priority.NORMAL, counters, cacheTag);
+ for (int i = 0; i < run.count; ++i) {
+ if (cached[i] != fresh[i]) {
Review Comment:
`putFileData` processes this array incrementally and can throw after earlier entries have already been inserted and locked (for example, when a cached buffer at a later offset has a different declared length). Because `owned` is set only after the whole call returns, `finishFetch` then treats those earlier cache-owned buffers as raw allocations and calls `allocator.deallocate` on them, leaving the cache pointing at freed memory. Track ownership after each successful insertion or use an insertion/rollback path that preserves the cache's ownership on partial failure.
##########
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java:
##########
@@ -446,179 +440,19 @@ private void checkEndOfRowGroup() throws IOException {
throw new IOException("expecting more rows but reached last block. Read "
+ rowsReturned + " out of " + totalRowCount);
}
- List<ColumnDescriptor> columns = requestedSchema.getColumns();
- List<Type> types = requestedSchema.getFields();
- columnReaders = new VectorizedColumnReader[columns.size()];
-
- if (!ColumnProjectionUtils.isReadAllColumns(jobConf)) {
- //certain queries like select count(*) from table do not have
- //any projected columns and still have isReadAllColumns as false
- //in such cases columnReaders are not needed
- //However, if colsToInclude is not empty we should initialize each columnReader
- if(!colsToInclude.isEmpty()) {
- for (int i = 0; i < types.size(); ++i) {
- columnReaders[i] =
- buildVectorizedParquetReader(
- columnTypesList.get(colsToInclude.get(i)), types.get(i),
- pages, requestedSchema.getColumns(), skipTimestampConversion, writerTimezone, skipProlepticConversion,
- legacyConversionEnabled, 0, 0
- );
- }
- }
- } else {
- for (int i = 0; i < types.size(); ++i) {
- columnReaders[i] = buildVectorizedParquetReader(columnTypesList.get(i), types.get(i), pages,
- requestedSchema.getColumns(), skipTimestampConversion, writerTimezone, skipProlepticConversion,
- legacyConversionEnabled, 0, 0);
- }
- }
+ // Delegate the (Hive-type-driven) column-reader construction to the shared helper so the
+ // LLAP cache-backed consumer can reuse the exact same logic. Behavior is unchanged.
+ columnReaders = new ParquetRowGroupDecoder(fileSchema, initialDefaults).buildColumnReaders(
+ pages, requestedSchema, columnTypesList, colsToInclude,
+ ColumnProjectionUtils.isReadAllColumns(jobConf), skipTimestampConversion, writerTimezone,
+ skipProlepticConversion, legacyConversionEnabled);
Review Comment:
The native consumer reaches this helper with the schema built by `ParquetEncodedDataReader`, but that path never calls `getRequestedSchemaWithRowLineageColumns`. The existing reader explicitly augments `requestedSchema` and `colsToInclude` for ROW_ID/LAST_UPDATED_SEQUENCE_NUMBER at `VectorizedParquetRecordReader.java:267-270`; with row-lineage enabled, the native path consequently omits those Parquet fields and emits null/default lineage values. Apply the same augmentation to the native pipeline or force row-lineage reads to the fallback reader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4046187916
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -4958,6 +4958,9 @@ public static enum ConfVars {
LLAP_IO_ENCODE_ENABLED("hive.llap.io.encode.enabled", true,
"Whether LLAP should try to re-encode and cache data for non-ORC formats. This is used\n" +
"on LLAP Server side to determine if the infrastructure for that is initialized."),
+ LLAP_IO_PARQUET_NATIVE_ENABLED("hive.llap.io.parquet.native.enabled", false,
Review Comment:
let's keep false for now, we need to retest after merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4155478996
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ParquetEncodedDataConsumer.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.decode;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.type.DataTypePhysicalVariation;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.impl.ColumnVectorBatch;
+import org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl;
+import org.apache.hadoop.hive.llap.io.decode.ColumnVectorProducer.Includes;
+import org.apache.hadoop.hive.llap.io.encoded.ParquetEncodedColumnBatch;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonIOMetrics;
+import org.apache.hadoop.hive.ql.exec.vector.ColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.orc.encoded.Consumer;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetRowGroupDecoder;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedColumnReader;
+import org.apache.hadoop.hive.serde2.ColumnProjectionUtils;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
+import org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.parquet.HadoopReadOptions;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.compression.CompressionCodecFactory;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.metadata.FileMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+
+import com.google.common.base.Strings;
+
+/**
+ * The Parquet counterpart of {@link OrcEncodedDataConsumer}: it turns one row group's worth of
+ * cached column-chunk buffers ({@link ParquetEncodedColumnBatch}) into {@link ColumnVectorBatch}es
+ * and hands them downstream.
+ *
+ * <p>Rather than reimplementing Parquet value decoding, this consumer builds a
+ * {@link ParquetCachedPageReadStore} directly over the cached buffers (pages are parsed in place and
+ * decompressed lazily, no ParquetFileReader and no whole-chunk copy) and drives the very same
+ * {@link VectorizedColumnReader}s the non-cached vectorized reader uses (via {@link ParquetRowGroupDecoder}).
+ */
+public class ParquetEncodedDataConsumer
+ extends EncodedDataConsumer<Object, ParquetEncodedColumnBatch> {
+
+ private final Configuration jobConf;
+ private final boolean useDecimal64ColumnVectors;
+ private final CompressionCodecFactory codecFactory;
+ private final ParquetMetadataConverter converter;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private Path path;
+ private Map<String, Object> initialDefaults;
+
+ // Derived lazily (once) from the footer + job conf, then reused across row groups.
+ private List<TypeInfo> columnTypesList;
+ private List<Integer> colsToInclude;
+ private boolean readAllColumns;
+ private boolean skipTimestampConversion;
+ private boolean skipProlepticConversion;
+ private boolean legacyConversionEnabled;
+ private ZoneId writerTimezone;
+ private boolean schemaInitialized = false;
+
+ public ParquetEncodedDataConsumer(Consumer<ColumnVectorBatch> consumer, Includes includes,
+ QueryFragmentCounters counters, LlapDaemonIOMetrics ioMetrics, Configuration jobConf) {
+ super(consumer, includes.getPhysicalColumnIds().size(), ioMetrics, counters);
+ this.jobConf = jobConf;
+ this.useDecimal64ColumnVectors = HiveConf.getVar(jobConf,
+ ConfVars.HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_ENABLED)
+ .equalsIgnoreCase(HiveConf.HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_DECIMAL_64);
+ ParquetReadOptions options = HadoopReadOptions.builder(jobConf).build();
+ this.codecFactory = options.getCodecFactory();
+ this.converter = new ParquetMetadataConverter(options);
+ }
+
+ public void setFileMetadata(ParquetMetadata footer, MessageType requestedSchema, Path path) {
+ this.footer = footer;
+ this.requestedSchema = requestedSchema;
+ this.path = path;
+ }
+
+ public void setInitialDefaults(Map<String, Object> initialDefaults) {
+ this.initialDefaults = initialDefaults;
+ }
+
+ /**
+ * Derives the Hive type / projection lists and the conversion flags the same way
+ * {@link org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase} does from the job conf,
+ * so the shared {@link ParquetRowGroupDecoder} produces identical readers.
+ */
+ private void initSchema() {
+ FileMetaData fileMetaData = footer.getFileMetaData();
+ Map<String, String> kvMeta = fileMetaData.getKeyValueMetaData();
+
+ this.columnTypesList =
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES));
+ this.colsToInclude = ColumnProjectionUtils.getReadColumnIDs(jobConf);
+ this.readAllColumns = ColumnProjectionUtils.isReadAllColumns(jobConf);
+
+ this.writerTimezone = DataWritableReadSupport.getWriterTimeZoneId(kvMeta);
+ if (HiveConf.getBoolVar(jobConf, ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION)) {
+ this.skipTimestampConversion =
+ !Strings.nullToEmpty(fileMetaData.getCreatedBy()).startsWith("parquet-mr");
+ } else {
+ this.skipTimestampConversion = false;
+ }
+ Boolean proleptic = DataWritableReadSupport.getWriterDateProleptic(kvMeta);
+ if (proleptic == null) {
+ proleptic = HiveConf.getBoolVar(jobConf, ConfVars.HIVE_PARQUET_DATE_PROLEPTIC_GREGORIAN_DEFAULT);
+ }
+ this.skipProlepticConversion = proleptic;
+ this.legacyConversionEnabled = DataWritableReadSupport.getZoneConversionLegacy(kvMeta, jobConf);
+
+ this.schemaInitialized = true;
+ }
+
+ @Override
+ protected void decodeBatch(ParquetEncodedColumnBatch batch,
+ Consumer<ColumnVectorBatch> downstreamConsumer) throws InterruptedException {
+ if (!schemaInitialized) {
+ initSchema();
+ }
+
+ long startTime = counters.startTimeCounter();
+ // NOSONAR - S2093 does not apply: codecFactory is a per-consumer field with a per-batch
+ // release() (not close()); try-with-resources would tie it to this scope, which is wrong.
+ try { // NOSONAR
+ PageReadStore pages = new ParquetCachedPageReadStore(footer, batch, codecFactory, converter);
+
+ VectorizedColumnReader[] columnReaders =
+ new ParquetRowGroupDecoder(footer.getFileMetaData().getSchema(), initialDefaults).buildColumnReaders(
+ pages, requestedSchema, columnTypesList, colsToInclude, readAllColumns,
+ new ParquetRowGroupDecoder.TimestampConversionOptions(skipTimestampConversion,
+ writerTimezone, skipProlepticConversion, legacyConversionEnabled));
+
+ long rowsLeft = pages.getRowCount();
+ int batches = 0;
+ while (rowsLeft > 0) {
+ int batchSize = (int) Math.min(VectorizedRowBatch.DEFAULT_SIZE, rowsLeft);
+
+ ColumnVectorBatch cvb = takeBatch(batchSize);
+
+ // columnReaders[i] is requestedSchema field i, i.e. the i-th projected column.
+ for (int i = 0; i < columnReaders.length; ++i) {
+ if (columnReaders[i] == null) {
+ continue;
+ }
+ TypeInfo columnType = columnTypesList.get(readAllColumns ? i : colsToInclude.get(i));
+ ColumnVector cv = prepareColumnVector(cvb, i, columnType, batchSize);
+ columnReaders[i].readBatch(batchSize, cv, columnType);
+ }
+
+ downstreamConsumer.consumeData(cvb);
+ counters.incrCounter(LlapIOCounters.ROWS_EMITTED, batchSize);
+ rowsLeft -= batchSize;
+ ++batches;
+ }
+ counters.incrWallClockCounter(LlapIOCounters.DECODE_TIME_NS, startTime);
+ counters.incrCounter(LlapIOCounters.NUM_VECTOR_BATCHES, batches);
+ counters.incrCounter(LlapIOCounters.NUM_DECODED_BATCHES);
+ } catch (IOException | RuntimeException e) {
+ // parquet reports decode failures as runtime ParquetDecodingException.
+ LlapIoImpl.LOG.error("Parquet decodeBatch failed for rowGroup " + batch.rowGroupIx() + " of " + path, e);
Review Comment:
same throwable is logged with its stack at WARN by setError (LlapRecordReader.java:648) and again when next() rethrows it and the task fails
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4155066626
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java:
##########
@@ -122,6 +128,7 @@ public class LlapIoImpl implements LlapIo<VectorizedRowBatch>, LlapIoDebugDump {
// TODO: later, we may have a map
private final ColumnVectorProducer orcCvp, genericCvp;
+ private final ColumnVectorProducer parquetCvp;
Review Comment:
could we refactor to keep class generic. llap-client sees LlapOrcIo / LlapParquetIo from the earlier sketch
````
/** One natively served format: its input-format class and the producer that decodes it. */
interface LlapFormatIo {
Class<? extends InputFormat> inputFormatClass();
@Nullable ColumnVectorProducer producer(); // null when this daemon cannot serve it natively
}
final class LlapOrcIoImpl implements LlapOrcIo, LlapFormatIo {
private final ColumnVectorProducer cvp; // OrcColumnVectorProducer, always present
// getTailFromCache(...), vectorizedReaderForPath(...) move here unchanged
}
final class LlapParquetIoImpl implements LlapParquetIo, LlapFormatIo {
private final ColumnVectorProducer cvp; // null unless useLowLevelCache (F5), optionally && daemon flag
// getFooterBuffersFromCache(...) moves here; vectorizedReaderForPath arrives with its Iceberg caller
}
````
LlapIoImpl then holds orc, parquet, genericCvp and formats = List.of(orc, parquet), and the instanceof chain collapses:
````
@Override
public InputFormat<NullWritable, VectorizedRowBatch> getInputFormat(InputFormat<?, ?> src, Deserializer serde) {
ColumnVectorProducer cvp = serde != null ? genericCvp : nativeProducerFor(src.getClass());
if (cvp == null) {
LOG.warn("No LLAP IO producer for {} (serde-based: {})", src.getClass(), serde != null);
return null; // wrapForLlap falls to injectLlapCaches as today
}
return new LlapInputFormat(src, serde, cvp, executor, daemonConf);
}
private ColumnVectorProducer nativeProducerFor(Class<?> ifClass) {
for (LlapFormatIo f : formats) {
if (f.inputFormatClass().isAssignableFrom(ifClass)) return f.producer();
}
return null;
}
@Override public LlapOrcIo orc() { return orc; }
@Override public LlapParquetIo parquet() { return parquet; }
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4155156348
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ParquetCachedPageReadStore.java:
##########
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.decode;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.llap.io.encoded.ParquetEncodedColumnBatch;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.DataPage;
+import org.apache.parquet.column.page.DataPageV1;
+import org.apache.parquet.column.page.DataPageV2;
+import org.apache.parquet.column.page.DictionaryPage;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.column.page.PageReader;
+import org.apache.parquet.compression.CompressionCodecFactory;
+import org.apache.parquet.compression.CompressionCodecFactory.BytesInputDecompressor;
+import org.apache.parquet.format.DataPageHeader;
+import org.apache.parquet.format.DataPageHeaderV2;
+import org.apache.parquet.format.DictionaryPageHeader;
+import org.apache.parquet.format.PageHeader;
+import org.apache.parquet.format.Util;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.schema.PrimitiveType;
+
+/**
+ * {@link PageReadStore} over one row group's cached column-chunk buffers, so that the vectorized
+ * column readers decode straight out of the LLAP cache: no ParquetFileReader, and no copy of the
+ * chunk bytes.
+ *
+ * <p><b>Why this is a copy.</b> Parquet does the same thing over a file, but none of it is
+ * reachable from here: {@code ParquetFileReader.Chunk.readAllPages} is private, the store it fills
+ * ({@code org.apache.parquet.hadoop.ColumnChunkPageReadStore}) is package-private and so is its
+ * {@code addColumn}, and the {@link PageReader} it puts in ({@code ColumnChunkPageReader}) is a
+ * package-private {@code final} class. Only the glue below is re-implemented; page header parsing,
+ * encoding and statistics conversion, the page value types and decompression are all parquet's own
+ * public API.
+ *
+ * <p><b>On a parquet upgrade.</b> Exactly two members track upstream code, each with a counterpart
+ * to diff against the new release:
+ * <ul>
+ * <li>{@link #readAllPages} mirrors {@code ParquetFileReader.Chunk.readAllPages}: the page-header
+ * walk, the page-type switch and the mapping of header fields onto {@link DataPageV1},
+ * {@link DataPageV2} and {@link DictionaryPage}. A new page type, a new header field or a
+ * changed page constructor lands here.</li>
+ * <li>{@link CachedChunkPageReader} mirrors {@code ColumnChunkPageReader}: page queueing and
+ * lazy decompression. New {@link PageReader} behaviour lands here.</li>
+ * </ul>
+ * Also check whether {@link PageReadStore} gained or changed a {@code default} method, since this
+ * class implements none of them ({@code getRowIndexOffset}, {@code getRowIndexes} and
+ * {@code close} are all inherited). {@code TestParquetCachedPageReadStore} compares this store page
+ * for page against the stock one and pins the parquet version, so the upgrade fails the build
+ * rather than passing with a divergence.
+ *
+ * <p>Deliberately unsupported, because LLAP's Parquet path does not use them: the offset index (so
+ * {@code getFirstRowIndex} and {@code getIndexRowCount} keep their no-offset-index defaults),
+ * column encryption, and page checksum verification.
+ *
+ * <p>How it works:
+ * <ol>
+ * <li><b>Input</b>: a {@link ParquetEncodedColumnBatch} holding the already-cached
+ * {@link MemoryBuffer}s for one row group's projected column chunks, plus the file footer.</li>
+ * <li><b>Stitching and parsing</b>: per chunk, {@link #chunkBuffers} turns the cache buffers into a
+ * list of {@link ByteBuffer} slices covering exactly the chunk's byte region; {@link #readAllPages}
+ * then walks that byte range with {@link Util#readPageHeader}, building {@link DataPageV1} /
+ * {@link DataPageV2} / {@link DictionaryPage} objects whose payloads are
+ * {@link ByteBufferInputStream} views over the cache buffers - the page bytes are never copied.</li>
+ * <li><b>Lazy decompression</b>: each {@link CachedChunkPageReader} keeps the raw bytes and its
+ * {@link BytesInputDecompressor}, and decompresses only when the vectorized column reader
+ * actually calls {@link PageReader#readPage()}, so pages that get pruned never pay the codec cost.</li>
+ * <li><b>Output</b>: the resulting {@code ColumnPath -> PageReader} map backs
+ * {@link #getPageReader(ColumnDescriptor)}, so parquet's VectorizedColumnReader never notices
+ * that it is not reading from a file.</li>
+ * </ol>
+ */
+class ParquetCachedPageReadStore implements PageReadStore {
+
+ private final Map<ColumnPath, PageReader> readers = new HashMap<>();
+ private final long rowCount;
+
+ ParquetCachedPageReadStore(ParquetMetadata footer, ParquetEncodedColumnBatch batch,
+ CompressionCodecFactory codecFactory, ParquetMetadataConverter converter) throws IOException {
+ BlockMetaData block = footer.getBlocks().get(batch.rowGroupIx());
+ this.rowCount = block.getRowCount();
+ String createdBy = footer.getFileMetaData().getCreatedBy();
+ ColumnChunkMetaData[] chunks = batch.chunks();
+ // pc is the projection index: chunks[pc] and the batch's three buffer arrays are all indexed by
+ // it, while the footer's block lists columns in file order. Keying on the path keeps the two
+ // apart, so a projection that drops columns from the middle of the schema still lines up.
+ for (int pc = 0; pc < chunks.length; ++pc) {
+ ColumnChunkMetaData chunk = chunks[pc];
+ readers.put(chunk.getPath(), readAllPages(chunk, chunkBuffers(batch, pc), createdBy,
+ codecFactory.getDecompressor(chunk.getCodec()), converter));
+ }
+ }
+
+ /**
+ * Slices of the cached buffers covering exactly the chunk's byte region, in file order.
+ *
+ * <p>This is the only place in the class that touches bytes, and the only place the cache appears
+ * at all: the {@link MemoryBuffer}s are already-populated LLAP cache memory, so there is no read
+ * here and nothing to decide - by the time this runs, a cache hit and a miss that
+ * {@code ParquetEncodedDataReader} had to fetch look exactly the same.
+ *
+ * <p>The buffers are cache ranges aligned to absolute file offsets, not to this chunk, so the
+ * first and last one usually overhang the chunk (the first may even start before it). That is what
+ * the trimming below is for.
+ *
+ * <p>The trimmed slices have to tile the chunk exactly, which is checked as they are built: a
+ * buffer that does not continue where the previous one ended, or one that reaches past the chunk,
+ * means the batch is not carrying the buffers the chunk was planned with. Without the check the
+ * page walk would read the wrong bytes as page bytes, and fail much later with a decoding error
+ * about a corrupt header.
+ */
+ private static List<ByteBuffer> chunkBuffers(ParquetEncodedColumnBatch batch, int pc)
Review Comment:
maybe `sliceChunk` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4157117476
##########
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetRowGroupDecoder.java:
##########
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.ql.io.parquet.vector;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.parquet.ParquetRuntimeException;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.InvalidSchemaException;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Reusable, stateless helper that builds the {@link VectorizedColumnReader} array for one Parquet
+ * row group, given an already-read {@link PageReadStore}. The logic here was lifted verbatim from
+ * {@link VectorizedParquetRecordReader} so that both the row-by-split reader and the LLAP
+ * cache-backed consumer ({@code ParquetEncodedDataConsumer}) can share a single, behavior-preserving
+ * implementation of the Hive-type-driven Parquet column-reader construction.
+ *
+ * <p>The instance only carries the {@code fileSchema} (used for the schema-evolution check). All
+ * per-row-group state lives in the supplied {@link PageReadStore}.
+ */
+public class ParquetRowGroupDecoder {
+
+ private static final int MAP_DEFINITION_LEVEL_MAX = 3;
+
+ /**
+ * Bag of writer-timezone / proleptic / legacy conversion flags forwarded to every primitive
+ * column reader. Grouped so the reader-construction entry point stays under the parameter cap.
+ */
+ public record TimestampConversionOptions(
+ boolean skipTimestampConversion,
+ ZoneId writerTimezone,
+ boolean skipProlepticConversion,
+ boolean legacyConversionEnabled) {
+ }
+
+ private final MessageType fileSchema;
+ private final Map<String, Object> initialDefaults;
+
+ public ParquetRowGroupDecoder(MessageType fileSchema, Map<String, Object> initialDefaults) {
+ this.fileSchema = fileSchema;
+ this.initialDefaults = initialDefaults;
+ }
+
+ /**
+ * Builds the per-(requested-)column {@link VectorizedColumnReader} array for a row group.
+ *
+ * @param pages the row group's page store (from {@code reader.readRowGroup(..)}
+ * / {@code readNextRowGroup()})
+ * @param requestedSchema the projected Parquet schema being read
+ * @param columnTypesList the Hive type infos for ALL table columns (indexed by table col id)
+ * @param colsToInclude the table column ids being read, in requested-schema field order;
+ * may be empty (e.g. {@code count(*)}), in which case all readers are null
+ * @param readAllColumns whether projection is "read all columns"
+ * @param options writer-timezone / proleptic / legacy conversion flags, see
+ * {@link TimestampConversionOptions}
+ * @return one reader per requested-schema field (null entries where no reader is needed)
+ */
+ public VectorizedColumnReader[] buildColumnReaders(
+ PageReadStore pages,
+ MessageType requestedSchema,
+ List<TypeInfo> columnTypesList,
+ List<Integer> colsToInclude,
+ boolean readAllColumns,
+ TimestampConversionOptions options) throws IOException {
+ List<Type> types = requestedSchema.getFields();
+ VectorizedColumnReader[] columnReaders = new VectorizedColumnReader[types.size()];
+
+ if (!readAllColumns) {
+ // certain queries like select count(*) from table do not have
+ // any projected columns and still have isReadAllColumns as false
+ // in such cases columnReaders are not needed
+ // However, if colsToInclude is not empty we should initialize each columnReader
+ if (!colsToInclude.isEmpty()) {
+ for (int i = 0; i < types.size(); ++i) {
+ columnReaders[i] = buildVectorizedParquetReader(
+ columnTypesList.get(colsToInclude.get(i)), types.get(i), pages,
+ requestedSchema.getColumns(), options, 0, 0);
+ }
+ }
+ } else {
+ for (int i = 0; i < types.size(); ++i) {
+ columnReaders[i] = buildVectorizedParquetReader(columnTypesList.get(i),
+ types.get(i), pages, requestedSchema.getColumns(), options, 0, 0);
+ }
+ }
+ return columnReaders;
+ }
+
+ private static List<ColumnDescriptor> getAllColumnDescriptorByType(
+ int depth,
+ Type type,
+ List<ColumnDescriptor> columns) throws ParquetRuntimeException {
+ List<ColumnDescriptor> res = new ArrayList<>();
+ for (ColumnDescriptor descriptor : columns) {
+ if (depth >= descriptor.getPath().length) {
+ throw new InvalidSchemaException("Corrupted Parquet schema");
+ }
+ if (type.getName().equals(descriptor.getPath()[depth])) {
+ res.add(descriptor);
+ }
+ }
+ return res;
+ }
+
+ // Nested types are unsupported here on purpose; callers detect that in advance
+ // (ParquetEncodedDataReader.projectsNestedTypes) and route the query to the non-native reader.
+ private static PrimitiveType getElementType(Type type) {
+ if (type.isPrimitive()) {
+ return type.asPrimitiveType();
+ }
+ if (type.asGroupType().getFields().size() > 1) {
+ throw new UnsupportedOperationException(
+ "Current Parquet Vectorization reader doesn't support nested type");
+ }
+
+ Type childType = type.asGroupType().getFields().getFirst();
+
+ // Parquet file generated using thrift may have child type as PrimitiveType
+ if (childType.isPrimitive()) {
+ return childType.asPrimitiveType();
+ } else {
+ return childType.asGroupType().getFields().getFirst().asPrimitiveType();
+ }
+ }
+
+ // Build VectorizedParquetColumnReader via Hive typeInfo and Parquet schema
+ private VectorizedColumnReader buildVectorizedParquetReader(
+ TypeInfo typeInfo,
+ Type type,
+ PageReadStore pages,
+ List<ColumnDescriptor> columnDescriptors,
+ TimestampConversionOptions options,
+ int depth, int currentDefLevel) throws IOException {
+ int typeDefLevel = currentDefLevel;
+ if (type.isRepetition(Type.Repetition.OPTIONAL) || type.isRepetition(Type.Repetition.REPEATED)) {
+ typeDefLevel++;
+ }
+ List<ColumnDescriptor> descriptors =
+ getAllColumnDescriptorByType(depth, type, columnDescriptors);
+ // Support for schema evolution: if the column from the current
+ // query schema is not present in the file schema, return a dummy
+ // reader that produces nulls. This allows queries to proceed even
+ // when new columns have been added after the file was written.
+ if (!fileSchema.getColumns().contains(descriptors.getFirst())) {
Review Comment:
The class is built per row group (VectorizedParquetRecordReader:445, ParquetEncodedDataConsumer:160) although it holds only per-file state, and line 175 calls fileSchema.getColumns() per requested column per row group, which allocates one ColumnDescriptor per leaf each time (getColumns() rebuilds everything on every call).
Build it once per file and keep fileSchema.getColumns() as a Set field. On a 500-column file that is the one O(requested × leaves) allocation left in the per-row-group path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4156039257
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ // Fallback path: no cache key, so we read status + footer straight from HDFS. Time both
+ // under HDFS_TIME_NS to match how ORC accounts for cache-miss footer reads.
+ final FileSystem fs = path.getFileSystem(jobConf);
+ long hdfsStart = counters.startTimeCounter();
+ final FileStatus stat = fs.getFileStatus(path);
+ counters.recordHdfsTime(hdfsStart);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ hdfsStart = counters.startTimeCounter();
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ counters.recordHdfsTime(hdfsStart);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ long startTime = counters.startTimeCounter();
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Exception e) {
+ // A shutdown-triggered InterruptedException must not be silently reported as an ordinary
+ // consumer error: preserve the interrupt on the thread so any caller sees it.
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ }
+ consumer.setError(e);
+ } finally {
+ counters.incrWallClockCounter(LlapIOCounters.TOTAL_IO_TIME_NS, startTime);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ // The LLAP IO summary keys off TABLE / FILE / STRIPES; set them the same way ORC does so a
+ // native Parquet fragment shows up in the summary with the same fields populated.
+ if (cacheTag != null) {
+ counters.setDesc(QueryFragmentCounters.Desc.TABLE, cacheTag.getTableName());
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.FILE, path
+ + (fileKey == null ? "" : " (" + fileKey + ")"));
+
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = selectRowGroups(blocks, fileSchema);
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ // STRIPES is the ORC term; for Parquet the analog is row groups. Reuse the same descriptor so
+ // the summary layout stays common and we don't have to teach the reporter about a new field.
+ counters.setDesc(QueryFragmentCounters.Desc.STRIPES, "0," + selected.size());
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /**
+ * The row groups this split has to read: the ones whose first data page falls inside the split, as
+ * Parquet's own input format assigns them, minus any the pushed-down predicate rules out on its
+ * statistics. Returned in file order, since the fetch pipeline reads them in that order.
+ */
+ private List<BlockMetaData> selectRowGroups(List<BlockMetaData> blocks, MessageType fileSchema)
+ throws IOException {
+ long splitStart = split.getStart();
+ long splitEnd = splitStart + split.getLength();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().getFirst().getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate == null) {
+ return selected;
+ }
+ return RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Whether the query needs row-lineage virtual columns ({@code ROW__LINEAGE__ID} /
+ * {@code LAST__UPDATED__SEQUENCE__NUMBER}) that the fallback reader augments the requested schema
+ * with via {@link RowLineageUtils#getRequestedSchemaWithRowLineageColumns}. The native pipeline
+ * doesn't propagate that augmentation through {@code Includes} yet, so we defer to the fallback
+ * reader instead of quietly emitting nulls in the lineage slots.
+ */
+ public boolean needsRowLineage() {
+ VectorizedRowBatchCtx rbCtx = Utilities.getVectorizedRowBatchCtx(jobConf);
+ if (rbCtx == null) {
+ return false;
+ }
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ return RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema, VirtualColumn.ROW_LINEAGE_ID)
+ || RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema,
+ VirtualColumn.LAST_UPDATED_SEQUENCE_NUMBER);
+ }
+
+ /**
+ * Leaf-column positions of the requested fields; {@code BlockMetaData.getColumns()} is a flat list
+ * of leaves in file-schema order, so a top-level primitive at ordinal {@code k} in the file schema
+ * may sit at a very different leaf index (e.g. a nested group of two leaves before it shifts it
+ * from {@code 1} to {@code 2}). We only reach this method when every requested field is a
+ * top-level primitive ({@link #projectsNestedTypes()} would have forced a fallback otherwise), so
+ * each requested field maps to exactly one leaf whose path is a single segment.
+ */
+ static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
+ List<ColumnDescriptor> allLeaves = fileSchema.getColumns();
+ List<Integer> selected = new ArrayList<>();
+ for (Type field : requestedSchema.getFields()) {
+ if (!fileSchema.containsField(field.getName())) {
+ continue;
+ }
+ for (int i = 0; i < allLeaves.size(); ++i) {
+ String[] path = allLeaves.get(i).getPath();
+ // Single-segment path == top-level primitive; skip leaves buried inside a group.
+ if (path.length == 1 && path[0].equals(field.getName())) {
+ selected.add(i);
+ break;
+ }
+ }
+ }
+ return selected.stream().mapToInt(Integer::intValue).toArray();
+ }
+
+ private static long bytes(int[] projected, BlockMetaData block) {
+ long total = 0;
+ for (int leaf : projected) {
+ total += block.getColumns().get(leaf).getTotalSize();
+ }
+ return total;
+ }
+
+ private static long bytes(Fetch fetch) {
+ long total = 0;
+ for (ColumnChunkMetaData chunk : fetch.batch.chunks()) {
+ total += chunk.getTotalSize();
+ }
+ return total;
+ }
+
+ /** One row group on its way in: buffers planned, missing ranges requested, not yet decoded. */
+ private static final class Fetch {
+ private final ParquetEncodedColumnBatch batch = new ParquetEncodedColumnBatch();
+ private final List<ColumnPlan> columns = new ArrayList<>();
+ private final List<Part> misses = new ArrayList<>();
+ private final List<Run> runs = new ArrayList<>();
+ }
+
+ /** The buffers covering one column chunk, in file order, and which stretches of them are new. */
+ private record ColumnPlan(List<Part> parts, List<MissRun> missRuns) {
+ ColumnPlan() {
+ this(new ArrayList<>(), new ArrayList<>());
+ }
+ }
+
+ /** One cache buffer's worth of a column chunk: a hit handed back by the cache, or a miss to fill. */
+ private static final class Part {
+ private MemoryBuffer buffer;
+ private final DiskRange range;
+ /** We hold one ref to release; until then a miss is a raw allocation to free. */
+ private boolean owned;
+
+ Part(MemoryBuffer buffer, DiskRange range, boolean owned) {
+ this.buffer = buffer;
+ this.range = range;
+ this.owned = owned;
+ }
+ }
+
+ /** {@code count} consecutive parts covering one contiguous missing sub-range; cached as a unit. */
+ private record MissRun(int firstPart, int count) {
+ }
+
+ /** One vectored range and the cache buffers it fills, in file order. */
+ private record Run(FileRange range, List<Part> parts) {
Review Comment:
ReadRange - holds a FileRange and the parts it fills?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4217263981
##########
ql/src/java/org/apache/hadoop/hive/llap/ParquetCacheLayout.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.llap;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.DataCache.DiskRangeListFactory;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+
+/**
+ * How Parquet column chunks are laid out in the LLAP cache and fetched from the file. Everything
+ * that caches a chunk goes through this, so a reader finding only part of one cached re-caches
+ * the gap on the boundaries the first reader used. A reader builds one from the daemon's allocator
+ * and configuration and uses it for every chunk, which is what keeps its layout consistent.
+ */
+public final class ParquetCacheLayout {
+
+ /**
+ * An object store fetches separate ranges concurrently but never splits one, so cutting buys
+ * parallelism while a large range amortises the request. Measured against S3 the knee is here:
+ * below 4 Mb throughput drops sharply, above 8 Mb it is flat. Trino also splits at 8 Mb.
+ */
+ private static final int MIN_RANGE_CAP_BYTES = 8 << 20;
+
+ public static final DiskRangeListFactory CACHE_CHUNK_FACTORY = CacheChunk::new;
+
+ private final int maxBuffer;
+ private final int minBuffer;
+ private final int maxRange;
+
+ /** The layout for a daemon: its allocator's maximum and the grain it caches non-ORC data at. */
+ public ParquetCacheLayout(Allocator allocator, Configuration conf) {
+ this(allocator.getMaxAllocation(), HiveConf.getSizeVar(conf, HiveConf.ConfVars.LLAP_IO_ENCODE_ALLOC_SIZE));
+ }
+
+ /**
+ * @param maxAlloc largest buffer the allocator hands out; rounded down to a power of two
+ * @param splitFloor size below which a remainder is left as one buffer rather than split further
+ */
+ public ParquetCacheLayout(int maxAlloc, long splitFloor) {
+ this.maxBuffer = Integer.highestOneBit(maxAlloc);
Review Comment:
Both guards have a config that reaches them, so I'd rather keep them.
`maxBuffer = Integer.highestOneBit(maxAlloc)`
`BuddyAllocator` does report a power of two, so this is a no-op there. But `SimpleAllocator`, which is what `hive.llap.io.memory.mode=none` installs, reports `Integer.MAX_VALUE` (2147483647). Without the rounding `maxBuffer` is that odd number, the emitted sizes stop being powers of two, and `bufferSizes` loses the property that any contiguous run of its output decomposes back into itself. That property is what lets this reader and `LlapCacheAwareFs` re-cache each other's evicted gaps under the same keys.
`minBuffer = min(splitFloor, maxBuffer)`: this is the same clamp `SerDeEncodedDataReader.determineAllocSize` already does for the other non-ORC reader. `hive.llap.io.encode.alloc.size` is declared with a bare `SizeValidator()` — no bounds at all, despite its own description promising "a power of two between `alloc.min` and `alloc.max`" — so nothing stops it being set above the allocator's maximum.
##########
ql/src/java/org/apache/hadoop/hive/llap/ParquetCacheLayout.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.llap;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.DataCache.DiskRangeListFactory;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+
+/**
+ * How Parquet column chunks are laid out in the LLAP cache and fetched from the file. Everything
+ * that caches a chunk goes through this, so a reader finding only part of one cached re-caches
+ * the gap on the boundaries the first reader used. A reader builds one from the daemon's allocator
+ * and configuration and uses it for every chunk, which is what keeps its layout consistent.
+ */
+public final class ParquetCacheLayout {
+
+ /**
+ * An object store fetches separate ranges concurrently but never splits one, so cutting buys
+ * parallelism while a large range amortises the request. Measured against S3 the knee is here:
+ * below 4 Mb throughput drops sharply, above 8 Mb it is flat. Trino also splits at 8 Mb.
+ */
+ private static final int MIN_RANGE_CAP_BYTES = 8 << 20;
+
+ public static final DiskRangeListFactory CACHE_CHUNK_FACTORY = CacheChunk::new;
+
+ private final int maxBuffer;
+ private final int minBuffer;
+ private final int maxRange;
+
+ /** The layout for a daemon: its allocator's maximum and the grain it caches non-ORC data at. */
+ public ParquetCacheLayout(Allocator allocator, Configuration conf) {
+ this(allocator.getMaxAllocation(), HiveConf.getSizeVar(conf, HiveConf.ConfVars.LLAP_IO_ENCODE_ALLOC_SIZE));
+ }
+
+ /**
+ * @param maxAlloc largest buffer the allocator hands out; rounded down to a power of two
+ * @param splitFloor size below which a remainder is left as one buffer rather than split further
+ */
+ public ParquetCacheLayout(int maxAlloc, long splitFloor) {
+ this.maxBuffer = Integer.highestOneBit(maxAlloc);
Review Comment:
Both guards have a config that reaches them, so I'd rather keep them.
`maxBuffer = Integer.highestOneBit(maxAlloc)`
`BuddyAllocator` does report a power of two, so this is a no-op there. But `SimpleAllocator`, which is what `hive.llap.io.memory.mode=none` installs, reports `Integer.MAX_VALUE` (2147483647). Without the rounding `maxBuffer` is that odd number, the emitted sizes stop being powers of two, and `bufferSizes` loses the property that any contiguous run of its output decomposes back into itself. That property is what lets this reader and `LlapCacheAwareFs` re-cache each other's evicted gaps under the same keys.
`minBuffer = min(splitFloor, maxBuffer)`: this is the same clamp `SerDeEncodedDataReader.determineAllocSize` already does for the other non-ORC reader. `hive.llap.io.encode.alloc.size` is declared with a bare `SizeValidator()` — no bounds at all, despite its own description promising "a power of two between `alloc.min` and `alloc.max`" — so nothing stops it being set above the allocator's maximum.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6793:
URL: https://github.com/apache/hive/pull/6793#issuecomment-6345317916
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6793) **Quality Gate passed**
Issues
 [89 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6793&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6793&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6793&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6793&metric=new_coverage&view=list)
 [0.1% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6793&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6793)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4194264362
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ParquetColumnVectorProducer.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.decode;
+
+import java.io.IOException;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.impl.ColumnVectorBatch;
+import org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl;
+import org.apache.hadoop.hive.llap.io.encoded.ParquetEncodedDataReader;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonCacheMetrics;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonIOMetrics;
+import org.apache.hadoop.hive.ql.io.orc.encoded.Consumer;
+import org.apache.hadoop.hive.ql.io.sarg.SearchArgument;
+import org.apache.hadoop.hive.ql.plan.PartitionDesc;
+import org.apache.hadoop.hive.serde2.Deserializer;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.InputFormat;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.Reporter;
+
+public class ParquetColumnVectorProducer implements ColumnVectorProducer {
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration conf;
+ private final LlapDaemonCacheMetrics cacheMetrics;
+ private final LlapDaemonIOMetrics ioMetrics;
+
+ public ParquetColumnVectorProducer(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration conf, LlapDaemonCacheMetrics cacheMetrics, LlapDaemonIOMetrics ioMetrics) {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.conf = conf;
+ this.cacheMetrics = cacheMetrics;
+ this.ioMetrics = ioMetrics;
+ }
+
+ @Override
+ public ReadPipeline createReadPipeline(Consumer<ColumnVectorBatch> consumer, FileSplit split,
+ Includes includes, SearchArgument sarg, QueryFragmentCounters counters,
+ SchemaEvolutionFactory sef, InputFormat<?, ?> sourceInputFormat, Deserializer sourceSerDe,
+ Reporter reporter, JobConf job, Map<Path, PartitionDesc> parts) {
+ try {
+ cacheMetrics.incrCacheReadRequests();
+ ParquetEncodedDataConsumer edc =
+ new ParquetEncodedDataConsumer(consumer, includes, counters, ioMetrics, job);
+ ParquetEncodedDataReader reader = new ParquetEncodedDataReader(
+ lowLevelCache, bufferManager, conf, job, split, edc, counters);
+ reader.loadFooter();
+ // Null makes LlapInputFormat fall back to the normal VectorizedParquetRecordReader.
+ if (reader.projectsNestedTypes()) {
+ LlapIoImpl.LOG.info("Parquet native cache: falling back to normal reader for {} due to "
+ + "nested types in the projection", split.getPath());
+ return null;
+ }
+ /*
+ * Fallback path still caches (footer via LlapProxy, data via LlapCacheAwareFs); what's lost
+ * for this split is vectored reads, row-group lookahead, and its LLAP IO summary counters.
+ * Guard fires per split only for Iceberg-Parquet files that physically carry _row_id /
+ * _last_updated_sequence_number while the query projects those virtual columns.
+ *
+ * Table shape Fires?
+ * ------------------------------------------------------------ ------
+ * Non-Iceberg Parquet never
+ * Iceberg-Parquet, row lineage disabled never
Review Comment:
we need some kind of guard here until HIVE-30103
TLDR: what I can do here is to improve the guard, remove the table, and add a unit test
this code is actually reachable yet: Iceberg tables can project `ROW__LINEAGE__ID` /
`LAST__UPDATED__SEQUENCE__NUMBER`, and Iceberg's Parquet reads don't reach this producer,
`HiveVectorizedReader` only routes its ORC branch into LLAP IO
HIVE-30103 is the change that points Parquet here too
without the guard it fails silently rather than loudly: so when this pipeline serves a batch, the lineage column's value is never actually read from the file, it's left at whatever `initializeRowLineageColumns` put there, which is -1
downstream code (`HiveBatchIterator`) can't distinguish between the cases when "the native path never looked" and "this file was actually written before lineage tracking existed" so it treats -1 as the second case and creates a value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4157278864
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ParquetEncodedDataConsumer.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.decode;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.type.DataTypePhysicalVariation;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.impl.ColumnVectorBatch;
+import org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl;
+import org.apache.hadoop.hive.llap.io.decode.ColumnVectorProducer.Includes;
+import org.apache.hadoop.hive.llap.io.encoded.ParquetEncodedColumnBatch;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonIOMetrics;
+import org.apache.hadoop.hive.ql.exec.vector.ColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.orc.encoded.Consumer;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetRowGroupDecoder;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedColumnReader;
+import org.apache.hadoop.hive.serde2.ColumnProjectionUtils;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
+import org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.parquet.HadoopReadOptions;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.compression.CompressionCodecFactory;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.metadata.FileMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+
+import com.google.common.base.Strings;
+
+/**
+ * The Parquet counterpart of {@link OrcEncodedDataConsumer}: it turns one row group's worth of
+ * cached column-chunk buffers ({@link ParquetEncodedColumnBatch}) into {@link ColumnVectorBatch}es
+ * and hands them downstream.
+ *
+ * <p>Rather than reimplementing Parquet value decoding, this consumer builds a
+ * {@link ParquetCachedPageReadStore} directly over the cached buffers (pages are parsed in place and
+ * decompressed lazily, no ParquetFileReader and no whole-chunk copy) and drives the very same
+ * {@link VectorizedColumnReader}s the non-cached vectorized reader uses (via {@link ParquetRowGroupDecoder}).
+ */
+public class ParquetEncodedDataConsumer
+ extends EncodedDataConsumer<Object, ParquetEncodedColumnBatch> {
+
+ private final Configuration jobConf;
+ private final boolean useDecimal64ColumnVectors;
+ private final CompressionCodecFactory codecFactory;
+ private final ParquetMetadataConverter converter;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private Path path;
+ private Map<String, Object> initialDefaults;
+
+ // Derived lazily (once) from the footer + job conf, then reused across row groups.
+ private List<TypeInfo> columnTypesList;
+ private List<Integer> colsToInclude;
+ private boolean readAllColumns;
+ private boolean skipTimestampConversion;
+ private boolean skipProlepticConversion;
+ private boolean legacyConversionEnabled;
+ private ZoneId writerTimezone;
+ private boolean schemaInitialized = false;
+
+ public ParquetEncodedDataConsumer(Consumer<ColumnVectorBatch> consumer, Includes includes,
+ QueryFragmentCounters counters, LlapDaemonIOMetrics ioMetrics, Configuration jobConf) {
+ super(consumer, includes.getPhysicalColumnIds().size(), ioMetrics, counters);
+ this.jobConf = jobConf;
+ this.useDecimal64ColumnVectors = HiveConf.getVar(jobConf,
+ ConfVars.HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_ENABLED)
+ .equalsIgnoreCase(HiveConf.HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_DECIMAL_64);
+ ParquetReadOptions options = HadoopReadOptions.builder(jobConf).build();
+ this.codecFactory = options.getCodecFactory();
+ this.converter = new ParquetMetadataConverter(options);
+ }
+
+ public void setFileMetadata(ParquetMetadata footer, MessageType requestedSchema, Path path) {
+ this.footer = footer;
+ this.requestedSchema = requestedSchema;
+ this.path = path;
+ }
+
+ public void setInitialDefaults(Map<String, Object> initialDefaults) {
+ this.initialDefaults = initialDefaults;
+ }
+
+ /**
+ * Derives the Hive type / projection lists and the conversion flags the same way
+ * {@link org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase} does from the job conf,
+ * so the shared {@link ParquetRowGroupDecoder} produces identical readers.
+ */
+ private void initSchema() {
+ FileMetaData fileMetaData = footer.getFileMetaData();
+ Map<String, String> kvMeta = fileMetaData.getKeyValueMetaData();
+
+ this.columnTypesList =
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES));
+ this.colsToInclude = ColumnProjectionUtils.getReadColumnIDs(jobConf);
+ this.readAllColumns = ColumnProjectionUtils.isReadAllColumns(jobConf);
+
+ this.writerTimezone = DataWritableReadSupport.getWriterTimeZoneId(kvMeta);
+ if (HiveConf.getBoolVar(jobConf, ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION)) {
+ this.skipTimestampConversion =
+ !Strings.nullToEmpty(fileMetaData.getCreatedBy()).startsWith("parquet-mr");
+ } else {
+ this.skipTimestampConversion = false;
+ }
+ Boolean proleptic = DataWritableReadSupport.getWriterDateProleptic(kvMeta);
+ if (proleptic == null) {
+ proleptic = HiveConf.getBoolVar(jobConf, ConfVars.HIVE_PARQUET_DATE_PROLEPTIC_GREGORIAN_DEFAULT);
+ }
+ this.skipProlepticConversion = proleptic;
+ this.legacyConversionEnabled = DataWritableReadSupport.getZoneConversionLegacy(kvMeta, jobConf);
+
+ this.schemaInitialized = true;
+ }
+
+ @Override
+ protected void decodeBatch(ParquetEncodedColumnBatch batch,
+ Consumer<ColumnVectorBatch> downstreamConsumer) throws InterruptedException {
+ if (!schemaInitialized) {
+ initSchema();
+ }
+
+ long startTime = counters.startTimeCounter();
+ // NOSONAR - S2093 does not apply: codecFactory is a per-consumer field with a per-batch
+ // release() (not close()); try-with-resources would tie it to this scope, which is wrong.
+ try { // NOSONAR
+ PageReadStore pages = new ParquetCachedPageReadStore(footer, batch, codecFactory, converter);
+
+ VectorizedColumnReader[] columnReaders =
+ new ParquetRowGroupDecoder(footer.getFileMetaData().getSchema(), initialDefaults).buildColumnReaders(
+ pages, requestedSchema, columnTypesList, colsToInclude, readAllColumns,
+ new ParquetRowGroupDecoder.TimestampConversionOptions(skipTimestampConversion,
+ writerTimezone, skipProlepticConversion, legacyConversionEnabled));
+
+ long rowsLeft = pages.getRowCount();
+ int batches = 0;
+ while (rowsLeft > 0) {
+ int batchSize = (int) Math.min(VectorizedRowBatch.DEFAULT_SIZE, rowsLeft);
+
+ ColumnVectorBatch cvb = takeBatch(batchSize);
+
+ // columnReaders[i] is requestedSchema field i, i.e. the i-th projected column.
+ for (int i = 0; i < columnReaders.length; ++i) {
+ if (columnReaders[i] == null) {
+ continue;
+ }
+ TypeInfo columnType = columnTypesList.get(readAllColumns ? i : colsToInclude.get(i));
+ ColumnVector cv = prepareColumnVector(cvb, i, columnType, batchSize);
+ columnReaders[i].readBatch(batchSize, cv, columnType);
+ }
+
+ downstreamConsumer.consumeData(cvb);
+ counters.incrCounter(LlapIOCounters.ROWS_EMITTED, batchSize);
+ rowsLeft -= batchSize;
+ ++batches;
+ }
+ counters.incrWallClockCounter(LlapIOCounters.DECODE_TIME_NS, startTime);
+ counters.incrCounter(LlapIOCounters.NUM_VECTOR_BATCHES, batches);
+ counters.incrCounter(LlapIOCounters.NUM_DECODED_BATCHES);
+ } catch (IOException | RuntimeException e) {
+ // parquet reports decode failures as runtime ParquetDecodingException.
+ LlapIoImpl.LOG.error("Parquet decodeBatch failed for rowGroup " + batch.rowGroupIx() + " of " + path, e);
+ downstreamConsumer.setError(e);
Review Comment:
it is possible: `DECODE_FAILED_ERR.format(batch.rowGroupIx(), path)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4155079140
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java:
##########
@@ -171,7 +178,8 @@ private LlapIoImpl(Configuration conf) throws IOException {
MetadataCache metadataCache = null;
SerDeLowLevelCacheImpl serdeCache = null; // TODO: extract interface when needed
- BufferUsageManager bufferManagerOrc = null, bufferManagerGeneric = null;
+ BufferUsageManager bufferManagerData = null;
Review Comment:
I think `bufferManagerData` is pure redundancy with the `bufferManager` field and should be removed in favor of passing bufferManager directly, while bufferManagerGeneric is legitimately distinct and should stay
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java:
##########
@@ -219,7 +227,9 @@ private LlapIoImpl(Configuration conf) throws IOException {
cachePolicyWrapper.setEvictionListener(e);
cacheImpl.startThreads(); // Start the cache threads.
- bufferManager = bufferManagerOrc = cacheImpl; // Cache also serves as buffer manager.
+ // Cache also serves as buffer manager for both data and generic (encoded/serde) paths.
+ bufferManagerData = cacheImpl;
Review Comment:
I think bufferManagerData is pure redundancy with the bufferManager field and should be removed in favor of passing bufferManager directly, while bufferManagerGeneric is legitimately distinct and should stay
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4155125927
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java:
##########
@@ -144,6 +143,19 @@ private void returnSourceData(BatchType data) {
protected abstract void decodeBatch(BatchType batch,
Consumer<ColumnVectorBatch> downstreamConsumer) throws InterruptedException;
+ /**
+ * Takes a batch from the pool and readies it for one decode pass. Pooled batches come back with
+ * whatever the last pass left on them - {@code resetBeforeOffer} deliberately resets nothing, so
+ * the column vectors can be reused - which makes clearing the filter context part of taking one:
+ * a selection left over from the previous batch would otherwise be read as this batch's own.
+ */
+ protected ColumnVectorBatch takeBatch(int batchSize) {
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4155602822
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
Review Comment:
maybe `readFooter` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4155989825
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ // Fallback path: no cache key, so we read status + footer straight from HDFS. Time both
+ // under HDFS_TIME_NS to match how ORC accounts for cache-miss footer reads.
+ final FileSystem fs = path.getFileSystem(jobConf);
+ long hdfsStart = counters.startTimeCounter();
+ final FileStatus stat = fs.getFileStatus(path);
+ counters.recordHdfsTime(hdfsStart);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ hdfsStart = counters.startTimeCounter();
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ counters.recordHdfsTime(hdfsStart);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ long startTime = counters.startTimeCounter();
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Exception e) {
+ // A shutdown-triggered InterruptedException must not be silently reported as an ordinary
+ // consumer error: preserve the interrupt on the thread so any caller sees it.
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ }
+ consumer.setError(e);
+ } finally {
+ counters.incrWallClockCounter(LlapIOCounters.TOTAL_IO_TIME_NS, startTime);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ // The LLAP IO summary keys off TABLE / FILE / STRIPES; set them the same way ORC does so a
+ // native Parquet fragment shows up in the summary with the same fields populated.
+ if (cacheTag != null) {
+ counters.setDesc(QueryFragmentCounters.Desc.TABLE, cacheTag.getTableName());
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.FILE, path
+ + (fileKey == null ? "" : " (" + fileKey + ")"));
+
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = selectRowGroups(blocks, fileSchema);
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ // STRIPES is the ORC term; for Parquet the analog is row groups. Reuse the same descriptor so
+ // the summary layout stays common and we don't have to teach the reporter about a new field.
+ counters.setDesc(QueryFragmentCounters.Desc.STRIPES, "0," + selected.size());
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /**
+ * The row groups this split has to read: the ones whose first data page falls inside the split, as
+ * Parquet's own input format assigns them, minus any the pushed-down predicate rules out on its
+ * statistics. Returned in file order, since the fetch pipeline reads them in that order.
+ */
+ private List<BlockMetaData> selectRowGroups(List<BlockMetaData> blocks, MessageType fileSchema)
+ throws IOException {
+ long splitStart = split.getStart();
+ long splitEnd = splitStart + split.getLength();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().getFirst().getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate == null) {
+ return selected;
+ }
+ return RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Whether the query needs row-lineage virtual columns ({@code ROW__LINEAGE__ID} /
+ * {@code LAST__UPDATED__SEQUENCE__NUMBER}) that the fallback reader augments the requested schema
+ * with via {@link RowLineageUtils#getRequestedSchemaWithRowLineageColumns}. The native pipeline
+ * doesn't propagate that augmentation through {@code Includes} yet, so we defer to the fallback
+ * reader instead of quietly emitting nulls in the lineage slots.
+ */
+ public boolean needsRowLineage() {
+ VectorizedRowBatchCtx rbCtx = Utilities.getVectorizedRowBatchCtx(jobConf);
+ if (rbCtx == null) {
+ return false;
+ }
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ return RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema, VirtualColumn.ROW_LINEAGE_ID)
+ || RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema,
+ VirtualColumn.LAST_UPDATED_SEQUENCE_NUMBER);
+ }
+
+ /**
+ * Leaf-column positions of the requested fields; {@code BlockMetaData.getColumns()} is a flat list
+ * of leaves in file-schema order, so a top-level primitive at ordinal {@code k} in the file schema
+ * may sit at a very different leaf index (e.g. a nested group of two leaves before it shifts it
+ * from {@code 1} to {@code 2}). We only reach this method when every requested field is a
+ * top-level primitive ({@link #projectsNestedTypes()} would have forced a fallback otherwise), so
+ * each requested field maps to exactly one leaf whose path is a single segment.
+ */
+ static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
Review Comment:
/** Leaf index in {@code BlockMetaData.getColumns()} of each requested field present in the file. */
static int[] findLeafIndexes(MessageType requestedSchema, MessageType fileSchema)
with the variable and the parameters renamed from projected to leafIndexes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4157010703
##########
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetRowGroupDecoder.java:
##########
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.ql.io.parquet.vector;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.parquet.ParquetRuntimeException;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.InvalidSchemaException;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Reusable, stateless helper that builds the {@link VectorizedColumnReader} array for one Parquet
+ * row group, given an already-read {@link PageReadStore}. The logic here was lifted verbatim from
+ * {@link VectorizedParquetRecordReader} so that both the row-by-split reader and the LLAP
+ * cache-backed consumer ({@code ParquetEncodedDataConsumer}) can share a single, behavior-preserving
+ * implementation of the Hive-type-driven Parquet column-reader construction.
+ *
+ * <p>The instance only carries the {@code fileSchema} (used for the schema-evolution check). All
+ * per-row-group state lives in the supplied {@link PageReadStore}.
+ */
+public class ParquetRowGroupDecoder {
+
+ private static final int MAP_DEFINITION_LEVEL_MAX = 3;
+
+ /**
+ * Bag of writer-timezone / proleptic / legacy conversion flags forwarded to every primitive
+ * column reader. Grouped so the reader-construction entry point stays under the parameter cap.
+ */
+ public record TimestampConversionOptions(
+ boolean skipTimestampConversion,
+ ZoneId writerTimezone,
+ boolean skipProlepticConversion,
+ boolean legacyConversionEnabled) {
+ }
+
+ private final MessageType fileSchema;
+ private final Map<String, Object> initialDefaults;
+
+ public ParquetRowGroupDecoder(MessageType fileSchema, Map<String, Object> initialDefaults) {
+ this.fileSchema = fileSchema;
+ this.initialDefaults = initialDefaults;
+ }
+
+ /**
+ * Builds the per-(requested-)column {@link VectorizedColumnReader} array for a row group.
+ *
+ * @param pages the row group's page store (from {@code reader.readRowGroup(..)}
+ * / {@code readNextRowGroup()})
+ * @param requestedSchema the projected Parquet schema being read
+ * @param columnTypesList the Hive type infos for ALL table columns (indexed by table col id)
+ * @param colsToInclude the table column ids being read, in requested-schema field order;
+ * may be empty (e.g. {@code count(*)}), in which case all readers are null
+ * @param readAllColumns whether projection is "read all columns"
+ * @param options writer-timezone / proleptic / legacy conversion flags, see
+ * {@link TimestampConversionOptions}
+ * @return one reader per requested-schema field (null entries where no reader is needed)
+ */
+ public VectorizedColumnReader[] buildColumnReaders(
+ PageReadStore pages,
+ MessageType requestedSchema,
+ List<TypeInfo> columnTypesList,
+ List<Integer> colsToInclude,
+ boolean readAllColumns,
+ TimestampConversionOptions options) throws IOException {
+ List<Type> types = requestedSchema.getFields();
+ VectorizedColumnReader[] columnReaders = new VectorizedColumnReader[types.size()];
+
+ if (!readAllColumns) {
+ // certain queries like select count(*) from table do not have
+ // any projected columns and still have isReadAllColumns as false
+ // in such cases columnReaders are not needed
+ // However, if colsToInclude is not empty we should initialize each columnReader
+ if (!colsToInclude.isEmpty()) {
+ for (int i = 0; i < types.size(); ++i) {
+ columnReaders[i] = buildVectorizedParquetReader(
+ columnTypesList.get(colsToInclude.get(i)), types.get(i), pages,
+ requestedSchema.getColumns(), options, 0, 0);
+ }
+ }
+ } else {
+ for (int i = 0; i < types.size(); ++i) {
+ columnReaders[i] = buildVectorizedParquetReader(columnTypesList.get(i),
+ types.get(i), pages, requestedSchema.getColumns(), options, 0, 0);
+ }
+ }
+ return columnReaders;
+ }
+
+ private static List<ColumnDescriptor> getAllColumnDescriptorByType(
+ int depth,
+ Type type,
+ List<ColumnDescriptor> columns) throws ParquetRuntimeException {
+ List<ColumnDescriptor> res = new ArrayList<>();
+ for (ColumnDescriptor descriptor : columns) {
+ if (depth >= descriptor.getPath().length) {
+ throw new InvalidSchemaException("Corrupted Parquet schema");
+ }
+ if (type.getName().equals(descriptor.getPath()[depth])) {
+ res.add(descriptor);
+ }
+ }
+ return res;
+ }
+
+ // Nested types are unsupported here on purpose; callers detect that in advance
+ // (ParquetEncodedDataReader.projectsNestedTypes) and route the query to the non-native reader.
+ private static PrimitiveType getElementType(Type type) {
Review Comment:
listElementType ? returns the primitive element type of a LIST's three-level encoding
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4184457373
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ParquetCachedPageReadStore.java:
##########
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.decode;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.llap.io.encoded.ParquetEncodedColumnBatch;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.DataPage;
+import org.apache.parquet.column.page.DataPageV1;
+import org.apache.parquet.column.page.DataPageV2;
+import org.apache.parquet.column.page.DictionaryPage;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.column.page.PageReader;
+import org.apache.parquet.compression.CompressionCodecFactory;
+import org.apache.parquet.compression.CompressionCodecFactory.BytesInputDecompressor;
+import org.apache.parquet.format.DataPageHeader;
+import org.apache.parquet.format.DataPageHeaderV2;
+import org.apache.parquet.format.DictionaryPageHeader;
+import org.apache.parquet.format.PageHeader;
+import org.apache.parquet.format.Util;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.schema.PrimitiveType;
+
+/**
+ * {@link PageReadStore} over one row group's cached column-chunk buffers, so that the vectorized
+ * column readers decode straight out of the LLAP cache: no ParquetFileReader, and no copy of the
+ * chunk bytes.
+ *
+ * <p><b>Why this is a copy.</b> Parquet does the same thing over a file, but none of it is
+ * reachable from here: {@code ParquetFileReader.Chunk.readAllPages} is private, the store it fills
+ * ({@code org.apache.parquet.hadoop.ColumnChunkPageReadStore}) is package-private and so is its
+ * {@code addColumn}, and the {@link PageReader} it puts in ({@code ColumnChunkPageReader}) is a
+ * package-private {@code final} class. Only the glue below is re-implemented; page header parsing,
+ * encoding and statistics conversion, the page value types and decompression are all parquet's own
+ * public API.
+ *
+ * <p><b>On a parquet upgrade.</b> Exactly two members track upstream code, each with a counterpart
+ * to diff against the new release:
+ * <ul>
+ * <li>{@link #readAllPages} mirrors {@code ParquetFileReader.Chunk.readAllPages}: the page-header
+ * walk, the page-type switch and the mapping of header fields onto {@link DataPageV1},
+ * {@link DataPageV2} and {@link DictionaryPage}. A new page type, a new header field or a
+ * changed page constructor lands here.</li>
+ * <li>{@link CachedChunkPageReader} mirrors {@code ColumnChunkPageReader}: page queueing and
+ * lazy decompression. New {@link PageReader} behaviour lands here.</li>
+ * </ul>
+ * Also check whether {@link PageReadStore} gained or changed a {@code default} method, since this
+ * class implements none of them ({@code getRowIndexOffset}, {@code getRowIndexes} and
+ * {@code close} are all inherited). {@code TestParquetCachedPageReadStore} compares this store page
+ * for page against the stock one and pins the parquet version, so the upgrade fails the build
+ * rather than passing with a divergence.
+ *
+ * <p>Deliberately unsupported, because LLAP's Parquet path does not use them: the offset index (so
+ * {@code getFirstRowIndex} and {@code getIndexRowCount} keep their no-offset-index defaults),
+ * column encryption, and page checksum verification.
+ *
+ * <p>How it works:
+ * <ol>
+ * <li><b>Input</b>: a {@link ParquetEncodedColumnBatch} holding the already-cached
+ * {@link MemoryBuffer}s for one row group's projected column chunks, plus the file footer.</li>
+ * <li><b>Stitching and parsing</b>: per chunk, {@link #chunkBuffers} turns the cache buffers into a
+ * list of {@link ByteBuffer} slices covering exactly the chunk's byte region; {@link #readAllPages}
+ * then walks that byte range with {@link Util#readPageHeader}, building {@link DataPageV1} /
+ * {@link DataPageV2} / {@link DictionaryPage} objects whose payloads are
+ * {@link ByteBufferInputStream} views over the cache buffers - the page bytes are never copied.</li>
+ * <li><b>Lazy decompression</b>: each {@link CachedChunkPageReader} keeps the raw bytes and its
+ * {@link BytesInputDecompressor}, and decompresses only when the vectorized column reader
+ * actually calls {@link PageReader#readPage()}, so pages that get pruned never pay the codec cost.</li>
+ * <li><b>Output</b>: the resulting {@code ColumnPath -> PageReader} map backs
+ * {@link #getPageReader(ColumnDescriptor)}, so parquet's VectorizedColumnReader never notices
+ * that it is not reading from a file.</li>
+ * </ol>
+ */
+class ParquetCachedPageReadStore implements PageReadStore {
+
+ private final Map<ColumnPath, PageReader> readers = new HashMap<>();
+ private final long rowCount;
+
+ ParquetCachedPageReadStore(ParquetMetadata footer, ParquetEncodedColumnBatch batch,
+ CompressionCodecFactory codecFactory, ParquetMetadataConverter converter) throws IOException {
+ BlockMetaData block = footer.getBlocks().get(batch.rowGroupIx());
+ this.rowCount = block.getRowCount();
+ String createdBy = footer.getFileMetaData().getCreatedBy();
+ ColumnChunkMetaData[] chunks = batch.chunks();
+ // pc is the projection index: chunks[pc] and the batch's three buffer arrays are all indexed by
+ // it, while the footer's block lists columns in file order. Keying on the path keeps the two
+ // apart, so a projection that drops columns from the middle of the schema still lines up.
+ for (int pc = 0; pc < chunks.length; ++pc) {
+ ColumnChunkMetaData chunk = chunks[pc];
+ readers.put(chunk.getPath(), readAllPages(chunk, chunkBuffers(batch, pc), createdBy,
+ codecFactory.getDecompressor(chunk.getCodec()), converter));
+ }
+ }
+
+ /**
+ * Slices of the cached buffers covering exactly the chunk's byte region, in file order.
+ *
+ * <p>This is the only place in the class that touches bytes, and the only place the cache appears
+ * at all: the {@link MemoryBuffer}s are already-populated LLAP cache memory, so there is no read
+ * here and nothing to decide - by the time this runs, a cache hit and a miss that
+ * {@code ParquetEncodedDataReader} had to fetch look exactly the same.
+ *
+ * <p>The buffers are cache ranges aligned to absolute file offsets, not to this chunk, so the
+ * first and last one usually overhang the chunk (the first may even start before it). That is what
+ * the trimming below is for.
+ *
+ * <p>The trimmed slices have to tile the chunk exactly, which is checked as they are built: a
+ * buffer that does not continue where the previous one ended, or one that reaches past the chunk,
+ * means the batch is not carrying the buffers the chunk was planned with. Without the check the
+ * page walk would read the wrong bytes as page bytes, and fail much later with a decoding error
+ * about a corrupt header.
+ */
+ private static List<ByteBuffer> chunkBuffers(ParquetEncodedColumnBatch batch, int pc)
+ throws IOException {
+ ColumnChunkMetaData chunk = batch.chunks()[pc];
+ long chunkStart = chunk.getStartingPos();
+ long chunkEnd = chunkStart + chunk.getTotalSize();
+ MemoryBuffer[] columnBuffers = batch.columnBuffers()[pc];
+ long[] bufferOffsets = batch.bufferOffsets()[pc];
+ int[] bufferLengths = batch.bufferLengths()[pc];
+ List<ByteBuffer> slices = new ArrayList<>(columnBuffers.length);
+ long covered = chunkStart;
+ for (int i = 0; i < columnBuffers.length; ++i) {
+ long bufferStart = bufferOffsets[i];
+ long bufferEnd = bufferStart + bufferLengths[i];
+ long sliceStart = Math.max(chunkStart, bufferStart);
+ long sliceEnd = Math.min(chunkEnd, bufferEnd);
+ if (sliceStart != covered || sliceEnd <= sliceStart) {
+ throw new IOException("Cached buffer " + (i + 1) + " of " + columnBuffers.length + " for column chunk "
+ + chunk.getPath() + " covers [" + sliceStart + ", " + sliceEnd + ") of the chunk at ["
+ + chunkStart + ", " + chunkEnd + "), which does not continue at " + covered);
+ }
+ covered = sliceEnd;
+ // A dup, because the cache buffer is shared: moving position/limit must not be visible to
+ // another reader of the same buffer. slice() then keeps a view of the trimmed region, so the
+ // chunk bytes are still the cache's bytes - nothing is copied here or below.
+ ByteBuffer bb = columnBuffers[i].getByteBufferDup();
+ bb.position(bb.position() + (int) (sliceStart - bufferStart));
+ bb.limit(bb.position() + (int) (sliceEnd - sliceStart));
+ slices.add(bb.slice());
+ }
+ if (covered != chunkEnd) {
+ throw new IOException("Cached buffers for column chunk " + chunk.getPath() + " cover ["
+ + chunkStart + ", " + covered + "), but the chunk ends at " + chunkEnd);
+ }
+ return slices;
+ }
+
+ private static PageReader readAllPages(ColumnChunkMetaData chunk, List<ByteBuffer> buffers,
+ String createdBy, BytesInputDecompressor decompressor, ParquetMetadataConverter converter)
+ throws IOException {
+ // Reads across the buffer boundaries as if the chunk were contiguous, which it is not: a page
+ // header or a payload can straddle two cache buffers. sliceBuffers below hands out views into
+ // these same buffers, so every page this method builds points at cache memory for as long as it
+ // lives - the batch has to keep its refs until the consumer is done decoding.
+ ByteBufferInputStream stream = ByteBufferInputStream.wrap(buffers);
+ PrimitiveType type = chunk.getPrimitiveType();
+ List<DataPage> pages = new ArrayList<>();
+ DictionaryPage dictionaryPage = null;
+ long valuesRead = 0;
+ // Only data pages count towards valuesRead, so the dictionary page and any page type skipped
+ // below do not end the walk early; the footer's value count is the only terminator.
+ while (valuesRead < chunk.getValueCount()) {
+ PageHeader header = Util.readPageHeader(stream);
+ int uncompressedSize = header.getUncompressed_page_size();
+ int compressedSize = header.getCompressed_page_size();
+ switch (header.getType()) {
+ case DICTIONARY_PAGE:
+ if (dictionaryPage != null) {
+ throw new ParquetDecodingException("more than one dictionary page in column " + chunk.getPath());
+ }
+ DictionaryPageHeader dictHeader = header.getDictionary_page_header();
+ dictionaryPage = new DictionaryPage(BytesInput.from(stream.sliceBuffers(compressedSize)),
+ uncompressedSize, dictHeader.getNum_values(), converter.getEncoding(dictHeader.getEncoding()));
+ break;
+ case DATA_PAGE:
+ DataPageHeader v1 = header.getData_page_header();
+ pages.add(new DataPageV1(BytesInput.from(stream.sliceBuffers(compressedSize)), v1.getNum_values(),
+ uncompressedSize, converter.fromParquetStatistics(createdBy, v1.getStatistics(), type),
+ converter.getEncoding(v1.getRepetition_level_encoding()),
+ converter.getEncoding(v1.getDefinition_level_encoding()),
+ converter.getEncoding(v1.getEncoding())));
+ valuesRead += v1.getNum_values();
+ break;
+ case DATA_PAGE_V2:
Review Comment:
yes, they are already there:
```
@Test
public void testDataPageV2Uncompressed() throws Exception {
assertParity(WriterVersion.PARQUET_2_0, CompressionCodecName.UNCOMPRESSED, BUFFER_GRAIN, SCHEMA);å
}
@Test
public void testDataPageV2Snappy() throws Exception {
assertParity(WriterVersion.PARQUET_2_0, CompressionCodecName.SNAPPY, BUFFER_GRAIN, SCHEMA);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4217579491
##########
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetRowGroupDecoder.java:
##########
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.ql.io.parquet.vector;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.parquet.ParquetRuntimeException;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.InvalidSchemaException;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Reusable, stateless helper that builds the {@link VectorizedColumnReader} array for one Parquet
+ * row group, given an already-read {@link PageReadStore}. The logic here was lifted verbatim from
+ * {@link VectorizedParquetRecordReader} so that both the row-by-split reader and the LLAP
+ * cache-backed consumer ({@code ParquetEncodedDataConsumer}) can share a single, behavior-preserving
+ * implementation of the Hive-type-driven Parquet column-reader construction.
+ *
+ * <p>The instance only carries the {@code fileSchema} (used for the schema-evolution check). All
+ * per-row-group state lives in the supplied {@link PageReadStore}.
+ */
+public class ParquetRowGroupDecoder {
+
+ private static final int MAP_DEFINITION_LEVEL_MAX = 3;
+
+ /**
+ * Bag of writer-timezone / proleptic / legacy conversion flags forwarded to every primitive
+ * column reader. Grouped so the reader-construction entry point stays under the parameter cap.
+ */
+ public record TimestampConversionOptions(
+ boolean skipTimestampConversion,
+ ZoneId writerTimezone,
+ boolean skipProlepticConversion,
+ boolean legacyConversionEnabled) {
+ }
+
+ private final MessageType fileSchema;
+ private final Map<String, Object> initialDefaults;
+
+ public ParquetRowGroupDecoder(MessageType fileSchema, Map<String, Object> initialDefaults) {
+ this.fileSchema = fileSchema;
+ this.initialDefaults = initialDefaults;
+ }
+
+ /**
+ * Builds the per-(requested-)column {@link VectorizedColumnReader} array for a row group.
+ *
+ * @param pages the row group's page store (from {@code reader.readRowGroup(..)}
+ * / {@code readNextRowGroup()})
+ * @param requestedSchema the projected Parquet schema being read
+ * @param columnTypesList the Hive type infos for ALL table columns (indexed by table col id)
+ * @param colsToInclude the table column ids being read, in requested-schema field order;
+ * may be empty (e.g. {@code count(*)}), in which case all readers are null
+ * @param readAllColumns whether projection is "read all columns"
+ * @param options writer-timezone / proleptic / legacy conversion flags, see
+ * {@link TimestampConversionOptions}
+ * @return one reader per requested-schema field (null entries where no reader is needed)
+ */
+ public VectorizedColumnReader[] buildColumnReaders(
+ PageReadStore pages,
+ MessageType requestedSchema,
+ List<TypeInfo> columnTypesList,
+ List<Integer> colsToInclude,
+ boolean readAllColumns,
+ TimestampConversionOptions options) throws IOException {
+ List<Type> types = requestedSchema.getFields();
+ VectorizedColumnReader[] columnReaders = new VectorizedColumnReader[types.size()];
+
+ if (!readAllColumns) {
+ // certain queries like select count(*) from table do not have
+ // any projected columns and still have isReadAllColumns as false
+ // in such cases columnReaders are not needed
+ // However, if colsToInclude is not empty we should initialize each columnReader
+ if (!colsToInclude.isEmpty()) {
+ for (int i = 0; i < types.size(); ++i) {
+ columnReaders[i] = buildVectorizedParquetReader(
+ columnTypesList.get(colsToInclude.get(i)), types.get(i), pages,
+ requestedSchema.getColumns(), options, 0, 0);
+ }
+ }
+ } else {
+ for (int i = 0; i < types.size(); ++i) {
+ columnReaders[i] = buildVectorizedParquetReader(columnTypesList.get(i),
+ types.get(i), pages, requestedSchema.getColumns(), options, 0, 0);
+ }
+ }
+ return columnReaders;
+ }
+
+ private static List<ColumnDescriptor> getAllColumnDescriptorByType(
+ int depth,
+ Type type,
+ List<ColumnDescriptor> columns) throws ParquetRuntimeException {
+ List<ColumnDescriptor> res = new ArrayList<>();
+ for (ColumnDescriptor descriptor : columns) {
+ if (depth >= descriptor.getPath().length) {
+ throw new InvalidSchemaException("Corrupted Parquet schema");
+ }
+ if (type.getName().equals(descriptor.getPath()[depth])) {
+ res.add(descriptor);
+ }
+ }
+ return res;
+ }
+
+ // Nested types are unsupported here on purpose; callers detect that in advance
+ // (ParquetEncodedDataReader.projectsNestedTypes) and route the query to the non-native reader.
+ private static PrimitiveType getElementType(Type type) {
+ if (type.isPrimitive()) {
+ return type.asPrimitiveType();
+ }
+ if (type.asGroupType().getFields().size() > 1) {
+ throw new UnsupportedOperationException(
+ "Current Parquet Vectorization reader doesn't support nested type");
+ }
+
+ Type childType = type.asGroupType().getFields().getFirst();
+
+ // Parquet file generated using thrift may have child type as PrimitiveType
+ if (childType.isPrimitive()) {
+ return childType.asPrimitiveType();
+ } else {
+ return childType.asGroupType().getFields().getFirst().asPrimitiveType();
+ }
+ }
+
+ // Build VectorizedParquetColumnReader via Hive typeInfo and Parquet schema
+ private VectorizedColumnReader buildVectorizedParquetReader(
+ TypeInfo typeInfo,
+ Type type,
+ PageReadStore pages,
+ List<ColumnDescriptor> columnDescriptors,
+ TimestampConversionOptions options,
+ int depth, int currentDefLevel) throws IOException {
+ int typeDefLevel = currentDefLevel;
+ if (type.isRepetition(Type.Repetition.OPTIONAL) || type.isRepetition(Type.Repetition.REPEATED)) {
+ typeDefLevel++;
+ }
+ List<ColumnDescriptor> descriptors =
+ getAllColumnDescriptorByType(depth, type, columnDescriptors);
+ // Support for schema evolution: if the column from the current
+ // query schema is not present in the file schema, return a dummy
+ // reader that produces nulls. This allows queries to proceed even
+ // when new columns have been added after the file was written.
+ if (!fileSchema.getColumns().contains(descriptors.getFirst())) {
Review Comment:
good catch, not it's a field:
```
this.fileColumns = new HashSet<>(fileSchema.getColumns());
```
##########
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetRowGroupDecoder.java:
##########
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.ql.io.parquet.vector;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.parquet.ParquetRuntimeException;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.InvalidSchemaException;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Reusable, stateless helper that builds the {@link VectorizedColumnReader} array for one Parquet
+ * row group, given an already-read {@link PageReadStore}. The logic here was lifted verbatim from
+ * {@link VectorizedParquetRecordReader} so that both the row-by-split reader and the LLAP
+ * cache-backed consumer ({@code ParquetEncodedDataConsumer}) can share a single, behavior-preserving
+ * implementation of the Hive-type-driven Parquet column-reader construction.
+ *
+ * <p>The instance only carries the {@code fileSchema} (used for the schema-evolution check). All
+ * per-row-group state lives in the supplied {@link PageReadStore}.
+ */
+public class ParquetRowGroupDecoder {
+
+ private static final int MAP_DEFINITION_LEVEL_MAX = 3;
+
+ /**
+ * Bag of writer-timezone / proleptic / legacy conversion flags forwarded to every primitive
+ * column reader. Grouped so the reader-construction entry point stays under the parameter cap.
+ */
+ public record TimestampConversionOptions(
+ boolean skipTimestampConversion,
+ ZoneId writerTimezone,
+ boolean skipProlepticConversion,
+ boolean legacyConversionEnabled) {
+ }
+
+ private final MessageType fileSchema;
+ private final Map<String, Object> initialDefaults;
+
+ public ParquetRowGroupDecoder(MessageType fileSchema, Map<String, Object> initialDefaults) {
+ this.fileSchema = fileSchema;
+ this.initialDefaults = initialDefaults;
+ }
+
+ /**
+ * Builds the per-(requested-)column {@link VectorizedColumnReader} array for a row group.
+ *
+ * @param pages the row group's page store (from {@code reader.readRowGroup(..)}
+ * / {@code readNextRowGroup()})
+ * @param requestedSchema the projected Parquet schema being read
+ * @param columnTypesList the Hive type infos for ALL table columns (indexed by table col id)
+ * @param colsToInclude the table column ids being read, in requested-schema field order;
+ * may be empty (e.g. {@code count(*)}), in which case all readers are null
+ * @param readAllColumns whether projection is "read all columns"
+ * @param options writer-timezone / proleptic / legacy conversion flags, see
+ * {@link TimestampConversionOptions}
+ * @return one reader per requested-schema field (null entries where no reader is needed)
+ */
+ public VectorizedColumnReader[] buildColumnReaders(
+ PageReadStore pages,
+ MessageType requestedSchema,
+ List<TypeInfo> columnTypesList,
+ List<Integer> colsToInclude,
+ boolean readAllColumns,
+ TimestampConversionOptions options) throws IOException {
+ List<Type> types = requestedSchema.getFields();
+ VectorizedColumnReader[] columnReaders = new VectorizedColumnReader[types.size()];
+
+ if (!readAllColumns) {
+ // certain queries like select count(*) from table do not have
+ // any projected columns and still have isReadAllColumns as false
+ // in such cases columnReaders are not needed
+ // However, if colsToInclude is not empty we should initialize each columnReader
+ if (!colsToInclude.isEmpty()) {
+ for (int i = 0; i < types.size(); ++i) {
+ columnReaders[i] = buildVectorizedParquetReader(
+ columnTypesList.get(colsToInclude.get(i)), types.get(i), pages,
+ requestedSchema.getColumns(), options, 0, 0);
+ }
+ }
+ } else {
+ for (int i = 0; i < types.size(); ++i) {
+ columnReaders[i] = buildVectorizedParquetReader(columnTypesList.get(i),
+ types.get(i), pages, requestedSchema.getColumns(), options, 0, 0);
+ }
+ }
+ return columnReaders;
+ }
+
+ private static List<ColumnDescriptor> getAllColumnDescriptorByType(
+ int depth,
+ Type type,
+ List<ColumnDescriptor> columns) throws ParquetRuntimeException {
+ List<ColumnDescriptor> res = new ArrayList<>();
+ for (ColumnDescriptor descriptor : columns) {
+ if (depth >= descriptor.getPath().length) {
+ throw new InvalidSchemaException("Corrupted Parquet schema");
+ }
+ if (type.getName().equals(descriptor.getPath()[depth])) {
+ res.add(descriptor);
+ }
+ }
+ return res;
+ }
+
+ // Nested types are unsupported here on purpose; callers detect that in advance
+ // (ParquetEncodedDataReader.projectsNestedTypes) and route the query to the non-native reader.
+ private static PrimitiveType getElementType(Type type) {
+ if (type.isPrimitive()) {
+ return type.asPrimitiveType();
+ }
+ if (type.asGroupType().getFields().size() > 1) {
+ throw new UnsupportedOperationException(
+ "Current Parquet Vectorization reader doesn't support nested type");
+ }
+
+ Type childType = type.asGroupType().getFields().getFirst();
+
+ // Parquet file generated using thrift may have child type as PrimitiveType
+ if (childType.isPrimitive()) {
+ return childType.asPrimitiveType();
+ } else {
+ return childType.asGroupType().getFields().getFirst().asPrimitiveType();
+ }
+ }
+
+ // Build VectorizedParquetColumnReader via Hive typeInfo and Parquet schema
+ private VectorizedColumnReader buildVectorizedParquetReader(
+ TypeInfo typeInfo,
+ Type type,
+ PageReadStore pages,
+ List<ColumnDescriptor> columnDescriptors,
+ TimestampConversionOptions options,
+ int depth, int currentDefLevel) throws IOException {
+ int typeDefLevel = currentDefLevel;
+ if (type.isRepetition(Type.Repetition.OPTIONAL) || type.isRepetition(Type.Repetition.REPEATED)) {
+ typeDefLevel++;
+ }
+ List<ColumnDescriptor> descriptors =
+ getAllColumnDescriptorByType(depth, type, columnDescriptors);
+ // Support for schema evolution: if the column from the current
+ // query schema is not present in the file schema, return a dummy
+ // reader that produces nulls. This allows queries to proceed even
+ // when new columns have been added after the file was written.
+ if (!fileSchema.getColumns().contains(descriptors.getFirst())) {
Review Comment:
good catch, now it's a field:
```
this.fileColumns = new HashSet<>(fileSchema.getColumns());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4035069559
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,560 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ColumnVectorProducer.Includes;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final Includes includes;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split, Includes includes,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.includes = includes;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ final FileSystem fs = path.getFileSystem(jobConf);
+ final FileStatus stat = fs.getFileStatus(path);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Throwable t) {
+ consumer.setError(t);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final int maxAlloc = allocator.getMaxAllocation();
+ final long splitStart = split.getStart(), splitEnd = splitStart + split.getLength();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().get(0).getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate != null) {
+ selected = RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, maxAlloc, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, maxAlloc, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /** File-schema positions of the requested fields; column chunks follow the schema order. */
+ private static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
+ List<Integer> leaves = new ArrayList<>();
+ for (Type field : requestedSchema.getFields()) {
+ if (fileSchema.containsField(field.getName())) {
+ leaves.add(fileSchema.getFieldIndex(field.getName()));
+ }
+ }
+ return leaves.stream().mapToInt(Integer::intValue).toArray();
+ }
+
+ private static long bytes(int[] projected, BlockMetaData block) {
+ long total = 0;
+ for (int leaf : projected) {
+ total += block.getColumns().get(leaf).getTotalSize();
+ }
+ return total;
+ }
+
+ private static long bytes(Fetch fetch) {
+ long total = 0;
+ for (ColumnChunkMetaData chunk : fetch.batch.chunks) {
+ total += chunk.getTotalSize();
+ }
+ return total;
+ }
+
+ /** One row group on its way in: buffers planned, missing ranges requested, not yet decoded. */
+ private static final class Fetch {
+ private final ParquetEncodedColumnBatch batch = new ParquetEncodedColumnBatch();
+ private final List<ColumnPlan> columns = new ArrayList<>();
+ private final List<Part> misses = new ArrayList<>();
+ private final List<Run> runs = new ArrayList<>();
+ }
+
+ /** The buffers covering one column chunk, in file order, and which stretches of them are new. */
+ private record ColumnPlan(List<Part> parts, List<MissRun> missRuns) {
+ ColumnPlan() {
+ this(new ArrayList<>(), new ArrayList<>());
+ }
+ }
+
+ /** One cache buffer's worth of a column chunk: a hit handed back by the cache, or a miss to fill. */
+ private static final class Part {
+ private MemoryBuffer buffer;
+ private final DiskRange range;
+ private final boolean miss;
+ /** We hold one ref to release; until then a miss is a raw allocation to free. */
+ private boolean owned;
+
+ Part(MemoryBuffer buffer, DiskRange range, boolean miss) {
+ this.buffer = buffer;
+ this.range = range;
+ this.miss = miss;
+ this.owned = !miss;
+ }
+ }
+
+ /** {@code count} consecutive parts covering one contiguous missing sub-range; cached as a unit. */
+ private record MissRun(int firstPart, int count) {
+ }
+
+ /** One vectored range and the cache buffers it fills, in file order. */
+ private record Run(FileRange range, List<Part> parts) {
+ }
+
+ private Fetch startFetch(FSDataInputStream fileStream, ParquetRangeBuffers buffers, Allocator allocator,
+ int maxAlloc, int[] projected, BlockMetaData block, int rg)
+ throws IOException {
+ Fetch fetch = new Fetch();
+ ColumnChunkMetaData[] chunks = new ColumnChunkMetaData[projected.length];
+ for (int pc = 0; pc < projected.length; ++pc) {
+ chunks[pc] = block.getColumns().get(projected[pc]);
+ }
+ fetch.batch.init(fileKey, rg, chunks);
+ try {
+ for (ColumnChunkMetaData chunk : chunks) {
+ ColumnPlan column = new ColumnPlan();
+ fetch.columns.add(column);
+ planColumnChunk(allocator, maxAlloc, chunk.getStartingPos(),
+ chunk.getStartingPos() + chunk.getTotalSize(), column, fetch.misses);
+ }
+ requestMisses(fileStream, buffers, fetch, layout.maxRangeBytes());
+ } catch (Throwable t) {
+ abandon(allocator, fetch);
+ throw t;
+ }
+ return fetch;
+ }
+
+ private void finishFetch(Allocator allocator, ParquetRangeBuffers buffers, Fetch fetch)
+ throws IOException, InterruptedException {
+ try {
+ receiveMisses(buffers, fetch);
+ for (ColumnPlan column : fetch.columns) {
+ putColumn(allocator, column);
+ }
+ for (int pc = 0; pc < fetch.columns.size(); ++pc) {
+ assemble(fetch.batch, pc, fetch.columns.get(pc).parts);
+ }
+ // consumeData returns the batch on success; after a throw it is still ours.
+ consumer.consumeData(fetch.batch);
+ } catch (Throwable t) {
+ abandon(allocator, fetch);
+ throw t;
+ }
+ }
+
+ /** Drops a fetch that will not be decoded: outstanding requests are cancelled, buffers released. */
+ private void abandon(Allocator allocator, Fetch fetch) {
+ for (Run run : fetch.runs) {
+ run.range.getData().cancel(true);
+ }
+ for (ColumnPlan column : fetch.columns) {
+ for (Part part : column.parts) {
+ if (part.owned) {
+ bufferManager.decRefBuffer(part.buffer);
+ } else {
+ allocator.deallocate(part.buffer);
+ }
+ }
+ }
+ }
+
+ /** Package-private so a test can observe the read pattern. */
+ FSDataInputStream openFile(FileSystem fs) throws IOException {
+ return fs.open(path);
+ }
+
+ /** Lets the allocator abandon a wait for memory once the fragment is cancelled. */
+ private void allocateMultiple(Allocator allocator, MemoryBuffer[] dest, int size) {
+ if (allocator instanceof StoppableAllocator) {
+ ((StoppableAllocator) allocator).allocateMultiple(dest, size, DATA_BUFFER_FACTORY, isStopped);
+ } else {
+ allocator.allocateMultiple(dest, size, DATA_BUFFER_FACTORY);
+ }
+ }
+
+ /**
+ * Works out which buffers cover column chunk {@code [start, end)} without touching the file: cache
+ * hits as returned by getFileData, and freshly allocated power-of-two buffers for everything
+ * missing. Misses are appended to {@code allMisses} so the whole row group can be read at once.
+ */
+ private void planColumnChunk(Allocator allocator, int maxAlloc, long start, long end,
+ ColumnPlan column, List<Part> allMisses) throws IOException {
+ DiskRangeList head = new DiskRangeList(start, end);
+ if (fileKey != null) {
+ head = lowLevelCache.getFileData(fileKey, head, 0, ParquetCacheLayout.CACHE_CHUNK_FACTORY,
+ counters, new BooleanRef());
+ }
+ DiskRangeList current = head;
+ try {
+ for (; current != null; current = current.next) {
+ if (current.hasData()) {
+ column.parts.add(new Part(((CacheChunk) current).getBuffer(), current, false));
+ continue;
+ }
+ LlapHiveUtils.throwIfCacheOnlyRead(cacheOnly);
+ int[] sizes = layout.bufferSizes(current.getEnd() - current.getOffset());
+ column.missRuns.add(new MissRun(column.parts.size(), sizes.length));
+ long partFrom = current.getOffset();
+ for (int size : sizes) {
+ MemoryBuffer[] one = new MemoryBuffer[1];
+ allocateMultiple(allocator, one, size);
+ // The cache accounts and serves the bytes up to the buffer's limit.
+ ByteBuffer raw = one[0].getByteBufferRaw();
+ raw.limit(raw.position() + size);
+ Part part = new Part(one[0], new DiskRange(partFrom, partFrom + size), true);
+ column.parts.add(part);
+ allMisses.add(part);
+ partFrom += size;
+ }
+ }
+ } catch (Throwable t) {
+ // Hits past the failure point are still locked by getFileData; the caller only knows about
+ // the parts already recorded.
+ for (current = current.next; current != null; current = current.next) {
+ if (current.hasData()) {
+ bufferManager.decRefBuffer(((CacheChunk) current).getBuffer());
+ }
+ }
+ throw t;
+ }
+ }
+
+ /**
+ * Requests every missing buffer of a row group in one vectored call, one range per run of
+ * adjacent buffers. Column chunks sit back to back in the file, so a projection that keeps
+ * neighbouring columns reads them together; runs are capped so the row group arrives as
+ * several concurrent requests rather than one. A filesystem without a vectored implementation
+ * reads the ranges in turn.
+ */
+ private static void requestMisses(FSDataInputStream fileStream, ParquetRangeBuffers buffers, Fetch fetch,
+ int maxRange) throws IOException {
+ List<Part> misses = fetch.misses;
+ if (misses.isEmpty()) {
+ return;
+ }
+ misses.sort(Comparator.comparingLong(part -> part.range.getOffset()));
+ List<Run> runs = new ArrayList<>();
+ for (int i = 0; i < misses.size(); ) {
+ long from = misses.get(i).range.getOffset();
+ long to = misses.get(i).range.getEnd();
+ int j = i + 1;
+ for (; j < misses.size(); ++j) {
+ DiskRange next = misses.get(j).range;
+ if (next.getOffset() != to || next.getEnd() - from > maxRange) {
+ break;
+ }
+ to = next.getEnd();
+ }
+ runs.add(new Run(FileRange.createFileRange(from, (int) (to - from)), misses.subList(i, j)));
+ i = j;
+ }
+ List<FileRange> ranges = new ArrayList<>(runs.size());
+ for (Run run : runs) {
+ ranges.add(run.range);
+ }
+ fileStream.readVectored(ranges, buffers::allocate, buffers::release);
+ // Only requests the stream accepted are the fetch's to wait for or cancel.
+ fetch.runs.addAll(runs);
+ }
+
+ /** Waits for the requested runs and copies each into the cache buffers it covers. */
+ private static void receiveMisses(ParquetRangeBuffers buffers, Fetch fetch) throws IOException {
+ for (Run run : fetch.runs) {
+ ByteBuffer data = FutureIO.awaitFuture(run.range.getData());
+ for (Part part : run.parts) {
+ int length = part.range.getLength();
+ ByteBuffer src = data.duplicate();
+ src.position(data.position() + (int) (part.range.getOffset() - run.range.getOffset()));
+ src.limit(src.position() + length);
+ part.buffer.getByteBufferRaw().duplicate().put(src);
+ }
+ buffers.release(data);
+ }
+ }
+
+ /** Hands one column's freshly read buffers to the cache; every part then carries a ref we own. */
+ private void putColumn(Allocator allocator, ColumnPlan column) {
+ for (MissRun run : column.missRuns) {
+ List<Part> parts = column.parts.subList(run.firstPart, run.firstPart + run.count);
+ if (fileKey == null) {
+ for (Part part : parts) {
+ bufferManager.incRefBuffer(part.buffer);
+ }
+ } else {
+ MemoryBuffer[] fresh = new MemoryBuffer[run.count];
+ MemoryBuffer[] cached = new MemoryBuffer[run.count];
+ DiskRange[] ranges = new DiskRange[run.count];
+ for (int i = 0; i < run.count; ++i) {
+ fresh[i] = cached[i] = parts.get(i).buffer;
+ ranges[i] = parts.get(i).range;
+ }
+ lowLevelCache.putFileData(fileKey, ranges, cached, 0, Priority.NORMAL, counters, cacheTag);
+ for (int i = 0; i < run.count; ++i) {
+ if (cached[i] != fresh[i]) {
Review Comment:
ack, solved, and filed [HIVE-30065](https://issues.apache.org/jira/browse/HIVE-30065) for the same problem in ORC
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6793:
URL: https://github.com/apache/hive/pull/6793#issuecomment-5714468236
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6793) **Quality Gate passed**
Issues
 [69 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6793&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6793&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6793&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6793&metric=new_coverage&view=list)
 [0.2% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6793&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6793)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4194264362
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ParquetColumnVectorProducer.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.decode;
+
+import java.io.IOException;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.impl.ColumnVectorBatch;
+import org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl;
+import org.apache.hadoop.hive.llap.io.encoded.ParquetEncodedDataReader;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonCacheMetrics;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonIOMetrics;
+import org.apache.hadoop.hive.ql.io.orc.encoded.Consumer;
+import org.apache.hadoop.hive.ql.io.sarg.SearchArgument;
+import org.apache.hadoop.hive.ql.plan.PartitionDesc;
+import org.apache.hadoop.hive.serde2.Deserializer;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.InputFormat;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.Reporter;
+
+public class ParquetColumnVectorProducer implements ColumnVectorProducer {
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration conf;
+ private final LlapDaemonCacheMetrics cacheMetrics;
+ private final LlapDaemonIOMetrics ioMetrics;
+
+ public ParquetColumnVectorProducer(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration conf, LlapDaemonCacheMetrics cacheMetrics, LlapDaemonIOMetrics ioMetrics) {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.conf = conf;
+ this.cacheMetrics = cacheMetrics;
+ this.ioMetrics = ioMetrics;
+ }
+
+ @Override
+ public ReadPipeline createReadPipeline(Consumer<ColumnVectorBatch> consumer, FileSplit split,
+ Includes includes, SearchArgument sarg, QueryFragmentCounters counters,
+ SchemaEvolutionFactory sef, InputFormat<?, ?> sourceInputFormat, Deserializer sourceSerDe,
+ Reporter reporter, JobConf job, Map<Path, PartitionDesc> parts) {
+ try {
+ cacheMetrics.incrCacheReadRequests();
+ ParquetEncodedDataConsumer edc =
+ new ParquetEncodedDataConsumer(consumer, includes, counters, ioMetrics, job);
+ ParquetEncodedDataReader reader = new ParquetEncodedDataReader(
+ lowLevelCache, bufferManager, conf, job, split, edc, counters);
+ reader.loadFooter();
+ // Null makes LlapInputFormat fall back to the normal VectorizedParquetRecordReader.
+ if (reader.projectsNestedTypes()) {
+ LlapIoImpl.LOG.info("Parquet native cache: falling back to normal reader for {} due to "
+ + "nested types in the projection", split.getPath());
+ return null;
+ }
+ /*
+ * Fallback path still caches (footer via LlapProxy, data via LlapCacheAwareFs); what's lost
+ * for this split is vectored reads, row-group lookahead, and its LLAP IO summary counters.
+ * Guard fires per split only for Iceberg-Parquet files that physically carry _row_id /
+ * _last_updated_sequence_number while the query projects those virtual columns.
+ *
+ * Table shape Fires?
+ * ------------------------------------------------------------ ------
+ * Non-Iceberg Parquet never
+ * Iceberg-Parquet, row lineage disabled never
Review Comment:
we need some kind of guard here until HIVE-30103
TLDR: what I can do here is to improve the guard, remove the table, and add a unit test
this code is actually not reachable yet: Iceberg tables can project `ROW__LINEAGE__ID` /
`LAST__UPDATED__SEQUENCE__NUMBER`, and Iceberg's Parquet reads don't reach this producer, because `HiveVectorizedReader` only routes its ORC branch into LLAP IO
HIVE-30103 is the change that points Parquet here too
without the guard it fails silently rather than loudly: so when this pipeline serves a batch, the lineage column's value is never actually read from the file, it's left at whatever `initializeRowLineageColumns` put there, which is -1
downstream code (`HiveBatchIterator`) can't distinguish between the cases when "the native path never looked" and "this file was actually written before lineage tracking existed" so it treats -1 as the second case and creates a value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4156041397
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ // Fallback path: no cache key, so we read status + footer straight from HDFS. Time both
+ // under HDFS_TIME_NS to match how ORC accounts for cache-miss footer reads.
+ final FileSystem fs = path.getFileSystem(jobConf);
+ long hdfsStart = counters.startTimeCounter();
+ final FileStatus stat = fs.getFileStatus(path);
+ counters.recordHdfsTime(hdfsStart);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ hdfsStart = counters.startTimeCounter();
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ counters.recordHdfsTime(hdfsStart);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ long startTime = counters.startTimeCounter();
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Exception e) {
+ // A shutdown-triggered InterruptedException must not be silently reported as an ordinary
+ // consumer error: preserve the interrupt on the thread so any caller sees it.
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ }
+ consumer.setError(e);
+ } finally {
+ counters.incrWallClockCounter(LlapIOCounters.TOTAL_IO_TIME_NS, startTime);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ // The LLAP IO summary keys off TABLE / FILE / STRIPES; set them the same way ORC does so a
+ // native Parquet fragment shows up in the summary with the same fields populated.
+ if (cacheTag != null) {
+ counters.setDesc(QueryFragmentCounters.Desc.TABLE, cacheTag.getTableName());
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.FILE, path
+ + (fileKey == null ? "" : " (" + fileKey + ")"));
+
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = selectRowGroups(blocks, fileSchema);
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ // STRIPES is the ORC term; for Parquet the analog is row groups. Reuse the same descriptor so
+ // the summary layout stays common and we don't have to teach the reporter about a new field.
+ counters.setDesc(QueryFragmentCounters.Desc.STRIPES, "0," + selected.size());
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /**
+ * The row groups this split has to read: the ones whose first data page falls inside the split, as
+ * Parquet's own input format assigns them, minus any the pushed-down predicate rules out on its
+ * statistics. Returned in file order, since the fetch pipeline reads them in that order.
+ */
+ private List<BlockMetaData> selectRowGroups(List<BlockMetaData> blocks, MessageType fileSchema)
+ throws IOException {
+ long splitStart = split.getStart();
+ long splitEnd = splitStart + split.getLength();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().getFirst().getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate == null) {
+ return selected;
+ }
+ return RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Whether the query needs row-lineage virtual columns ({@code ROW__LINEAGE__ID} /
+ * {@code LAST__UPDATED__SEQUENCE__NUMBER}) that the fallback reader augments the requested schema
+ * with via {@link RowLineageUtils#getRequestedSchemaWithRowLineageColumns}. The native pipeline
+ * doesn't propagate that augmentation through {@code Includes} yet, so we defer to the fallback
+ * reader instead of quietly emitting nulls in the lineage slots.
+ */
+ public boolean needsRowLineage() {
+ VectorizedRowBatchCtx rbCtx = Utilities.getVectorizedRowBatchCtx(jobConf);
+ if (rbCtx == null) {
+ return false;
+ }
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ return RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema, VirtualColumn.ROW_LINEAGE_ID)
+ || RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema,
+ VirtualColumn.LAST_UPDATED_SEQUENCE_NUMBER);
+ }
+
+ /**
+ * Leaf-column positions of the requested fields; {@code BlockMetaData.getColumns()} is a flat list
+ * of leaves in file-schema order, so a top-level primitive at ordinal {@code k} in the file schema
+ * may sit at a very different leaf index (e.g. a nested group of two leaves before it shifts it
+ * from {@code 1} to {@code 2}). We only reach this method when every requested field is a
+ * top-level primitive ({@link #projectsNestedTypes()} would have forced a fallback otherwise), so
+ * each requested field maps to exactly one leaf whose path is a single segment.
+ */
+ static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
+ List<ColumnDescriptor> allLeaves = fileSchema.getColumns();
+ List<Integer> selected = new ArrayList<>();
+ for (Type field : requestedSchema.getFields()) {
+ if (!fileSchema.containsField(field.getName())) {
+ continue;
+ }
+ for (int i = 0; i < allLeaves.size(); ++i) {
+ String[] path = allLeaves.get(i).getPath();
+ // Single-segment path == top-level primitive; skip leaves buried inside a group.
+ if (path.length == 1 && path[0].equals(field.getName())) {
+ selected.add(i);
+ break;
+ }
+ }
+ }
+ return selected.stream().mapToInt(Integer::intValue).toArray();
+ }
+
+ private static long bytes(int[] projected, BlockMetaData block) {
+ long total = 0;
+ for (int leaf : projected) {
+ total += block.getColumns().get(leaf).getTotalSize();
+ }
+ return total;
+ }
+
+ private static long bytes(Fetch fetch) {
+ long total = 0;
+ for (ColumnChunkMetaData chunk : fetch.batch.chunks()) {
+ total += chunk.getTotalSize();
+ }
+ return total;
+ }
+
+ /** One row group on its way in: buffers planned, missing ranges requested, not yet decoded. */
+ private static final class Fetch {
+ private final ParquetEncodedColumnBatch batch = new ParquetEncodedColumnBatch();
+ private final List<ColumnPlan> columns = new ArrayList<>();
+ private final List<Part> misses = new ArrayList<>();
+ private final List<Run> runs = new ArrayList<>();
+ }
+
+ /** The buffers covering one column chunk, in file order, and which stretches of them are new. */
+ private record ColumnPlan(List<Part> parts, List<MissRun> missRuns) {
+ ColumnPlan() {
+ this(new ArrayList<>(), new ArrayList<>());
+ }
+ }
+
+ /** One cache buffer's worth of a column chunk: a hit handed back by the cache, or a miss to fill. */
+ private static final class Part {
+ private MemoryBuffer buffer;
+ private final DiskRange range;
+ /** We hold one ref to release; until then a miss is a raw allocation to free. */
+ private boolean owned;
+
+ Part(MemoryBuffer buffer, DiskRange range, boolean owned) {
+ this.buffer = buffer;
+ this.range = range;
+ this.owned = owned;
+ }
+ }
+
+ /** {@code count} consecutive parts covering one contiguous missing sub-range; cached as a unit. */
+ private record MissRun(int firstPart, int count) {
Review Comment:
Gap / planGap, Two different things are called "run": MissRun is a contiguous missing sub-range of one chunk (cache-side, cached as a unit), Run is one vectored read range that may span gaps of several adjacent columns
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4156642554
##########
ql/src/java/org/apache/hadoop/hive/llap/ParquetCacheLayout.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.llap;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.DataCache.DiskRangeListFactory;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+
+/**
+ * How Parquet column chunks are laid out in the LLAP cache and fetched from the file. Everything
+ * that caches a chunk goes through this, so a reader finding only part of one cached re-caches
+ * the gap on the boundaries the first reader used. A reader builds one from the daemon's allocator
+ * and configuration and uses it for every chunk, which is what keeps its layout consistent.
+ */
+public final class ParquetCacheLayout {
+
+ /**
+ * An object store fetches separate ranges concurrently but never splits one, so cutting buys
+ * parallelism while a large range amortises the request. Measured against S3 the knee is here:
+ * below 4 Mb throughput drops sharply, above 8 Mb it is flat. Trino also splits at 8 Mb.
+ */
+ private static final int MIN_RANGE_CAP_BYTES = 8 << 20;
+
+ public static final DiskRangeListFactory CACHE_CHUNK_FACTORY = CacheChunk::new;
+
+ private final int maxBuffer;
+ private final int minBuffer;
+ private final int maxRange;
+
+ /** The layout for a daemon: its allocator's maximum and the grain it caches non-ORC data at. */
+ public ParquetCacheLayout(Allocator allocator, Configuration conf) {
+ this(allocator.getMaxAllocation(), HiveConf.getSizeVar(conf, HiveConf.ConfVars.LLAP_IO_ENCODE_ALLOC_SIZE));
+ }
+
+ /**
+ * @param maxAlloc largest buffer the allocator hands out; rounded down to a power of two
+ * @param splitFloor size below which a remainder is left as one buffer rather than split further
+ */
+ public ParquetCacheLayout(int maxAlloc, long splitFloor) {
+ this.maxBuffer = Integer.highestOneBit(maxAlloc);
Review Comment:
i think we can drop the defensive caps
````
this.maxBuffer = maxAlloc;
this.minBuffer = (int) splitFloor;
this.rangeCap = Math.max(MIN_RANGE_CAP_BYTES, maxAlloc);
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4155551227
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java:
##########
@@ -261,7 +261,8 @@ public OrcEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager buff
}
consumer.setUseDecimal64ColumnVectors(HiveConf.getVar(jobConf,
- ConfVars.HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_ENABLED).equalsIgnoreCase("decimal_64"));
+ ConfVars.HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_ENABLED)
+ .equalsIgnoreCase(HiveConf.HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_DECIMAL_64));
Review Comment:
minor: wonder if we should/could move that helper INPUT_FORMAT_SUPPORTS_DECIMAL_64 into conf object
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4156049773
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ // Fallback path: no cache key, so we read status + footer straight from HDFS. Time both
+ // under HDFS_TIME_NS to match how ORC accounts for cache-miss footer reads.
+ final FileSystem fs = path.getFileSystem(jobConf);
+ long hdfsStart = counters.startTimeCounter();
+ final FileStatus stat = fs.getFileStatus(path);
+ counters.recordHdfsTime(hdfsStart);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ hdfsStart = counters.startTimeCounter();
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ counters.recordHdfsTime(hdfsStart);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ long startTime = counters.startTimeCounter();
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Exception e) {
+ // A shutdown-triggered InterruptedException must not be silently reported as an ordinary
+ // consumer error: preserve the interrupt on the thread so any caller sees it.
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ }
+ consumer.setError(e);
+ } finally {
+ counters.incrWallClockCounter(LlapIOCounters.TOTAL_IO_TIME_NS, startTime);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ // The LLAP IO summary keys off TABLE / FILE / STRIPES; set them the same way ORC does so a
+ // native Parquet fragment shows up in the summary with the same fields populated.
+ if (cacheTag != null) {
+ counters.setDesc(QueryFragmentCounters.Desc.TABLE, cacheTag.getTableName());
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.FILE, path
+ + (fileKey == null ? "" : " (" + fileKey + ")"));
+
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = selectRowGroups(blocks, fileSchema);
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ // STRIPES is the ORC term; for Parquet the analog is row groups. Reuse the same descriptor so
+ // the summary layout stays common and we don't have to teach the reporter about a new field.
+ counters.setDesc(QueryFragmentCounters.Desc.STRIPES, "0," + selected.size());
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /**
+ * The row groups this split has to read: the ones whose first data page falls inside the split, as
+ * Parquet's own input format assigns them, minus any the pushed-down predicate rules out on its
+ * statistics. Returned in file order, since the fetch pipeline reads them in that order.
+ */
+ private List<BlockMetaData> selectRowGroups(List<BlockMetaData> blocks, MessageType fileSchema)
+ throws IOException {
+ long splitStart = split.getStart();
+ long splitEnd = splitStart + split.getLength();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().getFirst().getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate == null) {
+ return selected;
+ }
+ return RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Whether the query needs row-lineage virtual columns ({@code ROW__LINEAGE__ID} /
+ * {@code LAST__UPDATED__SEQUENCE__NUMBER}) that the fallback reader augments the requested schema
+ * with via {@link RowLineageUtils#getRequestedSchemaWithRowLineageColumns}. The native pipeline
+ * doesn't propagate that augmentation through {@code Includes} yet, so we defer to the fallback
+ * reader instead of quietly emitting nulls in the lineage slots.
+ */
+ public boolean needsRowLineage() {
+ VectorizedRowBatchCtx rbCtx = Utilities.getVectorizedRowBatchCtx(jobConf);
+ if (rbCtx == null) {
+ return false;
+ }
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ return RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema, VirtualColumn.ROW_LINEAGE_ID)
+ || RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema,
+ VirtualColumn.LAST_UPDATED_SEQUENCE_NUMBER);
+ }
+
+ /**
+ * Leaf-column positions of the requested fields; {@code BlockMetaData.getColumns()} is a flat list
+ * of leaves in file-schema order, so a top-level primitive at ordinal {@code k} in the file schema
+ * may sit at a very different leaf index (e.g. a nested group of two leaves before it shifts it
+ * from {@code 1} to {@code 2}). We only reach this method when every requested field is a
+ * top-level primitive ({@link #projectsNestedTypes()} would have forced a fallback otherwise), so
+ * each requested field maps to exactly one leaf whose path is a single segment.
+ */
+ static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
+ List<ColumnDescriptor> allLeaves = fileSchema.getColumns();
+ List<Integer> selected = new ArrayList<>();
+ for (Type field : requestedSchema.getFields()) {
+ if (!fileSchema.containsField(field.getName())) {
+ continue;
+ }
+ for (int i = 0; i < allLeaves.size(); ++i) {
+ String[] path = allLeaves.get(i).getPath();
+ // Single-segment path == top-level primitive; skip leaves buried inside a group.
+ if (path.length == 1 && path[0].equals(field.getName())) {
+ selected.add(i);
+ break;
+ }
+ }
+ }
+ return selected.stream().mapToInt(Integer::intValue).toArray();
+ }
+
+ private static long bytes(int[] projected, BlockMetaData block) {
+ long total = 0;
+ for (int leaf : projected) {
+ total += block.getColumns().get(leaf).getTotalSize();
+ }
+ return total;
+ }
+
+ private static long bytes(Fetch fetch) {
+ long total = 0;
+ for (ColumnChunkMetaData chunk : fetch.batch.chunks()) {
+ total += chunk.getTotalSize();
+ }
+ return total;
+ }
+
+ /** One row group on its way in: buffers planned, missing ranges requested, not yet decoded. */
+ private static final class Fetch {
+ private final ParquetEncodedColumnBatch batch = new ParquetEncodedColumnBatch();
+ private final List<ColumnPlan> columns = new ArrayList<>();
+ private final List<Part> misses = new ArrayList<>();
+ private final List<Run> runs = new ArrayList<>();
+ }
+
+ /** The buffers covering one column chunk, in file order, and which stretches of them are new. */
+ private record ColumnPlan(List<Part> parts, List<MissRun> missRuns) {
+ ColumnPlan() {
+ this(new ArrayList<>(), new ArrayList<>());
+ }
+ }
+
+ /** One cache buffer's worth of a column chunk: a hit handed back by the cache, or a miss to fill. */
+ private static final class Part {
+ private MemoryBuffer buffer;
+ private final DiskRange range;
+ /** We hold one ref to release; until then a miss is a raw allocation to free. */
+ private boolean owned;
+
+ Part(MemoryBuffer buffer, DiskRange range, boolean owned) {
+ this.buffer = buffer;
+ this.range = range;
+ this.owned = owned;
+ }
+ }
+
+ /** {@code count} consecutive parts covering one contiguous missing sub-range; cached as a unit. */
+ private record MissRun(int firstPart, int count) {
+ }
+
+ /** One vectored range and the cache buffers it fills, in file order. */
+ private record Run(FileRange range, List<Part> parts) {
+ }
+
+ private Fetch startFetch(FSDataInputStream fileStream, ParquetRangeBuffers buffers, Allocator allocator,
+ int[] projected, BlockMetaData block, int rg)
+ throws IOException {
+ Fetch fetch = new Fetch();
+ ColumnChunkMetaData[] chunks = new ColumnChunkMetaData[projected.length];
+ for (int pc = 0; pc < projected.length; ++pc) {
+ chunks[pc] = block.getColumns().get(projected[pc]);
+ }
+ fetch.batch.init(fileKey, rg, chunks);
+ try {
+ for (ColumnChunkMetaData chunk : chunks) {
+ ColumnPlan column = new ColumnPlan();
+ fetch.columns.add(column);
+ planColumnChunk(allocator, chunk.getStartingPos(),
+ chunk.getStartingPos() + chunk.getTotalSize(), column, fetch.misses);
+ }
+ // The vectored dispatch is where non-async FS impls actually read; count it under HDFS_TIME.
+ long hdfsStart = counters.startTimeCounter();
+ requestMisses(fileStream, buffers, fetch, layout.maxRangeBytes());
+ counters.recordHdfsTime(hdfsStart);
+ } catch (Exception e) {
+ abandon(allocator, fetch);
+ throw e;
+ }
+ return fetch;
+ }
+
+ private void finishFetch(Allocator allocator, ParquetRangeBuffers buffers, Fetch fetch)
+ throws IOException, InterruptedException {
+ try {
+ // Awaiting the vectored futures is where the miss bytes actually arrive from HDFS.
+ long hdfsStart = counters.startTimeCounter();
+ receiveMisses(buffers, fetch);
+ counters.recordHdfsTime(hdfsStart);
+ for (ColumnPlan column : fetch.columns) {
+ putColumn(allocator, column);
+ }
+ for (int pc = 0; pc < fetch.columns.size(); ++pc) {
+ assemble(fetch.batch, pc, fetch.columns.get(pc).parts);
+ }
+ // consumeData returns the batch on success; after a throw it is still ours.
+ consumer.consumeData(fetch.batch);
+ } catch (Exception e) {
+ abandon(allocator, fetch);
+ throw e;
+ }
+ }
+
+ /** Drops a fetch that will not be decoded: outstanding requests are cancelled, buffers released. */
+ private void abandon(Allocator allocator, Fetch fetch) {
+ for (Run run : fetch.runs) {
+ run.range.getData().cancel(true);
+ }
+ for (ColumnPlan column : fetch.columns) {
+ for (Part part : column.parts) {
+ if (part.owned) {
+ bufferManager.decRefBuffer(part.buffer);
+ } else {
+ allocator.deallocate(part.buffer);
+ }
+ }
+ }
+ }
+
+ /** Package-private so a test can observe the read pattern. */
+ FSDataInputStream openFile(FileSystem fs) throws IOException {
+ return fs.open(path);
+ }
+
+ /** Lets the allocator abandon a wait for memory once the fragment is cancelled. */
+ private void allocateMultiple(Allocator allocator, MemoryBuffer[] dest, int size) {
+ if (allocator instanceof StoppableAllocator stoppable) {
+ stoppable.allocateMultiple(dest, size, DATA_BUFFER_FACTORY, isStopped);
+ } else {
+ allocator.allocateMultiple(dest, size, DATA_BUFFER_FACTORY);
+ }
+ }
+
+ /**
+ * Works out which buffers cover column chunk {@code [start, end)} without touching the file: cache
+ * hits as returned by getFileData, and freshly allocated power-of-two buffers for everything
+ * missing. Misses are appended to {@code allMisses} so the whole row group can be read at once.
+ */
+ private void planColumnChunk(Allocator allocator, long start, long end,
Review Comment:
planChunk?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4156009905
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ // Fallback path: no cache key, so we read status + footer straight from HDFS. Time both
+ // under HDFS_TIME_NS to match how ORC accounts for cache-miss footer reads.
+ final FileSystem fs = path.getFileSystem(jobConf);
+ long hdfsStart = counters.startTimeCounter();
+ final FileStatus stat = fs.getFileStatus(path);
+ counters.recordHdfsTime(hdfsStart);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ hdfsStart = counters.startTimeCounter();
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ counters.recordHdfsTime(hdfsStart);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ long startTime = counters.startTimeCounter();
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Exception e) {
+ // A shutdown-triggered InterruptedException must not be silently reported as an ordinary
+ // consumer error: preserve the interrupt on the thread so any caller sees it.
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ }
+ consumer.setError(e);
+ } finally {
+ counters.incrWallClockCounter(LlapIOCounters.TOTAL_IO_TIME_NS, startTime);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ // The LLAP IO summary keys off TABLE / FILE / STRIPES; set them the same way ORC does so a
+ // native Parquet fragment shows up in the summary with the same fields populated.
+ if (cacheTag != null) {
+ counters.setDesc(QueryFragmentCounters.Desc.TABLE, cacheTag.getTableName());
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.FILE, path
+ + (fileKey == null ? "" : " (" + fileKey + ")"));
+
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = selectRowGroups(blocks, fileSchema);
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ // STRIPES is the ORC term; for Parquet the analog is row groups. Reuse the same descriptor so
+ // the summary layout stays common and we don't have to teach the reporter about a new field.
+ counters.setDesc(QueryFragmentCounters.Desc.STRIPES, "0," + selected.size());
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /**
+ * The row groups this split has to read: the ones whose first data page falls inside the split, as
+ * Parquet's own input format assigns them, minus any the pushed-down predicate rules out on its
+ * statistics. Returned in file order, since the fetch pipeline reads them in that order.
+ */
+ private List<BlockMetaData> selectRowGroups(List<BlockMetaData> blocks, MessageType fileSchema)
+ throws IOException {
+ long splitStart = split.getStart();
+ long splitEnd = splitStart + split.getLength();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().getFirst().getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate == null) {
+ return selected;
+ }
+ return RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Whether the query needs row-lineage virtual columns ({@code ROW__LINEAGE__ID} /
+ * {@code LAST__UPDATED__SEQUENCE__NUMBER}) that the fallback reader augments the requested schema
+ * with via {@link RowLineageUtils#getRequestedSchemaWithRowLineageColumns}. The native pipeline
+ * doesn't propagate that augmentation through {@code Includes} yet, so we defer to the fallback
+ * reader instead of quietly emitting nulls in the lineage slots.
+ */
+ public boolean needsRowLineage() {
+ VectorizedRowBatchCtx rbCtx = Utilities.getVectorizedRowBatchCtx(jobConf);
+ if (rbCtx == null) {
+ return false;
+ }
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ return RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema, VirtualColumn.ROW_LINEAGE_ID)
+ || RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema,
+ VirtualColumn.LAST_UPDATED_SEQUENCE_NUMBER);
+ }
+
+ /**
+ * Leaf-column positions of the requested fields; {@code BlockMetaData.getColumns()} is a flat list
+ * of leaves in file-schema order, so a top-level primitive at ordinal {@code k} in the file schema
+ * may sit at a very different leaf index (e.g. a nested group of two leaves before it shifts it
+ * from {@code 1} to {@code 2}). We only reach this method when every requested field is a
+ * top-level primitive ({@link #projectsNestedTypes()} would have forced a fallback otherwise), so
+ * each requested field maps to exactly one leaf whose path is a single segment.
+ */
+ static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
+ List<ColumnDescriptor> allLeaves = fileSchema.getColumns();
+ List<Integer> selected = new ArrayList<>();
+ for (Type field : requestedSchema.getFields()) {
+ if (!fileSchema.containsField(field.getName())) {
+ continue;
+ }
+ for (int i = 0; i < allLeaves.size(); ++i) {
+ String[] path = allLeaves.get(i).getPath();
+ // Single-segment path == top-level primitive; skip leaves buried inside a group.
+ if (path.length == 1 && path[0].equals(field.getName())) {
+ selected.add(i);
+ break;
+ }
+ }
+ }
+ return selected.stream().mapToInt(Integer::intValue).toArray();
+ }
+
+ private static long bytes(int[] projected, BlockMetaData block) {
+ long total = 0;
+ for (int leaf : projected) {
+ total += block.getColumns().get(leaf).getTotalSize();
+ }
+ return total;
+ }
+
+ private static long bytes(Fetch fetch) {
Review Comment:
private static long projectedChunkBytes(Fetch fetch)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4155170769
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ParquetCachedPageReadStore.java:
##########
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.decode;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.llap.io.encoded.ParquetEncodedColumnBatch;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.DataPage;
+import org.apache.parquet.column.page.DataPageV1;
+import org.apache.parquet.column.page.DataPageV2;
+import org.apache.parquet.column.page.DictionaryPage;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.column.page.PageReader;
+import org.apache.parquet.compression.CompressionCodecFactory;
+import org.apache.parquet.compression.CompressionCodecFactory.BytesInputDecompressor;
+import org.apache.parquet.format.DataPageHeader;
+import org.apache.parquet.format.DataPageHeaderV2;
+import org.apache.parquet.format.DictionaryPageHeader;
+import org.apache.parquet.format.PageHeader;
+import org.apache.parquet.format.Util;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.schema.PrimitiveType;
+
+/**
+ * {@link PageReadStore} over one row group's cached column-chunk buffers, so that the vectorized
+ * column readers decode straight out of the LLAP cache: no ParquetFileReader, and no copy of the
+ * chunk bytes.
+ *
+ * <p><b>Why this is a copy.</b> Parquet does the same thing over a file, but none of it is
+ * reachable from here: {@code ParquetFileReader.Chunk.readAllPages} is private, the store it fills
+ * ({@code org.apache.parquet.hadoop.ColumnChunkPageReadStore}) is package-private and so is its
+ * {@code addColumn}, and the {@link PageReader} it puts in ({@code ColumnChunkPageReader}) is a
+ * package-private {@code final} class. Only the glue below is re-implemented; page header parsing,
+ * encoding and statistics conversion, the page value types and decompression are all parquet's own
+ * public API.
+ *
+ * <p><b>On a parquet upgrade.</b> Exactly two members track upstream code, each with a counterpart
+ * to diff against the new release:
+ * <ul>
+ * <li>{@link #readAllPages} mirrors {@code ParquetFileReader.Chunk.readAllPages}: the page-header
+ * walk, the page-type switch and the mapping of header fields onto {@link DataPageV1},
+ * {@link DataPageV2} and {@link DictionaryPage}. A new page type, a new header field or a
+ * changed page constructor lands here.</li>
+ * <li>{@link CachedChunkPageReader} mirrors {@code ColumnChunkPageReader}: page queueing and
+ * lazy decompression. New {@link PageReader} behaviour lands here.</li>
+ * </ul>
+ * Also check whether {@link PageReadStore} gained or changed a {@code default} method, since this
+ * class implements none of them ({@code getRowIndexOffset}, {@code getRowIndexes} and
+ * {@code close} are all inherited). {@code TestParquetCachedPageReadStore} compares this store page
+ * for page against the stock one and pins the parquet version, so the upgrade fails the build
+ * rather than passing with a divergence.
+ *
+ * <p>Deliberately unsupported, because LLAP's Parquet path does not use them: the offset index (so
+ * {@code getFirstRowIndex} and {@code getIndexRowCount} keep their no-offset-index defaults),
+ * column encryption, and page checksum verification.
+ *
+ * <p>How it works:
+ * <ol>
+ * <li><b>Input</b>: a {@link ParquetEncodedColumnBatch} holding the already-cached
+ * {@link MemoryBuffer}s for one row group's projected column chunks, plus the file footer.</li>
+ * <li><b>Stitching and parsing</b>: per chunk, {@link #chunkBuffers} turns the cache buffers into a
+ * list of {@link ByteBuffer} slices covering exactly the chunk's byte region; {@link #readAllPages}
+ * then walks that byte range with {@link Util#readPageHeader}, building {@link DataPageV1} /
+ * {@link DataPageV2} / {@link DictionaryPage} objects whose payloads are
+ * {@link ByteBufferInputStream} views over the cache buffers - the page bytes are never copied.</li>
+ * <li><b>Lazy decompression</b>: each {@link CachedChunkPageReader} keeps the raw bytes and its
+ * {@link BytesInputDecompressor}, and decompresses only when the vectorized column reader
+ * actually calls {@link PageReader#readPage()}, so pages that get pruned never pay the codec cost.</li>
+ * <li><b>Output</b>: the resulting {@code ColumnPath -> PageReader} map backs
+ * {@link #getPageReader(ColumnDescriptor)}, so parquet's VectorizedColumnReader never notices
+ * that it is not reading from a file.</li>
+ * </ol>
+ */
+class ParquetCachedPageReadStore implements PageReadStore {
+
+ private final Map<ColumnPath, PageReader> readers = new HashMap<>();
+ private final long rowCount;
+
+ ParquetCachedPageReadStore(ParquetMetadata footer, ParquetEncodedColumnBatch batch,
+ CompressionCodecFactory codecFactory, ParquetMetadataConverter converter) throws IOException {
+ BlockMetaData block = footer.getBlocks().get(batch.rowGroupIx());
+ this.rowCount = block.getRowCount();
+ String createdBy = footer.getFileMetaData().getCreatedBy();
+ ColumnChunkMetaData[] chunks = batch.chunks();
+ // pc is the projection index: chunks[pc] and the batch's three buffer arrays are all indexed by
+ // it, while the footer's block lists columns in file order. Keying on the path keeps the two
+ // apart, so a projection that drops columns from the middle of the schema still lines up.
+ for (int pc = 0; pc < chunks.length; ++pc) {
+ ColumnChunkMetaData chunk = chunks[pc];
+ readers.put(chunk.getPath(), readAllPages(chunk, chunkBuffers(batch, pc), createdBy,
+ codecFactory.getDecompressor(chunk.getCodec()), converter));
+ }
+ }
+
+ /**
+ * Slices of the cached buffers covering exactly the chunk's byte region, in file order.
+ *
+ * <p>This is the only place in the class that touches bytes, and the only place the cache appears
+ * at all: the {@link MemoryBuffer}s are already-populated LLAP cache memory, so there is no read
+ * here and nothing to decide - by the time this runs, a cache hit and a miss that
+ * {@code ParquetEncodedDataReader} had to fetch look exactly the same.
+ *
+ * <p>The buffers are cache ranges aligned to absolute file offsets, not to this chunk, so the
+ * first and last one usually overhang the chunk (the first may even start before it). That is what
+ * the trimming below is for.
+ *
+ * <p>The trimmed slices have to tile the chunk exactly, which is checked as they are built: a
+ * buffer that does not continue where the previous one ended, or one that reaches past the chunk,
+ * means the batch is not carrying the buffers the chunk was planned with. Without the check the
+ * page walk would read the wrong bytes as page bytes, and fail much later with a decoding error
+ * about a corrupt header.
+ */
+ private static List<ByteBuffer> chunkBuffers(ParquetEncodedColumnBatch batch, int pc)
+ throws IOException {
+ ColumnChunkMetaData chunk = batch.chunks()[pc];
+ long chunkStart = chunk.getStartingPos();
+ long chunkEnd = chunkStart + chunk.getTotalSize();
+ MemoryBuffer[] columnBuffers = batch.columnBuffers()[pc];
+ long[] bufferOffsets = batch.bufferOffsets()[pc];
+ int[] bufferLengths = batch.bufferLengths()[pc];
+ List<ByteBuffer> slices = new ArrayList<>(columnBuffers.length);
+ long covered = chunkStart;
+ for (int i = 0; i < columnBuffers.length; ++i) {
+ long bufferStart = bufferOffsets[i];
+ long bufferEnd = bufferStart + bufferLengths[i];
+ long sliceStart = Math.max(chunkStart, bufferStart);
+ long sliceEnd = Math.min(chunkEnd, bufferEnd);
+ if (sliceStart != covered || sliceEnd <= sliceStart) {
+ throw new IOException("Cached buffer " + (i + 1) + " of " + columnBuffers.length + " for column chunk "
+ + chunk.getPath() + " covers [" + sliceStart + ", " + sliceEnd + ") of the chunk at ["
+ + chunkStart + ", " + chunkEnd + "), which does not continue at " + covered);
+ }
+ covered = sliceEnd;
+ // A dup, because the cache buffer is shared: moving position/limit must not be visible to
+ // another reader of the same buffer. slice() then keeps a view of the trimmed region, so the
+ // chunk bytes are still the cache's bytes - nothing is copied here or below.
+ ByteBuffer bb = columnBuffers[i].getByteBufferDup();
+ bb.position(bb.position() + (int) (sliceStart - bufferStart));
+ bb.limit(bb.position() + (int) (sliceEnd - sliceStart));
+ slices.add(bb.slice());
+ }
+ if (covered != chunkEnd) {
+ throw new IOException("Cached buffers for column chunk " + chunk.getPath() + " cover ["
+ + chunkStart + ", " + covered + "), but the chunk ends at " + chunkEnd);
+ }
+ return slices;
+ }
+
+ private static PageReader readAllPages(ColumnChunkMetaData chunk, List<ByteBuffer> buffers,
+ String createdBy, BytesInputDecompressor decompressor, ParquetMetadataConverter converter)
+ throws IOException {
+ // Reads across the buffer boundaries as if the chunk were contiguous, which it is not: a page
+ // header or a payload can straddle two cache buffers. sliceBuffers below hands out views into
+ // these same buffers, so every page this method builds points at cache memory for as long as it
+ // lives - the batch has to keep its refs until the consumer is done decoding.
+ ByteBufferInputStream stream = ByteBufferInputStream.wrap(buffers);
+ PrimitiveType type = chunk.getPrimitiveType();
+ List<DataPage> pages = new ArrayList<>();
+ DictionaryPage dictionaryPage = null;
+ long valuesRead = 0;
+ // Only data pages count towards valuesRead, so the dictionary page and any page type skipped
+ // below do not end the walk early; the footer's value count is the only terminator.
+ while (valuesRead < chunk.getValueCount()) {
+ PageHeader header = Util.readPageHeader(stream);
+ int uncompressedSize = header.getUncompressed_page_size();
+ int compressedSize = header.getCompressed_page_size();
+ switch (header.getType()) {
+ case DICTIONARY_PAGE:
+ if (dictionaryPage != null) {
+ throw new ParquetDecodingException("more than one dictionary page in column " + chunk.getPath());
+ }
+ DictionaryPageHeader dictHeader = header.getDictionary_page_header();
+ dictionaryPage = new DictionaryPage(BytesInput.from(stream.sliceBuffers(compressedSize)),
+ uncompressedSize, dictHeader.getNum_values(), converter.getEncoding(dictHeader.getEncoding()));
+ break;
+ case DATA_PAGE:
+ DataPageHeader v1 = header.getData_page_header();
+ pages.add(new DataPageV1(BytesInput.from(stream.sliceBuffers(compressedSize)), v1.getNum_values(),
+ uncompressedSize, converter.fromParquetStatistics(createdBy, v1.getStatistics(), type),
+ converter.getEncoding(v1.getRepetition_level_encoding()),
+ converter.getEncoding(v1.getDefinition_level_encoding()),
+ converter.getEncoding(v1.getEncoding())));
+ valuesRead += v1.getNum_values();
+ break;
+ case DATA_PAGE_V2:
Review Comment:
do we have any tests for parquet.writer.version=v2, PARQUET_2_0 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4155203960
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ParquetCachedPageReadStore.java:
##########
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.decode;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.llap.io.encoded.ParquetEncodedColumnBatch;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.DataPage;
+import org.apache.parquet.column.page.DataPageV1;
+import org.apache.parquet.column.page.DataPageV2;
+import org.apache.parquet.column.page.DictionaryPage;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.column.page.PageReader;
+import org.apache.parquet.compression.CompressionCodecFactory;
+import org.apache.parquet.compression.CompressionCodecFactory.BytesInputDecompressor;
+import org.apache.parquet.format.DataPageHeader;
+import org.apache.parquet.format.DataPageHeaderV2;
+import org.apache.parquet.format.DictionaryPageHeader;
+import org.apache.parquet.format.PageHeader;
+import org.apache.parquet.format.Util;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.schema.PrimitiveType;
+
+/**
+ * {@link PageReadStore} over one row group's cached column-chunk buffers, so that the vectorized
+ * column readers decode straight out of the LLAP cache: no ParquetFileReader, and no copy of the
+ * chunk bytes.
+ *
+ * <p><b>Why this is a copy.</b> Parquet does the same thing over a file, but none of it is
+ * reachable from here: {@code ParquetFileReader.Chunk.readAllPages} is private, the store it fills
+ * ({@code org.apache.parquet.hadoop.ColumnChunkPageReadStore}) is package-private and so is its
+ * {@code addColumn}, and the {@link PageReader} it puts in ({@code ColumnChunkPageReader}) is a
+ * package-private {@code final} class. Only the glue below is re-implemented; page header parsing,
+ * encoding and statistics conversion, the page value types and decompression are all parquet's own
+ * public API.
+ *
+ * <p><b>On a parquet upgrade.</b> Exactly two members track upstream code, each with a counterpart
+ * to diff against the new release:
+ * <ul>
+ * <li>{@link #readAllPages} mirrors {@code ParquetFileReader.Chunk.readAllPages}: the page-header
+ * walk, the page-type switch and the mapping of header fields onto {@link DataPageV1},
+ * {@link DataPageV2} and {@link DictionaryPage}. A new page type, a new header field or a
+ * changed page constructor lands here.</li>
+ * <li>{@link CachedChunkPageReader} mirrors {@code ColumnChunkPageReader}: page queueing and
+ * lazy decompression. New {@link PageReader} behaviour lands here.</li>
+ * </ul>
+ * Also check whether {@link PageReadStore} gained or changed a {@code default} method, since this
+ * class implements none of them ({@code getRowIndexOffset}, {@code getRowIndexes} and
+ * {@code close} are all inherited). {@code TestParquetCachedPageReadStore} compares this store page
+ * for page against the stock one and pins the parquet version, so the upgrade fails the build
+ * rather than passing with a divergence.
+ *
+ * <p>Deliberately unsupported, because LLAP's Parquet path does not use them: the offset index (so
+ * {@code getFirstRowIndex} and {@code getIndexRowCount} keep their no-offset-index defaults),
+ * column encryption, and page checksum verification.
+ *
+ * <p>How it works:
+ * <ol>
+ * <li><b>Input</b>: a {@link ParquetEncodedColumnBatch} holding the already-cached
+ * {@link MemoryBuffer}s for one row group's projected column chunks, plus the file footer.</li>
+ * <li><b>Stitching and parsing</b>: per chunk, {@link #chunkBuffers} turns the cache buffers into a
+ * list of {@link ByteBuffer} slices covering exactly the chunk's byte region; {@link #readAllPages}
+ * then walks that byte range with {@link Util#readPageHeader}, building {@link DataPageV1} /
+ * {@link DataPageV2} / {@link DictionaryPage} objects whose payloads are
+ * {@link ByteBufferInputStream} views over the cache buffers - the page bytes are never copied.</li>
+ * <li><b>Lazy decompression</b>: each {@link CachedChunkPageReader} keeps the raw bytes and its
+ * {@link BytesInputDecompressor}, and decompresses only when the vectorized column reader
+ * actually calls {@link PageReader#readPage()}, so pages that get pruned never pay the codec cost.</li>
+ * <li><b>Output</b>: the resulting {@code ColumnPath -> PageReader} map backs
+ * {@link #getPageReader(ColumnDescriptor)}, so parquet's VectorizedColumnReader never notices
+ * that it is not reading from a file.</li>
+ * </ol>
+ */
+class ParquetCachedPageReadStore implements PageReadStore {
+
+ private final Map<ColumnPath, PageReader> readers = new HashMap<>();
+ private final long rowCount;
+
+ ParquetCachedPageReadStore(ParquetMetadata footer, ParquetEncodedColumnBatch batch,
+ CompressionCodecFactory codecFactory, ParquetMetadataConverter converter) throws IOException {
+ BlockMetaData block = footer.getBlocks().get(batch.rowGroupIx());
+ this.rowCount = block.getRowCount();
+ String createdBy = footer.getFileMetaData().getCreatedBy();
+ ColumnChunkMetaData[] chunks = batch.chunks();
+ // pc is the projection index: chunks[pc] and the batch's three buffer arrays are all indexed by
+ // it, while the footer's block lists columns in file order. Keying on the path keeps the two
+ // apart, so a projection that drops columns from the middle of the schema still lines up.
+ for (int pc = 0; pc < chunks.length; ++pc) {
+ ColumnChunkMetaData chunk = chunks[pc];
+ readers.put(chunk.getPath(), readAllPages(chunk, chunkBuffers(batch, pc), createdBy,
+ codecFactory.getDecompressor(chunk.getCodec()), converter));
+ }
+ }
+
+ /**
+ * Slices of the cached buffers covering exactly the chunk's byte region, in file order.
+ *
+ * <p>This is the only place in the class that touches bytes, and the only place the cache appears
+ * at all: the {@link MemoryBuffer}s are already-populated LLAP cache memory, so there is no read
+ * here and nothing to decide - by the time this runs, a cache hit and a miss that
+ * {@code ParquetEncodedDataReader} had to fetch look exactly the same.
+ *
+ * <p>The buffers are cache ranges aligned to absolute file offsets, not to this chunk, so the
+ * first and last one usually overhang the chunk (the first may even start before it). That is what
+ * the trimming below is for.
+ *
+ * <p>The trimmed slices have to tile the chunk exactly, which is checked as they are built: a
+ * buffer that does not continue where the previous one ended, or one that reaches past the chunk,
+ * means the batch is not carrying the buffers the chunk was planned with. Without the check the
+ * page walk would read the wrong bytes as page bytes, and fail much later with a decoding error
+ * about a corrupt header.
+ */
+ private static List<ByteBuffer> chunkBuffers(ParquetEncodedColumnBatch batch, int pc)
+ throws IOException {
+ ColumnChunkMetaData chunk = batch.chunks()[pc];
+ long chunkStart = chunk.getStartingPos();
+ long chunkEnd = chunkStart + chunk.getTotalSize();
+ MemoryBuffer[] columnBuffers = batch.columnBuffers()[pc];
+ long[] bufferOffsets = batch.bufferOffsets()[pc];
+ int[] bufferLengths = batch.bufferLengths()[pc];
+ List<ByteBuffer> slices = new ArrayList<>(columnBuffers.length);
+ long covered = chunkStart;
+ for (int i = 0; i < columnBuffers.length; ++i) {
+ long bufferStart = bufferOffsets[i];
+ long bufferEnd = bufferStart + bufferLengths[i];
+ long sliceStart = Math.max(chunkStart, bufferStart);
+ long sliceEnd = Math.min(chunkEnd, bufferEnd);
+ if (sliceStart != covered || sliceEnd <= sliceStart) {
+ throw new IOException("Cached buffer " + (i + 1) + " of " + columnBuffers.length + " for column chunk "
+ + chunk.getPath() + " covers [" + sliceStart + ", " + sliceEnd + ") of the chunk at ["
+ + chunkStart + ", " + chunkEnd + "), which does not continue at " + covered);
+ }
+ covered = sliceEnd;
+ // A dup, because the cache buffer is shared: moving position/limit must not be visible to
+ // another reader of the same buffer. slice() then keeps a view of the trimmed region, so the
+ // chunk bytes are still the cache's bytes - nothing is copied here or below.
+ ByteBuffer bb = columnBuffers[i].getByteBufferDup();
+ bb.position(bb.position() + (int) (sliceStart - bufferStart));
+ bb.limit(bb.position() + (int) (sliceEnd - sliceStart));
+ slices.add(bb.slice());
+ }
+ if (covered != chunkEnd) {
+ throw new IOException("Cached buffers for column chunk " + chunk.getPath() + " cover ["
+ + chunkStart + ", " + covered + "), but the chunk ends at " + chunkEnd);
+ }
+ return slices;
+ }
+
+ private static PageReader readAllPages(ColumnChunkMetaData chunk, List<ByteBuffer> buffers,
+ String createdBy, BytesInputDecompressor decompressor, ParquetMetadataConverter converter)
+ throws IOException {
+ // Reads across the buffer boundaries as if the chunk were contiguous, which it is not: a page
+ // header or a payload can straddle two cache buffers. sliceBuffers below hands out views into
+ // these same buffers, so every page this method builds points at cache memory for as long as it
+ // lives - the batch has to keep its refs until the consumer is done decoding.
+ ByteBufferInputStream stream = ByteBufferInputStream.wrap(buffers);
+ PrimitiveType type = chunk.getPrimitiveType();
+ List<DataPage> pages = new ArrayList<>();
+ DictionaryPage dictionaryPage = null;
+ long valuesRead = 0;
+ // Only data pages count towards valuesRead, so the dictionary page and any page type skipped
+ // below do not end the walk early; the footer's value count is the only terminator.
+ while (valuesRead < chunk.getValueCount()) {
+ PageHeader header = Util.readPageHeader(stream);
+ int uncompressedSize = header.getUncompressed_page_size();
+ int compressedSize = header.getCompressed_page_size();
+ switch (header.getType()) {
+ case DICTIONARY_PAGE:
+ if (dictionaryPage != null) {
+ throw new ParquetDecodingException("more than one dictionary page in column " + chunk.getPath());
+ }
+ DictionaryPageHeader dictHeader = header.getDictionary_page_header();
+ dictionaryPage = new DictionaryPage(BytesInput.from(stream.sliceBuffers(compressedSize)),
+ uncompressedSize, dictHeader.getNum_values(), converter.getEncoding(dictHeader.getEncoding()));
+ break;
+ case DATA_PAGE:
+ DataPageHeader v1 = header.getData_page_header();
+ pages.add(new DataPageV1(BytesInput.from(stream.sliceBuffers(compressedSize)), v1.getNum_values(),
+ uncompressedSize, converter.fromParquetStatistics(createdBy, v1.getStatistics(), type),
+ converter.getEncoding(v1.getRepetition_level_encoding()),
+ converter.getEncoding(v1.getDefinition_level_encoding()),
+ converter.getEncoding(v1.getEncoding())));
+ valuesRead += v1.getNum_values();
+ break;
+ case DATA_PAGE_V2:
+ DataPageHeaderV2 v2 = header.getData_page_header_v2();
+ // In V2 the level bytes sit inside the page but are never compressed, so only what is left
+ // after them is codec output. The three slices have to be taken in this order.
+ int dataSize = compressedSize
+ - v2.getRepetition_levels_byte_length() - v2.getDefinition_levels_byte_length();
+ BytesInput repetitionLevels = BytesInput.from(stream.sliceBuffers(v2.getRepetition_levels_byte_length()));
+ BytesInput definitionLevels = BytesInput.from(stream.sliceBuffers(v2.getDefinition_levels_byte_length()));
+ BytesInput values = BytesInput.from(stream.sliceBuffers(dataSize));
+ pages.add(new DataPageV2(v2.getNum_rows(), v2.getNum_nulls(), v2.getNum_values(),
+ repetitionLevels, definitionLevels, converter.getEncoding(v2.getEncoding()), values,
+ uncompressedSize, converter.fromParquetStatistics(createdBy, v2.getStatistics(), type),
+ v2.isIs_compressed()));
+ valuesRead += v2.getNum_values();
+ break;
+ default:
+ // A page type this parquet release knows and we do not (an index page, say). Skipping it
+ // by its compressed size keeps the walk aligned on the next header, as parquet does.
+ stream.skipFully(compressedSize);
+ }
+ }
+ if (valuesRead != chunk.getValueCount()) {
+ throw new IOException("Expected " + chunk.getValueCount() + " values in column chunk " + chunk.getPath()
+ + " at offset " + chunk.getStartingPos() + " but got " + valuesRead + " over " + pages.size() + " pages");
+ }
+ return new CachedChunkPageReader(decompressor, pages, dictionaryPage);
+ }
+
+ @Override
+ public long getRowCount() {
+ return rowCount;
+ }
+
+ @Override
+ public PageReader getPageReader(ColumnDescriptor descriptor) {
+ return readers.get(ColumnPath.get(descriptor.getPath()));
+ }
+
+ private static final class CachedChunkPageReader implements PageReader {
+ private final BytesInputDecompressor decompressor;
+ private final Queue<DataPage> compressedPages;
+ private final DictionaryPage compressedDictionaryPage;
+ private final long valueCount;
+
+ CachedChunkPageReader(BytesInputDecompressor decompressor, List<DataPage> compressedPages,
+ DictionaryPage compressedDictionaryPage) {
+ this.decompressor = decompressor;
+ this.compressedPages = new ArrayDeque<>(compressedPages);
+ this.compressedDictionaryPage = compressedDictionaryPage;
+ long count = 0;
+ for (DataPage p : compressedPages) {
+ count += p.getValueCount();
+ }
+ this.valueCount = count;
+ }
+
+ @Override
+ public long getTotalValueCount() {
+ return valueCount;
+ }
+
+ /**
+ * Decompresses one page, on demand. This is where the cached bytes stop being the cache's: the
+ * decompressed copy is an ordinary heap/direct buffer that the column reader owns and drops after
+ * the batch. Pages the reader never asks for are never decompressed, so a pruned page costs
+ * nothing beyond the header parse.
+ */
+ @Override
+ public DataPage readPage() {
+ DataPage compressedPage = compressedPages.poll();
+ if (compressedPage == null) {
+ return null;
+ }
+ return compressedPage.accept(new DataPage.Visitor<DataPage>() {
+ @Override
+ public DataPage visit(DataPageV1 page) {
+ try {
+ return new DataPageV1(decompressor.decompress(page.getBytes(), page.getUncompressedSize()),
+ page.getValueCount(), page.getUncompressedSize(), page.getStatistics(),
+ page.getRlEncoding(), page.getDlEncoding(), page.getValueEncoding());
+ } catch (IOException e) {
+ throw new ParquetDecodingException("could not decompress page", e);
+ }
+ }
+
+ @Override
+ public DataPage visit(DataPageV2 page) {
+ if (!page.isCompressed()) {
+ // Handed straight through, still a view over the cache buffers. Safe for the same reason
+ // the whole class is: the batch holds the refs until the consumer finishes the row group.
+ return page;
+ }
+ // getUncompressedSize() covers the levels, which were not compressed, so the codec's own
+ // output size is what is left after them.
+ int uncompressedSize = Math.toIntExact(page.getUncompressedSize()
+ - page.getDefinitionLevels().size() - page.getRepetitionLevels().size());
+ BytesInput data;
+ try {
+ data = decompressor.decompress(page.getData(), uncompressedSize);
Review Comment:
minor: should we extract in readPage and readDictionaryPage ?
````
private BytesInput decompress(BytesInput bytes, int uncompressedSize, String what) {
try {
return decompressor.decompress(bytes, uncompressedSize);
} catch (IOException e) {
throw new ParquetDecodingException("could not decompress " + what, e);
}
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4155405756
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ParquetColumnVectorProducer.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.decode;
+
+import java.io.IOException;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.impl.ColumnVectorBatch;
+import org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl;
+import org.apache.hadoop.hive.llap.io.encoded.ParquetEncodedDataReader;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonCacheMetrics;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonIOMetrics;
+import org.apache.hadoop.hive.ql.io.orc.encoded.Consumer;
+import org.apache.hadoop.hive.ql.io.sarg.SearchArgument;
+import org.apache.hadoop.hive.ql.plan.PartitionDesc;
+import org.apache.hadoop.hive.serde2.Deserializer;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.InputFormat;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.Reporter;
+
+public class ParquetColumnVectorProducer implements ColumnVectorProducer {
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration conf;
+ private final LlapDaemonCacheMetrics cacheMetrics;
+ private final LlapDaemonIOMetrics ioMetrics;
+
+ public ParquetColumnVectorProducer(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration conf, LlapDaemonCacheMetrics cacheMetrics, LlapDaemonIOMetrics ioMetrics) {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.conf = conf;
+ this.cacheMetrics = cacheMetrics;
+ this.ioMetrics = ioMetrics;
+ }
+
+ @Override
+ public ReadPipeline createReadPipeline(Consumer<ColumnVectorBatch> consumer, FileSplit split,
+ Includes includes, SearchArgument sarg, QueryFragmentCounters counters,
+ SchemaEvolutionFactory sef, InputFormat<?, ?> sourceInputFormat, Deserializer sourceSerDe,
+ Reporter reporter, JobConf job, Map<Path, PartitionDesc> parts) {
+ try {
+ cacheMetrics.incrCacheReadRequests();
+ ParquetEncodedDataConsumer edc =
+ new ParquetEncodedDataConsumer(consumer, includes, counters, ioMetrics, job);
+ ParquetEncodedDataReader reader = new ParquetEncodedDataReader(
+ lowLevelCache, bufferManager, conf, job, split, edc, counters);
+ reader.loadFooter();
+ // Null makes LlapInputFormat fall back to the normal VectorizedParquetRecordReader.
+ if (reader.projectsNestedTypes()) {
+ LlapIoImpl.LOG.info("Parquet native cache: falling back to normal reader for {} due to "
+ + "nested types in the projection", split.getPath());
+ return null;
+ }
+ /*
+ * Fallback path still caches (footer via LlapProxy, data via LlapCacheAwareFs); what's lost
+ * for this split is vectored reads, row-group lookahead, and its LLAP IO summary counters.
+ * Guard fires per split only for Iceberg-Parquet files that physically carry _row_id /
+ * _last_updated_sequence_number while the query projects those virtual columns.
+ *
+ * Table shape Fires?
+ * ------------------------------------------------------------ ------
+ * Non-Iceberg Parquet never
+ * Iceberg-Parquet, row lineage disabled never
+ * Iceberg-Parquet, row lineage on, file has no _row_id yet never
+ * Iceberg-Parquet, row lineage on, file has lineage columns yes, per split
+ */
+ if (reader.needsRowLineage()) {
+ LlapIoImpl.LOG.info("Parquet native cache: falling back to normal reader for {} due to "
+ + "row-lineage virtual columns", split.getPath());
+ return null;
+ }
+ edc.init(reader, reader);
Review Comment:
preexisting, but messy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4046584677
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -4958,6 +4958,9 @@ public static enum ConfVars {
LLAP_IO_ENCODE_ENABLED("hive.llap.io.encode.enabled", true,
"Whether LLAP should try to re-encode and cache data for non-ORC formats. This is used\n" +
"on LLAP Server side to determine if the infrastructure for that is initialized."),
+ LLAP_IO_PARQUET_NATIVE_ENABLED("hive.llap.io.parquet.native.enabled", false,
Review Comment:
ack, feature is going to be present in the code anyway
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6793:
URL: https://github.com/apache/hive/pull/6793#issuecomment-5727443370
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6793) **Quality Gate passed**
Issues
 [69 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6793&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6793&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6793&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6793&metric=new_coverage&view=list)
 [0.2% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6793&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6793)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4044609073
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -4958,6 +4958,9 @@ public static enum ConfVars {
LLAP_IO_ENCODE_ENABLED("hive.llap.io.encode.enabled", true,
"Whether LLAP should try to re-encode and cache data for non-ORC formats. This is used\n" +
"on LLAP Server side to determine if the infrastructure for that is initialized."),
+ LLAP_IO_PARQUET_NATIVE_ENABLED("hive.llap.io.parquet.native.enabled", false,
Review Comment:
it could be, tpcds 10TB testing is already done, we might be confident enough to make it true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4155170769
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ParquetCachedPageReadStore.java:
##########
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.decode;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.llap.io.encoded.ParquetEncodedColumnBatch;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.DataPage;
+import org.apache.parquet.column.page.DataPageV1;
+import org.apache.parquet.column.page.DataPageV2;
+import org.apache.parquet.column.page.DictionaryPage;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.column.page.PageReader;
+import org.apache.parquet.compression.CompressionCodecFactory;
+import org.apache.parquet.compression.CompressionCodecFactory.BytesInputDecompressor;
+import org.apache.parquet.format.DataPageHeader;
+import org.apache.parquet.format.DataPageHeaderV2;
+import org.apache.parquet.format.DictionaryPageHeader;
+import org.apache.parquet.format.PageHeader;
+import org.apache.parquet.format.Util;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.schema.PrimitiveType;
+
+/**
+ * {@link PageReadStore} over one row group's cached column-chunk buffers, so that the vectorized
+ * column readers decode straight out of the LLAP cache: no ParquetFileReader, and no copy of the
+ * chunk bytes.
+ *
+ * <p><b>Why this is a copy.</b> Parquet does the same thing over a file, but none of it is
+ * reachable from here: {@code ParquetFileReader.Chunk.readAllPages} is private, the store it fills
+ * ({@code org.apache.parquet.hadoop.ColumnChunkPageReadStore}) is package-private and so is its
+ * {@code addColumn}, and the {@link PageReader} it puts in ({@code ColumnChunkPageReader}) is a
+ * package-private {@code final} class. Only the glue below is re-implemented; page header parsing,
+ * encoding and statistics conversion, the page value types and decompression are all parquet's own
+ * public API.
+ *
+ * <p><b>On a parquet upgrade.</b> Exactly two members track upstream code, each with a counterpart
+ * to diff against the new release:
+ * <ul>
+ * <li>{@link #readAllPages} mirrors {@code ParquetFileReader.Chunk.readAllPages}: the page-header
+ * walk, the page-type switch and the mapping of header fields onto {@link DataPageV1},
+ * {@link DataPageV2} and {@link DictionaryPage}. A new page type, a new header field or a
+ * changed page constructor lands here.</li>
+ * <li>{@link CachedChunkPageReader} mirrors {@code ColumnChunkPageReader}: page queueing and
+ * lazy decompression. New {@link PageReader} behaviour lands here.</li>
+ * </ul>
+ * Also check whether {@link PageReadStore} gained or changed a {@code default} method, since this
+ * class implements none of them ({@code getRowIndexOffset}, {@code getRowIndexes} and
+ * {@code close} are all inherited). {@code TestParquetCachedPageReadStore} compares this store page
+ * for page against the stock one and pins the parquet version, so the upgrade fails the build
+ * rather than passing with a divergence.
+ *
+ * <p>Deliberately unsupported, because LLAP's Parquet path does not use them: the offset index (so
+ * {@code getFirstRowIndex} and {@code getIndexRowCount} keep their no-offset-index defaults),
+ * column encryption, and page checksum verification.
+ *
+ * <p>How it works:
+ * <ol>
+ * <li><b>Input</b>: a {@link ParquetEncodedColumnBatch} holding the already-cached
+ * {@link MemoryBuffer}s for one row group's projected column chunks, plus the file footer.</li>
+ * <li><b>Stitching and parsing</b>: per chunk, {@link #chunkBuffers} turns the cache buffers into a
+ * list of {@link ByteBuffer} slices covering exactly the chunk's byte region; {@link #readAllPages}
+ * then walks that byte range with {@link Util#readPageHeader}, building {@link DataPageV1} /
+ * {@link DataPageV2} / {@link DictionaryPage} objects whose payloads are
+ * {@link ByteBufferInputStream} views over the cache buffers - the page bytes are never copied.</li>
+ * <li><b>Lazy decompression</b>: each {@link CachedChunkPageReader} keeps the raw bytes and its
+ * {@link BytesInputDecompressor}, and decompresses only when the vectorized column reader
+ * actually calls {@link PageReader#readPage()}, so pages that get pruned never pay the codec cost.</li>
+ * <li><b>Output</b>: the resulting {@code ColumnPath -> PageReader} map backs
+ * {@link #getPageReader(ColumnDescriptor)}, so parquet's VectorizedColumnReader never notices
+ * that it is not reading from a file.</li>
+ * </ol>
+ */
+class ParquetCachedPageReadStore implements PageReadStore {
+
+ private final Map<ColumnPath, PageReader> readers = new HashMap<>();
+ private final long rowCount;
+
+ ParquetCachedPageReadStore(ParquetMetadata footer, ParquetEncodedColumnBatch batch,
+ CompressionCodecFactory codecFactory, ParquetMetadataConverter converter) throws IOException {
+ BlockMetaData block = footer.getBlocks().get(batch.rowGroupIx());
+ this.rowCount = block.getRowCount();
+ String createdBy = footer.getFileMetaData().getCreatedBy();
+ ColumnChunkMetaData[] chunks = batch.chunks();
+ // pc is the projection index: chunks[pc] and the batch's three buffer arrays are all indexed by
+ // it, while the footer's block lists columns in file order. Keying on the path keeps the two
+ // apart, so a projection that drops columns from the middle of the schema still lines up.
+ for (int pc = 0; pc < chunks.length; ++pc) {
+ ColumnChunkMetaData chunk = chunks[pc];
+ readers.put(chunk.getPath(), readAllPages(chunk, chunkBuffers(batch, pc), createdBy,
+ codecFactory.getDecompressor(chunk.getCodec()), converter));
+ }
+ }
+
+ /**
+ * Slices of the cached buffers covering exactly the chunk's byte region, in file order.
+ *
+ * <p>This is the only place in the class that touches bytes, and the only place the cache appears
+ * at all: the {@link MemoryBuffer}s are already-populated LLAP cache memory, so there is no read
+ * here and nothing to decide - by the time this runs, a cache hit and a miss that
+ * {@code ParquetEncodedDataReader} had to fetch look exactly the same.
+ *
+ * <p>The buffers are cache ranges aligned to absolute file offsets, not to this chunk, so the
+ * first and last one usually overhang the chunk (the first may even start before it). That is what
+ * the trimming below is for.
+ *
+ * <p>The trimmed slices have to tile the chunk exactly, which is checked as they are built: a
+ * buffer that does not continue where the previous one ended, or one that reaches past the chunk,
+ * means the batch is not carrying the buffers the chunk was planned with. Without the check the
+ * page walk would read the wrong bytes as page bytes, and fail much later with a decoding error
+ * about a corrupt header.
+ */
+ private static List<ByteBuffer> chunkBuffers(ParquetEncodedColumnBatch batch, int pc)
+ throws IOException {
+ ColumnChunkMetaData chunk = batch.chunks()[pc];
+ long chunkStart = chunk.getStartingPos();
+ long chunkEnd = chunkStart + chunk.getTotalSize();
+ MemoryBuffer[] columnBuffers = batch.columnBuffers()[pc];
+ long[] bufferOffsets = batch.bufferOffsets()[pc];
+ int[] bufferLengths = batch.bufferLengths()[pc];
+ List<ByteBuffer> slices = new ArrayList<>(columnBuffers.length);
+ long covered = chunkStart;
+ for (int i = 0; i < columnBuffers.length; ++i) {
+ long bufferStart = bufferOffsets[i];
+ long bufferEnd = bufferStart + bufferLengths[i];
+ long sliceStart = Math.max(chunkStart, bufferStart);
+ long sliceEnd = Math.min(chunkEnd, bufferEnd);
+ if (sliceStart != covered || sliceEnd <= sliceStart) {
+ throw new IOException("Cached buffer " + (i + 1) + " of " + columnBuffers.length + " for column chunk "
+ + chunk.getPath() + " covers [" + sliceStart + ", " + sliceEnd + ") of the chunk at ["
+ + chunkStart + ", " + chunkEnd + "), which does not continue at " + covered);
+ }
+ covered = sliceEnd;
+ // A dup, because the cache buffer is shared: moving position/limit must not be visible to
+ // another reader of the same buffer. slice() then keeps a view of the trimmed region, so the
+ // chunk bytes are still the cache's bytes - nothing is copied here or below.
+ ByteBuffer bb = columnBuffers[i].getByteBufferDup();
+ bb.position(bb.position() + (int) (sliceStart - bufferStart));
+ bb.limit(bb.position() + (int) (sliceEnd - sliceStart));
+ slices.add(bb.slice());
+ }
+ if (covered != chunkEnd) {
+ throw new IOException("Cached buffers for column chunk " + chunk.getPath() + " cover ["
+ + chunkStart + ", " + covered + "), but the chunk ends at " + chunkEnd);
+ }
+ return slices;
+ }
+
+ private static PageReader readAllPages(ColumnChunkMetaData chunk, List<ByteBuffer> buffers,
+ String createdBy, BytesInputDecompressor decompressor, ParquetMetadataConverter converter)
+ throws IOException {
+ // Reads across the buffer boundaries as if the chunk were contiguous, which it is not: a page
+ // header or a payload can straddle two cache buffers. sliceBuffers below hands out views into
+ // these same buffers, so every page this method builds points at cache memory for as long as it
+ // lives - the batch has to keep its refs until the consumer is done decoding.
+ ByteBufferInputStream stream = ByteBufferInputStream.wrap(buffers);
+ PrimitiveType type = chunk.getPrimitiveType();
+ List<DataPage> pages = new ArrayList<>();
+ DictionaryPage dictionaryPage = null;
+ long valuesRead = 0;
+ // Only data pages count towards valuesRead, so the dictionary page and any page type skipped
+ // below do not end the walk early; the footer's value count is the only terminator.
+ while (valuesRead < chunk.getValueCount()) {
+ PageHeader header = Util.readPageHeader(stream);
+ int uncompressedSize = header.getUncompressed_page_size();
+ int compressedSize = header.getCompressed_page_size();
+ switch (header.getType()) {
+ case DICTIONARY_PAGE:
+ if (dictionaryPage != null) {
+ throw new ParquetDecodingException("more than one dictionary page in column " + chunk.getPath());
+ }
+ DictionaryPageHeader dictHeader = header.getDictionary_page_header();
+ dictionaryPage = new DictionaryPage(BytesInput.from(stream.sliceBuffers(compressedSize)),
+ uncompressedSize, dictHeader.getNum_values(), converter.getEncoding(dictHeader.getEncoding()));
+ break;
+ case DATA_PAGE:
+ DataPageHeader v1 = header.getData_page_header();
+ pages.add(new DataPageV1(BytesInput.from(stream.sliceBuffers(compressedSize)), v1.getNum_values(),
+ uncompressedSize, converter.fromParquetStatistics(createdBy, v1.getStatistics(), type),
+ converter.getEncoding(v1.getRepetition_level_encoding()),
+ converter.getEncoding(v1.getDefinition_level_encoding()),
+ converter.getEncoding(v1.getEncoding())));
+ valuesRead += v1.getNum_values();
+ break;
+ case DATA_PAGE_V2:
Review Comment:
do we have any tests for parquet.writer.version=v2, PARQUET_2_0 ?
TestParquetCachedPageReadStore?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4155614846
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ // Fallback path: no cache key, so we read status + footer straight from HDFS. Time both
+ // under HDFS_TIME_NS to match how ORC accounts for cache-miss footer reads.
+ final FileSystem fs = path.getFileSystem(jobConf);
+ long hdfsStart = counters.startTimeCounter();
+ final FileStatus stat = fs.getFileStatus(path);
+ counters.recordHdfsTime(hdfsStart);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ hdfsStart = counters.startTimeCounter();
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ counters.recordHdfsTime(hdfsStart);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ long startTime = counters.startTimeCounter();
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Exception e) {
+ // A shutdown-triggered InterruptedException must not be silently reported as an ordinary
+ // consumer error: preserve the interrupt on the thread so any caller sees it.
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ }
+ consumer.setError(e);
+ } finally {
+ counters.incrWallClockCounter(LlapIOCounters.TOTAL_IO_TIME_NS, startTime);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ // The LLAP IO summary keys off TABLE / FILE / STRIPES; set them the same way ORC does so a
+ // native Parquet fragment shows up in the summary with the same fields populated.
+ if (cacheTag != null) {
+ counters.setDesc(QueryFragmentCounters.Desc.TABLE, cacheTag.getTableName());
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.FILE, path
+ + (fileKey == null ? "" : " (" + fileKey + ")"));
+
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = selectRowGroups(blocks, fileSchema);
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ // STRIPES is the ORC term; for Parquet the analog is row groups. Reuse the same descriptor so
+ // the summary layout stays common and we don't have to teach the reporter about a new field.
+ counters.setDesc(QueryFragmentCounters.Desc.STRIPES, "0," + selected.size());
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /**
+ * The row groups this split has to read: the ones whose first data page falls inside the split, as
+ * Parquet's own input format assigns them, minus any the pushed-down predicate rules out on its
+ * statistics. Returned in file order, since the fetch pipeline reads them in that order.
+ */
+ private List<BlockMetaData> selectRowGroups(List<BlockMetaData> blocks, MessageType fileSchema)
+ throws IOException {
+ long splitStart = split.getStart();
+ long splitEnd = splitStart + split.getLength();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().getFirst().getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate == null) {
+ return selected;
+ }
+ return RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Whether the query needs row-lineage virtual columns ({@code ROW__LINEAGE__ID} /
+ * {@code LAST__UPDATED__SEQUENCE__NUMBER}) that the fallback reader augments the requested schema
+ * with via {@link RowLineageUtils#getRequestedSchemaWithRowLineageColumns}. The native pipeline
+ * doesn't propagate that augmentation through {@code Includes} yet, so we defer to the fallback
+ * reader instead of quietly emitting nulls in the lineage slots.
+ */
+ public boolean needsRowLineage() {
Review Comment:
i think that should go away from here and move to [HIVE-30103](https://issues.apache.org/jira/browse/HIVE-30103)
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ // Fallback path: no cache key, so we read status + footer straight from HDFS. Time both
+ // under HDFS_TIME_NS to match how ORC accounts for cache-miss footer reads.
+ final FileSystem fs = path.getFileSystem(jobConf);
+ long hdfsStart = counters.startTimeCounter();
+ final FileStatus stat = fs.getFileStatus(path);
+ counters.recordHdfsTime(hdfsStart);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ hdfsStart = counters.startTimeCounter();
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ counters.recordHdfsTime(hdfsStart);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ long startTime = counters.startTimeCounter();
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Exception e) {
+ // A shutdown-triggered InterruptedException must not be silently reported as an ordinary
+ // consumer error: preserve the interrupt on the thread so any caller sees it.
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ }
+ consumer.setError(e);
+ } finally {
+ counters.incrWallClockCounter(LlapIOCounters.TOTAL_IO_TIME_NS, startTime);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ // The LLAP IO summary keys off TABLE / FILE / STRIPES; set them the same way ORC does so a
+ // native Parquet fragment shows up in the summary with the same fields populated.
+ if (cacheTag != null) {
+ counters.setDesc(QueryFragmentCounters.Desc.TABLE, cacheTag.getTableName());
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.FILE, path
+ + (fileKey == null ? "" : " (" + fileKey + ")"));
+
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = selectRowGroups(blocks, fileSchema);
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ // STRIPES is the ORC term; for Parquet the analog is row groups. Reuse the same descriptor so
+ // the summary layout stays common and we don't have to teach the reporter about a new field.
+ counters.setDesc(QueryFragmentCounters.Desc.STRIPES, "0," + selected.size());
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /**
+ * The row groups this split has to read: the ones whose first data page falls inside the split, as
+ * Parquet's own input format assigns them, minus any the pushed-down predicate rules out on its
+ * statistics. Returned in file order, since the fetch pipeline reads them in that order.
+ */
+ private List<BlockMetaData> selectRowGroups(List<BlockMetaData> blocks, MessageType fileSchema)
+ throws IOException {
+ long splitStart = split.getStart();
+ long splitEnd = splitStart + split.getLength();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().getFirst().getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate == null) {
+ return selected;
+ }
+ return RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Whether the query needs row-lineage virtual columns ({@code ROW__LINEAGE__ID} /
+ * {@code LAST__UPDATED__SEQUENCE__NUMBER}) that the fallback reader augments the requested schema
+ * with via {@link RowLineageUtils#getRequestedSchemaWithRowLineageColumns}. The native pipeline
+ * doesn't propagate that augmentation through {@code Includes} yet, so we defer to the fallback
+ * reader instead of quietly emitting nulls in the lineage slots.
+ */
+ public boolean needsRowLineage() {
Review Comment:
i think that should go away from here and be moved to [HIVE-30103](https://issues.apache.org/jira/browse/HIVE-30103)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4155989825
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ // Fallback path: no cache key, so we read status + footer straight from HDFS. Time both
+ // under HDFS_TIME_NS to match how ORC accounts for cache-miss footer reads.
+ final FileSystem fs = path.getFileSystem(jobConf);
+ long hdfsStart = counters.startTimeCounter();
+ final FileStatus stat = fs.getFileStatus(path);
+ counters.recordHdfsTime(hdfsStart);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ hdfsStart = counters.startTimeCounter();
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ counters.recordHdfsTime(hdfsStart);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ long startTime = counters.startTimeCounter();
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Exception e) {
+ // A shutdown-triggered InterruptedException must not be silently reported as an ordinary
+ // consumer error: preserve the interrupt on the thread so any caller sees it.
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ }
+ consumer.setError(e);
+ } finally {
+ counters.incrWallClockCounter(LlapIOCounters.TOTAL_IO_TIME_NS, startTime);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ // The LLAP IO summary keys off TABLE / FILE / STRIPES; set them the same way ORC does so a
+ // native Parquet fragment shows up in the summary with the same fields populated.
+ if (cacheTag != null) {
+ counters.setDesc(QueryFragmentCounters.Desc.TABLE, cacheTag.getTableName());
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.FILE, path
+ + (fileKey == null ? "" : " (" + fileKey + ")"));
+
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = selectRowGroups(blocks, fileSchema);
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ // STRIPES is the ORC term; for Parquet the analog is row groups. Reuse the same descriptor so
+ // the summary layout stays common and we don't have to teach the reporter about a new field.
+ counters.setDesc(QueryFragmentCounters.Desc.STRIPES, "0," + selected.size());
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /**
+ * The row groups this split has to read: the ones whose first data page falls inside the split, as
+ * Parquet's own input format assigns them, minus any the pushed-down predicate rules out on its
+ * statistics. Returned in file order, since the fetch pipeline reads them in that order.
+ */
+ private List<BlockMetaData> selectRowGroups(List<BlockMetaData> blocks, MessageType fileSchema)
+ throws IOException {
+ long splitStart = split.getStart();
+ long splitEnd = splitStart + split.getLength();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().getFirst().getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate == null) {
+ return selected;
+ }
+ return RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Whether the query needs row-lineage virtual columns ({@code ROW__LINEAGE__ID} /
+ * {@code LAST__UPDATED__SEQUENCE__NUMBER}) that the fallback reader augments the requested schema
+ * with via {@link RowLineageUtils#getRequestedSchemaWithRowLineageColumns}. The native pipeline
+ * doesn't propagate that augmentation through {@code Includes} yet, so we defer to the fallback
+ * reader instead of quietly emitting nulls in the lineage slots.
+ */
+ public boolean needsRowLineage() {
+ VectorizedRowBatchCtx rbCtx = Utilities.getVectorizedRowBatchCtx(jobConf);
+ if (rbCtx == null) {
+ return false;
+ }
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ return RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema, VirtualColumn.ROW_LINEAGE_ID)
+ || RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema,
+ VirtualColumn.LAST_UPDATED_SEQUENCE_NUMBER);
+ }
+
+ /**
+ * Leaf-column positions of the requested fields; {@code BlockMetaData.getColumns()} is a flat list
+ * of leaves in file-schema order, so a top-level primitive at ordinal {@code k} in the file schema
+ * may sit at a very different leaf index (e.g. a nested group of two leaves before it shifts it
+ * from {@code 1} to {@code 2}). We only reach this method when every requested field is a
+ * top-level primitive ({@link #projectsNestedTypes()} would have forced a fallback otherwise), so
+ * each requested field maps to exactly one leaf whose path is a single segment.
+ */
+ static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
Review Comment:
/** Leaf index in {@code BlockMetaData.getColumns()} of each requested field present in the file. */
static int[] findLeafIndexes(MessageType requestedSchema, MessageType fileSchema)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4156085108
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ // Fallback path: no cache key, so we read status + footer straight from HDFS. Time both
+ // under HDFS_TIME_NS to match how ORC accounts for cache-miss footer reads.
+ final FileSystem fs = path.getFileSystem(jobConf);
+ long hdfsStart = counters.startTimeCounter();
+ final FileStatus stat = fs.getFileStatus(path);
+ counters.recordHdfsTime(hdfsStart);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ hdfsStart = counters.startTimeCounter();
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ counters.recordHdfsTime(hdfsStart);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ long startTime = counters.startTimeCounter();
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Exception e) {
+ // A shutdown-triggered InterruptedException must not be silently reported as an ordinary
+ // consumer error: preserve the interrupt on the thread so any caller sees it.
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ }
+ consumer.setError(e);
+ } finally {
+ counters.incrWallClockCounter(LlapIOCounters.TOTAL_IO_TIME_NS, startTime);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ // The LLAP IO summary keys off TABLE / FILE / STRIPES; set them the same way ORC does so a
+ // native Parquet fragment shows up in the summary with the same fields populated.
+ if (cacheTag != null) {
+ counters.setDesc(QueryFragmentCounters.Desc.TABLE, cacheTag.getTableName());
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.FILE, path
+ + (fileKey == null ? "" : " (" + fileKey + ")"));
+
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = selectRowGroups(blocks, fileSchema);
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ // STRIPES is the ORC term; for Parquet the analog is row groups. Reuse the same descriptor so
+ // the summary layout stays common and we don't have to teach the reporter about a new field.
+ counters.setDesc(QueryFragmentCounters.Desc.STRIPES, "0," + selected.size());
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /**
+ * The row groups this split has to read: the ones whose first data page falls inside the split, as
+ * Parquet's own input format assigns them, minus any the pushed-down predicate rules out on its
+ * statistics. Returned in file order, since the fetch pipeline reads them in that order.
+ */
+ private List<BlockMetaData> selectRowGroups(List<BlockMetaData> blocks, MessageType fileSchema)
+ throws IOException {
+ long splitStart = split.getStart();
+ long splitEnd = splitStart + split.getLength();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().getFirst().getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate == null) {
+ return selected;
+ }
+ return RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Whether the query needs row-lineage virtual columns ({@code ROW__LINEAGE__ID} /
+ * {@code LAST__UPDATED__SEQUENCE__NUMBER}) that the fallback reader augments the requested schema
+ * with via {@link RowLineageUtils#getRequestedSchemaWithRowLineageColumns}. The native pipeline
+ * doesn't propagate that augmentation through {@code Includes} yet, so we defer to the fallback
+ * reader instead of quietly emitting nulls in the lineage slots.
+ */
+ public boolean needsRowLineage() {
+ VectorizedRowBatchCtx rbCtx = Utilities.getVectorizedRowBatchCtx(jobConf);
+ if (rbCtx == null) {
+ return false;
+ }
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ return RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema, VirtualColumn.ROW_LINEAGE_ID)
+ || RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema,
+ VirtualColumn.LAST_UPDATED_SEQUENCE_NUMBER);
+ }
+
+ /**
+ * Leaf-column positions of the requested fields; {@code BlockMetaData.getColumns()} is a flat list
+ * of leaves in file-schema order, so a top-level primitive at ordinal {@code k} in the file schema
+ * may sit at a very different leaf index (e.g. a nested group of two leaves before it shifts it
+ * from {@code 1} to {@code 2}). We only reach this method when every requested field is a
+ * top-level primitive ({@link #projectsNestedTypes()} would have forced a fallback otherwise), so
+ * each requested field maps to exactly one leaf whose path is a single segment.
+ */
+ static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
+ List<ColumnDescriptor> allLeaves = fileSchema.getColumns();
+ List<Integer> selected = new ArrayList<>();
+ for (Type field : requestedSchema.getFields()) {
+ if (!fileSchema.containsField(field.getName())) {
+ continue;
+ }
+ for (int i = 0; i < allLeaves.size(); ++i) {
+ String[] path = allLeaves.get(i).getPath();
+ // Single-segment path == top-level primitive; skip leaves buried inside a group.
+ if (path.length == 1 && path[0].equals(field.getName())) {
+ selected.add(i);
+ break;
+ }
+ }
+ }
+ return selected.stream().mapToInt(Integer::intValue).toArray();
+ }
+
+ private static long bytes(int[] projected, BlockMetaData block) {
+ long total = 0;
+ for (int leaf : projected) {
+ total += block.getColumns().get(leaf).getTotalSize();
+ }
+ return total;
+ }
+
+ private static long bytes(Fetch fetch) {
+ long total = 0;
+ for (ColumnChunkMetaData chunk : fetch.batch.chunks()) {
+ total += chunk.getTotalSize();
+ }
+ return total;
+ }
+
+ /** One row group on its way in: buffers planned, missing ranges requested, not yet decoded. */
+ private static final class Fetch {
+ private final ParquetEncodedColumnBatch batch = new ParquetEncodedColumnBatch();
+ private final List<ColumnPlan> columns = new ArrayList<>();
+ private final List<Part> misses = new ArrayList<>();
+ private final List<Run> runs = new ArrayList<>();
+ }
+
+ /** The buffers covering one column chunk, in file order, and which stretches of them are new. */
+ private record ColumnPlan(List<Part> parts, List<MissRun> missRuns) {
+ ColumnPlan() {
+ this(new ArrayList<>(), new ArrayList<>());
+ }
+ }
+
+ /** One cache buffer's worth of a column chunk: a hit handed back by the cache, or a miss to fill. */
+ private static final class Part {
+ private MemoryBuffer buffer;
+ private final DiskRange range;
+ /** We hold one ref to release; until then a miss is a raw allocation to free. */
+ private boolean owned;
+
+ Part(MemoryBuffer buffer, DiskRange range, boolean owned) {
+ this.buffer = buffer;
+ this.range = range;
+ this.owned = owned;
+ }
+ }
+
+ /** {@code count} consecutive parts covering one contiguous missing sub-range; cached as a unit. */
+ private record MissRun(int firstPart, int count) {
+ }
+
+ /** One vectored range and the cache buffers it fills, in file order. */
+ private record Run(FileRange range, List<Part> parts) {
+ }
+
+ private Fetch startFetch(FSDataInputStream fileStream, ParquetRangeBuffers buffers, Allocator allocator,
+ int[] projected, BlockMetaData block, int rg)
+ throws IOException {
+ Fetch fetch = new Fetch();
+ ColumnChunkMetaData[] chunks = new ColumnChunkMetaData[projected.length];
+ for (int pc = 0; pc < projected.length; ++pc) {
+ chunks[pc] = block.getColumns().get(projected[pc]);
+ }
+ fetch.batch.init(fileKey, rg, chunks);
+ try {
+ for (ColumnChunkMetaData chunk : chunks) {
+ ColumnPlan column = new ColumnPlan();
+ fetch.columns.add(column);
+ planColumnChunk(allocator, chunk.getStartingPos(),
+ chunk.getStartingPos() + chunk.getTotalSize(), column, fetch.misses);
+ }
+ // The vectored dispatch is where non-async FS impls actually read; count it under HDFS_TIME.
+ long hdfsStart = counters.startTimeCounter();
+ requestMisses(fileStream, buffers, fetch, layout.maxRangeBytes());
+ counters.recordHdfsTime(hdfsStart);
+ } catch (Exception e) {
+ abandon(allocator, fetch);
+ throw e;
+ }
+ return fetch;
+ }
+
+ private void finishFetch(Allocator allocator, ParquetRangeBuffers buffers, Fetch fetch)
+ throws IOException, InterruptedException {
+ try {
+ // Awaiting the vectored futures is where the miss bytes actually arrive from HDFS.
+ long hdfsStart = counters.startTimeCounter();
+ receiveMisses(buffers, fetch);
+ counters.recordHdfsTime(hdfsStart);
+ for (ColumnPlan column : fetch.columns) {
+ putColumn(allocator, column);
+ }
+ for (int pc = 0; pc < fetch.columns.size(); ++pc) {
+ assemble(fetch.batch, pc, fetch.columns.get(pc).parts);
+ }
+ // consumeData returns the batch on success; after a throw it is still ours.
+ consumer.consumeData(fetch.batch);
+ } catch (Exception e) {
+ abandon(allocator, fetch);
+ throw e;
+ }
+ }
+
+ /** Drops a fetch that will not be decoded: outstanding requests are cancelled, buffers released. */
+ private void abandon(Allocator allocator, Fetch fetch) {
+ for (Run run : fetch.runs) {
+ run.range.getData().cancel(true);
+ }
+ for (ColumnPlan column : fetch.columns) {
+ for (Part part : column.parts) {
+ if (part.owned) {
+ bufferManager.decRefBuffer(part.buffer);
+ } else {
+ allocator.deallocate(part.buffer);
+ }
+ }
+ }
+ }
+
+ /** Package-private so a test can observe the read pattern. */
+ FSDataInputStream openFile(FileSystem fs) throws IOException {
+ return fs.open(path);
+ }
+
+ /** Lets the allocator abandon a wait for memory once the fragment is cancelled. */
+ private void allocateMultiple(Allocator allocator, MemoryBuffer[] dest, int size) {
+ if (allocator instanceof StoppableAllocator stoppable) {
+ stoppable.allocateMultiple(dest, size, DATA_BUFFER_FACTORY, isStopped);
+ } else {
+ allocator.allocateMultiple(dest, size, DATA_BUFFER_FACTORY);
+ }
+ }
+
+ /**
+ * Works out which buffers cover column chunk {@code [start, end)} without touching the file: cache
+ * hits as returned by getFileData, and freshly allocated power-of-two buffers for everything
+ * missing. Misses are appended to {@code allMisses} so the whole row group can be read at once.
+ */
+ private void planColumnChunk(Allocator allocator, long start, long end,
+ ColumnPlan column, List<Part> allMisses) throws IOException {
+ DiskRangeList head = new DiskRangeList(start, end);
+ if (fileKey != null) {
+ head = lowLevelCache.getFileData(fileKey, head, 0, ParquetCacheLayout.CACHE_CHUNK_FACTORY,
+ counters, new BooleanRef());
+ }
+ DiskRangeList current = head;
+ try {
+ for (; current != null; current = current.next) {
+ if (current.hasData()) {
+ column.parts.add(new Part(((CacheChunk) current).getBuffer(), current, true));
+ } else {
+ planMissRun(allocator, current, column, allMisses);
+ }
+ }
+ } catch (Exception e) {
+ // Hits past the failure point are still locked by getFileData; the caller only knows about
+ // the parts already recorded.
+ for (current = current.next; current != null; current = current.next) {
+ if (current.hasData()) {
+ bufferManager.decRefBuffer(((CacheChunk) current).getBuffer());
+ }
+ }
+ throw e;
+ }
+ }
+
+ /**
+ * Records one missing sub-range as a run of freshly allocated power-of-two buffers, and appends
+ * each part to {@code allMisses} so the caller can request them all in one vectored call.
+ */
+ private void planMissRun(Allocator allocator, DiskRangeList missing, ColumnPlan column,
+ List<Part> allMisses) throws IOException {
+ LlapHiveUtils.throwIfCacheOnlyRead(cacheOnly);
+ int[] sizes = layout.bufferSizes(missing.getEnd() - missing.getOffset());
+ column.missRuns.add(new MissRun(column.parts.size(), sizes.length));
+ long partFrom = missing.getOffset();
+ for (int size : sizes) {
+ MemoryBuffer[] one = new MemoryBuffer[1];
+ allocateMultiple(allocator, one, size);
+ // The cache accounts and serves the bytes up to the buffer's limit.
+ ByteBuffer raw = one[0].getByteBufferRaw();
+ raw.limit(raw.position() + size);
+ Part part = new Part(one[0], new DiskRange(partFrom, partFrom + size), false);
+ column.parts.add(part);
+ allMisses.add(part);
+ partFrom += size;
+ }
+ }
+
+ /**
+ * Requests every missing buffer of a row group in one vectored call, one range per run of
+ * adjacent buffers. Column chunks sit back to back in the file, so a projection that keeps
+ * neighbouring columns reads them together; runs are capped so the row group arrives as
+ * several concurrent requests rather than one. A filesystem without a vectored implementation
+ * reads the ranges in turn.
+ */
+ private static void requestMisses(FSDataInputStream fileStream, ParquetRangeBuffers buffers, Fetch fetch,
+ int maxRange) throws IOException {
+ List<Part> misses = fetch.misses;
+ if (misses.isEmpty()) {
+ return;
+ }
+ misses.sort(Comparator.comparingLong(part -> part.range.getOffset()));
+ List<Run> runs = new ArrayList<>();
+ int i = 0;
+ while (i < misses.size()) {
+ long from = misses.get(i).range.getOffset();
+ long to = misses.get(i).range.getEnd();
+ int j = i + 1;
+ while (j < misses.size()) {
+ DiskRange next = misses.get(j).range;
+ if (next.getOffset() != to || next.getEnd() - from > maxRange) {
+ break;
+ }
+ to = next.getEnd();
+ ++j;
+ }
+ runs.add(new Run(FileRange.createFileRange(from, (int) (to - from)), misses.subList(i, j)));
+ i = j;
+ }
+ List<FileRange> ranges = runs.stream().map(run -> run.range).collect(Collectors.toList());
+ fileStream.readVectored(ranges, buffers::allocate, buffers::release);
+ // Only requests the stream accepted are the fetch's to wait for or cancel.
+ fetch.runs.addAll(runs);
+ }
+
+ /** Waits for the requested runs and copies each into the cache buffers it covers. */
+ private static void receiveMisses(ParquetRangeBuffers buffers, Fetch fetch) throws IOException {
+ for (Run run : fetch.runs) {
+ ByteBuffer data = FutureIO.awaitFuture(run.range.getData());
+ for (Part part : run.parts) {
+ int length = part.range.getLength();
+ ByteBuffer src = data.duplicate();
+ src.position(data.position() + (int) (part.range.getOffset() - run.range.getOffset()));
+ src.limit(src.position() + length);
+ part.buffer.getByteBufferRaw().duplicate().put(src);
+ }
+ buffers.release(data);
+ }
+ }
+
+ /** Hands one column's freshly read buffers to the cache; every part then carries a ref we own. */
+ private void putColumn(Allocator allocator, ColumnPlan column) {
+ for (MissRun run : column.missRuns) {
+ List<Part> parts = column.parts.subList(run.firstPart, run.firstPart + run.count);
+ if (fileKey == null) {
+ for (Part part : parts) {
+ bufferManager.incRefBuffer(part.buffer);
+ part.owned = true;
+ }
+ // No cache key => no putFileData; each part is already on the ref-counted side
+ // (incRefBuffer + owned=true), so cleanup will decRef rather than deallocate. Skip the
+ // cache-put loop.
+ continue;
+ }
+ // One range at a time so a mid-run throw from putFileData (e.g. the length-mismatch guard
+ // in LowLevelCacheImpl) leaves ownership clean: parts already handled are marked owned so
+ // cleanup calls decRefBuffer on cache-owned memory, while the current and later ones stay
+ // raw allocations that cleanup can safely deallocate.
+ for (Part part : parts) {
+ MemoryBuffer fresh = part.buffer;
+ MemoryBuffer[] pair = new MemoryBuffer[] {fresh};
+ DiskRange[] range = new DiskRange[] {part.range};
+ lowLevelCache.putFileData(fileKey, range, pair, 0, Priority.NORMAL, counters, cacheTag);
+ if (pair[0] != fresh) {
+ // The cache kept its own buffer (locked for us) and unlocked ours without freeing it.
+ allocator.deallocate(fresh);
+ part.buffer = pair[0];
+ }
+ part.owned = true;
+ }
+ }
+ }
+
+ private static void assemble(ParquetEncodedColumnBatch batch, int pc, List<Part> parts) {
Review Comment:
setBatchColumn ? fills batch.columnBuffers/offsets/lengths[pc] from the parts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4156998497
##########
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetRowGroupDecoder.java:
##########
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.ql.io.parquet.vector;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.parquet.ParquetRuntimeException;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.InvalidSchemaException;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Reusable, stateless helper that builds the {@link VectorizedColumnReader} array for one Parquet
+ * row group, given an already-read {@link PageReadStore}. The logic here was lifted verbatim from
+ * {@link VectorizedParquetRecordReader} so that both the row-by-split reader and the LLAP
+ * cache-backed consumer ({@code ParquetEncodedDataConsumer}) can share a single, behavior-preserving
+ * implementation of the Hive-type-driven Parquet column-reader construction.
+ *
+ * <p>The instance only carries the {@code fileSchema} (used for the schema-evolution check). All
+ * per-row-group state lives in the supplied {@link PageReadStore}.
+ */
+public class ParquetRowGroupDecoder {
+
+ private static final int MAP_DEFINITION_LEVEL_MAX = 3;
Review Comment:
MAX_MAP_WRAPPER_GROUPS ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4156103977
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ // Fallback path: no cache key, so we read status + footer straight from HDFS. Time both
+ // under HDFS_TIME_NS to match how ORC accounts for cache-miss footer reads.
+ final FileSystem fs = path.getFileSystem(jobConf);
+ long hdfsStart = counters.startTimeCounter();
+ final FileStatus stat = fs.getFileStatus(path);
+ counters.recordHdfsTime(hdfsStart);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ hdfsStart = counters.startTimeCounter();
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ counters.recordHdfsTime(hdfsStart);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ long startTime = counters.startTimeCounter();
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Exception e) {
+ // A shutdown-triggered InterruptedException must not be silently reported as an ordinary
+ // consumer error: preserve the interrupt on the thread so any caller sees it.
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ }
+ consumer.setError(e);
+ } finally {
+ counters.incrWallClockCounter(LlapIOCounters.TOTAL_IO_TIME_NS, startTime);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ // The LLAP IO summary keys off TABLE / FILE / STRIPES; set them the same way ORC does so a
+ // native Parquet fragment shows up in the summary with the same fields populated.
+ if (cacheTag != null) {
+ counters.setDesc(QueryFragmentCounters.Desc.TABLE, cacheTag.getTableName());
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.FILE, path
+ + (fileKey == null ? "" : " (" + fileKey + ")"));
+
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = selectRowGroups(blocks, fileSchema);
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ // STRIPES is the ORC term; for Parquet the analog is row groups. Reuse the same descriptor so
+ // the summary layout stays common and we don't have to teach the reporter about a new field.
+ counters.setDesc(QueryFragmentCounters.Desc.STRIPES, "0," + selected.size());
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /**
+ * The row groups this split has to read: the ones whose first data page falls inside the split, as
+ * Parquet's own input format assigns them, minus any the pushed-down predicate rules out on its
+ * statistics. Returned in file order, since the fetch pipeline reads them in that order.
+ */
+ private List<BlockMetaData> selectRowGroups(List<BlockMetaData> blocks, MessageType fileSchema)
+ throws IOException {
+ long splitStart = split.getStart();
+ long splitEnd = splitStart + split.getLength();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().getFirst().getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate == null) {
+ return selected;
+ }
+ return RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Whether the query needs row-lineage virtual columns ({@code ROW__LINEAGE__ID} /
+ * {@code LAST__UPDATED__SEQUENCE__NUMBER}) that the fallback reader augments the requested schema
+ * with via {@link RowLineageUtils#getRequestedSchemaWithRowLineageColumns}. The native pipeline
+ * doesn't propagate that augmentation through {@code Includes} yet, so we defer to the fallback
+ * reader instead of quietly emitting nulls in the lineage slots.
+ */
+ public boolean needsRowLineage() {
+ VectorizedRowBatchCtx rbCtx = Utilities.getVectorizedRowBatchCtx(jobConf);
+ if (rbCtx == null) {
+ return false;
+ }
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ return RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema, VirtualColumn.ROW_LINEAGE_ID)
+ || RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema,
+ VirtualColumn.LAST_UPDATED_SEQUENCE_NUMBER);
+ }
+
+ /**
+ * Leaf-column positions of the requested fields; {@code BlockMetaData.getColumns()} is a flat list
+ * of leaves in file-schema order, so a top-level primitive at ordinal {@code k} in the file schema
+ * may sit at a very different leaf index (e.g. a nested group of two leaves before it shifts it
+ * from {@code 1} to {@code 2}). We only reach this method when every requested field is a
+ * top-level primitive ({@link #projectsNestedTypes()} would have forced a fallback otherwise), so
+ * each requested field maps to exactly one leaf whose path is a single segment.
+ */
+ static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
+ List<ColumnDescriptor> allLeaves = fileSchema.getColumns();
+ List<Integer> selected = new ArrayList<>();
+ for (Type field : requestedSchema.getFields()) {
+ if (!fileSchema.containsField(field.getName())) {
+ continue;
+ }
+ for (int i = 0; i < allLeaves.size(); ++i) {
+ String[] path = allLeaves.get(i).getPath();
+ // Single-segment path == top-level primitive; skip leaves buried inside a group.
+ if (path.length == 1 && path[0].equals(field.getName())) {
+ selected.add(i);
+ break;
+ }
+ }
+ }
+ return selected.stream().mapToInt(Integer::intValue).toArray();
+ }
+
+ private static long bytes(int[] projected, BlockMetaData block) {
+ long total = 0;
+ for (int leaf : projected) {
+ total += block.getColumns().get(leaf).getTotalSize();
+ }
+ return total;
+ }
+
+ private static long bytes(Fetch fetch) {
+ long total = 0;
+ for (ColumnChunkMetaData chunk : fetch.batch.chunks()) {
+ total += chunk.getTotalSize();
+ }
+ return total;
+ }
+
+ /** One row group on its way in: buffers planned, missing ranges requested, not yet decoded. */
+ private static final class Fetch {
+ private final ParquetEncodedColumnBatch batch = new ParquetEncodedColumnBatch();
+ private final List<ColumnPlan> columns = new ArrayList<>();
+ private final List<Part> misses = new ArrayList<>();
+ private final List<Run> runs = new ArrayList<>();
+ }
+
+ /** The buffers covering one column chunk, in file order, and which stretches of them are new. */
+ private record ColumnPlan(List<Part> parts, List<MissRun> missRuns) {
+ ColumnPlan() {
+ this(new ArrayList<>(), new ArrayList<>());
+ }
+ }
+
+ /** One cache buffer's worth of a column chunk: a hit handed back by the cache, or a miss to fill. */
+ private static final class Part {
+ private MemoryBuffer buffer;
+ private final DiskRange range;
+ /** We hold one ref to release; until then a miss is a raw allocation to free. */
+ private boolean owned;
+
+ Part(MemoryBuffer buffer, DiskRange range, boolean owned) {
+ this.buffer = buffer;
+ this.range = range;
+ this.owned = owned;
+ }
+ }
+
+ /** {@code count} consecutive parts covering one contiguous missing sub-range; cached as a unit. */
+ private record MissRun(int firstPart, int count) {
+ }
+
+ /** One vectored range and the cache buffers it fills, in file order. */
+ private record Run(FileRange range, List<Part> parts) {
+ }
+
+ private Fetch startFetch(FSDataInputStream fileStream, ParquetRangeBuffers buffers, Allocator allocator,
+ int[] projected, BlockMetaData block, int rg)
+ throws IOException {
+ Fetch fetch = new Fetch();
+ ColumnChunkMetaData[] chunks = new ColumnChunkMetaData[projected.length];
+ for (int pc = 0; pc < projected.length; ++pc) {
+ chunks[pc] = block.getColumns().get(projected[pc]);
+ }
+ fetch.batch.init(fileKey, rg, chunks);
+ try {
+ for (ColumnChunkMetaData chunk : chunks) {
+ ColumnPlan column = new ColumnPlan();
+ fetch.columns.add(column);
+ planColumnChunk(allocator, chunk.getStartingPos(),
+ chunk.getStartingPos() + chunk.getTotalSize(), column, fetch.misses);
+ }
+ // The vectored dispatch is where non-async FS impls actually read; count it under HDFS_TIME.
+ long hdfsStart = counters.startTimeCounter();
+ requestMisses(fileStream, buffers, fetch, layout.maxRangeBytes());
+ counters.recordHdfsTime(hdfsStart);
+ } catch (Exception e) {
+ abandon(allocator, fetch);
+ throw e;
+ }
+ return fetch;
+ }
+
+ private void finishFetch(Allocator allocator, ParquetRangeBuffers buffers, Fetch fetch)
+ throws IOException, InterruptedException {
+ try {
+ // Awaiting the vectored futures is where the miss bytes actually arrive from HDFS.
+ long hdfsStart = counters.startTimeCounter();
+ receiveMisses(buffers, fetch);
+ counters.recordHdfsTime(hdfsStart);
+ for (ColumnPlan column : fetch.columns) {
+ putColumn(allocator, column);
+ }
+ for (int pc = 0; pc < fetch.columns.size(); ++pc) {
+ assemble(fetch.batch, pc, fetch.columns.get(pc).parts);
+ }
+ // consumeData returns the batch on success; after a throw it is still ours.
+ consumer.consumeData(fetch.batch);
+ } catch (Exception e) {
+ abandon(allocator, fetch);
+ throw e;
+ }
+ }
+
+ /** Drops a fetch that will not be decoded: outstanding requests are cancelled, buffers released. */
+ private void abandon(Allocator allocator, Fetch fetch) {
+ for (Run run : fetch.runs) {
+ run.range.getData().cancel(true);
+ }
+ for (ColumnPlan column : fetch.columns) {
+ for (Part part : column.parts) {
+ if (part.owned) {
+ bufferManager.decRefBuffer(part.buffer);
+ } else {
+ allocator.deallocate(part.buffer);
+ }
+ }
+ }
+ }
+
+ /** Package-private so a test can observe the read pattern. */
+ FSDataInputStream openFile(FileSystem fs) throws IOException {
+ return fs.open(path);
+ }
+
+ /** Lets the allocator abandon a wait for memory once the fragment is cancelled. */
+ private void allocateMultiple(Allocator allocator, MemoryBuffer[] dest, int size) {
+ if (allocator instanceof StoppableAllocator stoppable) {
+ stoppable.allocateMultiple(dest, size, DATA_BUFFER_FACTORY, isStopped);
+ } else {
+ allocator.allocateMultiple(dest, size, DATA_BUFFER_FACTORY);
+ }
+ }
+
+ /**
+ * Works out which buffers cover column chunk {@code [start, end)} without touching the file: cache
+ * hits as returned by getFileData, and freshly allocated power-of-two buffers for everything
+ * missing. Misses are appended to {@code allMisses} so the whole row group can be read at once.
+ */
+ private void planColumnChunk(Allocator allocator, long start, long end,
+ ColumnPlan column, List<Part> allMisses) throws IOException {
+ DiskRangeList head = new DiskRangeList(start, end);
+ if (fileKey != null) {
+ head = lowLevelCache.getFileData(fileKey, head, 0, ParquetCacheLayout.CACHE_CHUNK_FACTORY,
+ counters, new BooleanRef());
+ }
+ DiskRangeList current = head;
+ try {
+ for (; current != null; current = current.next) {
+ if (current.hasData()) {
+ column.parts.add(new Part(((CacheChunk) current).getBuffer(), current, true));
+ } else {
+ planMissRun(allocator, current, column, allMisses);
+ }
+ }
+ } catch (Exception e) {
+ // Hits past the failure point are still locked by getFileData; the caller only knows about
+ // the parts already recorded.
+ for (current = current.next; current != null; current = current.next) {
+ if (current.hasData()) {
+ bufferManager.decRefBuffer(((CacheChunk) current).getBuffer());
+ }
+ }
+ throw e;
+ }
+ }
+
+ /**
+ * Records one missing sub-range as a run of freshly allocated power-of-two buffers, and appends
+ * each part to {@code allMisses} so the caller can request them all in one vectored call.
+ */
+ private void planMissRun(Allocator allocator, DiskRangeList missing, ColumnPlan column,
+ List<Part> allMisses) throws IOException {
+ LlapHiveUtils.throwIfCacheOnlyRead(cacheOnly);
+ int[] sizes = layout.bufferSizes(missing.getEnd() - missing.getOffset());
+ column.missRuns.add(new MissRun(column.parts.size(), sizes.length));
+ long partFrom = missing.getOffset();
+ for (int size : sizes) {
+ MemoryBuffer[] one = new MemoryBuffer[1];
+ allocateMultiple(allocator, one, size);
+ // The cache accounts and serves the bytes up to the buffer's limit.
+ ByteBuffer raw = one[0].getByteBufferRaw();
+ raw.limit(raw.position() + size);
+ Part part = new Part(one[0], new DiskRange(partFrom, partFrom + size), false);
+ column.parts.add(part);
+ allMisses.add(part);
+ partFrom += size;
+ }
+ }
+
+ /**
+ * Requests every missing buffer of a row group in one vectored call, one range per run of
+ * adjacent buffers. Column chunks sit back to back in the file, so a projection that keeps
+ * neighbouring columns reads them together; runs are capped so the row group arrives as
+ * several concurrent requests rather than one. A filesystem without a vectored implementation
+ * reads the ranges in turn.
+ */
+ private static void requestMisses(FSDataInputStream fileStream, ParquetRangeBuffers buffers, Fetch fetch,
+ int maxRange) throws IOException {
+ List<Part> misses = fetch.misses;
+ if (misses.isEmpty()) {
+ return;
+ }
+ misses.sort(Comparator.comparingLong(part -> part.range.getOffset()));
+ List<Run> runs = new ArrayList<>();
+ int i = 0;
+ while (i < misses.size()) {
+ long from = misses.get(i).range.getOffset();
+ long to = misses.get(i).range.getEnd();
+ int j = i + 1;
+ while (j < misses.size()) {
+ DiskRange next = misses.get(j).range;
+ if (next.getOffset() != to || next.getEnd() - from > maxRange) {
+ break;
+ }
+ to = next.getEnd();
+ ++j;
+ }
+ runs.add(new Run(FileRange.createFileRange(from, (int) (to - from)), misses.subList(i, j)));
+ i = j;
+ }
+ List<FileRange> ranges = runs.stream().map(run -> run.range).collect(Collectors.toList());
+ fileStream.readVectored(ranges, buffers::allocate, buffers::release);
+ // Only requests the stream accepted are the fetch's to wait for or cancel.
+ fetch.runs.addAll(runs);
+ }
+
+ /** Waits for the requested runs and copies each into the cache buffers it covers. */
+ private static void receiveMisses(ParquetRangeBuffers buffers, Fetch fetch) throws IOException {
+ for (Run run : fetch.runs) {
+ ByteBuffer data = FutureIO.awaitFuture(run.range.getData());
+ for (Part part : run.parts) {
+ int length = part.range.getLength();
+ ByteBuffer src = data.duplicate();
+ src.position(data.position() + (int) (part.range.getOffset() - run.range.getOffset()));
+ src.limit(src.position() + length);
+ part.buffer.getByteBufferRaw().duplicate().put(src);
+ }
+ buffers.release(data);
+ }
+ }
+
+ /** Hands one column's freshly read buffers to the cache; every part then carries a ref we own. */
+ private void putColumn(Allocator allocator, ColumnPlan column) {
Review Comment:
cacheChunk ? It caches one column chunk's new parts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4157224923
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
Review Comment:
Follow-up improvement:
Today the stock reader and the native reader each derive the same six per-file facts from the same inputs, in different places, and could drift
````
┌────────────────────────────────────────────┬──────────────────────────────────────────┬───────────────────────────────────────────────┐
│ Decision │ Stock │ Native │
├────────────────────────────────────────────┼──────────────────────────────────────────┼───────────────────────────────────────────────┤
│ requested schema │ VectorizedParquetRecordReader.initialize │ ParquetEncodedDataReader.loadFooter │
├────────────────────────────────────────────┼──────────────────────────────────────────┼───────────────────────────────────────────────┤
│ timestamp, proleptic, legacy-zone flags │ ParquetRecordReaderBase.getSplit:160-170 │ ParquetEncodedDataConsumer.initSchema:120-144 │
├────────────────────────────────────────────┼──────────────────────────────────────────┼───────────────────────────────────────────────┤
│ writer timezone │ VectorizedParquetRecordReader:255 │ consumer :129 │
├────────────────────────────────────────────┼──────────────────────────────────────────┼───────────────────────────────────────────────┤
│ row groups in split (first-data-page rule) │ ParquetRecordReaderBase:132-138 │ reader :262-268 │
├────────────────────────────────────────────┼──────────────────────────────────────────┼───────────────────────────────────────────────┤
│ SARG to predicate, statistics filter │ ParquetRecordReaderBase:144-146 │ reader :276-284 │
├────────────────────────────────────────────┼──────────────────────────────────────────┼───────────────────────────────────────────────┤
│ decimal64 │ plan ctx │ conf string (F4) │
└────────────────────────────────────────────┴──────────────────────────────────────────┴───────────────────────────────────────────────┘
````
The proposal is one class in ql, built once per split from the things both readers already have, whose fields are those decisions:
````
/** Everything a Parquet split read decides once from the footer and the job conf, shared by both readers. */
public final class ParquetSplitPlan {
public final MessageType requestedSchema;
public final List<BlockMetaData> rowGroups; // in-split, statistics-filtered
public final TimestampConversionOptions conversions;
public final FilterPredicate predicate; // may be null
public static ParquetSplitPlan of(JobConf conf, ParquetMetadata footer, long splitStart, long splitLength) { … }
}
````
VectorizedParquetRecordReader.initialize and ParquetEncodedDataReader.loadFooter both become "footer, then ParquetSplitPlan.of(...)", and ParquetEncodedDataConsumer.initSchema disappears because the consumer receives the plan. toFilterPredicate and the five DataWritableReadSupport statics go back behind it, so the public surface between ql and llap-server is one type instead of six methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #6793:
URL: https://github.com/apache/hive/pull/6793#issuecomment-5926538252
latest testing results:
## TPC-DS A/B: `hive.llap.io.parquet.native.enabled`
99 queries × 3 reps × 2 arms = **594 runs, 0 failures**, against `tpcds_partitioned_parquet_10`.
Metric is `Run DAG` from the Tez query summary (pure DAG execution, excluding
compile/submit/fetch). Cluster restarted before each arm.
**A/B validity:** `false` arm showed LLAP IO counters in 0/297 runs, `true` arm in 297/297 —
the flag really does switch the read path, so the comparison is valid.
### Correctness: no differences
| | queries |
|---|---|
| byte-identical output | 90 |
| identical, both empty | 6 |
| identical multiset, row order differs | 3 (q31, q65, q71) |
| **value or row-count differenc
The 3 ordering differences are read-path defects: for those
queries the row order also varies between repetitions with the flag held constant, in both
arms (`ORDER BY` on constant/tietie-breaker). q77 is a fourth such
query, which happened to order stably in the `false` arm.
### Performance: 9.1% faster warm, 19.5% faster on first execution
| | `false` | `true` | change |
|---|---:|---:|---:|
| total `Run DAG`, warm | 995.8s
| total `Run DAG`, first execution | 1605.4s | 1291.7s | **−19.5%** |
Median per-query change **−7.9%**; 84/99 queries faster by some margin. 32 robustly faster
(every warm sample beats every ` noise, 7 material but
indeterminate. The win is broad rather than a few outliers — the top 8 queries account for
only 46% of the total saving.
<details>
<summary>Caveats</summary>
Single-box run, 10 GB scale factor. Run-to-run spread exceeds 15% between a query's own two
warm reps for 25 (query, arm) pairs, so the aggregate and the robust counts are the
load-bearing numbers — an individual query's percentage is indicative only.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4217587072
##########
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetRowGroupDecoder.java:
##########
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.ql.io.parquet.vector;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.parquet.ParquetRuntimeException;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.InvalidSchemaException;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Reusable, stateless helper that builds the {@link VectorizedColumnReader} array for one Parquet
+ * row group, given an already-read {@link PageReadStore}. The logic here was lifted verbatim from
+ * {@link VectorizedParquetRecordReader} so that both the row-by-split reader and the LLAP
+ * cache-backed consumer ({@code ParquetEncodedDataConsumer}) can share a single, behavior-preserving
+ * implementation of the Hive-type-driven Parquet column-reader construction.
+ *
+ * <p>The instance only carries the {@code fileSchema} (used for the schema-evolution check). All
+ * per-row-group state lives in the supplied {@link PageReadStore}.
+ */
+public class ParquetRowGroupDecoder {
+
+ private static final int MAP_DEFINITION_LEVEL_MAX = 3;
+
+ /**
+ * Bag of writer-timezone / proleptic / legacy conversion flags forwarded to every primitive
+ * column reader. Grouped so the reader-construction entry point stays under the parameter cap.
+ */
+ public record TimestampConversionOptions(
+ boolean skipTimestampConversion,
+ ZoneId writerTimezone,
+ boolean skipProlepticConversion,
+ boolean legacyConversionEnabled) {
+ }
+
+ private final MessageType fileSchema;
+ private final Map<String, Object> initialDefaults;
+
+ public ParquetRowGroupDecoder(MessageType fileSchema, Map<String, Object> initialDefaults) {
+ this.fileSchema = fileSchema;
+ this.initialDefaults = initialDefaults;
+ }
+
+ /**
+ * Builds the per-(requested-)column {@link VectorizedColumnReader} array for a row group.
+ *
+ * @param pages the row group's page store (from {@code reader.readRowGroup(..)}
+ * / {@code readNextRowGroup()})
+ * @param requestedSchema the projected Parquet schema being read
+ * @param columnTypesList the Hive type infos for ALL table columns (indexed by table col id)
+ * @param colsToInclude the table column ids being read, in requested-schema field order;
+ * may be empty (e.g. {@code count(*)}), in which case all readers are null
+ * @param readAllColumns whether projection is "read all columns"
+ * @param options writer-timezone / proleptic / legacy conversion flags, see
+ * {@link TimestampConversionOptions}
+ * @return one reader per requested-schema field (null entries where no reader is needed)
+ */
+ public VectorizedColumnReader[] buildColumnReaders(
+ PageReadStore pages,
+ MessageType requestedSchema,
+ List<TypeInfo> columnTypesList,
+ List<Integer> colsToInclude,
+ boolean readAllColumns,
+ TimestampConversionOptions options) throws IOException {
+ List<Type> types = requestedSchema.getFields();
+ VectorizedColumnReader[] columnReaders = new VectorizedColumnReader[types.size()];
+
+ if (!readAllColumns) {
+ // certain queries like select count(*) from table do not have
+ // any projected columns and still have isReadAllColumns as false
+ // in such cases columnReaders are not needed
+ // However, if colsToInclude is not empty we should initialize each columnReader
+ if (!colsToInclude.isEmpty()) {
+ for (int i = 0; i < types.size(); ++i) {
+ columnReaders[i] = buildVectorizedParquetReader(
+ columnTypesList.get(colsToInclude.get(i)), types.get(i), pages,
+ requestedSchema.getColumns(), options, 0, 0);
+ }
+ }
+ } else {
+ for (int i = 0; i < types.size(); ++i) {
+ columnReaders[i] = buildVectorizedParquetReader(columnTypesList.get(i),
+ types.get(i), pages, requestedSchema.getColumns(), options, 0, 0);
+ }
+ }
+ return columnReaders;
+ }
+
+ private static List<ColumnDescriptor> getAllColumnDescriptorByType(
+ int depth,
+ Type type,
+ List<ColumnDescriptor> columns) throws ParquetRuntimeException {
+ List<ColumnDescriptor> res = new ArrayList<>();
+ for (ColumnDescriptor descriptor : columns) {
+ if (depth >= descriptor.getPath().length) {
+ throw new InvalidSchemaException("Corrupted Parquet schema");
+ }
+ if (type.getName().equals(descriptor.getPath()[depth])) {
+ res.add(descriptor);
+ }
+ }
+ return res;
+ }
+
+ // Nested types are unsupported here on purpose; callers detect that in advance
+ // (ParquetEncodedDataReader.projectsNestedTypes) and route the query to the non-native reader.
+ private static PrimitiveType getElementType(Type type) {
+ if (type.isPrimitive()) {
+ return type.asPrimitiveType();
+ }
+ if (type.asGroupType().getFields().size() > 1) {
+ throw new UnsupportedOperationException(
+ "Current Parquet Vectorization reader doesn't support nested type");
+ }
+
+ Type childType = type.asGroupType().getFields().getFirst();
+
+ // Parquet file generated using thrift may have child type as PrimitiveType
+ if (childType.isPrimitive()) {
+ return childType.asPrimitiveType();
+ } else {
+ return childType.asGroupType().getFields().getFirst().asPrimitiveType();
+ }
+ }
+
+ // Build VectorizedParquetColumnReader via Hive typeInfo and Parquet schema
+ private VectorizedColumnReader buildVectorizedParquetReader(
+ TypeInfo typeInfo,
+ Type type,
+ PageReadStore pages,
+ List<ColumnDescriptor> columnDescriptors,
+ TimestampConversionOptions options,
+ int depth, int currentDefLevel) throws IOException {
+ int typeDefLevel = currentDefLevel;
+ if (type.isRepetition(Type.Repetition.OPTIONAL) || type.isRepetition(Type.Repetition.REPEATED)) {
+ typeDefLevel++;
+ }
+ List<ColumnDescriptor> descriptors =
+ getAllColumnDescriptorByType(depth, type, columnDescriptors);
+ // Support for schema evolution: if the column from the current
+ // query schema is not present in the file schema, return a dummy
+ // reader that produces nulls. This allows queries to proceed even
+ // when new columns have been added after the file was written.
+ if (!fileSchema.getColumns().contains(descriptors.getFirst())) {
+ return new VectorizedDummyColumnReader(Optional.ofNullable(initialDefaults)
+ .map(defaults -> defaults.getOrDefault(descriptors.getFirst().getPath()[0], null)).orElse(null));
+ }
+ switch (typeInfo.getCategory()) {
+ case PRIMITIVE:
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6793:
URL: https://github.com/apache/hive/pull/6793#issuecomment-6269626284
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6793) **Quality Gate passed**
Issues
 [87 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6793&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6793&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6793&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6793&metric=new_coverage&view=list)
 [0.1% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6793&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6793)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6793:
URL: https://github.com/apache/hive/pull/6793#issuecomment-5703051570
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6793) **Quality Gate passed**
Issues
 [104 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6793&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6793&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6793&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6793&metric=new_coverage&view=list)
 [0.2% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6793&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6793)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4030097385
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ParquetCachedPageReadStore.java:
##########
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.llap.io.decode;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+
+import org.apache.hadoop.hive.llap.io.encoded.ParquetEncodedColumnBatch;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.DataPage;
+import org.apache.parquet.column.page.DataPageV1;
+import org.apache.parquet.column.page.DataPageV2;
+import org.apache.parquet.column.page.DictionaryPage;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.column.page.PageReader;
+import org.apache.parquet.compression.CompressionCodecFactory;
+import org.apache.parquet.compression.CompressionCodecFactory.BytesInputDecompressor;
+import org.apache.parquet.format.DataPageHeader;
+import org.apache.parquet.format.DataPageHeaderV2;
+import org.apache.parquet.format.DictionaryPageHeader;
+import org.apache.parquet.format.PageHeader;
+import org.apache.parquet.format.Util;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.schema.PrimitiveType;
+
+/**
+ * {@link PageReadStore} over one row group's cached column-chunk buffers: page headers are parsed in
+ * place, page bytes are views of the cache buffers and are decompressed lazily on readPage, mirroring
+ * parquet-mr's ParquetFileReader.Chunk.readAllPages and ColumnChunkPageReader (no offset index, no decryption).
+ */
+class ParquetCachedPageReadStore implements PageReadStore {
+
+ private final Map<ColumnPath, PageReader> readers = new HashMap<>();
+ private final long rowCount;
+
+ ParquetCachedPageReadStore(ParquetMetadata footer, ParquetEncodedColumnBatch batch,
+ CompressionCodecFactory codecFactory, ParquetMetadataConverter converter) throws IOException {
+ BlockMetaData block = footer.getBlocks().get(batch.rowGroupIx);
+ this.rowCount = block.getRowCount();
+ String createdBy = footer.getFileMetaData().getCreatedBy();
+ for (int pc = 0; pc < batch.chunks.length; ++pc) {
+ ColumnChunkMetaData chunk = batch.chunks[pc];
+ readers.put(chunk.getPath(), readAllPages(chunk, chunkBuffers(batch, pc), createdBy,
+ codecFactory.getDecompressor(chunk.getCodec()), converter));
+ }
+ }
+
+ /** Slices of the cached buffers covering exactly the chunk's byte region, in file order. */
+ private static List<ByteBuffer> chunkBuffers(ParquetEncodedColumnBatch batch, int pc) {
+ long start = batch.chunks[pc].getStartingPos(), end = start + batch.chunks[pc].getTotalSize();
+ List<ByteBuffer> slices = new ArrayList<>(batch.columnBuffers[pc].length);
+ for (int i = 0; i < batch.columnBuffers[pc].length; ++i) {
+ long offset = batch.bufferOffsets[pc][i];
+ long from = Math.max(start, offset), to = Math.min(end, offset + batch.bufferLengths[pc][i]);
Review Comment:
same, new line for to, maybe need better naming
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4030054129
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java:
##########
@@ -272,6 +279,10 @@ public void debugDumpShort(StringBuilder sb) {
metadataCache, dataCache, pathCache, bufferManagerOrc, conf, cacheMetrics, ioMetrics, tracePool);
this.genericCvp = isEncodeEnabled ? new GenericColumnVectorProducer(
serdeCache, bufferManagerGeneric, conf, cacheMetrics, ioMetrics, tracePool, encodeExecutor) : null;
+ // Native Parquet IO is gated per query by the job conf at the dispatch sites.
+ this.parquetCvp = dataCache != null
+ ? new ParquetColumnVectorProducer(dataCache, bufferManagerOrc, conf, cacheMetrics, ioMetrics)
Review Comment:
Should we drop `orc` from `bufferManagerOrc` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4030077865
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java:
##########
@@ -350,6 +353,11 @@ public void start() {
private boolean checkOrcSchemaEvolution() {
SchemaEvolution evolution = rp.getSchemaEvolution();
+ if (evolution == null) {
+ // No ORC-style schema evolution to validate (e.g. native parquet path);
Review Comment:
why we check non-ORC schema inside `checkOrcSchemaEvolution` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4155488512
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ParquetEncodedDataConsumer.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.decode;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.type.DataTypePhysicalVariation;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.impl.ColumnVectorBatch;
+import org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl;
+import org.apache.hadoop.hive.llap.io.decode.ColumnVectorProducer.Includes;
+import org.apache.hadoop.hive.llap.io.encoded.ParquetEncodedColumnBatch;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonIOMetrics;
+import org.apache.hadoop.hive.ql.exec.vector.ColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.orc.encoded.Consumer;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetRowGroupDecoder;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedColumnReader;
+import org.apache.hadoop.hive.serde2.ColumnProjectionUtils;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
+import org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.parquet.HadoopReadOptions;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.compression.CompressionCodecFactory;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.metadata.FileMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+
+import com.google.common.base.Strings;
+
+/**
+ * The Parquet counterpart of {@link OrcEncodedDataConsumer}: it turns one row group's worth of
+ * cached column-chunk buffers ({@link ParquetEncodedColumnBatch}) into {@link ColumnVectorBatch}es
+ * and hands them downstream.
+ *
+ * <p>Rather than reimplementing Parquet value decoding, this consumer builds a
+ * {@link ParquetCachedPageReadStore} directly over the cached buffers (pages are parsed in place and
+ * decompressed lazily, no ParquetFileReader and no whole-chunk copy) and drives the very same
+ * {@link VectorizedColumnReader}s the non-cached vectorized reader uses (via {@link ParquetRowGroupDecoder}).
+ */
+public class ParquetEncodedDataConsumer
+ extends EncodedDataConsumer<Object, ParquetEncodedColumnBatch> {
+
+ private final Configuration jobConf;
+ private final boolean useDecimal64ColumnVectors;
+ private final CompressionCodecFactory codecFactory;
+ private final ParquetMetadataConverter converter;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private Path path;
+ private Map<String, Object> initialDefaults;
+
+ // Derived lazily (once) from the footer + job conf, then reused across row groups.
+ private List<TypeInfo> columnTypesList;
+ private List<Integer> colsToInclude;
+ private boolean readAllColumns;
+ private boolean skipTimestampConversion;
+ private boolean skipProlepticConversion;
+ private boolean legacyConversionEnabled;
+ private ZoneId writerTimezone;
+ private boolean schemaInitialized = false;
+
+ public ParquetEncodedDataConsumer(Consumer<ColumnVectorBatch> consumer, Includes includes,
+ QueryFragmentCounters counters, LlapDaemonIOMetrics ioMetrics, Configuration jobConf) {
+ super(consumer, includes.getPhysicalColumnIds().size(), ioMetrics, counters);
+ this.jobConf = jobConf;
+ this.useDecimal64ColumnVectors = HiveConf.getVar(jobConf,
+ ConfVars.HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_ENABLED)
+ .equalsIgnoreCase(HiveConf.HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_DECIMAL_64);
+ ParquetReadOptions options = HadoopReadOptions.builder(jobConf).build();
+ this.codecFactory = options.getCodecFactory();
+ this.converter = new ParquetMetadataConverter(options);
+ }
+
+ public void setFileMetadata(ParquetMetadata footer, MessageType requestedSchema, Path path) {
+ this.footer = footer;
+ this.requestedSchema = requestedSchema;
+ this.path = path;
+ }
+
+ public void setInitialDefaults(Map<String, Object> initialDefaults) {
+ this.initialDefaults = initialDefaults;
+ }
+
+ /**
+ * Derives the Hive type / projection lists and the conversion flags the same way
+ * {@link org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase} does from the job conf,
+ * so the shared {@link ParquetRowGroupDecoder} produces identical readers.
+ */
+ private void initSchema() {
+ FileMetaData fileMetaData = footer.getFileMetaData();
+ Map<String, String> kvMeta = fileMetaData.getKeyValueMetaData();
+
+ this.columnTypesList =
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES));
+ this.colsToInclude = ColumnProjectionUtils.getReadColumnIDs(jobConf);
+ this.readAllColumns = ColumnProjectionUtils.isReadAllColumns(jobConf);
+
+ this.writerTimezone = DataWritableReadSupport.getWriterTimeZoneId(kvMeta);
+ if (HiveConf.getBoolVar(jobConf, ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION)) {
+ this.skipTimestampConversion =
+ !Strings.nullToEmpty(fileMetaData.getCreatedBy()).startsWith("parquet-mr");
+ } else {
+ this.skipTimestampConversion = false;
+ }
+ Boolean proleptic = DataWritableReadSupport.getWriterDateProleptic(kvMeta);
+ if (proleptic == null) {
+ proleptic = HiveConf.getBoolVar(jobConf, ConfVars.HIVE_PARQUET_DATE_PROLEPTIC_GREGORIAN_DEFAULT);
+ }
+ this.skipProlepticConversion = proleptic;
+ this.legacyConversionEnabled = DataWritableReadSupport.getZoneConversionLegacy(kvMeta, jobConf);
+
+ this.schemaInitialized = true;
+ }
+
+ @Override
+ protected void decodeBatch(ParquetEncodedColumnBatch batch,
+ Consumer<ColumnVectorBatch> downstreamConsumer) throws InterruptedException {
+ if (!schemaInitialized) {
+ initSchema();
+ }
+
+ long startTime = counters.startTimeCounter();
+ // NOSONAR - S2093 does not apply: codecFactory is a per-consumer field with a per-batch
+ // release() (not close()); try-with-resources would tie it to this scope, which is wrong.
+ try { // NOSONAR
+ PageReadStore pages = new ParquetCachedPageReadStore(footer, batch, codecFactory, converter);
+
+ VectorizedColumnReader[] columnReaders =
+ new ParquetRowGroupDecoder(footer.getFileMetaData().getSchema(), initialDefaults).buildColumnReaders(
+ pages, requestedSchema, columnTypesList, colsToInclude, readAllColumns,
+ new ParquetRowGroupDecoder.TimestampConversionOptions(skipTimestampConversion,
+ writerTimezone, skipProlepticConversion, legacyConversionEnabled));
+
+ long rowsLeft = pages.getRowCount();
+ int batches = 0;
+ while (rowsLeft > 0) {
+ int batchSize = (int) Math.min(VectorizedRowBatch.DEFAULT_SIZE, rowsLeft);
+
+ ColumnVectorBatch cvb = takeBatch(batchSize);
+
+ // columnReaders[i] is requestedSchema field i, i.e. the i-th projected column.
+ for (int i = 0; i < columnReaders.length; ++i) {
+ if (columnReaders[i] == null) {
+ continue;
+ }
+ TypeInfo columnType = columnTypesList.get(readAllColumns ? i : colsToInclude.get(i));
+ ColumnVector cv = prepareColumnVector(cvb, i, columnType, batchSize);
+ columnReaders[i].readBatch(batchSize, cv, columnType);
+ }
+
+ downstreamConsumer.consumeData(cvb);
+ counters.incrCounter(LlapIOCounters.ROWS_EMITTED, batchSize);
+ rowsLeft -= batchSize;
+ ++batches;
+ }
+ counters.incrWallClockCounter(LlapIOCounters.DECODE_TIME_NS, startTime);
+ counters.incrCounter(LlapIOCounters.NUM_VECTOR_BATCHES, batches);
+ counters.incrCounter(LlapIOCounters.NUM_DECODED_BATCHES);
+ } catch (IOException | RuntimeException e) {
+ // parquet reports decode failures as runtime ParquetDecodingException.
+ LlapIoImpl.LOG.error("Parquet decodeBatch failed for rowGroup " + batch.rowGroupIx() + " of " + path, e);
+ downstreamConsumer.setError(e);
Review Comment:
maybe
````
downstreamConsumer.setError(new IOException(
"Parquet decode failed for row group " + batch.rowGroupIx() + " of " + path, e));
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4156103977
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ // Fallback path: no cache key, so we read status + footer straight from HDFS. Time both
+ // under HDFS_TIME_NS to match how ORC accounts for cache-miss footer reads.
+ final FileSystem fs = path.getFileSystem(jobConf);
+ long hdfsStart = counters.startTimeCounter();
+ final FileStatus stat = fs.getFileStatus(path);
+ counters.recordHdfsTime(hdfsStart);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ hdfsStart = counters.startTimeCounter();
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ counters.recordHdfsTime(hdfsStart);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ long startTime = counters.startTimeCounter();
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Exception e) {
+ // A shutdown-triggered InterruptedException must not be silently reported as an ordinary
+ // consumer error: preserve the interrupt on the thread so any caller sees it.
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ }
+ consumer.setError(e);
+ } finally {
+ counters.incrWallClockCounter(LlapIOCounters.TOTAL_IO_TIME_NS, startTime);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ // The LLAP IO summary keys off TABLE / FILE / STRIPES; set them the same way ORC does so a
+ // native Parquet fragment shows up in the summary with the same fields populated.
+ if (cacheTag != null) {
+ counters.setDesc(QueryFragmentCounters.Desc.TABLE, cacheTag.getTableName());
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.FILE, path
+ + (fileKey == null ? "" : " (" + fileKey + ")"));
+
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = selectRowGroups(blocks, fileSchema);
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ // STRIPES is the ORC term; for Parquet the analog is row groups. Reuse the same descriptor so
+ // the summary layout stays common and we don't have to teach the reporter about a new field.
+ counters.setDesc(QueryFragmentCounters.Desc.STRIPES, "0," + selected.size());
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /**
+ * The row groups this split has to read: the ones whose first data page falls inside the split, as
+ * Parquet's own input format assigns them, minus any the pushed-down predicate rules out on its
+ * statistics. Returned in file order, since the fetch pipeline reads them in that order.
+ */
+ private List<BlockMetaData> selectRowGroups(List<BlockMetaData> blocks, MessageType fileSchema)
+ throws IOException {
+ long splitStart = split.getStart();
+ long splitEnd = splitStart + split.getLength();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().getFirst().getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate == null) {
+ return selected;
+ }
+ return RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Whether the query needs row-lineage virtual columns ({@code ROW__LINEAGE__ID} /
+ * {@code LAST__UPDATED__SEQUENCE__NUMBER}) that the fallback reader augments the requested schema
+ * with via {@link RowLineageUtils#getRequestedSchemaWithRowLineageColumns}. The native pipeline
+ * doesn't propagate that augmentation through {@code Includes} yet, so we defer to the fallback
+ * reader instead of quietly emitting nulls in the lineage slots.
+ */
+ public boolean needsRowLineage() {
+ VectorizedRowBatchCtx rbCtx = Utilities.getVectorizedRowBatchCtx(jobConf);
+ if (rbCtx == null) {
+ return false;
+ }
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ return RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema, VirtualColumn.ROW_LINEAGE_ID)
+ || RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema,
+ VirtualColumn.LAST_UPDATED_SEQUENCE_NUMBER);
+ }
+
+ /**
+ * Leaf-column positions of the requested fields; {@code BlockMetaData.getColumns()} is a flat list
+ * of leaves in file-schema order, so a top-level primitive at ordinal {@code k} in the file schema
+ * may sit at a very different leaf index (e.g. a nested group of two leaves before it shifts it
+ * from {@code 1} to {@code 2}). We only reach this method when every requested field is a
+ * top-level primitive ({@link #projectsNestedTypes()} would have forced a fallback otherwise), so
+ * each requested field maps to exactly one leaf whose path is a single segment.
+ */
+ static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
+ List<ColumnDescriptor> allLeaves = fileSchema.getColumns();
+ List<Integer> selected = new ArrayList<>();
+ for (Type field : requestedSchema.getFields()) {
+ if (!fileSchema.containsField(field.getName())) {
+ continue;
+ }
+ for (int i = 0; i < allLeaves.size(); ++i) {
+ String[] path = allLeaves.get(i).getPath();
+ // Single-segment path == top-level primitive; skip leaves buried inside a group.
+ if (path.length == 1 && path[0].equals(field.getName())) {
+ selected.add(i);
+ break;
+ }
+ }
+ }
+ return selected.stream().mapToInt(Integer::intValue).toArray();
+ }
+
+ private static long bytes(int[] projected, BlockMetaData block) {
+ long total = 0;
+ for (int leaf : projected) {
+ total += block.getColumns().get(leaf).getTotalSize();
+ }
+ return total;
+ }
+
+ private static long bytes(Fetch fetch) {
+ long total = 0;
+ for (ColumnChunkMetaData chunk : fetch.batch.chunks()) {
+ total += chunk.getTotalSize();
+ }
+ return total;
+ }
+
+ /** One row group on its way in: buffers planned, missing ranges requested, not yet decoded. */
+ private static final class Fetch {
+ private final ParquetEncodedColumnBatch batch = new ParquetEncodedColumnBatch();
+ private final List<ColumnPlan> columns = new ArrayList<>();
+ private final List<Part> misses = new ArrayList<>();
+ private final List<Run> runs = new ArrayList<>();
+ }
+
+ /** The buffers covering one column chunk, in file order, and which stretches of them are new. */
+ private record ColumnPlan(List<Part> parts, List<MissRun> missRuns) {
+ ColumnPlan() {
+ this(new ArrayList<>(), new ArrayList<>());
+ }
+ }
+
+ /** One cache buffer's worth of a column chunk: a hit handed back by the cache, or a miss to fill. */
+ private static final class Part {
+ private MemoryBuffer buffer;
+ private final DiskRange range;
+ /** We hold one ref to release; until then a miss is a raw allocation to free. */
+ private boolean owned;
+
+ Part(MemoryBuffer buffer, DiskRange range, boolean owned) {
+ this.buffer = buffer;
+ this.range = range;
+ this.owned = owned;
+ }
+ }
+
+ /** {@code count} consecutive parts covering one contiguous missing sub-range; cached as a unit. */
+ private record MissRun(int firstPart, int count) {
+ }
+
+ /** One vectored range and the cache buffers it fills, in file order. */
+ private record Run(FileRange range, List<Part> parts) {
+ }
+
+ private Fetch startFetch(FSDataInputStream fileStream, ParquetRangeBuffers buffers, Allocator allocator,
+ int[] projected, BlockMetaData block, int rg)
+ throws IOException {
+ Fetch fetch = new Fetch();
+ ColumnChunkMetaData[] chunks = new ColumnChunkMetaData[projected.length];
+ for (int pc = 0; pc < projected.length; ++pc) {
+ chunks[pc] = block.getColumns().get(projected[pc]);
+ }
+ fetch.batch.init(fileKey, rg, chunks);
+ try {
+ for (ColumnChunkMetaData chunk : chunks) {
+ ColumnPlan column = new ColumnPlan();
+ fetch.columns.add(column);
+ planColumnChunk(allocator, chunk.getStartingPos(),
+ chunk.getStartingPos() + chunk.getTotalSize(), column, fetch.misses);
+ }
+ // The vectored dispatch is where non-async FS impls actually read; count it under HDFS_TIME.
+ long hdfsStart = counters.startTimeCounter();
+ requestMisses(fileStream, buffers, fetch, layout.maxRangeBytes());
+ counters.recordHdfsTime(hdfsStart);
+ } catch (Exception e) {
+ abandon(allocator, fetch);
+ throw e;
+ }
+ return fetch;
+ }
+
+ private void finishFetch(Allocator allocator, ParquetRangeBuffers buffers, Fetch fetch)
+ throws IOException, InterruptedException {
+ try {
+ // Awaiting the vectored futures is where the miss bytes actually arrive from HDFS.
+ long hdfsStart = counters.startTimeCounter();
+ receiveMisses(buffers, fetch);
+ counters.recordHdfsTime(hdfsStart);
+ for (ColumnPlan column : fetch.columns) {
+ putColumn(allocator, column);
+ }
+ for (int pc = 0; pc < fetch.columns.size(); ++pc) {
+ assemble(fetch.batch, pc, fetch.columns.get(pc).parts);
+ }
+ // consumeData returns the batch on success; after a throw it is still ours.
+ consumer.consumeData(fetch.batch);
+ } catch (Exception e) {
+ abandon(allocator, fetch);
+ throw e;
+ }
+ }
+
+ /** Drops a fetch that will not be decoded: outstanding requests are cancelled, buffers released. */
+ private void abandon(Allocator allocator, Fetch fetch) {
+ for (Run run : fetch.runs) {
+ run.range.getData().cancel(true);
+ }
+ for (ColumnPlan column : fetch.columns) {
+ for (Part part : column.parts) {
+ if (part.owned) {
+ bufferManager.decRefBuffer(part.buffer);
+ } else {
+ allocator.deallocate(part.buffer);
+ }
+ }
+ }
+ }
+
+ /** Package-private so a test can observe the read pattern. */
+ FSDataInputStream openFile(FileSystem fs) throws IOException {
+ return fs.open(path);
+ }
+
+ /** Lets the allocator abandon a wait for memory once the fragment is cancelled. */
+ private void allocateMultiple(Allocator allocator, MemoryBuffer[] dest, int size) {
+ if (allocator instanceof StoppableAllocator stoppable) {
+ stoppable.allocateMultiple(dest, size, DATA_BUFFER_FACTORY, isStopped);
+ } else {
+ allocator.allocateMultiple(dest, size, DATA_BUFFER_FACTORY);
+ }
+ }
+
+ /**
+ * Works out which buffers cover column chunk {@code [start, end)} without touching the file: cache
+ * hits as returned by getFileData, and freshly allocated power-of-two buffers for everything
+ * missing. Misses are appended to {@code allMisses} so the whole row group can be read at once.
+ */
+ private void planColumnChunk(Allocator allocator, long start, long end,
+ ColumnPlan column, List<Part> allMisses) throws IOException {
+ DiskRangeList head = new DiskRangeList(start, end);
+ if (fileKey != null) {
+ head = lowLevelCache.getFileData(fileKey, head, 0, ParquetCacheLayout.CACHE_CHUNK_FACTORY,
+ counters, new BooleanRef());
+ }
+ DiskRangeList current = head;
+ try {
+ for (; current != null; current = current.next) {
+ if (current.hasData()) {
+ column.parts.add(new Part(((CacheChunk) current).getBuffer(), current, true));
+ } else {
+ planMissRun(allocator, current, column, allMisses);
+ }
+ }
+ } catch (Exception e) {
+ // Hits past the failure point are still locked by getFileData; the caller only knows about
+ // the parts already recorded.
+ for (current = current.next; current != null; current = current.next) {
+ if (current.hasData()) {
+ bufferManager.decRefBuffer(((CacheChunk) current).getBuffer());
+ }
+ }
+ throw e;
+ }
+ }
+
+ /**
+ * Records one missing sub-range as a run of freshly allocated power-of-two buffers, and appends
+ * each part to {@code allMisses} so the caller can request them all in one vectored call.
+ */
+ private void planMissRun(Allocator allocator, DiskRangeList missing, ColumnPlan column,
+ List<Part> allMisses) throws IOException {
+ LlapHiveUtils.throwIfCacheOnlyRead(cacheOnly);
+ int[] sizes = layout.bufferSizes(missing.getEnd() - missing.getOffset());
+ column.missRuns.add(new MissRun(column.parts.size(), sizes.length));
+ long partFrom = missing.getOffset();
+ for (int size : sizes) {
+ MemoryBuffer[] one = new MemoryBuffer[1];
+ allocateMultiple(allocator, one, size);
+ // The cache accounts and serves the bytes up to the buffer's limit.
+ ByteBuffer raw = one[0].getByteBufferRaw();
+ raw.limit(raw.position() + size);
+ Part part = new Part(one[0], new DiskRange(partFrom, partFrom + size), false);
+ column.parts.add(part);
+ allMisses.add(part);
+ partFrom += size;
+ }
+ }
+
+ /**
+ * Requests every missing buffer of a row group in one vectored call, one range per run of
+ * adjacent buffers. Column chunks sit back to back in the file, so a projection that keeps
+ * neighbouring columns reads them together; runs are capped so the row group arrives as
+ * several concurrent requests rather than one. A filesystem without a vectored implementation
+ * reads the ranges in turn.
+ */
+ private static void requestMisses(FSDataInputStream fileStream, ParquetRangeBuffers buffers, Fetch fetch,
+ int maxRange) throws IOException {
+ List<Part> misses = fetch.misses;
+ if (misses.isEmpty()) {
+ return;
+ }
+ misses.sort(Comparator.comparingLong(part -> part.range.getOffset()));
+ List<Run> runs = new ArrayList<>();
+ int i = 0;
+ while (i < misses.size()) {
+ long from = misses.get(i).range.getOffset();
+ long to = misses.get(i).range.getEnd();
+ int j = i + 1;
+ while (j < misses.size()) {
+ DiskRange next = misses.get(j).range;
+ if (next.getOffset() != to || next.getEnd() - from > maxRange) {
+ break;
+ }
+ to = next.getEnd();
+ ++j;
+ }
+ runs.add(new Run(FileRange.createFileRange(from, (int) (to - from)), misses.subList(i, j)));
+ i = j;
+ }
+ List<FileRange> ranges = runs.stream().map(run -> run.range).collect(Collectors.toList());
+ fileStream.readVectored(ranges, buffers::allocate, buffers::release);
+ // Only requests the stream accepted are the fetch's to wait for or cancel.
+ fetch.runs.addAll(runs);
+ }
+
+ /** Waits for the requested runs and copies each into the cache buffers it covers. */
+ private static void receiveMisses(ParquetRangeBuffers buffers, Fetch fetch) throws IOException {
+ for (Run run : fetch.runs) {
+ ByteBuffer data = FutureIO.awaitFuture(run.range.getData());
+ for (Part part : run.parts) {
+ int length = part.range.getLength();
+ ByteBuffer src = data.duplicate();
+ src.position(data.position() + (int) (part.range.getOffset() - run.range.getOffset()));
+ src.limit(src.position() + length);
+ part.buffer.getByteBufferRaw().duplicate().put(src);
+ }
+ buffers.release(data);
+ }
+ }
+
+ /** Hands one column's freshly read buffers to the cache; every part then carries a ref we own. */
+ private void putColumn(Allocator allocator, ColumnPlan column) {
Review Comment:
cacheChunk ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4156647117
##########
ql/src/java/org/apache/hadoop/hive/llap/ParquetCacheLayout.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.llap;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.DataCache.DiskRangeListFactory;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+
+/**
+ * How Parquet column chunks are laid out in the LLAP cache and fetched from the file. Everything
+ * that caches a chunk goes through this, so a reader finding only part of one cached re-caches
+ * the gap on the boundaries the first reader used. A reader builds one from the daemon's allocator
+ * and configuration and uses it for every chunk, which is what keeps its layout consistent.
+ */
+public final class ParquetCacheLayout {
+
+ /**
+ * An object store fetches separate ranges concurrently but never splits one, so cutting buys
+ * parallelism while a large range amortises the request. Measured against S3 the knee is here:
+ * below 4 Mb throughput drops sharply, above 8 Mb it is flat. Trino also splits at 8 Mb.
+ */
+ private static final int MIN_RANGE_CAP_BYTES = 8 << 20;
+
+ public static final DiskRangeListFactory CACHE_CHUNK_FACTORY = CacheChunk::new;
+
+ private final int maxBuffer;
+ private final int minBuffer;
+ private final int maxRange;
Review Comment:
rangeCap ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4154456235
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java:
##########
@@ -123,19 +125,15 @@ public RecordReader<NullWritable, VectorizedRowBatch> getRecordReader(
cvp, executor, sourceInputFormat, sourceSerDe, reporter, daemonConf);
if (rr == null) {
// Reader-specific incompatibility like SMB or schema evolution.
+ if (sourceInputFormat instanceof LlapCacheOnlyInputFormatInterface) {
+ LlapProxy.getIo().initCacheOnlyInputFormat(sourceInputFormat);
+ }
return sourceInputFormat.getRecordReader(split, job, reporter);
}
- // For non-vectorized operator case, wrap the reader if possible.
- RecordReader<NullWritable, VectorizedRowBatch> result = rr;
- if (!Utilities.getIsVectorized(job)) {
- result = null;
- if (HiveConf.getBoolVar(job, ConfVars.LLAP_IO_ROW_WRAPPER_ENABLED)) {
- result = wrapLlapReader(tableIncludedCols, rr, split);
- }
- if (result == null) {
- // Cannot wrap a reader for non-vectorized pipeline.
- return sourceInputFormat.getRecordReader(split, job, reporter);
- }
+ RecordReader<NullWritable, VectorizedRowBatch> result =
Review Comment:
wrapOrFallback never falls back!
what if we refactor
````
public RecordReader<NullWritable, VectorizedRowBatch> getRecordReader(
boolean vectorized = Utilities.getIsVectorized(job);
if (!vectorized && !canWrapToRows(job)) {
// The pipeline wants rows and no batch-to-row wrapper exists for this format; decide here,
// before a reader (and its read pipeline, footer fetch, counters) is built for nothing.
return fallback(split, job, reporter);
}
....
LlapRecordReader rr = LlapRecordReader.create(job, fileSplit, tableIncludedCols, hostName,
cvp, executor, sourceInputFormat, sourceSerDe, reporter, daemonConf);
if (rr == null) {
// Reader-specific incompatibility: SMB, schema evolution, or the producer declined the split.
return fallback(split, job, reporter);
}
RecordReader<NullWritable, VectorizedRowBatch> result =
vectorized ? rr : wrapToRows(rr, tableIncludedCols, split);
....
}
/** The non-vectorized pipeline can take LLAP batches only through the source format's row wrapper. */
private boolean canWrapToRows(JobConf job) {
return HiveConf.getBoolVar(job, ConfVars.LLAP_IO_ROW_WRAPPER_ENABLED)
&& sourceInputFormat instanceof BatchToRowInputFormat;
}
/** Only called when {@link #canWrapToRows} held, so the cast is safe. */
private RecordReader<NullWritable, VectorizedRowBatch> wrapToRows(
LlapRecordReader rr, List<Integer> includedCols, InputSplit split) throws IOException {
LlapIoImpl.LOG.info("Using batch-to-row converter for split: " + split);
return bogusCast(((BatchToRowInputFormat) sourceInputFormat).getWrapper(
rr, rr.getVectorizedRowBatchCtx(), includedCols));
}
/**
* Serves the split with the source format. A cache-only format still gets the LLAP caches
* injected, so a declined split reads through the same cache entries as the native path.
*/
private RecordReader<NullWritable, VectorizedRowBatch> fallback(
InputSplit split, JobConf job, Reporter reporter) throws IOException {
if (sourceInputFormat instanceof LlapCacheOnlyInputFormatInterface) {
HiveInputFormat.injectLlapCaches(sourceInputFormat, LlapProxy.getIo(), job);
}
return sourceInputFormat.getRecordReader(split, job, reporter);
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6793:
URL: https://github.com/apache/hive/pull/6793#issuecomment-5912759063
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6793) **Quality Gate passed**
Issues
 [130 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6793&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6793&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6793&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6793&metric=new_coverage&view=list)
 [1.1% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6793&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6793)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog merged PR #6793:
URL: https://github.com/apache/hive/pull/6793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6793:
URL: https://github.com/apache/hive/pull/6793#issuecomment-6370458276
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6793) **Quality Gate passed**
Issues
 [80 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6793&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6793&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6793&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6793&metric=new_coverage&view=list)
 [0.1% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6793&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6793)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4196254665
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ParquetColumnVectorProducer.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.decode;
+
+import java.io.IOException;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.impl.ColumnVectorBatch;
+import org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl;
+import org.apache.hadoop.hive.llap.io.encoded.ParquetEncodedDataReader;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonCacheMetrics;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonIOMetrics;
+import org.apache.hadoop.hive.ql.io.orc.encoded.Consumer;
+import org.apache.hadoop.hive.ql.io.sarg.SearchArgument;
+import org.apache.hadoop.hive.ql.plan.PartitionDesc;
+import org.apache.hadoop.hive.serde2.Deserializer;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.InputFormat;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.Reporter;
+
+public class ParquetColumnVectorProducer implements ColumnVectorProducer {
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration conf;
+ private final LlapDaemonCacheMetrics cacheMetrics;
+ private final LlapDaemonIOMetrics ioMetrics;
+
+ public ParquetColumnVectorProducer(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration conf, LlapDaemonCacheMetrics cacheMetrics, LlapDaemonIOMetrics ioMetrics) {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.conf = conf;
+ this.cacheMetrics = cacheMetrics;
+ this.ioMetrics = ioMetrics;
+ }
+
+ @Override
+ public ReadPipeline createReadPipeline(Consumer<ColumnVectorBatch> consumer, FileSplit split,
+ Includes includes, SearchArgument sarg, QueryFragmentCounters counters,
+ SchemaEvolutionFactory sef, InputFormat<?, ?> sourceInputFormat, Deserializer sourceSerDe,
+ Reporter reporter, JobConf job, Map<Path, PartitionDesc> parts) {
+ try {
+ cacheMetrics.incrCacheReadRequests();
+ ParquetEncodedDataConsumer edc =
+ new ParquetEncodedDataConsumer(consumer, includes, counters, ioMetrics, job);
+ ParquetEncodedDataReader reader = new ParquetEncodedDataReader(
+ lowLevelCache, bufferManager, conf, job, split, edc, counters);
+ reader.loadFooter();
+ // Null makes LlapInputFormat fall back to the normal VectorizedParquetRecordReader.
+ if (reader.projectsNestedTypes()) {
+ LlapIoImpl.LOG.info("Parquet native cache: falling back to normal reader for {} due to "
+ + "nested types in the projection", split.getPath());
+ return null;
+ }
+ /*
+ * Fallback path still caches (footer via LlapProxy, data via LlapCacheAwareFs); what's lost
+ * for this split is vectored reads, row-group lookahead, and its LLAP IO summary counters.
+ * Guard fires per split only for Iceberg-Parquet files that physically carry _row_id /
+ * _last_updated_sequence_number while the query projects those virtual columns.
+ *
+ * Table shape Fires?
+ * ------------------------------------------------------------ ------
+ * Non-Iceberg Parquet never
+ * Iceberg-Parquet, row lineage disabled never
+ * Iceberg-Parquet, row lineage on, file has no _row_id yet never
+ * Iceberg-Parquet, row lineage on, file has lineage columns yes, per split
+ */
+ if (reader.needsRowLineage()) {
+ LlapIoImpl.LOG.info("Parquet native cache: falling back to normal reader for {} due to "
+ + "row-lineage virtual columns", split.getPath());
+ return null;
+ }
+ edc.init(reader, reader);
Review Comment:
ack, created https://issues.apache.org/jira/browse/HIVE-30141
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4184048111
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java:
##########
@@ -123,19 +125,15 @@ public RecordReader<NullWritable, VectorizedRowBatch> getRecordReader(
cvp, executor, sourceInputFormat, sourceSerDe, reporter, daemonConf);
if (rr == null) {
// Reader-specific incompatibility like SMB or schema evolution.
+ if (sourceInputFormat instanceof LlapCacheOnlyInputFormatInterface) {
+ LlapProxy.getIo().initCacheOnlyInputFormat(sourceInputFormat);
+ }
return sourceInputFormat.getRecordReader(split, job, reporter);
}
- // For non-vectorized operator case, wrap the reader if possible.
- RecordReader<NullWritable, VectorizedRowBatch> result = rr;
- if (!Utilities.getIsVectorized(job)) {
- result = null;
- if (HiveConf.getBoolVar(job, ConfVars.LLAP_IO_ROW_WRAPPER_ENABLED)) {
- result = wrapLlapReader(tableIncludedCols, rr, split);
- }
- if (result == null) {
- // Cannot wrap a reader for non-vectorized pipeline.
- return sourceInputFormat.getRecordReader(split, job, reporter);
- }
+ RecordReader<NullWritable, VectorizedRowBatch> result =
Review Comment:
ack, it was quite misleading that `wrapOrFallback` didn't actually fall back, just induced circumstances to make the caller able to fallback, suggested refactor makes sense to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4216294766
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java:
##########
@@ -261,7 +261,8 @@ public OrcEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager buff
}
consumer.setUseDecimal64ColumnVectors(HiveConf.getVar(jobConf,
- ConfVars.HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_ENABLED).equalsIgnoreCase("decimal_64"));
+ ConfVars.HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_ENABLED)
+ .equalsIgnoreCase(HiveConf.HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_DECIMAL_64));
Review Comment:
ack, doing it! might be a follow-up, but this change is obvious, I don't mind if it makes the patch bigger a bit
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java:
##########
@@ -261,7 +261,8 @@ public OrcEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager buff
}
consumer.setUseDecimal64ColumnVectors(HiveConf.getVar(jobConf,
- ConfVars.HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_ENABLED).equalsIgnoreCase("decimal_64"));
+ ConfVars.HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_ENABLED)
+ .equalsIgnoreCase(HiveConf.HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_DECIMAL_64));
Review Comment:
ack, doing it now! might be a follow-up, but this change is obvious, I don't mind if it makes the patch bigger a bit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6793:
URL: https://github.com/apache/hive/pull/6793#issuecomment-6264217434
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6793) **Quality Gate passed**
Issues
 [93 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6793&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6793&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6793&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6793&metric=new_coverage&view=list)
 [0.1% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6793&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6793)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4217587072
##########
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetRowGroupDecoder.java:
##########
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.ql.io.parquet.vector;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.parquet.ParquetRuntimeException;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.InvalidSchemaException;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Reusable, stateless helper that builds the {@link VectorizedColumnReader} array for one Parquet
+ * row group, given an already-read {@link PageReadStore}. The logic here was lifted verbatim from
+ * {@link VectorizedParquetRecordReader} so that both the row-by-split reader and the LLAP
+ * cache-backed consumer ({@code ParquetEncodedDataConsumer}) can share a single, behavior-preserving
+ * implementation of the Hive-type-driven Parquet column-reader construction.
+ *
+ * <p>The instance only carries the {@code fileSchema} (used for the schema-evolution check). All
+ * per-row-group state lives in the supplied {@link PageReadStore}.
+ */
+public class ParquetRowGroupDecoder {
+
+ private static final int MAP_DEFINITION_LEVEL_MAX = 3;
+
+ /**
+ * Bag of writer-timezone / proleptic / legacy conversion flags forwarded to every primitive
+ * column reader. Grouped so the reader-construction entry point stays under the parameter cap.
+ */
+ public record TimestampConversionOptions(
+ boolean skipTimestampConversion,
+ ZoneId writerTimezone,
+ boolean skipProlepticConversion,
+ boolean legacyConversionEnabled) {
+ }
+
+ private final MessageType fileSchema;
+ private final Map<String, Object> initialDefaults;
+
+ public ParquetRowGroupDecoder(MessageType fileSchema, Map<String, Object> initialDefaults) {
+ this.fileSchema = fileSchema;
+ this.initialDefaults = initialDefaults;
+ }
+
+ /**
+ * Builds the per-(requested-)column {@link VectorizedColumnReader} array for a row group.
+ *
+ * @param pages the row group's page store (from {@code reader.readRowGroup(..)}
+ * / {@code readNextRowGroup()})
+ * @param requestedSchema the projected Parquet schema being read
+ * @param columnTypesList the Hive type infos for ALL table columns (indexed by table col id)
+ * @param colsToInclude the table column ids being read, in requested-schema field order;
+ * may be empty (e.g. {@code count(*)}), in which case all readers are null
+ * @param readAllColumns whether projection is "read all columns"
+ * @param options writer-timezone / proleptic / legacy conversion flags, see
+ * {@link TimestampConversionOptions}
+ * @return one reader per requested-schema field (null entries where no reader is needed)
+ */
+ public VectorizedColumnReader[] buildColumnReaders(
+ PageReadStore pages,
+ MessageType requestedSchema,
+ List<TypeInfo> columnTypesList,
+ List<Integer> colsToInclude,
+ boolean readAllColumns,
+ TimestampConversionOptions options) throws IOException {
+ List<Type> types = requestedSchema.getFields();
+ VectorizedColumnReader[] columnReaders = new VectorizedColumnReader[types.size()];
+
+ if (!readAllColumns) {
+ // certain queries like select count(*) from table do not have
+ // any projected columns and still have isReadAllColumns as false
+ // in such cases columnReaders are not needed
+ // However, if colsToInclude is not empty we should initialize each columnReader
+ if (!colsToInclude.isEmpty()) {
+ for (int i = 0; i < types.size(); ++i) {
+ columnReaders[i] = buildVectorizedParquetReader(
+ columnTypesList.get(colsToInclude.get(i)), types.get(i), pages,
+ requestedSchema.getColumns(), options, 0, 0);
+ }
+ }
+ } else {
+ for (int i = 0; i < types.size(); ++i) {
+ columnReaders[i] = buildVectorizedParquetReader(columnTypesList.get(i),
+ types.get(i), pages, requestedSchema.getColumns(), options, 0, 0);
+ }
+ }
+ return columnReaders;
+ }
+
+ private static List<ColumnDescriptor> getAllColumnDescriptorByType(
+ int depth,
+ Type type,
+ List<ColumnDescriptor> columns) throws ParquetRuntimeException {
+ List<ColumnDescriptor> res = new ArrayList<>();
+ for (ColumnDescriptor descriptor : columns) {
+ if (depth >= descriptor.getPath().length) {
+ throw new InvalidSchemaException("Corrupted Parquet schema");
+ }
+ if (type.getName().equals(descriptor.getPath()[depth])) {
+ res.add(descriptor);
+ }
+ }
+ return res;
+ }
+
+ // Nested types are unsupported here on purpose; callers detect that in advance
+ // (ParquetEncodedDataReader.projectsNestedTypes) and route the query to the non-native reader.
+ private static PrimitiveType getElementType(Type type) {
+ if (type.isPrimitive()) {
+ return type.asPrimitiveType();
+ }
+ if (type.asGroupType().getFields().size() > 1) {
+ throw new UnsupportedOperationException(
+ "Current Parquet Vectorization reader doesn't support nested type");
+ }
+
+ Type childType = type.asGroupType().getFields().getFirst();
+
+ // Parquet file generated using thrift may have child type as PrimitiveType
+ if (childType.isPrimitive()) {
+ return childType.asPrimitiveType();
+ } else {
+ return childType.asGroupType().getFields().getFirst().asPrimitiveType();
+ }
+ }
+
+ // Build VectorizedParquetColumnReader via Hive typeInfo and Parquet schema
+ private VectorizedColumnReader buildVectorizedParquetReader(
+ TypeInfo typeInfo,
+ Type type,
+ PageReadStore pages,
+ List<ColumnDescriptor> columnDescriptors,
+ TimestampConversionOptions options,
+ int depth, int currentDefLevel) throws IOException {
+ int typeDefLevel = currentDefLevel;
+ if (type.isRepetition(Type.Repetition.OPTIONAL) || type.isRepetition(Type.Repetition.REPEATED)) {
+ typeDefLevel++;
+ }
+ List<ColumnDescriptor> descriptors =
+ getAllColumnDescriptorByType(depth, type, columnDescriptors);
+ // Support for schema evolution: if the column from the current
+ // query schema is not present in the file schema, return a dummy
+ // reader that produces nulls. This allows queries to proceed even
+ // when new columns have been added after the file was written.
+ if (!fileSchema.getColumns().contains(descriptors.getFirst())) {
+ return new VectorizedDummyColumnReader(Optional.ofNullable(initialDefaults)
+ .map(defaults -> defaults.getOrDefault(descriptors.getFirst().getPath()[0], null)).orElse(null));
+ }
+ switch (typeInfo.getCategory()) {
+ case PRIMITIVE:
Review Comment:
fixed to:
```
switch (typeInfo.getCategory()) {
case PRIMITIVE:
```
##########
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetRowGroupDecoder.java:
##########
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.ql.io.parquet.vector;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.parquet.ParquetRuntimeException;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.InvalidSchemaException;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Reusable, stateless helper that builds the {@link VectorizedColumnReader} array for one Parquet
+ * row group, given an already-read {@link PageReadStore}. The logic here was lifted verbatim from
+ * {@link VectorizedParquetRecordReader} so that both the row-by-split reader and the LLAP
+ * cache-backed consumer ({@code ParquetEncodedDataConsumer}) can share a single, behavior-preserving
+ * implementation of the Hive-type-driven Parquet column-reader construction.
+ *
+ * <p>The instance only carries the {@code fileSchema} (used for the schema-evolution check). All
+ * per-row-group state lives in the supplied {@link PageReadStore}.
+ */
+public class ParquetRowGroupDecoder {
+
+ private static final int MAP_DEFINITION_LEVEL_MAX = 3;
+
+ /**
+ * Bag of writer-timezone / proleptic / legacy conversion flags forwarded to every primitive
+ * column reader. Grouped so the reader-construction entry point stays under the parameter cap.
+ */
+ public record TimestampConversionOptions(
+ boolean skipTimestampConversion,
+ ZoneId writerTimezone,
+ boolean skipProlepticConversion,
+ boolean legacyConversionEnabled) {
+ }
+
+ private final MessageType fileSchema;
+ private final Map<String, Object> initialDefaults;
+
+ public ParquetRowGroupDecoder(MessageType fileSchema, Map<String, Object> initialDefaults) {
+ this.fileSchema = fileSchema;
+ this.initialDefaults = initialDefaults;
+ }
+
+ /**
+ * Builds the per-(requested-)column {@link VectorizedColumnReader} array for a row group.
+ *
+ * @param pages the row group's page store (from {@code reader.readRowGroup(..)}
+ * / {@code readNextRowGroup()})
+ * @param requestedSchema the projected Parquet schema being read
+ * @param columnTypesList the Hive type infos for ALL table columns (indexed by table col id)
+ * @param colsToInclude the table column ids being read, in requested-schema field order;
+ * may be empty (e.g. {@code count(*)}), in which case all readers are null
+ * @param readAllColumns whether projection is "read all columns"
+ * @param options writer-timezone / proleptic / legacy conversion flags, see
+ * {@link TimestampConversionOptions}
+ * @return one reader per requested-schema field (null entries where no reader is needed)
+ */
+ public VectorizedColumnReader[] buildColumnReaders(
+ PageReadStore pages,
+ MessageType requestedSchema,
+ List<TypeInfo> columnTypesList,
+ List<Integer> colsToInclude,
+ boolean readAllColumns,
+ TimestampConversionOptions options) throws IOException {
+ List<Type> types = requestedSchema.getFields();
+ VectorizedColumnReader[] columnReaders = new VectorizedColumnReader[types.size()];
+
+ if (!readAllColumns) {
+ // certain queries like select count(*) from table do not have
+ // any projected columns and still have isReadAllColumns as false
+ // in such cases columnReaders are not needed
+ // However, if colsToInclude is not empty we should initialize each columnReader
+ if (!colsToInclude.isEmpty()) {
+ for (int i = 0; i < types.size(); ++i) {
+ columnReaders[i] = buildVectorizedParquetReader(
+ columnTypesList.get(colsToInclude.get(i)), types.get(i), pages,
+ requestedSchema.getColumns(), options, 0, 0);
+ }
+ }
+ } else {
+ for (int i = 0; i < types.size(); ++i) {
+ columnReaders[i] = buildVectorizedParquetReader(columnTypesList.get(i),
+ types.get(i), pages, requestedSchema.getColumns(), options, 0, 0);
+ }
+ }
+ return columnReaders;
+ }
+
+ private static List<ColumnDescriptor> getAllColumnDescriptorByType(
+ int depth,
+ Type type,
+ List<ColumnDescriptor> columns) throws ParquetRuntimeException {
+ List<ColumnDescriptor> res = new ArrayList<>();
+ for (ColumnDescriptor descriptor : columns) {
+ if (depth >= descriptor.getPath().length) {
+ throw new InvalidSchemaException("Corrupted Parquet schema");
+ }
+ if (type.getName().equals(descriptor.getPath()[depth])) {
+ res.add(descriptor);
+ }
+ }
+ return res;
+ }
+
+ // Nested types are unsupported here on purpose; callers detect that in advance
+ // (ParquetEncodedDataReader.projectsNestedTypes) and route the query to the non-native reader.
+ private static PrimitiveType getElementType(Type type) {
+ if (type.isPrimitive()) {
+ return type.asPrimitiveType();
+ }
+ if (type.asGroupType().getFields().size() > 1) {
+ throw new UnsupportedOperationException(
+ "Current Parquet Vectorization reader doesn't support nested type");
+ }
+
+ Type childType = type.asGroupType().getFields().getFirst();
+
+ // Parquet file generated using thrift may have child type as PrimitiveType
+ if (childType.isPrimitive()) {
+ return childType.asPrimitiveType();
+ } else {
+ return childType.asGroupType().getFields().getFirst().asPrimitiveType();
+ }
+ }
+
+ // Build VectorizedParquetColumnReader via Hive typeInfo and Parquet schema
+ private VectorizedColumnReader buildVectorizedParquetReader(
+ TypeInfo typeInfo,
+ Type type,
+ PageReadStore pages,
+ List<ColumnDescriptor> columnDescriptors,
+ TimestampConversionOptions options,
+ int depth, int currentDefLevel) throws IOException {
+ int typeDefLevel = currentDefLevel;
+ if (type.isRepetition(Type.Repetition.OPTIONAL) || type.isRepetition(Type.Repetition.REPEATED)) {
+ typeDefLevel++;
+ }
+ List<ColumnDescriptor> descriptors =
+ getAllColumnDescriptorByType(depth, type, columnDescriptors);
+ // Support for schema evolution: if the column from the current
+ // query schema is not present in the file schema, return a dummy
+ // reader that produces nulls. This allows queries to proceed even
+ // when new columns have been added after the file was written.
+ if (!fileSchema.getColumns().contains(descriptors.getFirst())) {
+ return new VectorizedDummyColumnReader(Optional.ofNullable(initialDefaults)
+ .map(defaults -> defaults.getOrDefault(descriptors.getFirst().getPath()[0], null)).orElse(null));
+ }
+ switch (typeInfo.getCategory()) {
+ case PRIMITIVE:
Review Comment:
fixed to:
```
switch (typeInfo.getCategory()) {
case PRIMITIVE:
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4216490639
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ // Fallback path: no cache key, so we read status + footer straight from HDFS. Time both
+ // under HDFS_TIME_NS to match how ORC accounts for cache-miss footer reads.
+ final FileSystem fs = path.getFileSystem(jobConf);
+ long hdfsStart = counters.startTimeCounter();
+ final FileStatus stat = fs.getFileStatus(path);
+ counters.recordHdfsTime(hdfsStart);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ hdfsStart = counters.startTimeCounter();
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ counters.recordHdfsTime(hdfsStart);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ long startTime = counters.startTimeCounter();
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Exception e) {
+ // A shutdown-triggered InterruptedException must not be silently reported as an ordinary
+ // consumer error: preserve the interrupt on the thread so any caller sees it.
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ }
+ consumer.setError(e);
+ } finally {
+ counters.incrWallClockCounter(LlapIOCounters.TOTAL_IO_TIME_NS, startTime);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ // The LLAP IO summary keys off TABLE / FILE / STRIPES; set them the same way ORC does so a
+ // native Parquet fragment shows up in the summary with the same fields populated.
+ if (cacheTag != null) {
+ counters.setDesc(QueryFragmentCounters.Desc.TABLE, cacheTag.getTableName());
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.FILE, path
+ + (fileKey == null ? "" : " (" + fileKey + ")"));
+
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = selectRowGroups(blocks, fileSchema);
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ // STRIPES is the ORC term; for Parquet the analog is row groups. Reuse the same descriptor so
+ // the summary layout stays common and we don't have to teach the reporter about a new field.
+ counters.setDesc(QueryFragmentCounters.Desc.STRIPES, "0," + selected.size());
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /**
+ * The row groups this split has to read: the ones whose first data page falls inside the split, as
+ * Parquet's own input format assigns them, minus any the pushed-down predicate rules out on its
+ * statistics. Returned in file order, since the fetch pipeline reads them in that order.
+ */
+ private List<BlockMetaData> selectRowGroups(List<BlockMetaData> blocks, MessageType fileSchema)
+ throws IOException {
+ long splitStart = split.getStart();
+ long splitEnd = splitStart + split.getLength();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().getFirst().getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate == null) {
+ return selected;
+ }
+ return RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Whether the query needs row-lineage virtual columns ({@code ROW__LINEAGE__ID} /
+ * {@code LAST__UPDATED__SEQUENCE__NUMBER}) that the fallback reader augments the requested schema
+ * with via {@link RowLineageUtils#getRequestedSchemaWithRowLineageColumns}. The native pipeline
+ * doesn't propagate that augmentation through {@code Includes} yet, so we defer to the fallback
+ * reader instead of quietly emitting nulls in the lineage slots.
+ */
+ public boolean needsRowLineage() {
Review Comment:
see my response: https://github.com/apache/hive/pull/6793/changes#r4194264362
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4034330064
##########
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java:
##########
@@ -446,179 +440,19 @@ private void checkEndOfRowGroup() throws IOException {
throw new IOException("expecting more rows but reached last block. Read "
+ rowsReturned + " out of " + totalRowCount);
}
- List<ColumnDescriptor> columns = requestedSchema.getColumns();
- List<Type> types = requestedSchema.getFields();
- columnReaders = new VectorizedColumnReader[columns.size()];
-
- if (!ColumnProjectionUtils.isReadAllColumns(jobConf)) {
- //certain queries like select count(*) from table do not have
- //any projected columns and still have isReadAllColumns as false
- //in such cases columnReaders are not needed
- //However, if colsToInclude is not empty we should initialize each columnReader
- if(!colsToInclude.isEmpty()) {
- for (int i = 0; i < types.size(); ++i) {
- columnReaders[i] =
- buildVectorizedParquetReader(
- columnTypesList.get(colsToInclude.get(i)), types.get(i),
- pages, requestedSchema.getColumns(), skipTimestampConversion, writerTimezone, skipProlepticConversion,
- legacyConversionEnabled, 0, 0
- );
- }
- }
- } else {
- for (int i = 0; i < types.size(); ++i) {
- columnReaders[i] = buildVectorizedParquetReader(columnTypesList.get(i), types.get(i), pages,
- requestedSchema.getColumns(), skipTimestampConversion, writerTimezone, skipProlepticConversion,
- legacyConversionEnabled, 0, 0);
- }
- }
+ // Delegate the (Hive-type-driven) column-reader construction to the shared helper so the
+ // LLAP cache-backed consumer can reuse the exact same logic. Behavior is unchanged.
+ columnReaders = new ParquetRowGroupDecoder(fileSchema, initialDefaults).buildColumnReaders(
+ pages, requestedSchema, columnTypesList, colsToInclude,
+ ColumnProjectionUtils.isReadAllColumns(jobConf), skipTimestampConversion, writerTimezone,
+ skipProlepticConversion, legacyConversionEnabled);
Review Comment:
added a guard with good explanation regarding when it fires and about the consequences
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4034551562
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,560 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ColumnVectorProducer.Includes;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final Includes includes;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split, Includes includes,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.includes = includes;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ final FileSystem fs = path.getFileSystem(jobConf);
+ final FileStatus stat = fs.getFileStatus(path);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Throwable t) {
+ consumer.setError(t);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final int maxAlloc = allocator.getMaxAllocation();
+ final long splitStart = split.getStart(), splitEnd = splitStart + split.getLength();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().get(0).getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate != null) {
+ selected = RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, maxAlloc, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, maxAlloc, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /** File-schema positions of the requested fields; column chunks follow the schema order. */
+ private static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
+ List<Integer> leaves = new ArrayList<>();
+ for (Type field : requestedSchema.getFields()) {
+ if (fileSchema.containsField(field.getName())) {
+ leaves.add(fileSchema.getFieldIndex(field.getName()));
+ }
+ }
+ return leaves.stream().mapToInt(Integer::intValue).toArray();
Review Comment:
good catch, this would totally mess up columns in case of schema like:
```
message hive_schema {
optional group nested {
optional int32 a;
optional int32 b;
}
optional int32 x;
}
```
where ` fileSchema.getFieldIndex("x")` = 1, but `fileSchema.getColumns() → [nested.a, nested.b, x] (three leaves, indices 0, 1, 2)`, so `block.getColumns().get(1)` was used for x, but it's actually `nested.b`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4154375677
##########
llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIo.java:
##########
@@ -74,11 +74,15 @@ InputFormat<NullWritable, T> getInputFormat(
* @param fileKey fileId of the Parquet file (either the Long fileId of HDFS or the SyntheticFileId).
* Optional, if it is not provided, it will be generated, see:
* org.apache.hadoop.hive.ql.io.HdfsUtils#getFileId()
+ * @param cacheHit optional out-parameter; when supplied, its {@code value} field is written
+ * with {@code true} on a cache hit and {@code false} on a miss, so the caller
+ * (which lives in llap-server and holds {@code QueryFragmentCounters}) can bump
+ * {@code METADATA_CACHE_HIT} / {@code METADATA_CACHE_MISS} for the LLAP IO summary.
* @return
* @throws IOException
*/
- MemoryBufferOrBuffers getParquetFooterBuffersFromCache(Path path, JobConf conf, @Nullable Object fileKey)
- throws IOException;
+ MemoryBufferOrBuffers getParquetFooterBuffersFromCache(Path path, JobConf conf, @Nullable Object fileKey,
Review Comment:
could we group format specific methods under sub-interfaces?
````
public interface LlapIo<T> {
InputFormat<NullWritable, T> getInputFormat(InputFormat<?, ?> src, @Nullable Deserializer serde);
… close, getMemoryInfo, purge, evictEntity, initCacheOnlyInputFormat, fetchCachedContentInfo, loadDataIntoCache, usingLowLevelCache …
LlapOrcIo orc();
LlapParquetIo parquet();
}
public interface LlapOrcIo {
OrcTail getTailFromCache(Path, Configuration, CacheTag, @Nullable Object fileKey) throws IOException;
RecordReader<NullWritable, VectorizedRowBatch> vectorizedReaderForPath(Object fileKey, Path, CacheTag,
List<Integer> tableIncludedCols, JobConf, long offset, long length, Reporter) throws IOException;
}
public interface LlapParquetIo {
MemoryBufferOrBuffers getFooterBuffersFromCache(Path, JobConf, @Nullable Object fileKey, @Nullable BooleanRef cacheHit) throws IOException;
// vectorizedReaderForPath(...) arrives with its Iceberg caller
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4155989825
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ // Fallback path: no cache key, so we read status + footer straight from HDFS. Time both
+ // under HDFS_TIME_NS to match how ORC accounts for cache-miss footer reads.
+ final FileSystem fs = path.getFileSystem(jobConf);
+ long hdfsStart = counters.startTimeCounter();
+ final FileStatus stat = fs.getFileStatus(path);
+ counters.recordHdfsTime(hdfsStart);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ hdfsStart = counters.startTimeCounter();
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ counters.recordHdfsTime(hdfsStart);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ long startTime = counters.startTimeCounter();
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Exception e) {
+ // A shutdown-triggered InterruptedException must not be silently reported as an ordinary
+ // consumer error: preserve the interrupt on the thread so any caller sees it.
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ }
+ consumer.setError(e);
+ } finally {
+ counters.incrWallClockCounter(LlapIOCounters.TOTAL_IO_TIME_NS, startTime);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ // The LLAP IO summary keys off TABLE / FILE / STRIPES; set them the same way ORC does so a
+ // native Parquet fragment shows up in the summary with the same fields populated.
+ if (cacheTag != null) {
+ counters.setDesc(QueryFragmentCounters.Desc.TABLE, cacheTag.getTableName());
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.FILE, path
+ + (fileKey == null ? "" : " (" + fileKey + ")"));
+
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = selectRowGroups(blocks, fileSchema);
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ // STRIPES is the ORC term; for Parquet the analog is row groups. Reuse the same descriptor so
+ // the summary layout stays common and we don't have to teach the reporter about a new field.
+ counters.setDesc(QueryFragmentCounters.Desc.STRIPES, "0," + selected.size());
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /**
+ * The row groups this split has to read: the ones whose first data page falls inside the split, as
+ * Parquet's own input format assigns them, minus any the pushed-down predicate rules out on its
+ * statistics. Returned in file order, since the fetch pipeline reads them in that order.
+ */
+ private List<BlockMetaData> selectRowGroups(List<BlockMetaData> blocks, MessageType fileSchema)
+ throws IOException {
+ long splitStart = split.getStart();
+ long splitEnd = splitStart + split.getLength();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().getFirst().getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate == null) {
+ return selected;
+ }
+ return RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Whether the query needs row-lineage virtual columns ({@code ROW__LINEAGE__ID} /
+ * {@code LAST__UPDATED__SEQUENCE__NUMBER}) that the fallback reader augments the requested schema
+ * with via {@link RowLineageUtils#getRequestedSchemaWithRowLineageColumns}. The native pipeline
+ * doesn't propagate that augmentation through {@code Includes} yet, so we defer to the fallback
+ * reader instead of quietly emitting nulls in the lineage slots.
+ */
+ public boolean needsRowLineage() {
+ VectorizedRowBatchCtx rbCtx = Utilities.getVectorizedRowBatchCtx(jobConf);
+ if (rbCtx == null) {
+ return false;
+ }
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ return RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema, VirtualColumn.ROW_LINEAGE_ID)
+ || RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema,
+ VirtualColumn.LAST_UPDATED_SEQUENCE_NUMBER);
+ }
+
+ /**
+ * Leaf-column positions of the requested fields; {@code BlockMetaData.getColumns()} is a flat list
+ * of leaves in file-schema order, so a top-level primitive at ordinal {@code k} in the file schema
+ * may sit at a very different leaf index (e.g. a nested group of two leaves before it shifts it
+ * from {@code 1} to {@code 2}). We only reach this method when every requested field is a
+ * top-level primitive ({@link #projectsNestedTypes()} would have forced a fallback otherwise), so
+ * each requested field maps to exactly one leaf whose path is a single segment.
+ */
+ static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
Review Comment:
/** Leaf index in {@code BlockMetaData.getColumns()} of each requested field present in the file. */
static int[] findLeafIndexes(MessageType requestedSchema, MessageType fileSchema) ?
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ // Fallback path: no cache key, so we read status + footer straight from HDFS. Time both
+ // under HDFS_TIME_NS to match how ORC accounts for cache-miss footer reads.
+ final FileSystem fs = path.getFileSystem(jobConf);
+ long hdfsStart = counters.startTimeCounter();
+ final FileStatus stat = fs.getFileStatus(path);
+ counters.recordHdfsTime(hdfsStart);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ hdfsStart = counters.startTimeCounter();
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ counters.recordHdfsTime(hdfsStart);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ long startTime = counters.startTimeCounter();
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Exception e) {
+ // A shutdown-triggered InterruptedException must not be silently reported as an ordinary
+ // consumer error: preserve the interrupt on the thread so any caller sees it.
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ }
+ consumer.setError(e);
+ } finally {
+ counters.incrWallClockCounter(LlapIOCounters.TOTAL_IO_TIME_NS, startTime);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ // The LLAP IO summary keys off TABLE / FILE / STRIPES; set them the same way ORC does so a
+ // native Parquet fragment shows up in the summary with the same fields populated.
+ if (cacheTag != null) {
+ counters.setDesc(QueryFragmentCounters.Desc.TABLE, cacheTag.getTableName());
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.FILE, path
+ + (fileKey == null ? "" : " (" + fileKey + ")"));
+
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = selectRowGroups(blocks, fileSchema);
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ // STRIPES is the ORC term; for Parquet the analog is row groups. Reuse the same descriptor so
+ // the summary layout stays common and we don't have to teach the reporter about a new field.
+ counters.setDesc(QueryFragmentCounters.Desc.STRIPES, "0," + selected.size());
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /**
+ * The row groups this split has to read: the ones whose first data page falls inside the split, as
+ * Parquet's own input format assigns them, minus any the pushed-down predicate rules out on its
+ * statistics. Returned in file order, since the fetch pipeline reads them in that order.
+ */
+ private List<BlockMetaData> selectRowGroups(List<BlockMetaData> blocks, MessageType fileSchema)
+ throws IOException {
+ long splitStart = split.getStart();
+ long splitEnd = splitStart + split.getLength();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().getFirst().getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate == null) {
+ return selected;
+ }
+ return RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Whether the query needs row-lineage virtual columns ({@code ROW__LINEAGE__ID} /
+ * {@code LAST__UPDATED__SEQUENCE__NUMBER}) that the fallback reader augments the requested schema
+ * with via {@link RowLineageUtils#getRequestedSchemaWithRowLineageColumns}. The native pipeline
+ * doesn't propagate that augmentation through {@code Includes} yet, so we defer to the fallback
+ * reader instead of quietly emitting nulls in the lineage slots.
+ */
+ public boolean needsRowLineage() {
+ VectorizedRowBatchCtx rbCtx = Utilities.getVectorizedRowBatchCtx(jobConf);
+ if (rbCtx == null) {
+ return false;
+ }
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ return RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema, VirtualColumn.ROW_LINEAGE_ID)
+ || RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema,
+ VirtualColumn.LAST_UPDATED_SEQUENCE_NUMBER);
+ }
+
+ /**
+ * Leaf-column positions of the requested fields; {@code BlockMetaData.getColumns()} is a flat list
+ * of leaves in file-schema order, so a top-level primitive at ordinal {@code k} in the file schema
+ * may sit at a very different leaf index (e.g. a nested group of two leaves before it shifts it
+ * from {@code 1} to {@code 2}). We only reach this method when every requested field is a
+ * top-level primitive ({@link #projectsNestedTypes()} would have forced a fallback otherwise), so
+ * each requested field maps to exactly one leaf whose path is a single segment.
+ */
+ static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
Review Comment:
````
/** Leaf index in {@code BlockMetaData.getColumns()} of each requested field present in the file. */
static int[] findLeafIndexes(MessageType requestedSchema, MessageType fileSchema)
````?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4155989825
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ // Fallback path: no cache key, so we read status + footer straight from HDFS. Time both
+ // under HDFS_TIME_NS to match how ORC accounts for cache-miss footer reads.
+ final FileSystem fs = path.getFileSystem(jobConf);
+ long hdfsStart = counters.startTimeCounter();
+ final FileStatus stat = fs.getFileStatus(path);
+ counters.recordHdfsTime(hdfsStart);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ hdfsStart = counters.startTimeCounter();
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ counters.recordHdfsTime(hdfsStart);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ long startTime = counters.startTimeCounter();
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Exception e) {
+ // A shutdown-triggered InterruptedException must not be silently reported as an ordinary
+ // consumer error: preserve the interrupt on the thread so any caller sees it.
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ }
+ consumer.setError(e);
+ } finally {
+ counters.incrWallClockCounter(LlapIOCounters.TOTAL_IO_TIME_NS, startTime);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ // The LLAP IO summary keys off TABLE / FILE / STRIPES; set them the same way ORC does so a
+ // native Parquet fragment shows up in the summary with the same fields populated.
+ if (cacheTag != null) {
+ counters.setDesc(QueryFragmentCounters.Desc.TABLE, cacheTag.getTableName());
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.FILE, path
+ + (fileKey == null ? "" : " (" + fileKey + ")"));
+
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = selectRowGroups(blocks, fileSchema);
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ // STRIPES is the ORC term; for Parquet the analog is row groups. Reuse the same descriptor so
+ // the summary layout stays common and we don't have to teach the reporter about a new field.
+ counters.setDesc(QueryFragmentCounters.Desc.STRIPES, "0," + selected.size());
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /**
+ * The row groups this split has to read: the ones whose first data page falls inside the split, as
+ * Parquet's own input format assigns them, minus any the pushed-down predicate rules out on its
+ * statistics. Returned in file order, since the fetch pipeline reads them in that order.
+ */
+ private List<BlockMetaData> selectRowGroups(List<BlockMetaData> blocks, MessageType fileSchema)
+ throws IOException {
+ long splitStart = split.getStart();
+ long splitEnd = splitStart + split.getLength();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().getFirst().getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate == null) {
+ return selected;
+ }
+ return RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Whether the query needs row-lineage virtual columns ({@code ROW__LINEAGE__ID} /
+ * {@code LAST__UPDATED__SEQUENCE__NUMBER}) that the fallback reader augments the requested schema
+ * with via {@link RowLineageUtils#getRequestedSchemaWithRowLineageColumns}. The native pipeline
+ * doesn't propagate that augmentation through {@code Includes} yet, so we defer to the fallback
+ * reader instead of quietly emitting nulls in the lineage slots.
+ */
+ public boolean needsRowLineage() {
+ VectorizedRowBatchCtx rbCtx = Utilities.getVectorizedRowBatchCtx(jobConf);
+ if (rbCtx == null) {
+ return false;
+ }
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ return RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema, VirtualColumn.ROW_LINEAGE_ID)
+ || RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema,
+ VirtualColumn.LAST_UPDATED_SEQUENCE_NUMBER);
+ }
+
+ /**
+ * Leaf-column positions of the requested fields; {@code BlockMetaData.getColumns()} is a flat list
+ * of leaves in file-schema order, so a top-level primitive at ordinal {@code k} in the file schema
+ * may sit at a very different leaf index (e.g. a nested group of two leaves before it shifts it
+ * from {@code 1} to {@code 2}). We only reach this method when every requested field is a
+ * top-level primitive ({@link #projectsNestedTypes()} would have forced a fallback otherwise), so
+ * each requested field maps to exactly one leaf whose path is a single segment.
+ */
+ static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
Review Comment:
````
/** Leaf index in {@code BlockMetaData.getColumns()} of each requested field present in the file. */
static int[] findLeafIndexes(MessageType requestedSchema, MessageType fileSchema)
````
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4156097197
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ // Fallback path: no cache key, so we read status + footer straight from HDFS. Time both
+ // under HDFS_TIME_NS to match how ORC accounts for cache-miss footer reads.
+ final FileSystem fs = path.getFileSystem(jobConf);
+ long hdfsStart = counters.startTimeCounter();
+ final FileStatus stat = fs.getFileStatus(path);
+ counters.recordHdfsTime(hdfsStart);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ hdfsStart = counters.startTimeCounter();
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ counters.recordHdfsTime(hdfsStart);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ long startTime = counters.startTimeCounter();
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Exception e) {
+ // A shutdown-triggered InterruptedException must not be silently reported as an ordinary
+ // consumer error: preserve the interrupt on the thread so any caller sees it.
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ }
+ consumer.setError(e);
+ } finally {
+ counters.incrWallClockCounter(LlapIOCounters.TOTAL_IO_TIME_NS, startTime);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ // The LLAP IO summary keys off TABLE / FILE / STRIPES; set them the same way ORC does so a
+ // native Parquet fragment shows up in the summary with the same fields populated.
+ if (cacheTag != null) {
+ counters.setDesc(QueryFragmentCounters.Desc.TABLE, cacheTag.getTableName());
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.FILE, path
+ + (fileKey == null ? "" : " (" + fileKey + ")"));
+
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = selectRowGroups(blocks, fileSchema);
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ // STRIPES is the ORC term; for Parquet the analog is row groups. Reuse the same descriptor so
+ // the summary layout stays common and we don't have to teach the reporter about a new field.
+ counters.setDesc(QueryFragmentCounters.Desc.STRIPES, "0," + selected.size());
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /**
+ * The row groups this split has to read: the ones whose first data page falls inside the split, as
+ * Parquet's own input format assigns them, minus any the pushed-down predicate rules out on its
+ * statistics. Returned in file order, since the fetch pipeline reads them in that order.
+ */
+ private List<BlockMetaData> selectRowGroups(List<BlockMetaData> blocks, MessageType fileSchema)
+ throws IOException {
+ long splitStart = split.getStart();
+ long splitEnd = splitStart + split.getLength();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().getFirst().getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate == null) {
+ return selected;
+ }
+ return RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Whether the query needs row-lineage virtual columns ({@code ROW__LINEAGE__ID} /
+ * {@code LAST__UPDATED__SEQUENCE__NUMBER}) that the fallback reader augments the requested schema
+ * with via {@link RowLineageUtils#getRequestedSchemaWithRowLineageColumns}. The native pipeline
+ * doesn't propagate that augmentation through {@code Includes} yet, so we defer to the fallback
+ * reader instead of quietly emitting nulls in the lineage slots.
+ */
+ public boolean needsRowLineage() {
+ VectorizedRowBatchCtx rbCtx = Utilities.getVectorizedRowBatchCtx(jobConf);
+ if (rbCtx == null) {
+ return false;
+ }
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ return RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema, VirtualColumn.ROW_LINEAGE_ID)
+ || RowLineageUtils.isRowLineageColumnPresent(rbCtx, fileSchema,
+ VirtualColumn.LAST_UPDATED_SEQUENCE_NUMBER);
+ }
+
+ /**
+ * Leaf-column positions of the requested fields; {@code BlockMetaData.getColumns()} is a flat list
+ * of leaves in file-schema order, so a top-level primitive at ordinal {@code k} in the file schema
+ * may sit at a very different leaf index (e.g. a nested group of two leaves before it shifts it
+ * from {@code 1} to {@code 2}). We only reach this method when every requested field is a
+ * top-level primitive ({@link #projectsNestedTypes()} would have forced a fallback otherwise), so
+ * each requested field maps to exactly one leaf whose path is a single segment.
+ */
+ static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
+ List<ColumnDescriptor> allLeaves = fileSchema.getColumns();
+ List<Integer> selected = new ArrayList<>();
+ for (Type field : requestedSchema.getFields()) {
+ if (!fileSchema.containsField(field.getName())) {
+ continue;
+ }
+ for (int i = 0; i < allLeaves.size(); ++i) {
+ String[] path = allLeaves.get(i).getPath();
+ // Single-segment path == top-level primitive; skip leaves buried inside a group.
+ if (path.length == 1 && path[0].equals(field.getName())) {
+ selected.add(i);
+ break;
+ }
+ }
+ }
+ return selected.stream().mapToInt(Integer::intValue).toArray();
+ }
+
+ private static long bytes(int[] projected, BlockMetaData block) {
+ long total = 0;
+ for (int leaf : projected) {
+ total += block.getColumns().get(leaf).getTotalSize();
+ }
+ return total;
+ }
+
+ private static long bytes(Fetch fetch) {
+ long total = 0;
+ for (ColumnChunkMetaData chunk : fetch.batch.chunks()) {
+ total += chunk.getTotalSize();
+ }
+ return total;
+ }
+
+ /** One row group on its way in: buffers planned, missing ranges requested, not yet decoded. */
+ private static final class Fetch {
+ private final ParquetEncodedColumnBatch batch = new ParquetEncodedColumnBatch();
+ private final List<ColumnPlan> columns = new ArrayList<>();
+ private final List<Part> misses = new ArrayList<>();
+ private final List<Run> runs = new ArrayList<>();
+ }
+
+ /** The buffers covering one column chunk, in file order, and which stretches of them are new. */
+ private record ColumnPlan(List<Part> parts, List<MissRun> missRuns) {
Review Comment:
ChunkPlan with fields parts, gaps ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4157224923
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.metadata.RowLineageUtils;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
Review Comment:
Follow-up improvement:
Today the stock reader and the native reader each derive the same six per-file facts from the same inputs, in different places, and could drift
````
┌────────────────────────────────────────────┬──────────────────────────────────────────┬───────────────────────────────────────────────┐
│ Decision │ Stock │ Native │
├────────────────────────────────────────────┼──────────────────────────────────────────┼───────────────────────────────────────────────┤
│ requested schema │ VectorizedParquetRecordReader.initialize │ ParquetEncodedDataReader.loadFooter │
├────────────────────────────────────────────┼──────────────────────────────────────────┼───────────────────────────────────────────────┤
│ timestamp, proleptic, legacy-zone flags │ ParquetRecordReaderBase.getSplit:160-170 │ ParquetEncodedDataConsumer.initSchema:120-144 │
├────────────────────────────────────────────┼──────────────────────────────────────────┼───────────────────────────────────────────────┤
│ writer timezone │ VectorizedParquetRecordReader:255 │ consumer :129 │
├────────────────────────────────────────────┼──────────────────────────────────────────┼───────────────────────────────────────────────┤
│ row groups in split (first-data-page rule) │ ParquetRecordReaderBase:132-138 │ reader :262-268 │
├────────────────────────────────────────────┼──────────────────────────────────────────┼───────────────────────────────────────────────┤
│ SARG to predicate, statistics filter │ ParquetRecordReaderBase:144-146 │ reader :276-284 │
├────────────────────────────────────────────┼──────────────────────────────────────────┼───────────────────────────────────────────────┤
│ decimal64 │ plan ctx │ conf string (F4) │
└────────────────────────────────────────────┴──────────────────────────────────────────┴───────────────────────────────────────────────┘
````
The proposal is one class in ql, built once per split from the things both readers already have, whose fields are those decisions:
````
/** Everything a Parquet split read decides once from the footer and the job conf, shared by both readers. */
public final class ParquetSplitPlan {
public final MessageType requestedSchema;
public final List<BlockMetaData> rowGroups; // in-split, statistics-filtered
public final TimestampConversionOptions conversions;
public final FilterPredicate predicate; // may be null
public static ParquetSplitPlan of(JobConf conf, ParquetMetadata footer, long splitStart, long splitLength) { … }
}
````
VectorizedParquetRecordReader.initialize and ParquetEncodedDataReader.loadFooter both become "footer, then ParquetSplitPlan.of(...)", and ParquetEncodedDataConsumer.initSchema disappears because the consumer receives the plan. toFilterPredicate and the five DataWritableReadSupport statics go back behind it, so the public surface between ql and llap-server is one type instead of six methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4156908050
##########
ql/src/java/org/apache/hadoop/hive/llap/ParquetRangeBuffers.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.llap;
+
+import java.nio.ByteBuffer;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.StreamCapabilities;
+import org.apache.hadoop.io.ElasticByteBufferPool;
+
+/**
+ * Heap buffers for one stream's vectored reads. A stream returning slices of a larger read says
+ * so ({@link StreamCapabilities#VECTOREDIO_BUFFERS_SLICED}); those alias each other and are left
+ * to the collector. Other streams' buffers are exclusive and pooled for reuse across the file's
+ * row groups. The pool is per stream, not per JVM: {@link ElasticByteBufferPool} never evicts
+ * and has one lock, so a shared one would pin every executor's peak for the daemon's life.
+ */
+public final class ParquetRangeBuffers {
+ private final ElasticByteBufferPool pool;
+
+ private ParquetRangeBuffers(boolean pooled) {
+ this.pool = pooled ? new ElasticByteBufferPool() : null;
+ }
+
+ public static ParquetRangeBuffers forStream(FSDataInputStream stream) {
Review Comment:
maybe refactor to cleaner
````
/** Pooled: the stream fills buffers we own, so they can be reused across the split's row groups. */
static VectoredReadBuffers pooled() {
return new VectoredReadBuffers(new ElasticByteBufferPool());
}
/** Unpooled: the stream may hand back slices of a larger buffer; slices alias, so they go to the GC. */
static VectoredReadBuffers unpooled() {
return new VectoredReadBuffers(null);
}
static VectoredReadBuffers forStream(FSDataInputStream stream) {
return stream.hasCapability(StreamCapabilities.VECTOREDIO_BUFFERS_SLICED) ? unpooled() : pooled();
}
private VectoredReadBuffers(@Nullable ElasticByteBufferPool pool) {
this.pool = pool;
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4157278864
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ParquetEncodedDataConsumer.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.hive.llap.io.decode;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.type.DataTypePhysicalVariation;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.impl.ColumnVectorBatch;
+import org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl;
+import org.apache.hadoop.hive.llap.io.decode.ColumnVectorProducer.Includes;
+import org.apache.hadoop.hive.llap.io.encoded.ParquetEncodedColumnBatch;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonIOMetrics;
+import org.apache.hadoop.hive.ql.exec.vector.ColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.orc.encoded.Consumer;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetRowGroupDecoder;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedColumnReader;
+import org.apache.hadoop.hive.serde2.ColumnProjectionUtils;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
+import org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.parquet.HadoopReadOptions;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.compression.CompressionCodecFactory;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.metadata.FileMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+
+import com.google.common.base.Strings;
+
+/**
+ * The Parquet counterpart of {@link OrcEncodedDataConsumer}: it turns one row group's worth of
+ * cached column-chunk buffers ({@link ParquetEncodedColumnBatch}) into {@link ColumnVectorBatch}es
+ * and hands them downstream.
+ *
+ * <p>Rather than reimplementing Parquet value decoding, this consumer builds a
+ * {@link ParquetCachedPageReadStore} directly over the cached buffers (pages are parsed in place and
+ * decompressed lazily, no ParquetFileReader and no whole-chunk copy) and drives the very same
+ * {@link VectorizedColumnReader}s the non-cached vectorized reader uses (via {@link ParquetRowGroupDecoder}).
+ */
+public class ParquetEncodedDataConsumer
+ extends EncodedDataConsumer<Object, ParquetEncodedColumnBatch> {
+
+ private final Configuration jobConf;
+ private final boolean useDecimal64ColumnVectors;
+ private final CompressionCodecFactory codecFactory;
+ private final ParquetMetadataConverter converter;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private Path path;
+ private Map<String, Object> initialDefaults;
+
+ // Derived lazily (once) from the footer + job conf, then reused across row groups.
+ private List<TypeInfo> columnTypesList;
+ private List<Integer> colsToInclude;
+ private boolean readAllColumns;
+ private boolean skipTimestampConversion;
+ private boolean skipProlepticConversion;
+ private boolean legacyConversionEnabled;
+ private ZoneId writerTimezone;
+ private boolean schemaInitialized = false;
+
+ public ParquetEncodedDataConsumer(Consumer<ColumnVectorBatch> consumer, Includes includes,
+ QueryFragmentCounters counters, LlapDaemonIOMetrics ioMetrics, Configuration jobConf) {
+ super(consumer, includes.getPhysicalColumnIds().size(), ioMetrics, counters);
+ this.jobConf = jobConf;
+ this.useDecimal64ColumnVectors = HiveConf.getVar(jobConf,
+ ConfVars.HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_ENABLED)
+ .equalsIgnoreCase(HiveConf.HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_DECIMAL_64);
+ ParquetReadOptions options = HadoopReadOptions.builder(jobConf).build();
+ this.codecFactory = options.getCodecFactory();
+ this.converter = new ParquetMetadataConverter(options);
+ }
+
+ public void setFileMetadata(ParquetMetadata footer, MessageType requestedSchema, Path path) {
+ this.footer = footer;
+ this.requestedSchema = requestedSchema;
+ this.path = path;
+ }
+
+ public void setInitialDefaults(Map<String, Object> initialDefaults) {
+ this.initialDefaults = initialDefaults;
+ }
+
+ /**
+ * Derives the Hive type / projection lists and the conversion flags the same way
+ * {@link org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase} does from the job conf,
+ * so the shared {@link ParquetRowGroupDecoder} produces identical readers.
+ */
+ private void initSchema() {
+ FileMetaData fileMetaData = footer.getFileMetaData();
+ Map<String, String> kvMeta = fileMetaData.getKeyValueMetaData();
+
+ this.columnTypesList =
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES));
+ this.colsToInclude = ColumnProjectionUtils.getReadColumnIDs(jobConf);
+ this.readAllColumns = ColumnProjectionUtils.isReadAllColumns(jobConf);
+
+ this.writerTimezone = DataWritableReadSupport.getWriterTimeZoneId(kvMeta);
+ if (HiveConf.getBoolVar(jobConf, ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION)) {
+ this.skipTimestampConversion =
+ !Strings.nullToEmpty(fileMetaData.getCreatedBy()).startsWith("parquet-mr");
+ } else {
+ this.skipTimestampConversion = false;
+ }
+ Boolean proleptic = DataWritableReadSupport.getWriterDateProleptic(kvMeta);
+ if (proleptic == null) {
+ proleptic = HiveConf.getBoolVar(jobConf, ConfVars.HIVE_PARQUET_DATE_PROLEPTIC_GREGORIAN_DEFAULT);
+ }
+ this.skipProlepticConversion = proleptic;
+ this.legacyConversionEnabled = DataWritableReadSupport.getZoneConversionLegacy(kvMeta, jobConf);
+
+ this.schemaInitialized = true;
+ }
+
+ @Override
+ protected void decodeBatch(ParquetEncodedColumnBatch batch,
+ Consumer<ColumnVectorBatch> downstreamConsumer) throws InterruptedException {
+ if (!schemaInitialized) {
+ initSchema();
+ }
+
+ long startTime = counters.startTimeCounter();
+ // NOSONAR - S2093 does not apply: codecFactory is a per-consumer field with a per-batch
+ // release() (not close()); try-with-resources would tie it to this scope, which is wrong.
+ try { // NOSONAR
+ PageReadStore pages = new ParquetCachedPageReadStore(footer, batch, codecFactory, converter);
+
+ VectorizedColumnReader[] columnReaders =
+ new ParquetRowGroupDecoder(footer.getFileMetaData().getSchema(), initialDefaults).buildColumnReaders(
+ pages, requestedSchema, columnTypesList, colsToInclude, readAllColumns,
+ new ParquetRowGroupDecoder.TimestampConversionOptions(skipTimestampConversion,
+ writerTimezone, skipProlepticConversion, legacyConversionEnabled));
+
+ long rowsLeft = pages.getRowCount();
+ int batches = 0;
+ while (rowsLeft > 0) {
+ int batchSize = (int) Math.min(VectorizedRowBatch.DEFAULT_SIZE, rowsLeft);
+
+ ColumnVectorBatch cvb = takeBatch(batchSize);
+
+ // columnReaders[i] is requestedSchema field i, i.e. the i-th projected column.
+ for (int i = 0; i < columnReaders.length; ++i) {
+ if (columnReaders[i] == null) {
+ continue;
+ }
+ TypeInfo columnType = columnTypesList.get(readAllColumns ? i : colsToInclude.get(i));
+ ColumnVector cv = prepareColumnVector(cvb, i, columnType, batchSize);
+ columnReaders[i].readBatch(batchSize, cv, columnType);
+ }
+
+ downstreamConsumer.consumeData(cvb);
+ counters.incrCounter(LlapIOCounters.ROWS_EMITTED, batchSize);
+ rowsLeft -= batchSize;
+ ++batches;
+ }
+ counters.incrWallClockCounter(LlapIOCounters.DECODE_TIME_NS, startTime);
+ counters.incrCounter(LlapIOCounters.NUM_VECTOR_BATCHES, batches);
+ counters.incrCounter(LlapIOCounters.NUM_DECODED_BATCHES);
+ } catch (IOException | RuntimeException e) {
+ // parquet reports decode failures as runtime ParquetDecodingException.
+ LlapIoImpl.LOG.error("Parquet decodeBatch failed for rowGroup " + batch.rowGroupIx() + " of " + path, e);
+ downstreamConsumer.setError(e);
Review Comment:
it is possible: `str.format(batch.rowGroupIx(), path)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6793:
URL: https://github.com/apache/hive/pull/6793#issuecomment-6361900184
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6793) **Quality Gate passed**
Issues
 [90 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6793&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6793&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6793&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6793&metric=new_coverage&view=list)
 [0.1% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6793&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6793)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #6793:
URL: https://github.com/apache/hive/pull/6793#issuecomment-6300071305
### TPC-DS validation: flag off / on / off / on
I ran all 99 TPC-DS queries against `tpcds_partitioned_parquet_10` on a k8s LLAP cluster. Each query ran four times in sequence, alternating `hive.llap.io.parquet.native.enabled` false / true / false / true, starting from a cold daemon. The query results cache was off.
| Run | Flag | Cache state | Summed `Run DAG` |
|---|---|---|---|
| r1 | false | cold | 1502.2 s |
| r2 | true | warm (filled by r1) | 764.4 s |
| r3 | false | warm | 916.9 s |
| r4 | true | warm | 757.7 s |
**Results**
- **Same answers:** 96/99 queries returned identical output in all four runs. The other three (q31, q65, q71) differed only in row order, and none of them has a total `ORDER BY`.
- **Shared cache works:** r2 ran at almost a 100% hit rate on bytes that the stock reader cached in r1, so both readers cut and key files the same way. Only q34 and q93 had a few misses.
- **Faster when warm:** with the same bytes served from the same cache, the native path is about 17% faster than the stock path (r3 vs r4: 916.9 s → 757.7 s). r2 and r4 are within 1% of each other, so the native path needed no warm-up of its own. The gain comes from decoding in place over the cache on the IO threads, with no copy into parquet's buffers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4034551562
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/ParquetEncodedDataReader.java:
##########
@@ -0,0 +1,560 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileRange;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.Allocator.BufferObjectFactory;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.common.io.DataCache.BooleanRef;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBufferOrBuffers;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.ParquetCacheLayout;
+import org.apache.hadoop.hive.llap.ParquetRangeBuffers;
+import org.apache.hadoop.hive.llap.LlapHiveUtils;
+import org.apache.hadoop.hive.llap.cache.BufferUsageManager;
+import org.apache.hadoop.hive.llap.cache.LlapDataBuffer;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.LlapProxy;
+import org.apache.hadoop.hive.llap.io.decode.ColumnVectorProducer.Includes;
+import org.apache.hadoop.hive.llap.io.decode.ParquetEncodedDataConsumer;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.SyntheticFileId;
+import org.apache.hadoop.hive.ql.io.orc.encoded.CacheChunk;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetFooterInputFromCache;
+import org.apache.hadoop.hive.ql.io.orc.encoded.StoppableAllocator;
+import org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.functional.FutureIO;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.filter2.compat.RowGroupFilter;
+import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.HadoopStreams;
+import org.apache.parquet.io.InputFile;
+import org.apache.parquet.io.SeekableInputStream;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.tez.common.CallableWithNdc;
+
+/**
+ * Reads one Parquet split through the LLAP cache on an IO thread. Row groups whose first data
+ * page falls in the split are selected and filtered by their statistics; for each one the
+ * projected column chunks are looked up in the cache, the missing ranges are requested in one
+ * vectored read, and the chunks are handed to the consumer to decode. The next row group's
+ * request is issued before the current one is decoded so its transfer overlaps the decode, as
+ * long as the two together stay within this thread's share of the cache. Every buffer in a
+ * consumed batch carries exactly one ref, owned by the batch until returnData.
+ */
+public class ParquetEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<ParquetEncodedColumnBatch> {
+
+ private static final BufferObjectFactory DATA_BUFFER_FACTORY = LlapDataBuffer::new;
+
+ private final LowLevelCache lowLevelCache;
+ private final BufferUsageManager bufferManager;
+ private final Configuration daemonConf;
+ private final ParquetCacheLayout layout;
+ private final JobConf jobConf;
+ private final FileSplit split;
+ private final Includes includes;
+ private final ParquetEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
+ private final Path path;
+ private final boolean cacheOnly;
+ /** Bytes of column chunks one IO thread may hold across the row group in decode and the next. */
+ private final long lookaheadBudget;
+
+ private Object fileKey;
+ private CacheTag cacheTag;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private final AtomicBoolean isStopped = new AtomicBoolean(false);
+
+ public ParquetEncodedDataReader(LowLevelCache lowLevelCache, BufferUsageManager bufferManager,
+ Configuration daemonConf, Configuration jobConf, FileSplit split, Includes includes,
+ ParquetEncodedDataConsumer consumer, QueryFragmentCounters counters) throws IOException {
+ this.lowLevelCache = lowLevelCache;
+ this.bufferManager = bufferManager;
+ this.daemonConf = daemonConf;
+ this.layout = new ParquetCacheLayout(bufferManager.getAllocator(), daemonConf);
+ this.jobConf = (JobConf) jobConf;
+ this.split = split;
+ this.includes = includes;
+ this.consumer = consumer;
+ this.counters = counters;
+ this.path = split.getPath();
+ this.ugi = UserGroupInformation.getCurrentUser();
+ this.cacheOnly = HiveConf.getBoolVar(jobConf, ConfVars.LLAP_IO_CACHE_ONLY);
+ this.lookaheadBudget = HiveConf.getSizeVar(daemonConf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE)
+ / Math.max(1, HiveConf.getIntVar(daemonConf, ConfVars.LLAP_IO_THREADPOOL_SIZE));
+ }
+
+ /** Reads the footer once (through the LLAP footer cache when the file has a usable key). */
+ public ParquetMetadata loadFooter() throws IOException {
+ fileKey = SyntheticFileId.fromJobConf(jobConf);
+ if (fileKey == null) {
+ fileKey = LlapHiveUtils.createFileIdUsingFS(path.getFileSystem(jobConf), path, daemonConf);
+ }
+ if (fileKey != null) {
+ cacheTag = VectorizedParquetRecordReader.cacheTagOfParquetFile(path, daemonConf, jobConf);
+ // Bumps METADATA_CACHE_HIT / METADATA_CACHE_MISS so the LLAP IO summary accounts for the
+ // Parquet footer lookup the same way it does for ORC's file tail.
+ BooleanRef cacheHit = new BooleanRef();
+ MemoryBufferOrBuffers footerData =
+ LlapProxy.getIo().getParquetFooterBuffersFromCache(path, jobConf, fileKey, cacheHit);
+ counters.incrCounter(cacheHit.value
+ ? LlapIOCounters.METADATA_CACHE_HIT : LlapIOCounters.METADATA_CACHE_MISS);
+ footer = ParquetFileReader.readFooter(
+ new ParquetFooterInputFromCache(footerData), ParquetMetadataConverter.NO_FILTER);
+ } else {
+ final FileSystem fs = path.getFileSystem(jobConf);
+ final FileStatus stat = fs.getFileStatus(path);
+ InputFile inputFile = new InputFile() {
+ @Override
+ public SeekableInputStream newStream() throws IOException {
+ return HadoopStreams.wrap(fs.open(path));
+ }
+ @Override
+ public long getLength() {
+ return stat.getLen();
+ }
+ };
+ footer = ParquetFileReader.readFooter(inputFile, ParquetMetadataConverter.NO_FILTER);
+ }
+ requestedSchema = DataWritableReadSupport.getRequestedSchema(
+ jobConf.getBoolean(DataWritableReadSupport.PARQUET_COLUMN_INDEX_ACCESS, false),
+ DataWritableReadSupport.getColumnNames(jobConf.get(IOConstants.COLUMNS)),
+ DataWritableReadSupport.getColumnTypes(jobConf.get(IOConstants.COLUMNS_TYPES)),
+ footer.getFileMetaData().getSchema(), jobConf);
+ return footer;
+ }
+
+ @Override
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs((PrivilegedExceptionAction<Void>) () -> {
+ try {
+ performDataRead();
+ consumer.setDone();
+ } catch (Throwable t) {
+ consumer.setError(t);
+ }
+ return null;
+ });
+ }
+
+ private void performDataRead() throws IOException, InterruptedException {
+ MessageType fileSchema = footer.getFileMetaData().getSchema();
+ int[] projected = projectedLeaves(requestedSchema, fileSchema);
+ consumer.setFileMetadata(footer, requestedSchema, path);
+
+ final Allocator allocator = bufferManager.getAllocator();
+ final int maxAlloc = allocator.getMaxAllocation();
+ final long splitStart = split.getStart(), splitEnd = splitStart + split.getLength();
+ final List<BlockMetaData> blocks = footer.getBlocks();
+ List<BlockMetaData> selected = new ArrayList<>();
+ for (BlockMetaData block : blocks) {
+ long firstDataPage = block.getColumns().get(0).getFirstDataPageOffset();
+ if (firstDataPage >= splitStart && firstDataPage < splitEnd) {
+ selected.add(block);
+ }
+ }
+ FilterPredicate predicate = ParquetRecordReaderBase.toFilterPredicate(jobConf, fileSchema);
+ if (predicate != null) {
+ selected = RowGroupFilter.filterRowGroups(FilterCompat.get(predicate), selected, fileSchema);
+ }
+ Map<BlockMetaData, Integer> rowGroupOf = new IdentityHashMap<>();
+ for (int i = 0; i < blocks.size(); ++i) {
+ rowGroupOf.put(blocks.get(i), i);
+ }
+ counters.incrCounter(LlapIOCounters.SELECTED_ROWGROUPS, selected.size());
+
+ FileSystem fs = path.getFileSystem(jobConf);
+ try (FSDataInputStream fileStream = openFile(fs)) {
+ ParquetRangeBuffers buffers = ParquetRangeBuffers.forStream(fileStream);
+ Deque<Fetch> inFlight = new ArrayDeque<>();
+ try {
+ for (int i = 0; i < selected.size() && !isStopped.get(); ++i) {
+ if (inFlight.isEmpty()) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, maxAlloc, projected, selected.get(i),
+ rowGroupOf.get(selected.get(i))));
+ }
+ // The next row group's requests go out now so its transfer overlaps this one's decode.
+ if (i + 1 < selected.size() && !isStopped.get()
+ && bytes(inFlight.peek()) + bytes(projected, selected.get(i + 1)) <= lookaheadBudget) {
+ inFlight.add(startFetch(fileStream, buffers, allocator, maxAlloc, projected, selected.get(i + 1),
+ rowGroupOf.get(selected.get(i + 1))));
+ }
+ finishFetch(allocator, buffers, inFlight.poll());
+ }
+ } finally {
+ for (Fetch fetch : inFlight) {
+ abandon(allocator, fetch);
+ }
+ }
+ }
+ }
+
+ /** Whether the projection reaches into a group type, which this reader does not decode. */
+ public boolean projectsNestedTypes() {
+ for (Type field : requestedSchema.getFields()) {
+ if (!field.isPrimitive()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /** File-schema positions of the requested fields; column chunks follow the schema order. */
+ private static int[] projectedLeaves(MessageType requestedSchema, MessageType fileSchema) {
+ List<Integer> leaves = new ArrayList<>();
+ for (Type field : requestedSchema.getFields()) {
+ if (fileSchema.containsField(field.getName())) {
+ leaves.add(fileSchema.getFieldIndex(field.getName()));
+ }
+ }
+ return leaves.stream().mapToInt(Integer::intValue).toArray();
Review Comment:
good catch, this would totally mess up columns in case of schema like:
```
message hive_schema {
optional group nested {
optional int32 a;
optional int32 b;
}
optional int32 x;
}
```
where ` fileSchema.getFieldIndex("x")` = 1, but `fileSchema.getColumns() → [nested.a, nested.b, x] (three leaves, indices 0, 1, 2).`, so `block.getColumns().get(1)` was used for x, but it's actually `nested.b.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6793:
URL: https://github.com/apache/hive/pull/6793#issuecomment-5731003795
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6793) **Quality Gate passed**
Issues
 [28 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6793&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6793&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6793&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6793&metric=new_coverage&view=list)
 [0.2% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6793&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6793)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4030125485
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ParquetEncodedDataConsumer.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.llap.io.decode;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.type.DataTypePhysicalVariation;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.llap.counters.LlapIOCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.io.api.impl.ColumnVectorBatch;
+import org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl;
+import org.apache.hadoop.hive.llap.io.decode.ColumnVectorProducer.Includes;
+import org.apache.hadoop.hive.llap.io.encoded.ParquetEncodedColumnBatch;
+import org.apache.hadoop.hive.llap.metrics.LlapDaemonIOMetrics;
+import org.apache.hadoop.hive.ql.exec.vector.ColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.ql.io.orc.encoded.Consumer;
+import org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport;
+import org.apache.hadoop.hive.ql.io.parquet.vector.ParquetRowGroupDecoder;
+import org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedColumnReader;
+import org.apache.hadoop.hive.serde2.ColumnProjectionUtils;
+import org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.orc.TypeDescription;
+import org.apache.orc.impl.SchemaEvolution;
+import org.apache.parquet.HadoopReadOptions;
+import org.apache.parquet.ParquetReadOptions;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.compression.CompressionCodecFactory;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+
+import com.google.common.base.Strings;
+
+/**
+ * The Parquet counterpart of {@link OrcEncodedDataConsumer}: it turns one row group's worth of
+ * cached column-chunk buffers ({@link ParquetEncodedColumnBatch}) into {@link ColumnVectorBatch}es
+ * and hands them downstream.
+ *
+ * <p>Rather than reimplementing Parquet value decoding, this consumer builds a
+ * {@link ParquetCachedPageReadStore} directly over the cached buffers (pages are parsed in place and
+ * decompressed lazily, no ParquetFileReader and no whole-chunk copy) and drives the very same
+ * {@link VectorizedColumnReader}s the non-cached vectorized reader uses (via {@link ParquetRowGroupDecoder}).
+ */
+public class ParquetEncodedDataConsumer
+ extends EncodedDataConsumer<Object, ParquetEncodedColumnBatch> {
+
+ private final Configuration jobConf;
+ private final boolean useDecimal64ColumnVectors;
+ private final CompressionCodecFactory codecFactory;
+ private final ParquetMetadataConverter converter;
+ private ParquetMetadata footer;
+ private MessageType requestedSchema;
+ private Path path;
+ private Map<String, Object> initialDefaults;
+
+ // Derived lazily (once) from the footer + job conf, then reused across row groups.
+ private List<TypeInfo> columnTypesList;
+ private List<Integer> colsToInclude;
+ private boolean readAllColumns;
+ private boolean skipTimestampConversion;
+ private boolean skipProlepticConversion;
+ private boolean legacyConversionEnabled;
+ private ZoneId writerTimezone;
+ private boolean schemaInitialized = false;
+
+ public ParquetEncodedDataConsumer(Consumer<ColumnVectorBatch> consumer, Includes includes,
+ QueryFragmentCounters counters, LlapDaemonIOMetrics ioMetrics, Configuration jobConf) {
+ super(consumer, includes.getPhysicalColumnIds().size(), ioMetrics, counters);
+ this.jobConf = jobConf;
+ this.useDecimal64ColumnVectors = HiveConf.getVar(jobConf,
+ ConfVars.HIVE_VECTORIZED_INPUT_FORMAT_SUPPORTS_ENABLED).equalsIgnoreCase("decimal_64");
Review Comment:
do we have constant for "decimal_64" ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30059: LLAP IO Cache: native parquet data cache [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6793:
URL: https://github.com/apache/hive/pull/6793#discussion_r4030091140
##########
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ParquetCachedPageReadStore.java:
##########
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.llap.io.decode;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+
+import org.apache.hadoop.hive.llap.io.encoded.ParquetEncodedColumnBatch;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.DataPage;
+import org.apache.parquet.column.page.DataPageV1;
+import org.apache.parquet.column.page.DataPageV2;
+import org.apache.parquet.column.page.DictionaryPage;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.column.page.PageReader;
+import org.apache.parquet.compression.CompressionCodecFactory;
+import org.apache.parquet.compression.CompressionCodecFactory.BytesInputDecompressor;
+import org.apache.parquet.format.DataPageHeader;
+import org.apache.parquet.format.DataPageHeaderV2;
+import org.apache.parquet.format.DictionaryPageHeader;
+import org.apache.parquet.format.PageHeader;
+import org.apache.parquet.format.Util;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.schema.PrimitiveType;
+
+/**
+ * {@link PageReadStore} over one row group's cached column-chunk buffers: page headers are parsed in
+ * place, page bytes are views of the cache buffers and are decompressed lazily on readPage, mirroring
+ * parquet-mr's ParquetFileReader.Chunk.readAllPages and ColumnChunkPageReader (no offset index, no decryption).
+ */
+class ParquetCachedPageReadStore implements PageReadStore {
+
+ private final Map<ColumnPath, PageReader> readers = new HashMap<>();
+ private final long rowCount;
+
+ ParquetCachedPageReadStore(ParquetMetadata footer, ParquetEncodedColumnBatch batch,
+ CompressionCodecFactory codecFactory, ParquetMetadataConverter converter) throws IOException {
+ BlockMetaData block = footer.getBlocks().get(batch.rowGroupIx);
+ this.rowCount = block.getRowCount();
+ String createdBy = footer.getFileMetaData().getCreatedBy();
+ for (int pc = 0; pc < batch.chunks.length; ++pc) {
+ ColumnChunkMetaData chunk = batch.chunks[pc];
+ readers.put(chunk.getPath(), readAllPages(chunk, chunkBuffers(batch, pc), createdBy,
+ codecFactory.getDecompressor(chunk.getCodec()), converter));
+ }
+ }
+
+ /** Slices of the cached buffers covering exactly the chunk's byte region, in file order. */
+ private static List<ByteBuffer> chunkBuffers(ParquetEncodedColumnBatch batch, int pc) {
+ long start = batch.chunks[pc].getStartingPos(), end = start + batch.chunks[pc].getTotalSize();
Review Comment:
please move `env` var on a new line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org