You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/02/05 13:16:52 UTC
[GitHub] [hudi] nsivabalan commented on a change in pull request #2494: [HUDI-1552] Improve performance of key lookups from base file in Metadata Table.
nsivabalan commented on a change in pull request #2494:
URL: https://github.com/apache/hudi/pull/2494#discussion_r570955118
##########
File path: hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java
##########
@@ -147,23 +184,21 @@ private void initIfNeeded() {
}
}
timings.add(timer.endTimer());
- LOG.info(String.format("Metadata read for key %s took [open, baseFileRead, logMerge] %s ms", key, timings));
+ LOG.info(String.format("Metadata read for key %s took [baseFileRead, logMerge] %s ms", key, timings));
return Option.ofNullable(hoodieRecord);
} catch (IOException ioe) {
throw new HoodieIOException("Error merging records from metadata table for key :" + key, ioe);
- } finally {
- closeIfNeeded();
}
}
/**
- * Open readers to the base and log files.
+ * Returns the readers to the base and log files.
+ *
+ * If reuse is allowed then cached readers are returned. Otherwise new readers are opened.
Review comment:
please correct me if I am wrong. within this method, we don't care about reuse enabled or not. So, wondering if we need to remove this comment?
##########
File path: hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java
##########
@@ -147,23 +184,21 @@ private void initIfNeeded() {
}
}
timings.add(timer.endTimer());
- LOG.info(String.format("Metadata read for key %s took [open, baseFileRead, logMerge] %s ms", key, timings));
+ LOG.info(String.format("Metadata read for key %s took [baseFileRead, logMerge] %s ms", key, timings));
return Option.ofNullable(hoodieRecord);
} catch (IOException ioe) {
throw new HoodieIOException("Error merging records from metadata table for key :" + key, ioe);
- } finally {
- closeIfNeeded();
}
}
/**
- * Open readers to the base and log files.
+ * Returns the readers to the base and log files.
+ *
+ * If reuse is allowed then cached readers are returned. Otherwise new readers are opened.
*/
- private synchronized void openFileSliceIfNeeded() throws IOException {
- if (metadataConfig.enableReuse() && baseFileReader != null) {
- // we will reuse what's open.
- return;
- }
+ private Pair<HoodieFileReader, HoodieMetadataMergedLogRecordScanner> openReaders() throws IOException {
+ HoodieFileReader baseFileReader = null;
Review comment:
may be we could name instance vars as cachedBaseFileReader and cachedLogFileReader. wdyt?
##########
File path: hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java
##########
@@ -147,23 +184,21 @@ private void initIfNeeded() {
}
}
timings.add(timer.endTimer());
- LOG.info(String.format("Metadata read for key %s took [open, baseFileRead, logMerge] %s ms", key, timings));
+ LOG.info(String.format("Metadata read for key %s took [baseFileRead, logMerge] %s ms", key, timings));
return Option.ofNullable(hoodieRecord);
} catch (IOException ioe) {
throw new HoodieIOException("Error merging records from metadata table for key :" + key, ioe);
- } finally {
- closeIfNeeded();
}
}
/**
- * Open readers to the base and log files.
+ * Returns the readers to the base and log files.
+ *
+ * If reuse is allowed then cached readers are returned. Otherwise new readers are opened.
*/
- private synchronized void openFileSliceIfNeeded() throws IOException {
- if (metadataConfig.enableReuse() && baseFileReader != null) {
- // we will reuse what's open.
- return;
- }
+ private Pair<HoodieFileReader, HoodieMetadataMergedLogRecordScanner> openReaders() throws IOException {
+ HoodieFileReader baseFileReader = null;
Review comment:
this is confusing to have same var names for both local variables and instance variables. Can you please change it. I was reviewing code assuming openReaders sets instance vars only to realize after a long time that these are local vars.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org