You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2020/06/09 16:00:49 UTC
[GitHub] [hive] rbalamohan opened a new pull request #1081: HIVE-23597: VectorizedOrcAcidRowBatchReader::ColumnizedDeleteEventReg…
rbalamohan opened a new pull request #1081:
URL: https://github.com/apache/hive/pull/1081
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] closed pull request #1081: HIVE-23597: VectorizedOrcAcidRowBatchReader::ColumnizedDeleteEventReg…
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #1081:
URL: https://github.com/apache/hive/pull/1081
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] rbalamohan commented on a change in pull request #1081: HIVE-23597: VectorizedOrcAcidRowBatchReader::ColumnizedDeleteEventReg…
Posted by GitBox <gi...@apache.org>.
rbalamohan commented on a change in pull request #1081:
URL: https://github.com/apache/hive/pull/1081#discussion_r438677418
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
##########
@@ -1561,24 +1572,22 @@ public int compareTo(CompressedOwid other) {
try {
final Path[] deleteDeltaDirs = getDeleteDeltaDirsFromSplit(orcSplit);
if (deleteDeltaDirs.length > 0) {
+ FileSystem fs = orcSplit.getPath().getFileSystem(conf);
+ AcidOutputFormat.Options orcSplitMinMaxWriteIds =
+ AcidUtils.parseBaseOrDeltaBucketFilename(orcSplit.getPath(), conf);
int totalDeleteEventCount = 0;
for (Path deleteDeltaDir : deleteDeltaDirs) {
- FileSystem fs = deleteDeltaDir.getFileSystem(conf);
+ if (!isQualifiedDeleteDeltaForSplit(orcSplitMinMaxWriteIds, deleteDeltaDir)) {
+ continue;
+ }
Path[] deleteDeltaFiles = OrcRawRecordMerger.getDeltaFiles(deleteDeltaDir, bucket,
new OrcRawRecordMerger.Options().isCompacting(false), null);
for (Path deleteDeltaFile : deleteDeltaFiles) {
- // NOTE: Calling last flush length below is more for future-proofing when we have
- // streaming deletes. But currently we don't support streaming deletes, and this can
- // be removed if this becomes a performance issue.
- long length = OrcAcidUtils.getLastFlushLength(fs, deleteDeltaFile);
+ // NOTE: When streaming deletes are supported, consider using OrcAcidUtils.getLastFlushLength(fs, deleteDeltaFile)
// NOTE: A check for existence of deleteDeltaFile is required because we may not have
// deletes for the bucket being taken into consideration for this split processing.
- if (length != -1 && fs.exists(deleteDeltaFile)) {
- /**
- * todo: we have OrcSplit.orcTail so we should be able to get stats from there
- */
- Reader deleteDeltaReader = OrcFile.createReader(deleteDeltaFile,
- OrcFile.readerOptions(conf).maxLength(length));
+ if (fs.exists(deleteDeltaFile)) {
Review comment:
Addressed the file.exists issue in recent commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #1081: HIVE-23597: VectorizedOrcAcidRowBatchReader::ColumnizedDeleteEventReg…
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #1081:
URL: https://github.com/apache/hive/pull/1081#discussion_r437191512
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
##########
@@ -1561,24 +1572,22 @@ public int compareTo(CompressedOwid other) {
try {
final Path[] deleteDeltaDirs = getDeleteDeltaDirsFromSplit(orcSplit);
if (deleteDeltaDirs.length > 0) {
+ FileSystem fs = orcSplit.getPath().getFileSystem(conf);
+ AcidOutputFormat.Options orcSplitMinMaxWriteIds =
+ AcidUtils.parseBaseOrDeltaBucketFilename(orcSplit.getPath(), conf);
int totalDeleteEventCount = 0;
for (Path deleteDeltaDir : deleteDeltaDirs) {
- FileSystem fs = deleteDeltaDir.getFileSystem(conf);
+ if (!isQualifiedDeleteDeltaForSplit(orcSplitMinMaxWriteIds, deleteDeltaDir)) {
+ continue;
+ }
Path[] deleteDeltaFiles = OrcRawRecordMerger.getDeltaFiles(deleteDeltaDir, bucket,
new OrcRawRecordMerger.Options().isCompacting(false), null);
for (Path deleteDeltaFile : deleteDeltaFiles) {
- // NOTE: Calling last flush length below is more for future-proofing when we have
- // streaming deletes. But currently we don't support streaming deletes, and this can
- // be removed if this becomes a performance issue.
- long length = OrcAcidUtils.getLastFlushLength(fs, deleteDeltaFile);
+ // NOTE: When streaming deletes are supported, consider using OrcAcidUtils.getLastFlushLength(fs, deleteDeltaFile)
// NOTE: A check for existence of deleteDeltaFile is required because we may not have
// deletes for the bucket being taken into consideration for this split processing.
- if (length != -1 && fs.exists(deleteDeltaFile)) {
- /**
- * todo: we have OrcSplit.orcTail so we should be able to get stats from there
- */
- Reader deleteDeltaReader = OrcFile.createReader(deleteDeltaFile,
- OrcFile.readerOptions(conf).maxLength(length));
+ if (fs.exists(deleteDeltaFile)) {
Review comment:
We might wan to get rid of this exists to save NN calls, and just handle FileNotFoundException in case of missing file (exists does the same inside it's method)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] commented on pull request #1081: HIVE-23597: VectorizedOrcAcidRowBatchReader::ColumnizedDeleteEventReg…
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #1081:
URL: https://github.com/apache/hive/pull/1081#issuecomment-684124193
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] rbalamohan commented on a change in pull request #1081: HIVE-23597: VectorizedOrcAcidRowBatchReader::ColumnizedDeleteEventReg…
Posted by GitBox <gi...@apache.org>.
rbalamohan commented on a change in pull request #1081:
URL: https://github.com/apache/hive/pull/1081#discussion_r447302965
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
##########
@@ -1605,6 +1618,46 @@ public int compareTo(CompressedOwid other) {
throw e; // rethrow the exception so that the caller can handle.
}
}
+
+ /**
+ * Create delete delta reader. Caching orc tail to avoid FS lookup/reads for repeated scans.
+ *
+ * @param deleteDeltaFile
+ * @param conf
+ * @param fs FileSystem
+ * @return delete file reader
+ * @throws IOException
+ */
+ private Reader getDeleteDeltaReader(Path deleteDeltaFile, JobConf conf, FileSystem fs) throws IOException {
+ OrcTail deleteDeltaTail = deleteDeltaOrcTailCache.getIfPresent(deleteDeltaFile);
Review comment:
Yes, it will not change for the file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #1081: HIVE-23597: VectorizedOrcAcidRowBatchReader::ColumnizedDeleteEventReg…
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #1081:
URL: https://github.com/apache/hive/pull/1081#discussion_r438762099
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
##########
@@ -1561,24 +1572,22 @@ public int compareTo(CompressedOwid other) {
try {
final Path[] deleteDeltaDirs = getDeleteDeltaDirsFromSplit(orcSplit);
if (deleteDeltaDirs.length > 0) {
+ FileSystem fs = orcSplit.getPath().getFileSystem(conf);
+ AcidOutputFormat.Options orcSplitMinMaxWriteIds =
+ AcidUtils.parseBaseOrDeltaBucketFilename(orcSplit.getPath(), conf);
int totalDeleteEventCount = 0;
for (Path deleteDeltaDir : deleteDeltaDirs) {
- FileSystem fs = deleteDeltaDir.getFileSystem(conf);
+ if (!isQualifiedDeleteDeltaForSplit(orcSplitMinMaxWriteIds, deleteDeltaDir)) {
+ continue;
+ }
Path[] deleteDeltaFiles = OrcRawRecordMerger.getDeltaFiles(deleteDeltaDir, bucket,
new OrcRawRecordMerger.Options().isCompacting(false), null);
for (Path deleteDeltaFile : deleteDeltaFiles) {
- // NOTE: Calling last flush length below is more for future-proofing when we have
- // streaming deletes. But currently we don't support streaming deletes, and this can
- // be removed if this becomes a performance issue.
- long length = OrcAcidUtils.getLastFlushLength(fs, deleteDeltaFile);
+ // NOTE: When streaming deletes are supported, consider using OrcAcidUtils.getLastFlushLength(fs, deleteDeltaFile)
// NOTE: A check for existence of deleteDeltaFile is required because we may not have
// deletes for the bucket being taken into consideration for this split processing.
- if (length != -1 && fs.exists(deleteDeltaFile)) {
- /**
- * todo: we have OrcSplit.orcTail so we should be able to get stats from there
- */
- Reader deleteDeltaReader = OrcFile.createReader(deleteDeltaFile,
- OrcFile.readerOptions(conf).maxLength(length));
+ if (fs.exists(deleteDeltaFile)) {
Review comment:
Looks good, let's get a green run :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kgyrtkirk closed pull request #1081: HIVE-23597: VectorizedOrcAcidRowBatchReader::ColumnizedDeleteEventReg…
Posted by GitBox <gi...@apache.org>.
kgyrtkirk closed pull request #1081:
URL: https://github.com/apache/hive/pull/1081
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] rbalamohan commented on a change in pull request #1081: HIVE-23597: VectorizedOrcAcidRowBatchReader::ColumnizedDeleteEventReg…
Posted by GitBox <gi...@apache.org>.
rbalamohan commented on a change in pull request #1081:
URL: https://github.com/apache/hive/pull/1081#discussion_r437820538
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
##########
@@ -1561,24 +1572,22 @@ public int compareTo(CompressedOwid other) {
try {
final Path[] deleteDeltaDirs = getDeleteDeltaDirsFromSplit(orcSplit);
if (deleteDeltaDirs.length > 0) {
+ FileSystem fs = orcSplit.getPath().getFileSystem(conf);
+ AcidOutputFormat.Options orcSplitMinMaxWriteIds =
+ AcidUtils.parseBaseOrDeltaBucketFilename(orcSplit.getPath(), conf);
int totalDeleteEventCount = 0;
for (Path deleteDeltaDir : deleteDeltaDirs) {
- FileSystem fs = deleteDeltaDir.getFileSystem(conf);
+ if (!isQualifiedDeleteDeltaForSplit(orcSplitMinMaxWriteIds, deleteDeltaDir)) {
+ continue;
+ }
Path[] deleteDeltaFiles = OrcRawRecordMerger.getDeltaFiles(deleteDeltaDir, bucket,
new OrcRawRecordMerger.Options().isCompacting(false), null);
for (Path deleteDeltaFile : deleteDeltaFiles) {
- // NOTE: Calling last flush length below is more for future-proofing when we have
- // streaming deletes. But currently we don't support streaming deletes, and this can
- // be removed if this becomes a performance issue.
- long length = OrcAcidUtils.getLastFlushLength(fs, deleteDeltaFile);
+ // NOTE: When streaming deletes are supported, consider using OrcAcidUtils.getLastFlushLength(fs, deleteDeltaFile)
// NOTE: A check for existence of deleteDeltaFile is required because we may not have
// deletes for the bucket being taken into consideration for this split processing.
- if (length != -1 && fs.exists(deleteDeltaFile)) {
- /**
- * todo: we have OrcSplit.orcTail so we should be able to get stats from there
- */
- Reader deleteDeltaReader = OrcFile.createReader(deleteDeltaFile,
- OrcFile.readerOptions(conf).maxLength(length));
+ if (fs.exists(deleteDeltaFile)) {
Review comment:
Yes, also runtime can be further reduced if DeleteReader values are in memory as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #1081: HIVE-23597: VectorizedOrcAcidRowBatchReader::ColumnizedDeleteEventReg…
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #1081:
URL: https://github.com/apache/hive/pull/1081#discussion_r446784234
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
##########
@@ -1605,6 +1618,46 @@ public int compareTo(CompressedOwid other) {
throw e; // rethrow the exception so that the caller can handle.
}
}
+
+ /**
+ * Create delete delta reader. Caching orc tail to avoid FS lookup/reads for repeated scans.
+ *
+ * @param deleteDeltaFile
+ * @param conf
+ * @param fs FileSystem
+ * @return delete file reader
+ * @throws IOException
+ */
+ private Reader getDeleteDeltaReader(Path deleteDeltaFile, JobConf conf, FileSystem fs) throws IOException {
+ OrcTail deleteDeltaTail = deleteDeltaOrcTailCache.getIfPresent(deleteDeltaFile);
Review comment:
Is the OrcTail thread safe? If I understandcorrectly the tail will be read by multiple LLAP threads concurrenly
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org