You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@druid.apache.org by ji...@apache.org on 2018/07/09 20:24:56 UTC
[incubator-druid] branch 0.12.2 updated: HdfsDataSegmentPusher:
Close tmpIndexFile before copying it. (#5873) (#5968)
This is an automated email from the ASF dual-hosted git repository.
jihoonson pushed a commit to branch 0.12.2
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/0.12.2 by this push:
new 0b7bf68 HdfsDataSegmentPusher: Close tmpIndexFile before copying it. (#5873) (#5968)
0b7bf68 is described below
commit 0b7bf6859eea9f0eb78fbcb6e42a019ace1586c5
Author: Jihoon Son <ji...@apache.org>
AuthorDate: Mon Jul 9 13:24:53 2018 -0700
HdfsDataSegmentPusher: Close tmpIndexFile before copying it. (#5873) (#5968)
It seems that copy-before-close works OK on HDFS, but it doesn't work
on all filesystems. In particular, we observed this not working properly
with Google Cloud Storage. And anyway, it's better hygiene to close files
before attempting to copy them somewhere else.
---
.../src/main/java/io/druid/storage/hdfs/HdfsDataSegmentPusher.java | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/extensions-core/hdfs-storage/src/main/java/io/druid/storage/hdfs/HdfsDataSegmentPusher.java b/extensions-core/hdfs-storage/src/main/java/io/druid/storage/hdfs/HdfsDataSegmentPusher.java
index 94b41e7..61e194b 100644
--- a/extensions-core/hdfs-storage/src/main/java/io/druid/storage/hdfs/HdfsDataSegmentPusher.java
+++ b/extensions-core/hdfs-storage/src/main/java/io/druid/storage/hdfs/HdfsDataSegmentPusher.java
@@ -114,8 +114,11 @@ public class HdfsDataSegmentPusher implements DataSegmentPusher
final long size;
final DataSegment dataSegment;
- try (FSDataOutputStream out = fs.create(tmpIndexFile)) {
- size = CompressionUtils.zip(inDir, out);
+ try {
+ try (FSDataOutputStream out = fs.create(tmpIndexFile)) {
+ size = CompressionUtils.zip(inDir, out);
+ }
+
final String uniquePrefix = useUniquePath ? DataSegmentPusher.generateUniquePath() + "_" : "";
final Path outIndexFile = new Path(StringUtils.format(
"%s/%s/%d_%sindex.zip",
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
For additional commands, e-mail: dev-help@druid.apache.org