You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@orc.apache.org by omalley <gi...@git.apache.org> on 2017/01/19 00:57:32 UTC
[GitHub] orc pull request #84: ORC-132. Implement a merge file method and fix the num...
GitHub user omalley opened a pull request:
https://github.com/apache/orc/pull/84
ORC-132. Implement a merge file method and fix the number of rows wri\u2026
\u2026tten
during merge. (omalley)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/omalley/orc orc-132
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/orc/pull/84.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #84
----
commit d6b673972feeb9869e693d1d07af86a7e23b4536
Author: Owen O'Malley <om...@apache.org>
Date: 2017-01-19T00:55:17Z
ORC-132. Implement a merge file method and fix the number of rows written
during merge. (omalley)
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc pull request #84: ORC-132. Implement a merge file method and fix the num...
Posted by omalley <gi...@git.apache.org>.
Github user omalley commented on a diff in the pull request:
https://github.com/apache/orc/pull/84#discussion_r96791802
--- Diff: java/core/src/test/org/apache/orc/TestVectorOrcFile.java ---
@@ -2944,4 +2948,164 @@ public void testWriterVersion() throws Exception {
assertEquals(OrcFile.WriterVersion.ORIGINAL, OrcFile.WriterVersion.from(0));
assertEquals(OrcFile.WriterVersion.HIVE_4243, OrcFile.WriterVersion.from(2));
}
+
+ /**
+ * Test whether the file versions are translated correctly
+ * @throws Exception
+ */
+ @Test
+ public void testFileVersion() throws Exception {
+ assertEquals(OrcFile.Version.V_0_11, ReaderImpl.getFileVersion(null));
+ assertEquals(OrcFile.Version.V_0_11, ReaderImpl.getFileVersion(new ArrayList<Integer>()));
+ assertEquals(OrcFile.Version.V_0_11,
+ ReaderImpl.getFileVersion(Arrays.asList(new Integer[]{0, 11})));
+ assertEquals(OrcFile.Version.V_0_12,
+ ReaderImpl.getFileVersion(Arrays.asList(new Integer[]{0, 12})));
+ assertEquals(OrcFile.Version.FUTURE,
+ ReaderImpl.getFileVersion(Arrays.asList(new Integer[]{9999, 0})));
+ }
+
+ @Test
+ public void testMergeUnderstood() throws Exception {
+ Path p = new Path("test.orc");
+ Reader futureVersion = Mockito.mock(Reader.class);
+ Mockito.when(futureVersion.getFileVersion()).thenReturn(OrcFile.Version.FUTURE);
+ Mockito.when(futureVersion.getWriterVersion()).thenReturn(OrcFile.WriterVersion.HIVE_4243);
+ assertEquals(false, OrcFile.understandFormat(p, futureVersion));
+ Reader futureWriter = Mockito.mock(Reader.class);
+ Mockito.when(futureWriter.getFileVersion()).thenReturn(OrcFile.Version.V_0_11);
+ Mockito.when(futureWriter.getWriterVersion()).thenReturn(OrcFile.WriterVersion.FUTURE);
+ assertEquals(false, OrcFile.understandFormat(p, futureWriter));
+ Reader current = Mockito.mock(Reader.class);
+ Mockito.when(current.getFileVersion()).thenReturn(OrcFile.Version.CURRENT);
+ Mockito.when(current.getWriterVersion()).thenReturn(OrcFile.CURRENT_WRITER);
+ assertEquals(true, OrcFile.understandFormat(p, current));
+ }
+
+ static ByteBuffer fromString(String s) {
+ return ByteBuffer.wrap(s.getBytes(StandardCharsets.UTF_8));
+ }
+
+ static byte[] fromInt(int x) {
+ return Integer.toHexString(x).getBytes(StandardCharsets.UTF_8);
+ }
+
+ @Test
+ public void testGoodMerge() throws Exception {
--- End diff --
The second merge in testGoodMerge, I did have two files that are dropped for being incompatible.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc pull request #84: ORC-132. Implement a merge file method and fix the num...
Posted by prasanthj <gi...@git.apache.org>.
Github user prasanthj commented on a diff in the pull request:
https://github.com/apache/orc/pull/84#discussion_r96788747
--- Diff: java/core/src/test/org/apache/orc/TestVectorOrcFile.java ---
@@ -2944,4 +2948,164 @@ public void testWriterVersion() throws Exception {
assertEquals(OrcFile.WriterVersion.ORIGINAL, OrcFile.WriterVersion.from(0));
assertEquals(OrcFile.WriterVersion.HIVE_4243, OrcFile.WriterVersion.from(2));
}
+
+ /**
+ * Test whether the file versions are translated correctly
+ * @throws Exception
+ */
+ @Test
+ public void testFileVersion() throws Exception {
+ assertEquals(OrcFile.Version.V_0_11, ReaderImpl.getFileVersion(null));
+ assertEquals(OrcFile.Version.V_0_11, ReaderImpl.getFileVersion(new ArrayList<Integer>()));
+ assertEquals(OrcFile.Version.V_0_11,
+ ReaderImpl.getFileVersion(Arrays.asList(new Integer[]{0, 11})));
+ assertEquals(OrcFile.Version.V_0_12,
+ ReaderImpl.getFileVersion(Arrays.asList(new Integer[]{0, 12})));
+ assertEquals(OrcFile.Version.FUTURE,
+ ReaderImpl.getFileVersion(Arrays.asList(new Integer[]{9999, 0})));
+ }
+
+ @Test
+ public void testMergeUnderstood() throws Exception {
+ Path p = new Path("test.orc");
+ Reader futureVersion = Mockito.mock(Reader.class);
+ Mockito.when(futureVersion.getFileVersion()).thenReturn(OrcFile.Version.FUTURE);
+ Mockito.when(futureVersion.getWriterVersion()).thenReturn(OrcFile.WriterVersion.HIVE_4243);
+ assertEquals(false, OrcFile.understandFormat(p, futureVersion));
+ Reader futureWriter = Mockito.mock(Reader.class);
+ Mockito.when(futureWriter.getFileVersion()).thenReturn(OrcFile.Version.V_0_11);
+ Mockito.when(futureWriter.getWriterVersion()).thenReturn(OrcFile.WriterVersion.FUTURE);
+ assertEquals(false, OrcFile.understandFormat(p, futureWriter));
+ Reader current = Mockito.mock(Reader.class);
+ Mockito.when(current.getFileVersion()).thenReturn(OrcFile.Version.CURRENT);
+ Mockito.when(current.getWriterVersion()).thenReturn(OrcFile.CURRENT_WRITER);
+ assertEquals(true, OrcFile.understandFormat(p, current));
+ }
+
+ static ByteBuffer fromString(String s) {
+ return ByteBuffer.wrap(s.getBytes(StandardCharsets.UTF_8));
+ }
+
+ static byte[] fromInt(int x) {
+ return Integer.toHexString(x).getBytes(StandardCharsets.UTF_8);
+ }
+
+ @Test
+ public void testGoodMerge() throws Exception {
--- End diff --
Will be good to have some tests for bad merge (incompatibility tests)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc pull request #84: ORC-132. Implement a merge file method and fix the num...
Posted by omalley <gi...@git.apache.org>.
Github user omalley commented on a diff in the pull request:
https://github.com/apache/orc/pull/84#discussion_r96791649
--- Diff: java/core/src/java/org/apache/orc/OrcFile.java ---
@@ -642,4 +671,187 @@ public static Writer createWriter(Path path,
return new WriterImpl(fs, path, opts);
}
+ /**
+ * Do we understand the version in the reader?
+ * @param path the path of the file
+ * @param reader the ORC file reader
+ * @return is the version understood by this writer?
+ */
+ static boolean understandFormat(Path path, Reader reader) {
+ if (reader.getFileVersion() == Version.FUTURE) {
+ LOG.info("Can't merge {} because it has a future version.", path);
+ return false;
+ }
+ if (reader.getWriterVersion() == WriterVersion.FUTURE) {
+ LOG.info("Can't merge {} because it has a future writerVersion.", path);
+ return false;
+ }
+ return true;
+ }
+
+ /**
+ * Is the new reader compatible with the file that is being written?
+ * @param schema the writer schema
+ * @param fileVersion the writer fileVersion
+ * @param writerVersion the writer writerVersion
+ * @param compressionSize the compression buffer size
+ * @param rowIndexStride the row index stride
+ * @param compression the compression that was used
+ * @param userMetadata the user metadata
+ * @param path the new path name for warning messages
+ * @param reader the new reader
+ * @return is the reader compatible with the previous ones?
+ */
+ static boolean readerIsCompatible(TypeDescription schema,
+ Version fileVersion,
+ WriterVersion writerVersion,
+ int compressionSize,
+ int rowIndexStride,
+ CompressionKind compression,
+ Map<String, ByteBuffer> userMetadata,
+ Path path,
+ Reader reader) {
+ // now we have to check compatibility
+ if (!reader.getSchema().equals(schema)) {
+ LOG.info("Can't merge {} because of different schemas {} vs {}",
+ path, reader.getSchema(), schema);
+ return false;
+ }
+ if (reader.getCompressionKind() != compression) {
+ LOG.info("Can't merge {} because of different compression {} vs {}",
+ path, reader.getCompressionKind(), compression);
+ return false;
+ }
+ if (compression != CompressionKind.NONE &&
+ reader.getCompressionSize() != compressionSize) {
+ LOG.info("Can't merge {} because of different compression sizes {} vs {}",
+ path, reader.getCompressionSize(), compressionSize);
+ return false;
+ }
+ if (reader.getFileVersion() != fileVersion) {
+ LOG.info("Can't merge {} because of different file versions {} vs {}",
+ path, reader.getFileVersion(), fileVersion);
+ return false;
+ }
+ if (reader.getWriterVersion() != writerVersion) {
+ LOG.info("Can't merge {} because of different writer versions {} vs {}",
+ path, reader.getFileVersion(), fileVersion);
+ return false;
+ }
+ if (reader.getRowIndexStride() != rowIndexStride) {
+ LOG.info("Can't merge {} because of different row index strides {} vs {}",
+ path, reader.getRowIndexStride(), rowIndexStride);
+ return false;
+ }
+ for(String key: reader.getMetadataKeys()) {
+ if (userMetadata.containsKey(key)) {
+ ByteBuffer currentValue = userMetadata.get(key);
+ ByteBuffer newValue = reader.getMetadataValue(key);
+ if (!newValue.equals(currentValue)) {
+ LOG.info("Can't merge {} because of different user metadata {}", path,
+ key);
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ static void mergeMetadata(Map<String,ByteBuffer> metadata,
+ Reader reader) {
+ for(String key: reader.getMetadataKeys()) {
+ metadata.put(key, reader.getMetadataValue(key));
+ }
+ }
+
+ /**
+ * Merges multiple ORC files that all have the same schema to produce
+ * a single ORC file.
+ * The merge will reject files that aren't compatible with the merged file
+ * so the output list may be shorter than the input list.
+ * The stripes are copied as serialized byte buffers.
+ * The user metadata are merged and files that disagree on the value
+ * associated with a key will be rejected.
+ *
+ * @param outputPath the output file
+ * @param options the options for writing with although the options related
+ * to the input files' encodings are overridden
+ * @param inputFiles the list of files to merge
+ * @return the list of files that were successfully merged
+ * @throws IOException
+ */
+ public static List<Path> mergeFiles(Path outputPath,
+ WriterOptions options,
+ Path... inputFiles) throws IOException {
--- End diff --
Really? It isn't clear to me that List<Path> is that much easier than a Path[] and the Path[] requires 1 fewer objects. But Ok.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc pull request #84: ORC-132. Implement a merge file method and fix the num...
Posted by prasanthj <gi...@git.apache.org>.
Github user prasanthj commented on a diff in the pull request:
https://github.com/apache/orc/pull/84#discussion_r96788146
--- Diff: java/core/src/java/org/apache/orc/OrcFile.java ---
@@ -642,4 +671,187 @@ public static Writer createWriter(Path path,
return new WriterImpl(fs, path, opts);
}
+ /**
+ * Do we understand the version in the reader?
+ * @param path the path of the file
+ * @param reader the ORC file reader
+ * @return is the version understood by this writer?
+ */
+ static boolean understandFormat(Path path, Reader reader) {
+ if (reader.getFileVersion() == Version.FUTURE) {
+ LOG.info("Can't merge {} because it has a future version.", path);
+ return false;
+ }
+ if (reader.getWriterVersion() == WriterVersion.FUTURE) {
+ LOG.info("Can't merge {} because it has a future writerVersion.", path);
+ return false;
+ }
+ return true;
+ }
+
+ /**
+ * Is the new reader compatible with the file that is being written?
+ * @param schema the writer schema
+ * @param fileVersion the writer fileVersion
+ * @param writerVersion the writer writerVersion
+ * @param compressionSize the compression buffer size
+ * @param rowIndexStride the row index stride
+ * @param compression the compression that was used
+ * @param userMetadata the user metadata
+ * @param path the new path name for warning messages
+ * @param reader the new reader
+ * @return is the reader compatible with the previous ones?
+ */
+ static boolean readerIsCompatible(TypeDescription schema,
+ Version fileVersion,
+ WriterVersion writerVersion,
+ int compressionSize,
+ int rowIndexStride,
+ CompressionKind compression,
+ Map<String, ByteBuffer> userMetadata,
+ Path path,
+ Reader reader) {
+ // now we have to check compatibility
+ if (!reader.getSchema().equals(schema)) {
+ LOG.info("Can't merge {} because of different schemas {} vs {}",
+ path, reader.getSchema(), schema);
+ return false;
+ }
+ if (reader.getCompressionKind() != compression) {
+ LOG.info("Can't merge {} because of different compression {} vs {}",
+ path, reader.getCompressionKind(), compression);
+ return false;
+ }
+ if (compression != CompressionKind.NONE &&
+ reader.getCompressionSize() != compressionSize) {
+ LOG.info("Can't merge {} because of different compression sizes {} vs {}",
+ path, reader.getCompressionSize(), compressionSize);
+ return false;
+ }
+ if (reader.getFileVersion() != fileVersion) {
+ LOG.info("Can't merge {} because of different file versions {} vs {}",
+ path, reader.getFileVersion(), fileVersion);
+ return false;
+ }
+ if (reader.getWriterVersion() != writerVersion) {
+ LOG.info("Can't merge {} because of different writer versions {} vs {}",
+ path, reader.getFileVersion(), fileVersion);
+ return false;
+ }
+ if (reader.getRowIndexStride() != rowIndexStride) {
+ LOG.info("Can't merge {} because of different row index strides {} vs {}",
+ path, reader.getRowIndexStride(), rowIndexStride);
+ return false;
+ }
+ for(String key: reader.getMetadataKeys()) {
+ if (userMetadata.containsKey(key)) {
+ ByteBuffer currentValue = userMetadata.get(key);
+ ByteBuffer newValue = reader.getMetadataValue(key);
+ if (!newValue.equals(currentValue)) {
+ LOG.info("Can't merge {} because of different user metadata {}", path,
+ key);
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ static void mergeMetadata(Map<String,ByteBuffer> metadata,
+ Reader reader) {
+ for(String key: reader.getMetadataKeys()) {
+ metadata.put(key, reader.getMetadataValue(key));
+ }
+ }
+
+ /**
+ * Merges multiple ORC files that all have the same schema to produce
+ * a single ORC file.
+ * The merge will reject files that aren't compatible with the merged file
+ * so the output list may be shorter than the input list.
+ * The stripes are copied as serialized byte buffers.
+ * The user metadata are merged and files that disagree on the value
+ * associated with a key will be rejected.
+ *
+ * @param outputPath the output file
+ * @param options the options for writing with although the options related
+ * to the input files' encodings are overridden
+ * @param inputFiles the list of files to merge
+ * @return the list of files that were successfully merged
+ * @throws IOException
+ */
+ public static List<Path> mergeFiles(Path outputPath,
+ WriterOptions options,
+ Path... inputFiles) throws IOException {
--- End diff --
List instead of varargs? Or another interface that converts List to varargs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc pull request #84: ORC-132. Implement a merge file method and fix the num...
Posted by prasanthj <gi...@git.apache.org>.
Github user prasanthj commented on a diff in the pull request:
https://github.com/apache/orc/pull/84#discussion_r96799080
--- Diff: java/core/src/java/org/apache/orc/OrcFile.java ---
@@ -642,4 +671,187 @@ public static Writer createWriter(Path path,
return new WriterImpl(fs, path, opts);
}
+ /**
+ * Do we understand the version in the reader?
+ * @param path the path of the file
+ * @param reader the ORC file reader
+ * @return is the version understood by this writer?
+ */
+ static boolean understandFormat(Path path, Reader reader) {
+ if (reader.getFileVersion() == Version.FUTURE) {
+ LOG.info("Can't merge {} because it has a future version.", path);
+ return false;
+ }
+ if (reader.getWriterVersion() == WriterVersion.FUTURE) {
+ LOG.info("Can't merge {} because it has a future writerVersion.", path);
+ return false;
+ }
+ return true;
+ }
+
+ /**
+ * Is the new reader compatible with the file that is being written?
+ * @param schema the writer schema
+ * @param fileVersion the writer fileVersion
+ * @param writerVersion the writer writerVersion
+ * @param compressionSize the compression buffer size
+ * @param rowIndexStride the row index stride
+ * @param compression the compression that was used
+ * @param userMetadata the user metadata
+ * @param path the new path name for warning messages
+ * @param reader the new reader
+ * @return is the reader compatible with the previous ones?
+ */
+ static boolean readerIsCompatible(TypeDescription schema,
+ Version fileVersion,
+ WriterVersion writerVersion,
+ int compressionSize,
+ int rowIndexStride,
+ CompressionKind compression,
+ Map<String, ByteBuffer> userMetadata,
+ Path path,
+ Reader reader) {
+ // now we have to check compatibility
+ if (!reader.getSchema().equals(schema)) {
+ LOG.info("Can't merge {} because of different schemas {} vs {}",
+ path, reader.getSchema(), schema);
+ return false;
+ }
+ if (reader.getCompressionKind() != compression) {
+ LOG.info("Can't merge {} because of different compression {} vs {}",
+ path, reader.getCompressionKind(), compression);
+ return false;
+ }
+ if (compression != CompressionKind.NONE &&
+ reader.getCompressionSize() != compressionSize) {
+ LOG.info("Can't merge {} because of different compression sizes {} vs {}",
+ path, reader.getCompressionSize(), compressionSize);
+ return false;
+ }
+ if (reader.getFileVersion() != fileVersion) {
+ LOG.info("Can't merge {} because of different file versions {} vs {}",
+ path, reader.getFileVersion(), fileVersion);
+ return false;
+ }
+ if (reader.getWriterVersion() != writerVersion) {
+ LOG.info("Can't merge {} because of different writer versions {} vs {}",
+ path, reader.getFileVersion(), fileVersion);
+ return false;
+ }
+ if (reader.getRowIndexStride() != rowIndexStride) {
+ LOG.info("Can't merge {} because of different row index strides {} vs {}",
+ path, reader.getRowIndexStride(), rowIndexStride);
+ return false;
+ }
+ for(String key: reader.getMetadataKeys()) {
+ if (userMetadata.containsKey(key)) {
+ ByteBuffer currentValue = userMetadata.get(key);
+ ByteBuffer newValue = reader.getMetadataValue(key);
+ if (!newValue.equals(currentValue)) {
+ LOG.info("Can't merge {} because of different user metadata {}", path,
+ key);
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ static void mergeMetadata(Map<String,ByteBuffer> metadata,
+ Reader reader) {
+ for(String key: reader.getMetadataKeys()) {
+ metadata.put(key, reader.getMetadataValue(key));
+ }
+ }
+
+ /**
+ * Merges multiple ORC files that all have the same schema to produce
+ * a single ORC file.
+ * The merge will reject files that aren't compatible with the merged file
+ * so the output list may be shorter than the input list.
+ * The stripes are copied as serialized byte buffers.
+ * The user metadata are merged and files that disagree on the value
+ * associated with a key will be rejected.
+ *
+ * @param outputPath the output file
+ * @param options the options for writing with although the options related
+ * to the input files' encodings are overridden
+ * @param inputFiles the list of files to merge
+ * @return the list of files that were successfully merged
+ * @throws IOException
+ */
+ public static List<Path> mergeFiles(Path outputPath,
+ WriterOptions options,
+ Path... inputFiles) throws IOException {
+ Writer output = null;
+ byte[] buffer = new byte[0];
+ Configuration conf = options.getConfiguration();
+ TypeDescription schema = null;
+ CompressionKind compression = null;
+ int bufferSize = 0;
+ Version fileVersion = null;
+ WriterVersion writerVersion = null;
+ int rowIndexStride = 0;
+ List<Path> result = new ArrayList<>(inputFiles.length);
+ Map<String, ByteBuffer> userMetadata = new HashMap<>();
+
+ for (Path input: inputFiles) {
+ FileSystem fs = input.getFileSystem(conf);
+ Reader reader = createReader(input,
+ readerOptions(options.getConfiguration()).filesystem(fs));
+
+ if (!understandFormat(input, reader)) {
+ continue;
+ } else if (schema == null) {
+ // if this is the first file that we are including, grab the values
+ schema = reader.getSchema();
+ compression = reader.getCompressionKind();
+ bufferSize = reader.getCompressionSize();
+ rowIndexStride = reader.getRowIndexStride();
+ fileVersion = reader.getFileVersion();
+ writerVersion = reader.getWriterVersion();
+ options.blockSize(bufferSize)
+ .version(fileVersion)
+ .writerVersion(writerVersion)
+ .compress(compression)
+ .rowIndexStride(rowIndexStride)
+ .setSchema(schema);
+ if (compression != CompressionKind.NONE) {
+ options.enforceBufferSize().bufferSize(bufferSize);
+ }
+ mergeMetadata(userMetadata, reader);
+ output = createWriter(outputPath, options);
+ } else if (!readerIsCompatible(schema, fileVersion, writerVersion,
+ bufferSize, rowIndexStride, compression, userMetadata, input,
+ reader)) {
+ continue;
+ } else {
+ mergeMetadata(userMetadata, reader);
+ }
+ List<OrcProto.StripeStatistics> statList =
+ reader.getOrcProtoStripeStatistics();
+ FSDataInputStream inputStream = fs.open(input);
+ int stripeNum = 0;
+ result.add(input);
+
+ for(StripeInformation stripe: reader.getStripes()) {
+ int length = (int) stripe.getLength();
+ if (buffer.length < length) {
+ buffer = new byte[length];
+ }
+ long offset = stripe.getOffset();
+ inputStream.readFully(offset, buffer, 0, length);
--- End diff --
Any exception in the entire code can leave the output file open and not readable. Isn't it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc pull request #84: ORC-132. Implement a merge file method and fix the num...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/orc/pull/84
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc issue #84: ORC-132. Implement a merge file method and fix the number of ...
Posted by omalley <gi...@git.apache.org>.
Github user omalley commented on the issue:
https://github.com/apache/orc/pull/84
The new version of the patch:
* uses the maximum buffer size of the input files rather than reject them.
* uses List rather than Path[] as the input parameter
* tests that the buffer size is updated correctly
* renames the testGoodMerge to testMerge since it tests both good and bad merges
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc pull request #84: ORC-132. Implement a merge file method and fix the num...
Posted by prasanthj <gi...@git.apache.org>.
Github user prasanthj commented on a diff in the pull request:
https://github.com/apache/orc/pull/84#discussion_r96939713
--- Diff: java/core/src/java/org/apache/orc/OrcFile.java ---
@@ -642,4 +671,187 @@ public static Writer createWriter(Path path,
return new WriterImpl(fs, path, opts);
}
+ /**
+ * Do we understand the version in the reader?
+ * @param path the path of the file
+ * @param reader the ORC file reader
+ * @return is the version understood by this writer?
+ */
+ static boolean understandFormat(Path path, Reader reader) {
+ if (reader.getFileVersion() == Version.FUTURE) {
+ LOG.info("Can't merge {} because it has a future version.", path);
+ return false;
+ }
+ if (reader.getWriterVersion() == WriterVersion.FUTURE) {
+ LOG.info("Can't merge {} because it has a future writerVersion.", path);
+ return false;
+ }
+ return true;
+ }
+
+ /**
+ * Is the new reader compatible with the file that is being written?
+ * @param schema the writer schema
+ * @param fileVersion the writer fileVersion
+ * @param writerVersion the writer writerVersion
+ * @param compressionSize the compression buffer size
+ * @param rowIndexStride the row index stride
+ * @param compression the compression that was used
+ * @param userMetadata the user metadata
+ * @param path the new path name for warning messages
+ * @param reader the new reader
+ * @return is the reader compatible with the previous ones?
+ */
+ static boolean readerIsCompatible(TypeDescription schema,
+ Version fileVersion,
+ WriterVersion writerVersion,
+ int compressionSize,
+ int rowIndexStride,
+ CompressionKind compression,
+ Map<String, ByteBuffer> userMetadata,
+ Path path,
+ Reader reader) {
+ // now we have to check compatibility
+ if (!reader.getSchema().equals(schema)) {
+ LOG.info("Can't merge {} because of different schemas {} vs {}",
+ path, reader.getSchema(), schema);
+ return false;
+ }
+ if (reader.getCompressionKind() != compression) {
+ LOG.info("Can't merge {} because of different compression {} vs {}",
+ path, reader.getCompressionKind(), compression);
+ return false;
+ }
+ if (compression != CompressionKind.NONE &&
+ reader.getCompressionSize() != compressionSize) {
+ LOG.info("Can't merge {} because of different compression sizes {} vs {}",
+ path, reader.getCompressionSize(), compressionSize);
+ return false;
+ }
+ if (reader.getFileVersion() != fileVersion) {
+ LOG.info("Can't merge {} because of different file versions {} vs {}",
+ path, reader.getFileVersion(), fileVersion);
+ return false;
+ }
+ if (reader.getWriterVersion() != writerVersion) {
+ LOG.info("Can't merge {} because of different writer versions {} vs {}",
+ path, reader.getFileVersion(), fileVersion);
+ return false;
+ }
+ if (reader.getRowIndexStride() != rowIndexStride) {
+ LOG.info("Can't merge {} because of different row index strides {} vs {}",
+ path, reader.getRowIndexStride(), rowIndexStride);
+ return false;
+ }
+ for(String key: reader.getMetadataKeys()) {
+ if (userMetadata.containsKey(key)) {
+ ByteBuffer currentValue = userMetadata.get(key);
+ ByteBuffer newValue = reader.getMetadataValue(key);
+ if (!newValue.equals(currentValue)) {
+ LOG.info("Can't merge {} because of different user metadata {}", path,
+ key);
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ static void mergeMetadata(Map<String,ByteBuffer> metadata,
+ Reader reader) {
+ for(String key: reader.getMetadataKeys()) {
+ metadata.put(key, reader.getMetadataValue(key));
+ }
+ }
+
+ /**
+ * Merges multiple ORC files that all have the same schema to produce
+ * a single ORC file.
+ * The merge will reject files that aren't compatible with the merged file
+ * so the output list may be shorter than the input list.
+ * The stripes are copied as serialized byte buffers.
+ * The user metadata are merged and files that disagree on the value
+ * associated with a key will be rejected.
+ *
+ * @param outputPath the output file
+ * @param options the options for writing with although the options related
+ * to the input files' encodings are overridden
+ * @param inputFiles the list of files to merge
+ * @return the list of files that were successfully merged
+ * @throws IOException
+ */
+ public static List<Path> mergeFiles(Path outputPath,
+ WriterOptions options,
+ Path... inputFiles) throws IOException {
+ Writer output = null;
+ byte[] buffer = new byte[0];
+ Configuration conf = options.getConfiguration();
+ TypeDescription schema = null;
+ CompressionKind compression = null;
+ int bufferSize = 0;
+ Version fileVersion = null;
+ WriterVersion writerVersion = null;
+ int rowIndexStride = 0;
+ List<Path> result = new ArrayList<>(inputFiles.length);
+ Map<String, ByteBuffer> userMetadata = new HashMap<>();
+
+ for (Path input: inputFiles) {
+ FileSystem fs = input.getFileSystem(conf);
+ Reader reader = createReader(input,
+ readerOptions(options.getConfiguration()).filesystem(fs));
+
+ if (!understandFormat(input, reader)) {
+ continue;
+ } else if (schema == null) {
+ // if this is the first file that we are including, grab the values
+ schema = reader.getSchema();
+ compression = reader.getCompressionKind();
+ bufferSize = reader.getCompressionSize();
+ rowIndexStride = reader.getRowIndexStride();
+ fileVersion = reader.getFileVersion();
+ writerVersion = reader.getWriterVersion();
+ options.blockSize(bufferSize)
+ .version(fileVersion)
+ .writerVersion(writerVersion)
+ .compress(compression)
+ .rowIndexStride(rowIndexStride)
+ .setSchema(schema);
+ if (compression != CompressionKind.NONE) {
+ options.enforceBufferSize().bufferSize(bufferSize);
+ }
+ mergeMetadata(userMetadata, reader);
+ output = createWriter(outputPath, options);
+ } else if (!readerIsCompatible(schema, fileVersion, writerVersion,
+ bufferSize, rowIndexStride, compression, userMetadata, input,
+ reader)) {
+ continue;
+ } else {
+ mergeMetadata(userMetadata, reader);
+ }
+ List<OrcProto.StripeStatistics> statList =
+ reader.getOrcProtoStripeStatistics();
+ FSDataInputStream inputStream = fs.open(input);
+ int stripeNum = 0;
+ result.add(input);
+
+ for(StripeInformation stripe: reader.getStripes()) {
+ int length = (int) stripe.getLength();
+ if (buffer.length < length) {
+ buffer = new byte[length];
+ }
+ long offset = stripe.getOffset();
+ inputStream.readFully(offset, buffer, 0, length);
--- End diff --
Appending stripes throwing exception could be for different reasons than writer and some of them are still avoidable. For example: Merging v11 files without stripe statistics will return null for statList throwing NPE. This could be skipped or avoided without leaving corrupted output file.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc pull request #84: ORC-132. Implement a merge file method and fix the num...
Posted by prasanthj <gi...@git.apache.org>.
Github user prasanthj commented on a diff in the pull request:
https://github.com/apache/orc/pull/84#discussion_r96788487
--- Diff: java/core/src/java/org/apache/orc/OrcFile.java ---
@@ -642,4 +671,187 @@ public static Writer createWriter(Path path,
return new WriterImpl(fs, path, opts);
}
+ /**
+ * Do we understand the version in the reader?
+ * @param path the path of the file
+ * @param reader the ORC file reader
+ * @return is the version understood by this writer?
+ */
+ static boolean understandFormat(Path path, Reader reader) {
+ if (reader.getFileVersion() == Version.FUTURE) {
+ LOG.info("Can't merge {} because it has a future version.", path);
+ return false;
+ }
+ if (reader.getWriterVersion() == WriterVersion.FUTURE) {
+ LOG.info("Can't merge {} because it has a future writerVersion.", path);
+ return false;
+ }
+ return true;
+ }
+
+ /**
+ * Is the new reader compatible with the file that is being written?
+ * @param schema the writer schema
+ * @param fileVersion the writer fileVersion
+ * @param writerVersion the writer writerVersion
+ * @param compressionSize the compression buffer size
+ * @param rowIndexStride the row index stride
+ * @param compression the compression that was used
+ * @param userMetadata the user metadata
+ * @param path the new path name for warning messages
+ * @param reader the new reader
+ * @return is the reader compatible with the previous ones?
+ */
+ static boolean readerIsCompatible(TypeDescription schema,
+ Version fileVersion,
+ WriterVersion writerVersion,
+ int compressionSize,
+ int rowIndexStride,
+ CompressionKind compression,
+ Map<String, ByteBuffer> userMetadata,
+ Path path,
+ Reader reader) {
+ // now we have to check compatibility
+ if (!reader.getSchema().equals(schema)) {
+ LOG.info("Can't merge {} because of different schemas {} vs {}",
+ path, reader.getSchema(), schema);
+ return false;
+ }
+ if (reader.getCompressionKind() != compression) {
+ LOG.info("Can't merge {} because of different compression {} vs {}",
+ path, reader.getCompressionKind(), compression);
+ return false;
+ }
+ if (compression != CompressionKind.NONE &&
+ reader.getCompressionSize() != compressionSize) {
+ LOG.info("Can't merge {} because of different compression sizes {} vs {}",
+ path, reader.getCompressionSize(), compressionSize);
+ return false;
+ }
+ if (reader.getFileVersion() != fileVersion) {
+ LOG.info("Can't merge {} because of different file versions {} vs {}",
+ path, reader.getFileVersion(), fileVersion);
+ return false;
+ }
+ if (reader.getWriterVersion() != writerVersion) {
+ LOG.info("Can't merge {} because of different writer versions {} vs {}",
+ path, reader.getFileVersion(), fileVersion);
+ return false;
+ }
+ if (reader.getRowIndexStride() != rowIndexStride) {
+ LOG.info("Can't merge {} because of different row index strides {} vs {}",
+ path, reader.getRowIndexStride(), rowIndexStride);
+ return false;
+ }
+ for(String key: reader.getMetadataKeys()) {
+ if (userMetadata.containsKey(key)) {
+ ByteBuffer currentValue = userMetadata.get(key);
+ ByteBuffer newValue = reader.getMetadataValue(key);
+ if (!newValue.equals(currentValue)) {
+ LOG.info("Can't merge {} because of different user metadata {}", path,
+ key);
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ static void mergeMetadata(Map<String,ByteBuffer> metadata,
+ Reader reader) {
+ for(String key: reader.getMetadataKeys()) {
+ metadata.put(key, reader.getMetadataValue(key));
+ }
+ }
+
+ /**
+ * Merges multiple ORC files that all have the same schema to produce
+ * a single ORC file.
+ * The merge will reject files that aren't compatible with the merged file
+ * so the output list may be shorter than the input list.
+ * The stripes are copied as serialized byte buffers.
+ * The user metadata are merged and files that disagree on the value
+ * associated with a key will be rejected.
+ *
+ * @param outputPath the output file
+ * @param options the options for writing with although the options related
+ * to the input files' encodings are overridden
+ * @param inputFiles the list of files to merge
+ * @return the list of files that were successfully merged
+ * @throws IOException
+ */
+ public static List<Path> mergeFiles(Path outputPath,
+ WriterOptions options,
+ Path... inputFiles) throws IOException {
+ Writer output = null;
+ byte[] buffer = new byte[0];
+ Configuration conf = options.getConfiguration();
+ TypeDescription schema = null;
+ CompressionKind compression = null;
+ int bufferSize = 0;
+ Version fileVersion = null;
+ WriterVersion writerVersion = null;
+ int rowIndexStride = 0;
+ List<Path> result = new ArrayList<>(inputFiles.length);
+ Map<String, ByteBuffer> userMetadata = new HashMap<>();
+
+ for (Path input: inputFiles) {
+ FileSystem fs = input.getFileSystem(conf);
+ Reader reader = createReader(input,
+ readerOptions(options.getConfiguration()).filesystem(fs));
+
+ if (!understandFormat(input, reader)) {
+ continue;
+ } else if (schema == null) {
+ // if this is the first file that we are including, grab the values
+ schema = reader.getSchema();
+ compression = reader.getCompressionKind();
+ bufferSize = reader.getCompressionSize();
+ rowIndexStride = reader.getRowIndexStride();
+ fileVersion = reader.getFileVersion();
+ writerVersion = reader.getWriterVersion();
+ options.blockSize(bufferSize)
+ .version(fileVersion)
+ .writerVersion(writerVersion)
+ .compress(compression)
+ .rowIndexStride(rowIndexStride)
+ .setSchema(schema);
+ if (compression != CompressionKind.NONE) {
+ options.enforceBufferSize().bufferSize(bufferSize);
+ }
+ mergeMetadata(userMetadata, reader);
+ output = createWriter(outputPath, options);
+ } else if (!readerIsCompatible(schema, fileVersion, writerVersion,
+ bufferSize, rowIndexStride, compression, userMetadata, input,
+ reader)) {
+ continue;
+ } else {
+ mergeMetadata(userMetadata, reader);
+ }
+ List<OrcProto.StripeStatistics> statList =
+ reader.getOrcProtoStripeStatistics();
+ FSDataInputStream inputStream = fs.open(input);
+ int stripeNum = 0;
+ result.add(input);
+
+ for(StripeInformation stripe: reader.getStripes()) {
+ int length = (int) stripe.getLength();
+ if (buffer.length < length) {
+ buffer = new byte[length];
+ }
+ long offset = stripe.getOffset();
+ inputStream.readFully(offset, buffer, 0, length);
--- End diff --
wrap this inside try .. catch? readFully throws EOFException
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc pull request #84: ORC-132. Implement a merge file method and fix the num...
Posted by omalley <gi...@git.apache.org>.
Github user omalley commented on a diff in the pull request:
https://github.com/apache/orc/pull/84#discussion_r96932371
--- Diff: java/core/src/java/org/apache/orc/OrcFile.java ---
@@ -642,4 +671,187 @@ public static Writer createWriter(Path path,
return new WriterImpl(fs, path, opts);
}
+ /**
+ * Do we understand the version in the reader?
+ * @param path the path of the file
+ * @param reader the ORC file reader
+ * @return is the version understood by this writer?
+ */
+ static boolean understandFormat(Path path, Reader reader) {
+ if (reader.getFileVersion() == Version.FUTURE) {
+ LOG.info("Can't merge {} because it has a future version.", path);
+ return false;
+ }
+ if (reader.getWriterVersion() == WriterVersion.FUTURE) {
+ LOG.info("Can't merge {} because it has a future writerVersion.", path);
+ return false;
+ }
+ return true;
+ }
+
+ /**
+ * Is the new reader compatible with the file that is being written?
+ * @param schema the writer schema
+ * @param fileVersion the writer fileVersion
+ * @param writerVersion the writer writerVersion
+ * @param compressionSize the compression buffer size
+ * @param rowIndexStride the row index stride
+ * @param compression the compression that was used
+ * @param userMetadata the user metadata
+ * @param path the new path name for warning messages
+ * @param reader the new reader
+ * @return is the reader compatible with the previous ones?
+ */
+ static boolean readerIsCompatible(TypeDescription schema,
+ Version fileVersion,
+ WriterVersion writerVersion,
+ int compressionSize,
+ int rowIndexStride,
+ CompressionKind compression,
+ Map<String, ByteBuffer> userMetadata,
+ Path path,
+ Reader reader) {
+ // now we have to check compatibility
+ if (!reader.getSchema().equals(schema)) {
+ LOG.info("Can't merge {} because of different schemas {} vs {}",
+ path, reader.getSchema(), schema);
+ return false;
+ }
+ if (reader.getCompressionKind() != compression) {
+ LOG.info("Can't merge {} because of different compression {} vs {}",
+ path, reader.getCompressionKind(), compression);
+ return false;
+ }
+ if (compression != CompressionKind.NONE &&
+ reader.getCompressionSize() != compressionSize) {
+ LOG.info("Can't merge {} because of different compression sizes {} vs {}",
+ path, reader.getCompressionSize(), compressionSize);
+ return false;
+ }
+ if (reader.getFileVersion() != fileVersion) {
+ LOG.info("Can't merge {} because of different file versions {} vs {}",
+ path, reader.getFileVersion(), fileVersion);
+ return false;
+ }
+ if (reader.getWriterVersion() != writerVersion) {
+ LOG.info("Can't merge {} because of different writer versions {} vs {}",
+ path, reader.getFileVersion(), fileVersion);
+ return false;
+ }
+ if (reader.getRowIndexStride() != rowIndexStride) {
+ LOG.info("Can't merge {} because of different row index strides {} vs {}",
+ path, reader.getRowIndexStride(), rowIndexStride);
+ return false;
+ }
+ for(String key: reader.getMetadataKeys()) {
+ if (userMetadata.containsKey(key)) {
+ ByteBuffer currentValue = userMetadata.get(key);
+ ByteBuffer newValue = reader.getMetadataValue(key);
+ if (!newValue.equals(currentValue)) {
+ LOG.info("Can't merge {} because of different user metadata {}", path,
+ key);
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ static void mergeMetadata(Map<String,ByteBuffer> metadata,
+ Reader reader) {
+ for(String key: reader.getMetadataKeys()) {
+ metadata.put(key, reader.getMetadataValue(key));
+ }
+ }
+
+ /**
+ * Merges multiple ORC files that all have the same schema to produce
+ * a single ORC file.
+ * The merge will reject files that aren't compatible with the merged file
+ * so the output list may be shorter than the input list.
+ * The stripes are copied as serialized byte buffers.
+ * The user metadata are merged and files that disagree on the value
+ * associated with a key will be rejected.
+ *
+ * @param outputPath the output file
+ * @param options the options for writing with although the options related
+ * to the input files' encodings are overridden
+ * @param inputFiles the list of files to merge
+ * @return the list of files that were successfully merged
+ * @throws IOException
+ */
+ public static List<Path> mergeFiles(Path outputPath,
+ WriterOptions options,
+ Path... inputFiles) throws IOException {
+ Writer output = null;
+ byte[] buffer = new byte[0];
+ Configuration conf = options.getConfiguration();
+ TypeDescription schema = null;
+ CompressionKind compression = null;
+ int bufferSize = 0;
+ Version fileVersion = null;
+ WriterVersion writerVersion = null;
+ int rowIndexStride = 0;
+ List<Path> result = new ArrayList<>(inputFiles.length);
+ Map<String, ByteBuffer> userMetadata = new HashMap<>();
+
+ for (Path input: inputFiles) {
+ FileSystem fs = input.getFileSystem(conf);
+ Reader reader = createReader(input,
+ readerOptions(options.getConfiguration()).filesystem(fs));
+
+ if (!understandFormat(input, reader)) {
+ continue;
+ } else if (schema == null) {
+ // if this is the first file that we are including, grab the values
+ schema = reader.getSchema();
+ compression = reader.getCompressionKind();
+ bufferSize = reader.getCompressionSize();
+ rowIndexStride = reader.getRowIndexStride();
+ fileVersion = reader.getFileVersion();
+ writerVersion = reader.getWriterVersion();
+ options.blockSize(bufferSize)
+ .version(fileVersion)
+ .writerVersion(writerVersion)
+ .compress(compression)
+ .rowIndexStride(rowIndexStride)
+ .setSchema(schema);
+ if (compression != CompressionKind.NONE) {
+ options.enforceBufferSize().bufferSize(bufferSize);
+ }
+ mergeMetadata(userMetadata, reader);
+ output = createWriter(outputPath, options);
+ } else if (!readerIsCompatible(schema, fileVersion, writerVersion,
+ bufferSize, rowIndexStride, compression, userMetadata, input,
+ reader)) {
+ continue;
+ } else {
+ mergeMetadata(userMetadata, reader);
+ }
+ List<OrcProto.StripeStatistics> statList =
+ reader.getOrcProtoStripeStatistics();
+ FSDataInputStream inputStream = fs.open(input);
+ int stripeNum = 0;
+ result.add(input);
+
+ for(StripeInformation stripe: reader.getStripes()) {
+ int length = (int) stripe.getLength();
+ if (buffer.length < length) {
+ buffer = new byte[length];
+ }
+ long offset = stripe.getOffset();
+ inputStream.readFully(offset, buffer, 0, length);
--- End diff --
True, but the same happens if you have a Writer that throws an exception.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc pull request #84: ORC-132. Implement a merge file method and fix the num...
Posted by omalley <gi...@git.apache.org>.
Github user omalley commented on a diff in the pull request:
https://github.com/apache/orc/pull/84#discussion_r96791717
--- Diff: java/core/src/java/org/apache/orc/OrcFile.java ---
@@ -642,4 +671,187 @@ public static Writer createWriter(Path path,
return new WriterImpl(fs, path, opts);
}
+ /**
+ * Do we understand the version in the reader?
+ * @param path the path of the file
+ * @param reader the ORC file reader
+ * @return is the version understood by this writer?
+ */
+ static boolean understandFormat(Path path, Reader reader) {
+ if (reader.getFileVersion() == Version.FUTURE) {
+ LOG.info("Can't merge {} because it has a future version.", path);
+ return false;
+ }
+ if (reader.getWriterVersion() == WriterVersion.FUTURE) {
+ LOG.info("Can't merge {} because it has a future writerVersion.", path);
+ return false;
+ }
+ return true;
+ }
+
+ /**
+ * Is the new reader compatible with the file that is being written?
+ * @param schema the writer schema
+ * @param fileVersion the writer fileVersion
+ * @param writerVersion the writer writerVersion
+ * @param compressionSize the compression buffer size
+ * @param rowIndexStride the row index stride
+ * @param compression the compression that was used
+ * @param userMetadata the user metadata
+ * @param path the new path name for warning messages
+ * @param reader the new reader
+ * @return is the reader compatible with the previous ones?
+ */
+ static boolean readerIsCompatible(TypeDescription schema,
+ Version fileVersion,
+ WriterVersion writerVersion,
+ int compressionSize,
+ int rowIndexStride,
+ CompressionKind compression,
+ Map<String, ByteBuffer> userMetadata,
+ Path path,
+ Reader reader) {
+ // now we have to check compatibility
+ if (!reader.getSchema().equals(schema)) {
+ LOG.info("Can't merge {} because of different schemas {} vs {}",
+ path, reader.getSchema(), schema);
+ return false;
+ }
+ if (reader.getCompressionKind() != compression) {
+ LOG.info("Can't merge {} because of different compression {} vs {}",
+ path, reader.getCompressionKind(), compression);
+ return false;
+ }
+ if (compression != CompressionKind.NONE &&
+ reader.getCompressionSize() != compressionSize) {
+ LOG.info("Can't merge {} because of different compression sizes {} vs {}",
+ path, reader.getCompressionSize(), compressionSize);
+ return false;
+ }
+ if (reader.getFileVersion() != fileVersion) {
+ LOG.info("Can't merge {} because of different file versions {} vs {}",
+ path, reader.getFileVersion(), fileVersion);
+ return false;
+ }
+ if (reader.getWriterVersion() != writerVersion) {
+ LOG.info("Can't merge {} because of different writer versions {} vs {}",
+ path, reader.getFileVersion(), fileVersion);
+ return false;
+ }
+ if (reader.getRowIndexStride() != rowIndexStride) {
+ LOG.info("Can't merge {} because of different row index strides {} vs {}",
+ path, reader.getRowIndexStride(), rowIndexStride);
+ return false;
+ }
+ for(String key: reader.getMetadataKeys()) {
+ if (userMetadata.containsKey(key)) {
+ ByteBuffer currentValue = userMetadata.get(key);
+ ByteBuffer newValue = reader.getMetadataValue(key);
+ if (!newValue.equals(currentValue)) {
+ LOG.info("Can't merge {} because of different user metadata {}", path,
+ key);
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ static void mergeMetadata(Map<String,ByteBuffer> metadata,
+ Reader reader) {
+ for(String key: reader.getMetadataKeys()) {
+ metadata.put(key, reader.getMetadataValue(key));
+ }
+ }
+
+ /**
+ * Merges multiple ORC files that all have the same schema to produce
+ * a single ORC file.
+ * The merge will reject files that aren't compatible with the merged file
+ * so the output list may be shorter than the input list.
+ * The stripes are copied as serialized byte buffers.
+ * The user metadata are merged and files that disagree on the value
+ * associated with a key will be rejected.
+ *
+ * @param outputPath the output file
+ * @param options the options for writing with although the options related
+ * to the input files' encodings are overridden
+ * @param inputFiles the list of files to merge
+ * @return the list of files that were successfully merged
+ * @throws IOException
+ */
+ public static List<Path> mergeFiles(Path outputPath,
+ WriterOptions options,
+ Path... inputFiles) throws IOException {
+ Writer output = null;
+ byte[] buffer = new byte[0];
+ Configuration conf = options.getConfiguration();
+ TypeDescription schema = null;
+ CompressionKind compression = null;
+ int bufferSize = 0;
+ Version fileVersion = null;
+ WriterVersion writerVersion = null;
+ int rowIndexStride = 0;
+ List<Path> result = new ArrayList<>(inputFiles.length);
+ Map<String, ByteBuffer> userMetadata = new HashMap<>();
+
+ for (Path input: inputFiles) {
+ FileSystem fs = input.getFileSystem(conf);
+ Reader reader = createReader(input,
+ readerOptions(options.getConfiguration()).filesystem(fs));
+
+ if (!understandFormat(input, reader)) {
+ continue;
+ } else if (schema == null) {
+ // if this is the first file that we are including, grab the values
+ schema = reader.getSchema();
+ compression = reader.getCompressionKind();
+ bufferSize = reader.getCompressionSize();
+ rowIndexStride = reader.getRowIndexStride();
+ fileVersion = reader.getFileVersion();
+ writerVersion = reader.getWriterVersion();
+ options.blockSize(bufferSize)
+ .version(fileVersion)
+ .writerVersion(writerVersion)
+ .compress(compression)
+ .rowIndexStride(rowIndexStride)
+ .setSchema(schema);
+ if (compression != CompressionKind.NONE) {
+ options.enforceBufferSize().bufferSize(bufferSize);
+ }
+ mergeMetadata(userMetadata, reader);
+ output = createWriter(outputPath, options);
+ } else if (!readerIsCompatible(schema, fileVersion, writerVersion,
+ bufferSize, rowIndexStride, compression, userMetadata, input,
+ reader)) {
+ continue;
+ } else {
+ mergeMetadata(userMetadata, reader);
+ }
+ List<OrcProto.StripeStatistics> statList =
+ reader.getOrcProtoStripeStatistics();
+ FSDataInputStream inputStream = fs.open(input);
+ int stripeNum = 0;
+ result.add(input);
+
+ for(StripeInformation stripe: reader.getStripes()) {
+ int length = (int) stripe.getLength();
+ if (buffer.length < length) {
+ buffer = new byte[length];
+ }
+ long offset = stripe.getOffset();
+ inputStream.readFully(offset, buffer, 0, length);
--- End diff --
No, I'd rather have the whole method blow out than hide the exception. It can only happen in the case of a broken ORC file.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc pull request #84: ORC-132. Implement a merge file method and fix the num...
Posted by omalley <gi...@git.apache.org>.
Github user omalley commented on a diff in the pull request:
https://github.com/apache/orc/pull/84#discussion_r96961337
--- Diff: java/core/src/java/org/apache/orc/OrcFile.java ---
@@ -642,4 +671,187 @@ public static Writer createWriter(Path path,
return new WriterImpl(fs, path, opts);
}
+ /**
+ * Do we understand the version in the reader?
+ * @param path the path of the file
+ * @param reader the ORC file reader
+ * @return is the version understood by this writer?
+ */
+ static boolean understandFormat(Path path, Reader reader) {
+ if (reader.getFileVersion() == Version.FUTURE) {
+ LOG.info("Can't merge {} because it has a future version.", path);
+ return false;
+ }
+ if (reader.getWriterVersion() == WriterVersion.FUTURE) {
+ LOG.info("Can't merge {} because it has a future writerVersion.", path);
+ return false;
+ }
+ return true;
+ }
+
+ /**
+ * Is the new reader compatible with the file that is being written?
+ * @param schema the writer schema
+ * @param fileVersion the writer fileVersion
+ * @param writerVersion the writer writerVersion
+ * @param compressionSize the compression buffer size
+ * @param rowIndexStride the row index stride
+ * @param compression the compression that was used
+ * @param userMetadata the user metadata
+ * @param path the new path name for warning messages
+ * @param reader the new reader
+ * @return is the reader compatible with the previous ones?
+ */
+ static boolean readerIsCompatible(TypeDescription schema,
+ Version fileVersion,
+ WriterVersion writerVersion,
+ int compressionSize,
+ int rowIndexStride,
+ CompressionKind compression,
+ Map<String, ByteBuffer> userMetadata,
+ Path path,
+ Reader reader) {
+ // now we have to check compatibility
+ if (!reader.getSchema().equals(schema)) {
+ LOG.info("Can't merge {} because of different schemas {} vs {}",
+ path, reader.getSchema(), schema);
+ return false;
+ }
+ if (reader.getCompressionKind() != compression) {
+ LOG.info("Can't merge {} because of different compression {} vs {}",
+ path, reader.getCompressionKind(), compression);
+ return false;
+ }
+ if (compression != CompressionKind.NONE &&
+ reader.getCompressionSize() != compressionSize) {
+ LOG.info("Can't merge {} because of different compression sizes {} vs {}",
+ path, reader.getCompressionSize(), compressionSize);
+ return false;
+ }
+ if (reader.getFileVersion() != fileVersion) {
+ LOG.info("Can't merge {} because of different file versions {} vs {}",
+ path, reader.getFileVersion(), fileVersion);
+ return false;
+ }
+ if (reader.getWriterVersion() != writerVersion) {
+ LOG.info("Can't merge {} because of different writer versions {} vs {}",
+ path, reader.getFileVersion(), fileVersion);
+ return false;
+ }
+ if (reader.getRowIndexStride() != rowIndexStride) {
+ LOG.info("Can't merge {} because of different row index strides {} vs {}",
+ path, reader.getRowIndexStride(), rowIndexStride);
+ return false;
+ }
+ for(String key: reader.getMetadataKeys()) {
+ if (userMetadata.containsKey(key)) {
+ ByteBuffer currentValue = userMetadata.get(key);
+ ByteBuffer newValue = reader.getMetadataValue(key);
+ if (!newValue.equals(currentValue)) {
+ LOG.info("Can't merge {} because of different user metadata {}", path,
+ key);
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ static void mergeMetadata(Map<String,ByteBuffer> metadata,
+ Reader reader) {
+ for(String key: reader.getMetadataKeys()) {
+ metadata.put(key, reader.getMetadataValue(key));
+ }
+ }
+
+ /**
+ * Merges multiple ORC files that all have the same schema to produce
+ * a single ORC file.
+ * The merge will reject files that aren't compatible with the merged file
+ * so the output list may be shorter than the input list.
+ * The stripes are copied as serialized byte buffers.
+ * The user metadata are merged and files that disagree on the value
+ * associated with a key will be rejected.
+ *
+ * @param outputPath the output file
+ * @param options the options for writing with although the options related
+ * to the input files' encodings are overridden
+ * @param inputFiles the list of files to merge
+ * @return the list of files that were successfully merged
+ * @throws IOException
+ */
+ public static List<Path> mergeFiles(Path outputPath,
+ WriterOptions options,
+ Path... inputFiles) throws IOException {
+ Writer output = null;
+ byte[] buffer = new byte[0];
+ Configuration conf = options.getConfiguration();
+ TypeDescription schema = null;
+ CompressionKind compression = null;
+ int bufferSize = 0;
+ Version fileVersion = null;
+ WriterVersion writerVersion = null;
+ int rowIndexStride = 0;
+ List<Path> result = new ArrayList<>(inputFiles.length);
+ Map<String, ByteBuffer> userMetadata = new HashMap<>();
+
+ for (Path input: inputFiles) {
+ FileSystem fs = input.getFileSystem(conf);
+ Reader reader = createReader(input,
+ readerOptions(options.getConfiguration()).filesystem(fs));
+
+ if (!understandFormat(input, reader)) {
+ continue;
+ } else if (schema == null) {
+ // if this is the first file that we are including, grab the values
+ schema = reader.getSchema();
+ compression = reader.getCompressionKind();
+ bufferSize = reader.getCompressionSize();
+ rowIndexStride = reader.getRowIndexStride();
+ fileVersion = reader.getFileVersion();
+ writerVersion = reader.getWriterVersion();
+ options.blockSize(bufferSize)
+ .version(fileVersion)
+ .writerVersion(writerVersion)
+ .compress(compression)
+ .rowIndexStride(rowIndexStride)
+ .setSchema(schema);
+ if (compression != CompressionKind.NONE) {
+ options.enforceBufferSize().bufferSize(bufferSize);
+ }
+ mergeMetadata(userMetadata, reader);
+ output = createWriter(outputPath, options);
+ } else if (!readerIsCompatible(schema, fileVersion, writerVersion,
+ bufferSize, rowIndexStride, compression, userMetadata, input,
+ reader)) {
+ continue;
+ } else {
+ mergeMetadata(userMetadata, reader);
+ }
+ List<OrcProto.StripeStatistics> statList =
+ reader.getOrcProtoStripeStatistics();
+ FSDataInputStream inputStream = fs.open(input);
+ int stripeNum = 0;
+ result.add(input);
+
+ for(StripeInformation stripe: reader.getStripes()) {
+ int length = (int) stripe.getLength();
+ if (buffer.length < length) {
+ buffer = new byte[length];
+ }
+ long offset = stripe.getOffset();
+ inputStream.readFully(offset, buffer, 0, length);
--- End diff --
The current WriterImpl.appendStripe() will fail if it is called on files that don't have stripe statistic, so that feels like we can handle it in a follow up bug.
If the merge can't complete, throwing an exception seems like the best course. I guess we can delete the partially written output file if that feels cleaner to you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc pull request #84: ORC-132. Implement a merge file method and fix the num...
Posted by prasanthj <gi...@git.apache.org>.
Github user prasanthj commented on a diff in the pull request:
https://github.com/apache/orc/pull/84#discussion_r96799177
--- Diff: java/core/src/test/org/apache/orc/TestVectorOrcFile.java ---
@@ -2944,4 +2948,164 @@ public void testWriterVersion() throws Exception {
assertEquals(OrcFile.WriterVersion.ORIGINAL, OrcFile.WriterVersion.from(0));
assertEquals(OrcFile.WriterVersion.HIVE_4243, OrcFile.WriterVersion.from(2));
}
+
+ /**
+ * Test whether the file versions are translated correctly
+ * @throws Exception
+ */
+ @Test
+ public void testFileVersion() throws Exception {
+ assertEquals(OrcFile.Version.V_0_11, ReaderImpl.getFileVersion(null));
+ assertEquals(OrcFile.Version.V_0_11, ReaderImpl.getFileVersion(new ArrayList<Integer>()));
+ assertEquals(OrcFile.Version.V_0_11,
+ ReaderImpl.getFileVersion(Arrays.asList(new Integer[]{0, 11})));
+ assertEquals(OrcFile.Version.V_0_12,
+ ReaderImpl.getFileVersion(Arrays.asList(new Integer[]{0, 12})));
+ assertEquals(OrcFile.Version.FUTURE,
+ ReaderImpl.getFileVersion(Arrays.asList(new Integer[]{9999, 0})));
+ }
+
+ @Test
+ public void testMergeUnderstood() throws Exception {
+ Path p = new Path("test.orc");
+ Reader futureVersion = Mockito.mock(Reader.class);
+ Mockito.when(futureVersion.getFileVersion()).thenReturn(OrcFile.Version.FUTURE);
+ Mockito.when(futureVersion.getWriterVersion()).thenReturn(OrcFile.WriterVersion.HIVE_4243);
+ assertEquals(false, OrcFile.understandFormat(p, futureVersion));
+ Reader futureWriter = Mockito.mock(Reader.class);
+ Mockito.when(futureWriter.getFileVersion()).thenReturn(OrcFile.Version.V_0_11);
+ Mockito.when(futureWriter.getWriterVersion()).thenReturn(OrcFile.WriterVersion.FUTURE);
+ assertEquals(false, OrcFile.understandFormat(p, futureWriter));
+ Reader current = Mockito.mock(Reader.class);
+ Mockito.when(current.getFileVersion()).thenReturn(OrcFile.Version.CURRENT);
+ Mockito.when(current.getWriterVersion()).thenReturn(OrcFile.CURRENT_WRITER);
+ assertEquals(true, OrcFile.understandFormat(p, current));
+ }
+
+ static ByteBuffer fromString(String s) {
+ return ByteBuffer.wrap(s.getBytes(StandardCharsets.UTF_8));
+ }
+
+ static byte[] fromInt(int x) {
+ return Integer.toHexString(x).getBytes(StandardCharsets.UTF_8);
+ }
+
+ @Test
+ public void testGoodMerge() throws Exception {
--- End diff --
ah! ok. Missed that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] orc pull request #84: ORC-132. Implement a merge file method and fix the num...
Posted by omalley <gi...@git.apache.org>.
Github user omalley commented on a diff in the pull request:
https://github.com/apache/orc/pull/84#discussion_r96791733
--- Diff: java/core/src/test/org/apache/orc/TestVectorOrcFile.java ---
@@ -2944,4 +2948,164 @@ public void testWriterVersion() throws Exception {
assertEquals(OrcFile.WriterVersion.ORIGINAL, OrcFile.WriterVersion.from(0));
assertEquals(OrcFile.WriterVersion.HIVE_4243, OrcFile.WriterVersion.from(2));
}
+
+ /**
+ * Test whether the file versions are translated correctly
+ * @throws Exception
+ */
+ @Test
+ public void testFileVersion() throws Exception {
+ assertEquals(OrcFile.Version.V_0_11, ReaderImpl.getFileVersion(null));
+ assertEquals(OrcFile.Version.V_0_11, ReaderImpl.getFileVersion(new ArrayList<Integer>()));
+ assertEquals(OrcFile.Version.V_0_11,
+ ReaderImpl.getFileVersion(Arrays.asList(new Integer[]{0, 11})));
+ assertEquals(OrcFile.Version.V_0_12,
+ ReaderImpl.getFileVersion(Arrays.asList(new Integer[]{0, 12})));
+ assertEquals(OrcFile.Version.FUTURE,
+ ReaderImpl.getFileVersion(Arrays.asList(new Integer[]{9999, 0})));
+ }
+
+ @Test
+ public void testMergeUnderstood() throws Exception {
+ Path p = new Path("test.orc");
+ Reader futureVersion = Mockito.mock(Reader.class);
+ Mockito.when(futureVersion.getFileVersion()).thenReturn(OrcFile.Version.FUTURE);
+ Mockito.when(futureVersion.getWriterVersion()).thenReturn(OrcFile.WriterVersion.HIVE_4243);
+ assertEquals(false, OrcFile.understandFormat(p, futureVersion));
+ Reader futureWriter = Mockito.mock(Reader.class);
+ Mockito.when(futureWriter.getFileVersion()).thenReturn(OrcFile.Version.V_0_11);
+ Mockito.when(futureWriter.getWriterVersion()).thenReturn(OrcFile.WriterVersion.FUTURE);
+ assertEquals(false, OrcFile.understandFormat(p, futureWriter));
+ Reader current = Mockito.mock(Reader.class);
+ Mockito.when(current.getFileVersion()).thenReturn(OrcFile.Version.CURRENT);
+ Mockito.when(current.getWriterVersion()).thenReturn(OrcFile.CURRENT_WRITER);
+ assertEquals(true, OrcFile.understandFormat(p, current));
+ }
+
+ static ByteBuffer fromString(String s) {
+ return ByteBuffer.wrap(s.getBytes(StandardCharsets.UTF_8));
+ }
+
+ static byte[] fromInt(int x) {
+ return Integer.toHexString(x).getBytes(StandardCharsets.UTF_8);
+ }
+
+ @Test
+ public void testGoodMerge() throws Exception {
--- End diff --
Ok.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---